Design for remote parameter updater using cclient #2330

jacquesqiao · 2017-06-01T00:04:19Z

测试方法：
1，编译。
2，编译完成之后运行 go run ./go/cmd/pserver/pserver.go -port 3000 会在本地启动一个parameter server，监听3000端口。
3，python paddle/go/pserver/cclient/test/test_train.py

helinwang · 2017-06-01T19:44:17Z

doc/design/cluster_train/remote_parameter_updater.md

+
+Remote Parameter Updater manage parameters through remote parameter server with the client that communicate with pserver([The Client Library of Parameter Server Design Doc](pserver_client.md))
+
+In PaddlePaddle Python V2 API, trainer is implemented in python, and the trainer will hold a member of parameter updater and call it's functions directly. In this design, we should also expose the api of RemoteParameterUpdater to python with swig.


hold a member -> hold the instance

helinwang · 2017-06-01T19:46:05Z

doc/design/cluster_train/remote_parameter_updater.md

+
+Remote Parameter Updater manage parameters through remote parameter server with the client that communicate with pserver([The Client Library of Parameter Server Design Doc](pserver_client.md))
+
+In PaddlePaddle Python V2 API, trainer is implemented in python, and the trainer will hold a member of parameter updater and call it's functions directly. In this design, we should also expose the api of RemoteParameterUpdater to python with swig.


we should -> we will
感觉“we will"的语气更加确定，更合适design doc。"we should"在英语里我理解是我们应当做（但不一定立刻会做）。

helinwang · 2017-06-01T19:47:18Z

CI failed, probably need to use pre commit hook.

… parameterupdater

dzhwinter

👍 such a fast implement!

dzhwinter · 2017-06-06T09:08:46Z

go/pserver/cclient/CMakeLists.txt

@@ -9,5 +9,14 @@ project(cxx_go C Go)
 include(golang)
 include(flags)

-go_library(client STATIC)
+go_library(paddle_cclient STATIC)


maybe named with pserver_cclient will be better? since it is only used by the new trainer module and communicate with parameter server. paddle_cclient would lead to an inference client at first glance.

Spend a lot of time working on CMake to make it work properly.@@

This is a good suggestion, I will rename it~

dzhwinter · 2017-06-06T09:19:27Z

paddle/trainer/NewRemoteParameterUpdater.cpp

+
+  // create parameter server client.
+  parameterClient_ =
+      paddle_new_pserver_client((char *)pserverSpec_.c_str(), FLAGS_trainer_id);


there is a little different with current RemoteUpdater here. we don't have a trainer id in paddle cloud.

dzhwinter · 2017-06-06T09:31:23Z

paddle/trainer/NewRemoteParameterUpdater.cpp

+  LOG(INFO) << "finishBatch in, cost: " << cost;
+
+  // send gradient to parameter server.
+  paddle_send_grads(parameterClient_, *newGradients_, parameterSize());


variable cost is useless here. cannot find anywhere update the newGradients_

helinwang

Overall great!

helinwang · 2017-06-07T00:23:35Z

go/cmake/golang.cmake

@@ -32,17 +32,21 @@ function(GO_LIBRARY NAME BUILD_TYPE)
  # will use the local changes in Paddle rather than checkout Paddle
  # in github.
  add_custom_target(copyPaddle
-    COMMAND ln -sf ${PADDLE_DIR} ${PADDLE_IN_GOPATH})
+    COMMAND rm -rf ${PADDLE_IN_GOPATH}/Paddle


Strange, -f in ln command force the link, so we probably don't need to remove the old link.

这个地方我换了一种link方式，在最后面指定了目录名为Paddle，因为目前的做法，如果paddle所在目录改名字了，link的名字也会变，这样 go get就找不到目录了。
然后修改之后，在mac上两次编译会报Operation not permitted，因为他试图在Paddle这个链接下再建一个链接，循环了。

helinwang · 2017-06-07T00:27:19Z

go/cmake/golang.cmake

    ${CMAKE_GO_FLAGS} ${GO_SOURCE}
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})

  add_custom_target(${NAME} ALL DEPENDS ${OUTPUT_DIR}/.timestamp ${ARGN})
-  add_dependencies(${NAME} goGet)
+  add_dependencies(${NAME} copyPaddle)
+#  add_dependencies(${NAME} goGet)


It need to depend on go get to get all dependencies, are there anything wrong with this line?

helinwang · 2017-06-07T00:36:05Z

paddle/trainer/NewRemoteParameterUpdater.cpp

+
+  // create parameter server client.
+  parameterClient_ =
+      paddle_new_pserver_client((char *)pserverSpec_.c_str(), FLAGS_trainer_id);


paddle_new_pserver_client in cclient does not take any argument. I am curious how does this compile?

https://github.com/PaddlePaddle/Paddle/blob/develop/go/pserver/cclient/cclient.go#L103

func paddle_new_pserver_client(addrs *C.char, selected int) C.client {

That's new, sorry I got confused.

helinwang · 2017-06-07T00:39:43Z

paddle/trainer/NewRemoteParameterUpdater.cpp

+    paddle_get_params(
+        parameterClient_, names_, newParameters_, (int)parameters_.size());
+  }
+  paddle_finish_init_params(parameterClient_);


only the trainer that inits params should call paddle_finish_init_params. Please put this line to line 56.
Please see the graph in the linked section for reference: https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/cluster_train/pserver_client.md#trainer-selection-process

helinwang · 2017-06-07T00:42:37Z

python/paddle/v2/optimizer.py

+                    num_passes, use_sparse_updater)
+            else:
+                parameter_updater = self.__create_new_remote_updater__(
+                    pserver_spec)


The cclient don't need pserver spec (the only required spec is parameter config), please see: https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/cluster_train/pserver_client.md#trainer-selection-process
Maybe you have an outdated version of cclient?

I see, the current code may be old here?
https://github.com/PaddlePaddle/Paddle/blob/develop/go/pserver/cclient/cclient.go#L103

func paddle_new_pserver_client(addrs *C.char, selected int) C.client {

That's new, sorry I got confused.

jacquesqiao · 2017-06-07T15:46:39Z

closed and open a new pr #2413

init design for remote parameter updater

9f300b9

jacquesqiao changed the title ~~init design for remote parameter updater using cclient~~ Design for remote parameter updater using cclient Jun 1, 2017

helinwang reviewed Jun 1, 2017

View reviewed changes

jacquesqiao added 7 commits June 2, 2017 23:18

update new remote parameterupdater

119e0d3

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

46a8ecb

… parameterupdater

fix cmake problem, can build and run

001b52c

refine code

c357712

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

b8608a9

… parameterupdater

code clean for CMakeLists.txt

499bca7

style check

5522a56

jacquesqiao requested a review from dzhwinter June 6, 2017 05:58

fix compile problem

68107b7

dzhwinter requested changes Jun 6, 2017

View reviewed changes

jacquesqiao added 6 commits June 6, 2017 20:42

fix compile problem

a7d8e53

remove -rf the go link

6d00e71

add pthread link lib for cclient/test/main

529431b

add no-PIC for compile

1c8b5a6

cgo recompile with -fPIC

513660b

another try

482a30a

helinwang requested changes Jun 7, 2017

View reviewed changes

add paddle_pserver_cclient_lib deps for paddle_api

97277e6

jacquesqiao closed this Jun 7, 2017

jacquesqiao mentioned this pull request Jun 7, 2017

new parameterupdater use paddle pserver cclient of go #2413

Merged

heavengate pushed a commit to heavengate/Paddle that referenced this pull request Aug 16, 2021

fix obj365 label and config (PaddlePaddle#2330)

aa2500c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design for remote parameter updater using cclient #2330

Design for remote parameter updater using cclient #2330

jacquesqiao commented Jun 1, 2017 •

edited

Loading

helinwang Jun 1, 2017

jacquesqiao Jun 6, 2017

helinwang Jun 1, 2017

jacquesqiao Jun 6, 2017

helinwang commented Jun 1, 2017 •

edited

Loading

dzhwinter left a comment

dzhwinter Jun 6, 2017

jacquesqiao Jun 6, 2017

dzhwinter Jun 6, 2017

dzhwinter Jun 6, 2017 •

edited

Loading

helinwang left a comment

helinwang Jun 7, 2017

jacquesqiao Jun 7, 2017

helinwang Jun 7, 2017

helinwang Jun 7, 2017

jacquesqiao Jun 7, 2017

helinwang Jun 7, 2017

helinwang Jun 7, 2017

jacquesqiao Jun 7, 2017

helinwang Jun 7, 2017 •

edited

Loading

jacquesqiao Jun 7, 2017

helinwang Jun 7, 2017

jacquesqiao commented Jun 7, 2017


		Remote Parameter Updater manage parameters through remote parameter server with the client that communicate with pserver([The Client Library of Parameter Server Design Doc](pserver_client.md))

		In PaddlePaddle Python V2 API, trainer is implemented in python, and the trainer will hold a member of parameter updater and call it's functions directly. In this design, we should also expose the api of RemoteParameterUpdater to python with swig.

Design for remote parameter updater using cclient #2330

Design for remote parameter updater using cclient #2330

Conversation

jacquesqiao commented Jun 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang commented Jun 1, 2017 • edited Loading

dzhwinter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dzhwinter Jun 6, 2017 • edited Loading

Choose a reason for hiding this comment

helinwang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacquesqiao commented Jun 7, 2017

jacquesqiao commented Jun 1, 2017 •

edited

Loading

helinwang commented Jun 1, 2017 •

edited

Loading

dzhwinter Jun 6, 2017 •

edited

Loading

helinwang Jun 7, 2017 •

edited

Loading