Design Doc: The Client Library of Parameter Server #2075

helinwang · 2017-05-10T00:38:48Z

Maybe here is better for review.

gongweibao · 2017-05-10T00:58:17Z

doc/design/cluster_train/trainer.md

+// to all parameter servers.
+func (*PServerClient) Save(path string) error
+```
+Please see [master server design doc](master_server.md) for the definition of `master.Task`.


It's a dead link

The file is in a PR. Will be a good link after the PR being merged.

gongweibao · 2017-05-10T01:09:12Z

doc/design/cluster_train/trainer.md

+
+// GetTask gets a new task by telling the master server the finished task.
+// Use nil as the finished task when getting the task for the first time.
+func (*MasterClient) GetTask(finished master.Task) (master.Task, error)


Finish a task和Get a new task是否分成两个函数比较好？

我把Master部分删了，下次再发一个PR，会分成两个函数。

gongweibao · 2017-05-10T01:10:21Z

doc/design/cluster_train/trainer.md

+// Parameter is a piece of data to sync with the parameter server.
+type Parameter struct {
+	Name        string
+	ElementType ElementType


变量名和变量的类型是否需要区分一下？

这倒不影响，因为ElementType作为变量名的调用方式是Parameter.ElementType，不会与全局scope的ElementType冲突。

typhoonzero · 2017-05-10T01:40:48Z

doc/design/cluster_train/trainer.md

+
+The Go interface is the basic abstraction of communications with the master server and parameter servers. We will add another layer on top (add retry logic, polish interface with C idiom) before exposing the library with a [C interface](#c-interface).
+
+```go


Are these interfaces kind of duplicate to #1964? The communication interfaces are export by the server and will be called remotely by the client. So the server and the client need to use the same piece of interface definition code.

The client library is not just a simple wrapper on top of RPC. For example ParamInitChan will use an etcd lock to figure out who will init the parameters (does not call parameter server RPC).

Indeed, the API here highly depends on the parameter server RPC API. So this PR and #1964 depends on each other. This in the API in my mind, I will be discussing with @dzhwinter and everyone to form the same API, borrowing ideas from both PRs.

typhoonzero · 2017-05-10T01:50:10Z

Well, in general, there are two ways of implement a parameter server:

Pre-define all the possible optimization algorithms, and implement these algorithms inside parameter server. So the client just calls the "algorithm-id" with gradients and some arguments.
Define basic OPs inside parameter server like "tensor add" or "tensor multiply" etc. The client then send gradients and an expression describing the optimization algorithm, eg. error := SendGrad("momentum(param, grads, lita, ...)", grads)

It's the first way in this document. Is the second choice possible? All trainers will use the same code, so passing a different expression to parameter server seems not a problem?

@typhoonzero The second way is possible to implement, but it's hard for me to see the benefit of it:

I think what user wants is to choose an update algorithm,
so implementing 1 makes the logic simple - the user just needs to specify a description of the predefined algorithm.
Implementing 2 is complex: the expressiveness of a single string is limited. For example, how can one express the update rule that is not stateless (e.g., some state is saved between gradient updates) with a single string? To me, it feels like 2 make the system a lot more complex, but I could not see a clear benefit (maybe I am wrong).

Helin

@helinwang @typhoonzero
we will go to Operator/kernel implement at the same time, we need to consider it in the trainer process.
Dong zhihong

wangkuiyi · 2017-05-10T01:27:26Z

doc/design/cluster_train/trainer.md

@@ -0,0 +1,82 @@
+# Design Doc: Trainer Communication Library


Design Doc: Trainer Communication Library

==>

Design Doc: The Client Library of Parameter Server

wangkuiyi · 2017-05-10T01:28:59Z

doc/design/cluster_train/trainer.md

@@ -0,0 +1,82 @@
+# Design Doc: Trainer Communication Library
+
+For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the trainer's communication library, which will manage communication with parameter servers and the [master server](master_server.md). The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file.


The client library should focus on talking to parameter server, not to the master process, because it is the client library of the parameter servers.

Agree. I removed the master communication part.

wangkuiyi · 2017-05-10T01:29:18Z

doc/design/cluster_train/trainer.md

+
+For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the trainer's communication library, which will manage communication with parameter servers and the [master server](master_server.md). The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file.
+
+## Go Interface


Go Interface

==>

Go Implementation

I replaced the Go API with C API. Go Implementation can be discussed later.

wangkuiyi · 2017-05-10T01:30:37Z

doc/design/cluster_train/trainer.md

+
+## Go Interface
+
+The Go interface is the basic abstraction of communications with the master server and parameter servers. We will add another layer on top (add retry logic, polish interface with C idiom) before exposing the library with a [C interface](#c-interface).


We write this client library in Go, because the parameter server is written in Go. To make the client library callable from trainers, which is written using C/C++, we need a C wrapper of this Go library.

I replaced the Go API with C API.

wangkuiyi · 2017-05-10T01:32:15Z

doc/design/cluster_train/trainer.md

+type Parameter struct {
+	Name        string
+	ElementType ElementType
+	Buffer      []byte


Buffer => Content ?

Changed to content (in C API, Go API has been removed.)

gongweibao · 2017-05-11T01:13:09Z

doc/design/cluster_train/pserver_client.md

+## C Interface
+
+```c
+#define PADDLE_ELEMENT_TYPE_INT32   0


define => enum?

Thanks! Done.

gongweibao · 2017-05-11T01:16:46Z

doc/design/cluster_train/pserver_client.md

+#define PADDLE_ELEMENT_TYPE_FLOAT32 4
+#define PADDLE_ELEMENT_TYPE_FLOAT64 5
+
+typedef struct {


Do we need to specify a namespace?

c里面貌似没有namespace，一般用前缀来防止link时候的冲突，比如我们用了paddle_。

dzhwinter · 2017-05-11T02:28:11Z

doc/design/cluster_train/pserver_client.md

+ * @param learning_rate the learning rate for the gradients.
+ * @return 0 if successful, otherwise -1.
+ */
+int paddle_send_grads(paddle_pserver_client* client, const paddle_gradient* grads, int total, double learning_rate);


maybe use the same interface with "set_parameter" better? otherwise, the set_parameters only used once.

Removed the set_parameter interface, only keeping paddle_send_grads.
Here are the reasons:
As far as I know, set_parameter is only used for saving the learning rate (or optimizer related stuff) so that next the user starts the training job, the learning rate can be the value when the model is saved. I think such use case does not justify the complication added to the system:

What should we do when saving parameters, do we saves these parameters as well?

Make the interface harder to understand.

If the user wants to use a different learning rate the next time he starts a training job, he can print the learning rate to screen and change the python code accordingly.

If we think this use case is important, maybe we can discuss and add later? For now I think we need to choose the simple way.

Maybe there are other use cases for set_parameter that I am unaware of?

dzhwinter · 2017-05-11T02:31:07Z

doc/design/cluster_train/pserver_client.md

+ * @return 0 if successful, otherwise -1.
+ */
+int paddle_send_grads(paddle_pserver_client* client, const paddle_gradient* grads, int total, double learning_rate);
+


in fact, I want to know where the Updater/optimizer should be.
The trainer need to calculate gradients before send grads, and the pserver also need optimizer to store the training state

Actually here it's only sending the learning rate, not the update method. The update method is sent here using Proto Buffer.

reyoung · 2017-05-11T06:48:15Z

请问下这个PR的设计里是否考虑了稀疏更新和正则化的问题？这两点在PServer端实现会比较复杂。

helinwang · 2017-05-12T01:02:45Z

@reyoung 谢谢提醒！如何支持Sparse更新以及正则化的支持我都加到design doc里了，请看更新版本。
另外还总结了些，写到了：#2106 (comment)

dzhwinter · 2017-05-12T01:54:09Z

doc/design/cluster_train/pserver_client.md

+
+## Parameter Partition
+
+Each parameter will be partitioned into parameter chunks to make the parameters evenly distributed on parameter servers. The partition is done automatically by the client library. The *sparse parameter* require a little different treatment:


Hmm, we should unify the Parameter Partition name with ParameterServer, I just wrote ParameterBlock as the Paddle already have.

Done. Renamed parameter chunks to parameter blocks.

typhoonzero

LGTM

gongweibao

LGTM

Yancey1989

LGTM!

Trainer Communication Library design doc, the Go interface part

50602e6

helinwang requested review from reyoung, Yancey1989, wangkuiyi, dzhwinter, gongweibao and typhoonzero May 10, 2017 00:38

Helin Wang added 2 commits May 9, 2017 17:56

update grammar

0ed68da

add SetParam

cd978b7

gongweibao requested changes May 10, 2017

View reviewed changes

typhoonzero reviewed May 10, 2017

View reviewed changes

wangkuiyi reviewed May 10, 2017

View reviewed changes

jacquesqiao self-requested a review May 10, 2017 23:16

Helin Wang added 4 commits May 10, 2017 16:41

change to C API

66a3cfe

update C API

0064fcb

update documentation

1525216

fix grammar

31654e2

helinwang changed the title ~~Trainer Communication Library design doc, the Go interface part~~ Design Doc: The Client Library of Parameter Server May 11, 2017

gongweibao approved these changes May 11, 2017

View reviewed changes

gongweibao requested changes May 11, 2017

View reviewed changes

Helin Wang added 3 commits May 10, 2017 18:33

add illustration

f1b89f9

change img positioning

c06d417

add subtitle

c583447

dzhwinter reviewed May 11, 2017

View reviewed changes

Helin Wang added 4 commits May 11, 2017 17:37

add more sections

00d4d89

use enum instead of define

3da6c85

polish wording

747994d

polish wording

7d74738

wangkuiyi approved these changes May 12, 2017

View reviewed changes

dzhwinter reviewed May 12, 2017

View reviewed changes

typhoonzero approved these changes May 12, 2017

View reviewed changes

rename parameter chunks to parameter blocks

5081c69

gongweibao approved these changes May 13, 2017

View reviewed changes

Yancey1989 approved these changes May 13, 2017

View reviewed changes

helinwang merged commit 1ba8206 into PaddlePaddle:develop May 13, 2017

helinwang deleted the trainer_design branch May 13, 2017 05:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design Doc: The Client Library of Parameter Server #2075

Design Doc: The Client Library of Parameter Server #2075

helinwang commented May 10, 2017 •

edited

Loading

gongweibao May 10, 2017

helinwang May 10, 2017 •

edited

Loading

gongweibao May 10, 2017

helinwang May 11, 2017

gongweibao May 10, 2017

helinwang May 10, 2017

typhoonzero May 10, 2017

helinwang May 10, 2017 •

edited

Loading

typhoonzero commented May 10, 2017 •

edited by dzhwinter

Loading

wangkuiyi May 10, 2017

helinwang May 11, 2017

wangkuiyi May 10, 2017

helinwang May 11, 2017

wangkuiyi May 10, 2017

helinwang May 11, 2017

wangkuiyi May 10, 2017

helinwang May 11, 2017

wangkuiyi May 10, 2017

helinwang May 11, 2017

gongweibao May 11, 2017

helinwang May 12, 2017

gongweibao May 11, 2017

helinwang May 11, 2017

dzhwinter May 11, 2017

helinwang May 12, 2017 •

edited

Loading

dzhwinter May 11, 2017

helinwang May 11, 2017

reyoung commented May 11, 2017 •

edited

Loading

helinwang commented May 12, 2017

dzhwinter May 12, 2017

helinwang May 12, 2017

typhoonzero left a comment

gongweibao left a comment

Yancey1989 left a comment


		The Go interface is the basic abstraction of communications with the master server and parameter servers. We will add another layer on top (add retry logic, polish interface with C idiom) before exposing the library with a [C interface](#c-interface).

		```go

		@@ -0,0 +1,82 @@
		# Design Doc: Trainer Communication Library

		For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the trainer's communication library, which will manage communication with parameter servers and the [master server](master_server.md). The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file.


		For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the trainer's communication library, which will manage communication with parameter servers and the [master server](master_server.md). The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file.

		## Go Interface


		## Parameter Partition

		Each parameter will be partitioned into parameter chunks to make the parameters evenly distributed on parameter servers. The partition is done automatically by the client library. The sparse parameter require a little different treatment:

Design Doc: The Client Library of Parameter Server #2075

Design Doc: The Client Library of Parameter Server #2075

Conversation

helinwang commented May 10, 2017 • edited Loading

Choose a reason for hiding this comment

helinwang May 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang May 10, 2017 • edited Loading

Choose a reason for hiding this comment

typhoonzero commented May 10, 2017 • edited by dzhwinter Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang May 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyoung commented May 11, 2017 • edited Loading

helinwang commented May 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

typhoonzero left a comment

Choose a reason for hiding this comment

gongweibao left a comment

Choose a reason for hiding this comment

Yancey1989 left a comment

Choose a reason for hiding this comment

helinwang commented May 10, 2017 •

edited

Loading

helinwang May 10, 2017 •

edited

Loading

helinwang May 10, 2017 •

edited

Loading

typhoonzero commented May 10, 2017 •

edited by dzhwinter

Loading

helinwang May 12, 2017 •

edited

Loading

reyoung commented May 11, 2017 •

edited

Loading