Checkpoint M2: lookup table checkpoint #11490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jacquesqiao
reviewed
Jun 22, 2018
| @@ -276,8 +320,8 @@ void AsyncGRPCServer::TryToRegisterNewOne(const std::string& rpc_name, | |||
| return; | |||
| } | |||
|
|
|||
| VLOG(4) << "register send rpc_name:" << rpc_name | |||
| << ", handler:" << rpc_call_map_[kRequestSend]; | |||
| LOG(INFO) << "TryToRegisterNewOne on RPC NAME: " << rpc_name | |||
There was a problem hiding this comment.
这个日志之前是错误的， 不管接收到的rpc_name是什么，只打印 handler:" << rpc_call_map_[kRequestSend]
jacquesqiao
reviewed
Jun 22, 2018
| @@ -243,28 +247,40 @@ void ListenAndServOp::RunImpl(const framework::Scope &scope, | |||
|
|
|||
| PADDLE_ENFORCE(!rpc_service_); | |||
| std::string endpoint = Attr<std::string>("endpoint"); | |||
| int checkpoint_notify_id = Attr<int>(kCheckpointBlockId); | |||
There was a problem hiding this comment.
checkpoint_notify_id 这个名字不太好
There was a problem hiding this comment.
rename to checkpoint_notify_block_id
jacquesqiao
reviewed
Jun 22, 2018
| import os | ||
|
|
||
| pserver_program.global_block().create_var( | ||
| name="loopup_table_path", |
jacquesqiao
reviewed
Jun 22, 2018
| checkpoint_save_block = pserver_program.create_block(pre_block_idx) | ||
| checkpoint_save_block.append_op( | ||
| type='save', | ||
| inputs={'X': [self.table_name]}, |
There was a problem hiding this comment.
table_name 为啥是input，而且应该也不是var吧？
There was a problem hiding this comment.
self.table_name是var, 已修改attr
jacquesqiao
reviewed
Jun 22, 2018
python/paddle/fluid/io.py
Outdated
| @@ -1068,6 +1187,29 @@ def save_trainer_args(dirname, trainer_id, trainer_args): | |||
|
|
|||
|
|
|||
| def load_trainer_args(checkpoint_dir, serial, trainer_id, trainer_args): | |||
| """ | |||
| trainer will load some args from it's independent directory, | |||
jacquesqiao
reviewed
Jun 22, 2018
| dirname(str): The directory path | ||
| main_program(Program): Find the variable named table_name in main_program | ||
| pserver_id(int): the serial number in pserver_endpoints list | ||
| table_name(str): lookup table name |
jacquesqiao
reviewed
Jun 22, 2018
paddle/fluid/operators/save_op.cc
Outdated
| AddComment(R"DOC( | ||
| Save operator | ||
|
|
||
| This operator will serialize and write a tensor variable to file on disk. | ||
| This operator will serialize and write a tensor/selected rows variable to file on disk. |
gongweibao
reviewed
Jun 22, 2018
| @@ -224,6 +266,8 @@ void AsyncGRPCServer::StartServer() { | |||
| reqs.reserve(kRequestBufSize); | |||
|
|
|||
| for (int i = 0; i < kRequestBufSize; i++) { | |||
| LOG(INFO) << "TryToRegisterNewOne on RPC NAME: " << rpc_name | |||
jacquesqiao
reviewed
Jun 25, 2018
| inputs={'X': [self.table_name]}, | ||
| outputs={}, | ||
| attrs={ | ||
| 'file_path': |
There was a problem hiding this comment.
'file_path': ""put this 'file_path' do not be used in save lookup table variable into comment
jacquesqiao
reviewed
Jun 25, 2018
|
|
||
| LOG(INFO) << "sync_mode:" << sync_mode << ", fan_in:" << fan_in | ||
| << ", end_point:" << endpoint; | ||
| << ", end_point:" << endpoint | ||
| << ", CheckpointNotify Id: " << checkpoint_notify_block_id; |
There was a problem hiding this comment.
"checkpoint_notify_block_id" => "checkpoint_block_id"
"CheckpointNotify Id" => "checkpoint_block_id "
jacquesqiao
reviewed
Jun 25, 2018
| std::shared_ptr<framework::ExecutorPrepareContext> ckpt_pre_context = nullptr; | ||
| if (checkpoint_notify_block_id != -1) { | ||
| auto ctx = executor.Prepare(*program, checkpoint_notify_block_id); | ||
| ckpt_pre_context = std::move(ctx); |
jacquesqiao
reviewed
Jun 25, 2018
paddle/fluid/operators/save_op.cc
Outdated
| #include "paddle/fluid/platform/device_context.h" | ||
|
|
||
| namespace paddle { | ||
| namespace operators { | ||
|
|
||
| // define LOOKUP_TABLE_PATH for checkpoint notify to save lookup table variables | ||
| // to directory specified. | ||
| constexpr char LOOKUP_TABLE_PATH[] = "lookup_table_path"; |
There was a problem hiding this comment.
https://google.github.io/styleguide/cppguide.html#Constant_Names
LOOKUP_TABLE_PATH => kLookupTablePath
jacquesqiao
previously approved these changes
Jun 26, 2018
gongweibao
approved these changes
Jun 26, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A new feature for #11410, it does not complete currently.
project: #10868