Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Binary Record Fixe #260

Merged
merged 4 commits into from
Feb 6, 2018
Merged

Binary Record Fixe #260

merged 4 commits into from
Feb 6, 2018

Conversation

cs2be
Copy link
Collaborator

@cs2be cs2be commented Feb 6, 2018

  • Fixed issues with binary_record.h filename generator method. std::hash is not consistent across OS. It will generate different hash's when ran on different OS on the same string. Change the filename generator to use a static counter. It will be more ideal to use UUID generator available in boost 1.4.2, however, boost 1.4.2 has problems compiling on c++11

  • Fix issue with pytorch demo where it logs to the absolute path /workspace

  • Fix performance issue on server when server fails to load image file. Currently server retries an API call 3 times before it fails, at which point it'll sleep the thread for 2 secs. This becomes problematic when there are alot of SDK errors. I'll create seperate issue to refine the retry logic to not be so expensive on the server.

…nerator method. std::hash is not consistent across OS. It will generate different hash's when ran on different OS on the same string. Change the filename generator to use a static counter. It will be more ideal to use UUID generator available in boost 1.4.2, however, boost 1.4.2 has problems compiling on c++11 \n* Fix issue with pytorch demo where it logs to the absolute path /workspace \n * Fix performance issue on server when server fails to load image file. Currently server retries an API call 3 times before it fails, at which point it'll sleep the thread for 2 secs. This becomes problematic when there are alot of SDK errors. I'll create seperate issue to refine the retry logic to not be so expensive on the server.
@@ -52,42 +51,59 @@ struct BinaryRecord {
file.write(data_.data(), data_.size());
}

const std::string& hash() { return hash_; }
static std::string generateFileName() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do

// which impacts performance.
if (!filename_.empty()) {
std::string path = dir_ + "/" + filename_;
std::cout << "READING FILE: " << path << std::endl;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this line, in the training period, cout with an endl is low performance.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ops sorry, forgot to remove this line, will remove it.

Copy link
Contributor

@Superjomn Superjomn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cs2be cs2be merged commit ef48ef3 into PaddlePaddle:develop Feb 6, 2018
@cs2be cs2be deleted the BINARY_RECORD_FIX branch February 6, 2018 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants