Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No cross-platform compatibility of logs #244

Closed
cs2be opened this issue Feb 1, 2018 · 3 comments
Closed

No cross-platform compatibility of logs #244

cs2be opened this issue Feb 1, 2018 · 3 comments
Assignees
Labels

Comments

@cs2be
Copy link
Collaborator

cs2be commented Feb 1, 2018

visualDL has errors when reading logs from another system. For example If logs were generate on linux, and user reads the logs on macOS, the visualDL server will run into alot of errors.

@cs2be
Copy link
Collaborator Author

cs2be commented Feb 2, 2018

The issue seems to be with "std::hashstd::string". The hash method is returning different values on Linux and OSX.

binary_record.h:72:
std::string hash() { return std::to_string(hasher(data)); }

So when we read the binary record, we compare the file hash and throw an error if the hash doesn't match:

binary_record.h:85:
CHECK_EQ(hash(), hash_) << "data broken: " << path;

I created a test in "test_binary_record.cc"

  std::string message = "hello world";
  BinaryRecord rcd("./", std::move(message));
  rcd.tofile();

  BinaryRecordReader reader("./", rcd.hash());
  LOG(INFO) << reader.data;
  ASSERT_EQ(reader.data, "hello world");

  std::hash<std::string> hasher;
  std::cout << "HASH: " << hasher(message) << std::endl;
}

And ran test on both Ubuntu and OSX:

Ubuntu:

root@cc62a709b30a:/visualdl/build2# ./vl_test 
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BinaryRecord
[ RUN      ] BinaryRecord.init
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0202 19:39:27.768194  5961 test_binary_record.cc:27] hello world
HASH: 5577293430985752569
[       OK ] BinaryRecord.init (5 ms)
[----------] 1 test from BinaryRecord (5 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (5 ms total)
[  PASSED  ] 1 test.

OSX:

(venv) Thuans-MacBook-Pro-2:build thuan$ ./vl_test 
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BinaryRecord
[ RUN      ] BinaryRecord.init
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0202 11:38:29.180557 4007486400 test_binary_record.cc:27] hello world
HASH: 12386028635079221413
[       OK ] BinaryRecord.init (1 ms)
[----------] 1 test from BinaryRecord (1 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1 ms total)
[  PASSED  ] 1 test.

@Superjomn I'm not an expert in C++, do you know if there is a more consistant hashing method for cross platform?

@varunarora varunarora changed the title visualDL has errors when reading logs from another system. For example If logs were generate on linux, and user reads the logs on macOS, the visualDL server will run into alot of errors. No cross-platform compatibility of logs Feb 2, 2018
@varunarora varunarora added the bug label Feb 2, 2018
@Superjomn
Copy link
Contributor

Superjomn commented Feb 3, 2018

Hash here just works as a filename generator for a BinaryRecord, and it has some bugs, for example, if two image components add two same image records, these two records will be saved to one same file called daffa(a hash code), if any image component update this record, it will delete the daffa file even another image component still needs it.

So I think a simple but bug-free way is to define a filename generator that does not depend on the content of the record, for example, just a global counter(static) seems work.

@cs2be
Copy link
Collaborator Author

cs2be commented Feb 8, 2018

Fixed in #260

@cs2be cs2be closed this as completed Feb 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants