Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[data-compaction][leo_object_storage] compaction|diagnose may get slow down when a garbage exist at the end of a avs #523

Closed
mocchira opened this issue Nov 17, 2016 · 2 comments

Comments

@mocchira
Copy link
Member

because file:pread always fail if the size of a invalid key|data is gt; the size of a avs file like this.

[E]     storage_0@127.0.0.1     2016-11-17 02:03:53.19699 +0000 1479348233      leo_file:pread/4        284      [{errors,[[{location,1552729},{expected_len,9039025},{actual_len,8933225},{diff,105800}][300/9360],unexpected_len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:04:01.807434 +0000        1479348241      leo_file:pread/4284      [{errors,[[{location,2043066},{expected_len,9360062},{actual_len,8442888},{diff,917174}]]},{result
,unexpected_len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:04:27.973403 +0000        1479348267      leo_file:pread/4284      [{errors,[[{location,3205264},{expected_len,7916219},{actual_len,7280690},{diff,635529}]]},{result
,unexpected_len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:04:34.670135 +0000        1479348274      leo_file:pread/4284      [{errors,[[{location,3393823},{expected_len,9135541},{actual_len,7092131},{diff,2043410}]]},{resul
t,unexpected_len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:04:41.831239 +0000        1479348281      leo_file:pread/4284      [{errors,[[{location,3721763},{expected_len,10085277},{actual_len,6764191},{diff,3321086}]]},{resu
lt,unexpected_len_and_eof}]

### ~~~ snip ~~~
### file:pread(s) failed) repeatedly 

[E]     [email protected]     2016-11-17 02:07:10.297342 +0000        1479348430      leo_file:pread/4284      [{errors,[[{location,10485950},{expected_len,128},{actual_len,4},{diff,124}]]},{result,unexpected_
len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:07:10.299366 +0000        1479348430      leo_file:pread/4284      [{errors,[[{location,10485951},{expected_len,128},{actual_len,3},{diff,125}]]},{result,unexpected_
len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:07:10.302033 +0000        1479348430      leo_file:pread/4284      [{errors,[[{location,10485952},{expected_len,128},{actual_len,2},{diff,126}]]},{result,unexpected_
len_and_eof}]
[E]     storage_0@127.0.0.1     2016-11-17 02:07:10.303306 +0000        1479348430      leo_file:pread/4284      [{errors,[[{location,10485953},{expected_len,128},{actual_len,1},{diff,127}]]},{result,unexpected_
len_and_eof}]
[W]     storage_0@127.0.0.1     2016-11-17 02:07:10.303677 +0000        1479348430      leo_compact_fsm_worker:execute_1/4       1048    [{obj_container_path,"/home/ubuntu/dev/leofs/package/leo_storage/avs/objec
t/0.avs_63646567195"},{error_pos_start,194},{error_pos_end,10485954},{errors,[{invalid_format,unexpected_time_format},{877,"invalid data"},unexpected_len_and_eof,{811,"invalid data"}]}]

In order to avoid this, need to get the size of a avs beforehand and skip pread if the size of a key|data is gt; its file size.

@mocchira
Copy link
Member Author

As file:pread is called with the raw mode,
https://github.com/erlang/otp/blob/OTP-18.3.4.4/erts/preloaded/src/prim_file.erl is called directly then as a result,
https://github.com/erlang/otp/blob/OTP-18.3.4.4/erts/emulator/drivers/unix/unix_efile.c#L706 is called
by a port driver on xunix (our targets).
According to http://pubs.opengroup.org/onlinepubs/7908799/xsh/read.html
(p)read may return less bytes than nbyte if

  • the number of bytes left in the file is less than nbyte ( in case of this issue )
  • the read() request was interrupted by a signal ( but this is handled by erlang itself so we can ignore )
  • the file is a pipe or FIFO or special file and has fewer than nbyte bytes immediately available for reading ( as we handle only regular files so we can ignore )

From these things,
we can regard unexpected_len_and_eof as a sign of close to EOF and
also judge there is no more files if # of bytes returned is less than the AVS header size (128).
so in this case we can stop compaction|diagnose and reduce subsequent file:pread(s) and outputs to error.log as shown above.

@mocchira
Copy link
Member Author

#378 should be rollbacked.

@yosukehara yosukehara changed the title [leo_object_storage] compaction|diagnose may get slow down when a garbage exist at the end of a avs [data-compaction][leo_object_storage] compaction|diagnose may get slow down when a garbage exist at the end of a avs Feb 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants