[leo_object_storage] More strictly checking the header, less file:pread(s) for reading a body

[mocchira]

As there are only two checks (timestamp, dsize) for converting a binary to a header,
In many case unnecessary file:pread to retrieve its body happen during skipping a garbage.
These can be relaxed by checking the header more strictly ( like checking every fields ) .

[yosukehara]

I've recognized this issue, every item should be checked to avoid remaining a broken object in a container I think.

• leo_object_storage_transformer:header_bin_to_metadata/1
• leo_object_storage_haystack:get_obj_for_new_cntnr/4

[mocchira]

UPDATE: Fixed clock to 4633552912011362 as it's not unix timestamp/microsec but the erlang specific one
UPDATE: Fixed msize to 4096 as it may include MDC related information
UPDATE: Fixed a part of spec based on AVS spec https://github.com/leo-project/leo_object_storage/blob/1.3.0/include/leo_object_storage.hrl#L226-L242

check specification in details

• ksize
  • 4096 should be enough upper limit according to http://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers since a key must be included in url
• dsize (nop)
  • since already checked at leo_object_storage_haystack
• msize
  • 4096 should be enough upper limit according to http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html and also take care MDC related
• csize
  • should be same with the limit of dsize defined at https://github.com/leo-project/leo_object_storage/blob/1.3.0/include/leo_object_storage.hrl#L202
• cnumber/cindex(nop)
  • since it's difficult to set the upper because avs spec is a enough small
• offset
  • 16TB should be enough according to https://kernelnewbies.org/Ext4/#head-4e13646f1961f41bf99152eaf137d309723ac524 (xfs's upper limit is 8EB)
• clock (os:timestamp it's not unix timestamp/microsec but the erlang specific one)
  • 4633552912011362 should be enough ( calculated by os:timestamp() + 60 * 60 * 24 * 365 * 100 * 1e+6 )
• timestamp (nop)
  • since already checked at leo_object_storage_transformer
• checksum/ring_hash (nop)
  • since it's difficult to set the upper because of its characteristics
• del (nop)
  • already checked as it's stored into 1bit

@yosukehara could you check the above spec?
I'll go ahead with this spec if no problem.

[yosukehara]

@mocchira Thanks for summarizing the check items.

We actually use custom-metadata as two kind of uses, a metadata of the multi data center replication and user defined metadata(s). So we cannot fix length of the msize which is 2048 bytes but we can define the length, ( udf-metadata:2048 bytes + mdc-metadata.max_length)

• leo_sync_remote_cluster: stack_fun/2

• msize
  • 2048 should be enough upper limit according to http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

[mocchira]

@yosukehara thanks.

https://github.com/leo-project/leo_storage/blob/develop/src/leo_sync_remote_cluster.erl#L349 with length(CMeta) == 2048 should be the upper limit.
I'd measure the actual value and use it as the upper limit for msize.

[mocchira]

(storage_0@127.0.0.1)3> R = leo_object_storage_transformer:list_to_cmeta_bin([
(storage_0@127.0.0.1)3> {cluster_id, hoge},
(storage_0@127.0.0.1)3> {num_of_replicas, 3},
(storage_0@127.0.0.1)3> {udm, crypto:rand_bytes(2048)}
(storage_0@127.0.0.1)3> ]).
<<131,108,0,0,0,4,104,2,100,0,10,99,108,117,115,116,101,
  114,95,105,100,100,0,4,104,111,103,101,104,...>>
(storage_0@127.0.0.1)4> byte_size(R).
2122
(storage_0@127.0.0.1)5>

so I'd like to choose 4096 for safe.

[mocchira]

@yosukehara
as it turned out that os:timestamp/0 used in leo_date:clock/1 is NOT unix timestamp/microsecs ( around 10^3 times bigger than that ),
I'd adjust a bit.

[mocchira]

Skipping time has gotten around 2x faster than the 1.3.0 by this fix.