Add fetch_by_key_in_parallel to fix #764

[mocchira]

No description provided.

[mocchira]

@yosukehara Also please check the comment here: leo-project/leofs#764 (comment)

@windkit Can you benchmark the delete-bucket performance on RIT env with the patch below for leo_storage.

diff --git a/apps/leo_storage/src/leo_storage_handler_object.erl b/apps/leo_storage/src/leo_storage_handler_object.erl
index bad8275..a94aee9 100644
--- a/apps/leo_storage/src/leo_storage_handler_object.erl
+++ b/apps/leo_storage/src/leo_storage_handler_object.erl
@@ -1076,7 +1076,7 @@ prefix_search_and_remove_objects(MQId, ParentDir, EnqueuedAt) ->
                           end
                   end
           end,
-    case catch leo_object_storage_api:fetch_by_key(ParentDir, Fun) of
+    case catch leo_object_storage_api:fetch_by_key_in_parallel(ParentDir, Fun, undefined) of
         {'EXIT', Cause} ->
             {error, Cause};
         {ok,_} ->

It should be almost same as without PR however I have a bit concern especially in case the number of AVS files is huge (256 - 512)

[yosukehara]

@mocchira WIP

[windkit]

@mocchira When I try to compile with

1. Get current develop branch of leofs
2. update the apps/leo_storage/src/leo_storage_handler_object.erl
3. Get this pull request on deps/leo_object_storage with git pull origin pull/22/head

and I cannot put any object to LeoFS. Do you think any things I am missing here?

[W]     [email protected]       2018-01-11 09:22:49.343422 +0900        1515630169      leo_storage_replicator:replicate_fun/2  249     [{key,<<"test/test">>},{node,local},{req_id,127006226},{cause,{noproc,{gen_server,call,[undefined,{get,<<"test/test">>},30000]}}}]
[E]     [email protected]       2018-01-11 09:22:49.352349 +0900        1515630169      leo_storage_handler_object:put/2        388     [{from,gateway},{method,put},{key,<<"test/test">>},{req_id,127006226},{cause,"Replicate failure"}]

[mocchira]

@windkit how about running ./git_checkout develop between step 1 and 2?

[windkit]

@mocchira Thank you, now it is working.

[windkit]

@mocchira From my testing, the initial enqueuing phase is quicker, but afterwards the consumption phase seems to be run in the same speed, does the above commit change that?

[windkit]

Version 1.3.8

PR (before the d17c49e

[mocchira]

@windkit

@mocchira From my testing, the initial enqueuing phase is quicker, but afterwards the consumption phase seems to be run in the same speed, does the above commit change that?

It should keep the same behavior. (maybe initial enqueuing phase gets a bit slow down due to the additional cpu cost to make the distribution as uniform as possible although quicker than 1.3.8)

anyway, there is still room to make the distribution more uniform by tuning some hard coded values so I'll ask you the benchmark again once the final optimization finished.

[mocchira]

@windkit Please benchmark again with the latest PR. also just in case let me confirm that what is set to num_of_containers? (256 or 512 would be desirable as I mentioned above)

[windkit]

PR (avs, 256)

@mocchira I am not seeing much difference with number of avs = 8 and 1.3.8

[mocchira]

@windkit Thanks. It looks good to me.
@yosukehara please start to review.

[windkit]

@mocchira it makes me wonder where is the bottleneck here, neither CPU / IO seems to be fully utilized.

[mocchira]

@windkit The original problem can be found on here leo-project/leofs#725 (comment).
In short, This PR tries to solve not only fully utilizing CPU/IO but also making timeout/slow processing for requests from gateway happen as less often as possible by distributing delete-bucket messages evenly into each leo_object_storage_server (and corresponding leo_backend_db_server).
Let me know if you have any question.

[yosukehara]

I've reviewed this PR. Can you confirm the comment?

[yosukehara]

5, as a trivial thing, I think that its value should be a constant.

[mocchira]

@yosukehara Fixed. PTAL.

[yosukehara]

@mocchira Merged, thanks

[windkit]

@mocchira but the result still shows that neither CPU nor IO got fully utilized, are there other bottlenecks?

[mocchira]

@windkit Yes probably software locks happened on eleveldb thread pool management/erlang runtime and/or OS thread scheduling especially for ones spawned by eleveldb thread pool. I think we can improve CPU/IO utilization in case dealing with many AVS files by tuning https://github.com/basho/eleveldb/blob/develop/c_src/eleveldb.cc#L223 which has been used as default(71) on leofs so changing it to the number of AVS files could improve CPU/IO utilization.
I will file it as another issue, thanks.

[mocchira]

filed on leo-project/leofs#970