[leo_storage][leo_object_storage] Refactor slow processing notification to avoid bottlenecks

[mocchira]

As reported on #966 (comment), the current erlang process structure to handle slow processing notification doesn't scale well so need to refactor (ex: getting rid of intermediate gen_server(s). instead directly CRUD the specific ets talbe created with {write_concurrency, true} and logging through a library module)

[mocchira]

I'm going to deprecate leo_object_storage_event_notifier/leo_object_storage_event in leo_object_storage and leo_storage_msg_collector in leo_storage. Instead add the following module to realize the same functionality while scaling pretty well even in case many slow processing/timeout(s) happen.

-module(leo_object_storage_msg_collector).

%% @doc Init the module (create the ets table for collecting messages)
%% Being invoked from leo_storage application startup phase
-spec(init(Enabled) -> ok | {error, any()}).

%% @doc Clear all messages
%%  Being invoked from https://github.com/leo-project/leofs/blob/develop/apps/leo_storage/src/leo_storage_watchdog_msgs.erl
-spec(clear() -> ok | {error, any()}).

%% @doc Retrieve all messages
%%  Being invoked from https://github.com/leo-project/leofs/blob/develop/apps/leo_storage/src/leo_storage_watchdog_msgs.erl
-spec(get() -> {ok, list()} | {error, any()}).

%% @doc Notify the timeout request
%%  Being invoked from leo_storage_replicator and leo_storage_repairer
-spec(notify(Msg, Method, Key) ->
             ok | {error, any()} when Msg::atom(),
                                      Method::put|get|delete,
                                      Key::binary()).

%% @doc Notify the slow processing request
%%  Being invoked from leo_object_storage_server
-spec(notify(Msg, Method, Key, ProcessingTime) ->
             ok | {error, any()} when Msg::atom(),
                                      Method::put|get|delete,
                                      Key::binary(),
                                      ProcessingTime::non_neg_integer()).

@yosukehara Take a look for review. Once the review finished, I will get to implement.

[yosukehara]

@mocchira I've reviewed your proposal and agreed to it.