[leo_storage][leo_mq] A replication request can be blocked for 30 seconds

[mocchira]

Found through discussions on #702.
Reference: #702 (comment)

This can be reduced by

• Checking the liveness of dependent nodes while receiving at https://github.com/leo-project/leofs/blob/1.3.4/apps/leo_storage/src/leo_storage_replicator.erl#L107-L127 preodically
• Exiting when detecting dependent nodes that still don't send a reply back go down

[mocchira]

It turned out that now

• No info which nodes send a successful reply
• SubParent calling the loop function holds the result and state (how many successes are there and so on).

That said, more tweaks will be needed to fix this issue.

• Grasp which nodes send a successful reply
• Add the node info to the message sent from callees

[mocchira]

Now https://github.com/leo-project/leo_konsul (leo_redundant_manager replacement) has been under active development and once this works finished, we may be able to fix this issue in a more easy way like below

    % Add the code below into where waiting for responses from recipients
    {ok, Ref} = leo_konsul:subscribe_on_node_state_change(self()),
    receive
        {Ref, node_state_change, Node, _PrevState, stop} ->
            % Give up waiting for the response from Node ...
    after ->
        % ...
    end,

We will postpone this issue to 1.4.1 once it turns out that the above idea can be feasible.

[mocchira]

Since leo_konsul will be introduced on 2.0.0, this issue depending on it also should postpone to 2.0.0.