[leo_storage] Improve the performance of the recover-node

[yosukehara]

We've found the performance of the recover-node is not good so we need to improve its performance to immediately fix inconsistent objects.

In current LeoFS' version - v1.2.21, after executing the recover-node, leo_storage assigns recovery messages to all storage nodes to avoid lost inconsistent objects, which means a recovered object is duplicated which depends on a routing-table(ring).

I'll improve that leo_storage reduces recovery messages and is able to do consensus between storage-nodes to ensure solving inconsistency.

[yosukehara]

I've improved the performance but there is still room for improvement.
Tomorrow, I'll improve leo_mq's consumer to be able to increase a number of consumption messages per a minute more and more.

[windkit]

Now the recovery process is more or less bound by disk I/O

[yosukehara]

The performance has been dramatically increased. 1M objects was recovered from 12hours to 7.5hours, finally 20min.

[yosukehara]

In this morning, we've tested the recover-node again. There is no issue. So I've closed this.

[yosukehara]

We've found a leo_mq's issue as below:

[E] storage_1@127.0.0.1 2016-05-10 12:49:08.421060 +0900    1462852148  
        leo_mq_server:handle_call/3 203 
        {noproc,{gen_server,call,[mq_persistent_node_message_0,first,30000]}}