Fix high memory usage of PeerManagerActor:: add_route_back

[pmnoxx]

While investigating memory leaks in Rust code I noticed that memory usage by PeerManager keeps growing.
PeerManager leak is within: (rust)func chain/network/src/lib.rs <near_network::peer_manager::PeerManagerActor as actix::handler::Handler<near_network::types::RoutedMessageFrom>>::handle 5577784d767f: 38.31MiB/22467

After 8-10 hours I saw 22k allocations, which were 38mb total. All those allocations are caused by PeerManaagerActor::routing_table.add_route_back. ROUTE_BACK_CACHE_SIZE size is limited to 1m, which means that by the time it becomes full, it will use around 1.7gb of memory, which is suboptimal.

[pmnoxx]

@bowenwang1996 @SkidanovAlex Should we decrease ROUTE_BACK_CACHE_SIZE?

[bowenwang1996]

@mfornet is this expected?

[mfornet]

@mfornet is this expected?

I don't doubt it.

We are currently relying in this cache for most of the routed messages!
We will stop overusing this cache after #3088 lands, but this is not in my scope right now. If this is high priority I can try to allocate some time to finish the PR. The core logic is implemented, but some effort is required to make it work with latest changes regarding upgradability.

In the short term, if this is a pressing issue, it should be safe to decrease the size of the cache temporarily, it is very likely that oldest entries are messages from peers that are currently disconnected.

[bowenwang1996]

Okay let's shrink the size to 100k then. @mfornet this shouldn't cause any performance issues right?

[mfornet]

Okay let's shrink the size to 100k then. @mfornet this shouldn't cause any performance issues right?

No, it should be fine!