Excessive memory consumption (>40G) after upgrading from 1.27.0 to 1.30.3

[Ragnaruk]

Description
After upgrading Nethermind from 1.27.0 to 1.30.3 it started consuming (leaking?) memory at a pretty alarming rate.

I am using Nethermind as an RPC node, and it looks like it is at least somewhat related ­— without any load memory consumption seems stable.

Rate of requests if needed:

eth_getCode: ~2k/min
eth_getTransactionReceipt: ~2k/min
trace_transaction: ~600/min
eth_blockNumber: ~20/min
eth_getBlockByNumber: ~20/min

Screenshots

Desktop (please complete the following information):

• Operating System: Rocky Linux
• Version: 8.10 (Green Obsidian)
• Installation Method: Docker
• Consensus Client: Prysm v5.2.0

[Ragnaruk]

Looks like the problem is with the trace_transaction requests.

Disabling them stabilized memory usage, but didn't reduce it. After a restart, everything returned to normal.

Looks like either the caching is too aggressive or the memory simply isn't released.

[kamilchodola]

Hi @Ragnaruk !

Thanks for the report! We will take a look at that as a priority to fix it asap.

[kamilchodola]

@brbrr Please reproduce it on our infra.

@benaadams Did we had any performance optimizations for tarce calls in last versions?