High memory usage with RocksDB (ParityDB is fine)

[nazar-pc]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Yet another thing I noticed after recent Substrate upgrade is that syncing a chain with ~50G of actual history (mostly full blocks in terms of size, but little to no compute) node consumes a lot of RAM.

Specifically on my dev machine with 128G of RAM node consumes ~17.5G of RAM. Turns out it consumes much less on machines with less ram (4.1G on 16G machine), so must be some optimistic caching then.

I was trying to figure out the source of that and started node with ParityDB just to try it out and found that with ParityDB node only uses 2.5G of RAM in 128G machine, which is great!

I know that RocksDB has internal caches and such, but I still found such a high usage quite surprising, so maybe something can be tuned about it on Substrate (or rather kvdb) level?

Steps to reproduce

No response

[nazar-pc]

Actually, while ParityDB seems to use less RAM generally, I still saw strange behaviors depending on how nodes are connected.

I saw than on the same kind of machine one node used ~1.8G/16G of RAM and another ~8.9G/16G. Syncing the same chain at the same time being connected together on a small network.

Will do more research, but I think this is distinct from #11503 (also happens during sync when there are no blocks produced or transactions submitted).

[bkchr]

Did you tried running with --database-cache-size 0?

[nazar-pc]

No and it will only affect RocksDB, I think there must be something DB-independent, I'll share more info once I have it.

[bkchr]

--database-cache-size is only used by RocksDB, but okay that it didn't helped.