Snap server rebase

[garyschulte]

PR description

Initial implementation of snap server using streaming flat database. Obseletes #5887

This PR adds snap server functionality behind a feature flag --Xsnapsync-server-enabled which enables besu to serve snap/1 protocol fully.

Snap server has a specific set of storage requirements:

• --data-storage-format=BONSAI, snap server only works with bonsai storage
• --Xbonsai-code-using-code-hash-enabled, snap server nodes need to store code by code hash in order to serve snap code requests
• --Xsnapsync-synchronizer-flat-db-healing-enabled, Snap server nodes must be snap synced with full flat db enabled, (or the node has to have been full-synced to have a fully-flattened db)

Given two of these requrirements (code-by-code-hash and flat-db-healing), most nodes will have to resync in order to serve snap data, with these parameters in the config file:

data-storage-format="BONSAI"
Xsnapsync-server-enabled=true
Xbonsai-code-using-code-hash-enabled=true
Xsnapsync-synchronizer-flat-db-healing-enabled=true

Performance Impact

Testing the initial implementation has shown serving snap data has slight impact to execution performance in normal network conditions, on the order of 10%

Load Test

To get an idea of the cost to execution performance in less than ideal conditions, besu was tested with the load of 2 static syncing peers (besu serving all of the sync load to both clients simulatenously). In this case besu showed roughly 2x the block execution times of besu nodes which were not serving snap data.

Normally besu would only serve some fraction of the load of syncing clients. We could push this further with more static syncing clients, but this was sufficient to get signal about the performance impact.

Once the snap clients completed their worldstate sync, besu performance returned to the mean immediately

Read vs write during this time:

Testing Clients

Besu 23.4.x
Geth 13.14
Nethermind 1.25.4

What's Next

• Optimizations. This is an initial implementation of snap server and now that we are satisfied in its correctness, we will turn to optimizing to minimize its impact on block execution times.

• A follow-on PR will add a storage subcommand to convert existing databases to fully flattened and code-by-code-hash storage, without the need for a resync.

Fixed Issue(s)

fixes #3165

Thanks for sending a pull request! Have you done the following?

• Checked out our contribution guidelines?
• Considered documentation and added the doc-change-required label to this PR if updates are required.
• Considered the changelog and included an update if required.
• For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Most advanced CI tests are deferred until PR approval, but you could:

• locally run all unit tests via: ./gradlew build
• locally run all acceptance tests via: ./gradlew acceptanceTest
• locally run all integration tests via: ./gradlew integrationTest
• locally run all reference tests via: ./gradlew ethereum:referenceTests:referenceTests

[matkt]

Added some questions but in general I like the implementation 💯

[matkt]

what we are doing in this case ? not sure to understand if we are stopping the snapserver ?

[garyschulte]

for now, snap server will operate as normal with just a warning. Essentially we will just serve empty code requests if the storage hasn't migrated. If we think it is necessary we can refuse to start snap server if we are not in code-by-code hash storage mode. I was thinking that we would address this corner case via the storage migration subcommand PR

[matkt]

I think we need to limite the number of trie nodes to serve by request as Geth. Because it seems to be a possible dos vector

[garyschulte]

I will 👀 what geth is doing here. Just to clarify, what about trie nodes queries makes you think they are disproportionately expensive? I would think range requests would represent more resource consumption, especially considering we are not directly using an iterator but rather collecting in memory to serve the requests.

[matkt]

I ask you 30000 trie nodes that does not exist. You will still have to read in base to find out if the trie node exists. So I can spam you as much as I want. The Range is not the same I ask you for a range and the peers will control what it returns in this range according to the number of leaves. To test. it is an intuition that I have it could be wrong

[macfarla]

few nits mostly for logging

[macfarla]

nit: would be nice for logs to be consistent with snap server vs SnapServer

[macfarla]

nit: this log message is duplicated, could be refactored to a method

[garyschulte]

I am going to skip this one for now since all of the return messages at least subtly different.

[macfarla]

can this annotation be removed now

[garyschulte]

apparently yes 👍