Fluffy and the EVM

[kdeme]

While a functional state network in Portal is still far away, we can already investigate how easy it is to link the EVM code to Fluffy.
This task is about investigating how convenient it is to add current EVM functionality to Fluffy.
No input needs to come from the network (there isn't one), it is rather to see of we can just drop in the EVM in Fluffy in its current state.

[mratsim]

Ideally, fluffy just uses the EVMC API ( https://github.com/status-im/nim-evmc ) so that it's easier to spec alternative Portal Network implementations.

[bhartnett]

This task is no longer required because the current plan is to connect the Fluffy state bridge to a running execution client. This would be required to keep up with the latest state changes for the latest blocks and therefore we don't need to run the EVM inside the Fluffy bridge. We will likely call trace_replayBlockTransactions to get the stateDiffs for each block from the Fluffy bridge in order to collect the state changes.

[kdeme]

While not core functionality of a pure Portal node, when we want to build a light client with full EL JSON-RPC API, we will need the EVM to execute certain txs (e.g. for estimateGas)?

[bhartnett]

Yes, you are right we would need the EVM for the following endpoints which are listed as part of the portal network spec:
eth_call
eth_estimateGas

Thanks for the reminder. I was thinking about the state bridge when closing this task. I'm going to reopen it with the goal of including the Nimbus EVM in Fluffy at some point.

[bhartnett]

Ideally we should pull out the Nimbus EVM into a separate library that we can then import into Fluffy either directly or using nim-evmc.

The database layer would need to be swapped out in order to get the data from the portal state network. In order to do this, I believe we should aim to make the Nimbus EVM more modular. I would suggest that we decouple the account ledger and database from the layers above by defining interfaces between them. We could then create another implementation of the state or database layer for the portal network.

For example we could have interfaces:
EVM executor -> state interface -> database interface

In Nimbus we have something like this:
EVM executor -> accounts ledger -> rocksdb

After creating interfaces Fluffy look something like:
EVM executor -> state interface -> portal state network

Revm has a modular architecture which we could get some inspiration from.
See the documentation for the revm state and database layers here:

• https://bluealloy.github.io/revm/crates/revm/state.html
• https://bluealloy.github.io/revm/crates/primitives/database.html

[bhartnett]

One design consideration we need to think about as we integrate the EVM into Fluffy is what database should we use for the temporary state that may be created when writing to contract storage while executing a transaction. At present we only need to implement eth_call and eth_estimateGas which are read only and therefore should throw away any temporary state changes but even so the evm will still need to store temporary updates in a db. Perhaps we should use an in memory database for this purpose to avoid pulling in RocksDb as is used in Nimbus.

[arnetheduck]

+1 on evmc as a starting point

temporary state that may be created when writing to contract storage while executing a transaction

technically, this already happens - the way eth1 works is that it collects all changes in memory before writing to disk, before a final commit or rollback moves the accumulated change set atomically to disk - apart from pure spec requirements (in case of transaction errors etc), the easiest way to imagine why this is so is that if we were to process a block in which some of the transactions are broken, we don't want that speculative execution to be written to disk - the same applies here.

[bhartnett]

+1 on evmc as a starting point

What would be the benefit of using evmc? My understanding is that it would allow us to link in different EVMs into Fluffy or Nimbus if we wanted to but I'm not sure that helps us much with this task because Fluffy will likely need a special purpose EVM that reads state from the portal network.

temporary state that may be created when writing to contract storage while executing a transaction

technically, this already happens - the way eth1 works is that it collects all changes in memory before writing to disk, before a final commit or rollback moves the accumulated change set atomically to disk - apart from pure spec requirements (in case of transaction errors etc), the easiest way to imagine why this is so is that if we were to process a block in which some of the transactions are broken, we don't want that speculative execution to be written to disk - the same applies here.

Yes true but if I was to pull in the Nimbus EVM as implemented then I would need to pull RocksDb into Fluffy which isn't ideal. We already use Sqlite and we would like to keep Fluffy as light weight as possible. For this reason I'll likely need to introduce some interface or similar into the Nimbus EVM (for now just in a feature branch) so that we can configure it to not use RocksDb and instead fetch state data from portal network but also write temporary state changes in memory somehow.

[arnetheduck]

What would be the benefit of using evmc?

it does more or less what you want, which is to isolate the nimbus evm from its database - it's also implemented as a compile-time flag which means that it doesn't burden nimbus with abstraction / interface cost at runtime - each interface and abstraction has an associated cost and if there's one thing nimbus has right now, it's too many layers of abstraction and interfaces that don't have a specific reason to exist.

That said, there's something called a memory backend for nimbus which also disables the rocksdb backend already - look for newCoreDbRef DefaultDbMemory in tests and code - if you really want a database abstraction layer, this is it.

[bhartnett]

What would be the benefit of using evmc?

it does more or less what you want, which is to isolate the nimbus evm from its database - it's also implemented as a compile-time flag which means that it doesn't burden nimbus with abstraction / interface cost at runtime - each interface and abstraction has an associated cost and if there's one thing nimbus has right now, it's too many layers of abstraction and interfaces that don't have a specific reason to exist.

That said, there's something called a memory backend for nimbus which also disables the rocksdb backend already - look for newCoreDbRef DefaultDbMemory in tests and code - if you really want a database abstraction layer, this is it.

Thanks, I see what you mean now and I agree this is a good idea. I agree about the abstractions as well and I think it's better that we don't add more if not needed.

Using the evmc host interface in Fluffy we should be able to return data from portal using synchronous calls as the API is not an async API. This will be a good starting point but I guess at some point we might want to update the Nimbus EVM to execute in an async way perhaps using another compile time flag so that state access over the portal network don't block. For example if we have a transaction that accesses multiple state values we should be able to do this concurrently if we wrap the evmc host interface calls to make them async.

[bhartnett]

I've run into some issues while trying to use the Nimbus EVM via the EVMC API. It appears to me that even when the evmc_enabled flag is set, the Nimbus EVM isn't implemented correctly in a way where it only accesses the standard EVMC interfaces which means (if I'm not mistaken) I can't use it as planned in it's current state.

For example in evmc_vm_glue.nim there is an evmcExecute proc which is meant to be used to execute an evmc_message but in this code the message isn't used (is ignored), see here: https://github.com/status-im/nimbus-eth1/blob/master/nimbus/transaction/evmc_vm_glue.nim#L19

[kdeme]

Hmm, yes, it has this big todo written there also: https://github.com/status-im/nimbus-eth1/blob/master/nimbus/transaction/evmc_vm_glue.nim#L25-L26

Can you make an issue regarding this? Seems our implementation is incomplete. All I found was this issue regarding testing: #475

[bhartnett]

Yes I noticed that as well.

The other even bigger problem is that our EVM doesn't always use the EVMC interface when accessing the database is many places so it appears that fixing it would be a decent amount of work. I discussed this with the team in the weekly meeting yesterday and the conclusion was that for now we don't want to make changes to the Nimbus EVM until we release Nimbus to mainnet. With that in mind my current plan is to use evmone via the EVMC interface in Fluffy. In the future we can update the Nimbus EVM to fully follow the EVMC spec and then swap out evmone for Nimbus EVM in Fluffy.

Sure, I'll create a new issue describing the remaining work to be done on EVMC for Nimbus.

[bhartnett]

New issue created, see here: #2803

[bhartnett]

I've found a useful tool in the evmone project which can be used to test a EVMC EVM. I've tested it on both evmone and Nimbus EVM.

$ ./evmc-vmtester /home/user/development/ethereum/evmone/build/lib/libevmone.so
EVMC VM Tester 12.0.0
Testing /home/user/development/ethereum/evmone/build/lib/libevmone.so

[==========] Running 10 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 10 tests from evmc_vm_test
[ RUN      ] evmc_vm_test.abi_version_match
[       OK ] evmc_vm_test.abi_version_match (0 ms)
[ RUN      ] evmc_vm_test.name
[       OK ] evmc_vm_test.name (0 ms)
[ RUN      ] evmc_vm_test.version
[       OK ] evmc_vm_test.version (0 ms)
[ RUN      ] evmc_vm_test.capabilities
[       OK ] evmc_vm_test.capabilities (0 ms)
[ RUN      ] evmc_vm_test.execute_call
[       OK ] evmc_vm_test.execute_call (0 ms)
[ RUN      ] evmc_vm_test.execute_create
[       OK ] evmc_vm_test.execute_create (0 ms)
[ RUN      ] evmc_vm_test.set_option_unknown_name
[       OK ] evmc_vm_test.set_option_unknown_name (0 ms)
[ RUN      ] evmc_vm_test.set_option_empty_value
[       OK ] evmc_vm_test.set_option_empty_value (0 ms)
[ RUN      ] evmc_vm_test.set_option_unknown_value
[       OK ] evmc_vm_test.set_option_unknown_value (0 ms)
[ RUN      ] evmc_vm_test.precompile_test
[       OK ] evmc_vm_test.precompile_test (0 ms)
[----------] 10 tests from evmc_vm_test (0 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 1 test suite ran. (0 ms total)
[  PASSED  ] 10 tests.

When I run it against the nimbus evm I get this:

$ ./evmc-vmtester /home/user/development/status-im/nimbus-eth1/libevm.so
EVMC VM Tester 12.0.0
Testing /home/user/development/status-im/nimbus-eth1/libnimbusevm.so
Segmentation fault (core dumped)

This is to be expected I guess because the Nimbus EVM doesn't yet support being loaded as a stand alone EVMC EVM.

[bhartnett]

I've made some progress on this one. eth_call is now working in this feature branch: https://github.com/status-im/nimbus-eth1/tree/fluffy-evmc-eth_call

Both the history network and state network are required to be enabled as well as the evm. Note that this is just a proof of concept so far and the code still needs to be refined and improved a bit more.

It works by first looking up the block header via the history network, then the state root is passed into the PortalEvmState type which is used to look up each piece of state on demand from the portal state network while the evm is executing the bytecode. The evm runs fully in memory and is ephemeral for that reason. For each transaction/call each piece of state is only fetched from the state network once, after which the in memory value is used and updated.

The existing content cache which is already running in Fluffy also provides some nice performance benefits for eth_call because the history and state content is cached across requests so when issuing requests that may lookup the same or similar data then less network requests are required.

This current implementation uses the EVMC interface and for now we are using evmone. The remaining work is:

• Make passing in an EVMC evm shared library configurable on the command line so we can in theory use any compatible EVMC vm.
• Testing: Run the EVM state tests against the Fluffy EVM to verify the implementation works for all scenarios (at least supporting calls for now)
• Fix the Nimbus EVM so that it is fully EVMC compatible as a EVMC VM.
• Make the Nimbus EVM the default in Fluffy if the cli parameter is not specified.