Database caching for uncompressed data #1137
Assignees
Comments
|
use rocksdb lru cache |
xgreenx
added a commit
that referenced
this issue
Apr 27, 2023
Closes #1137 The PR duplicates most of the changes from the #1157 but for the `master` branch. It adds two e2e tests, but they fail because of new changes in the `fuel-vm`(the change to the `PC` register). It is not critical, but replacing them with the correct scripts later would be nice. - One test uploads the contract with a huge large state and executes it. The test takes a lot of time to execute and highlights that we have performance problems with the state calculation: #1143 - Another test is more of a base for spamming the testnets. We can put any hex-encoded transaction and dry run it on the network. It contains a simple transfer transaction right now. The change improves the `fuel-core snapshot` command to accept the arguments, and you can either snap the whole state of the blockchain via the `everyone` command, or the state only of the contract. In the follow-up PRs we can snapping of the coins and messages too. Optimizations: - Use a lru cache to store the uncompressed data in the DB. The size is configurable via CLI `max-database-cache-size`. - Added a `DryRun` execution type that skips the coinbase `Mint` transaction. In the future, we can skip some non-actual steps too. - Updated the `PrometheusExtension` to not use the lock. - Removed usage of the tracing extension in the graph QL because we have `PrometheusExtension` to gather the statistic about requests. #1152 removes the honeycomb, so I didn't duplicate it here. - Replaced the `tokio::task::spawn_blocking` + `Semaphore` by `tokio_rayon::spawn_fifo`. - Replaced the concatenated `column_key` with the array for separate hash maps per column in the `MemoryStore` and `MemoryTransactionView`. - Replaced the `Vec<u8>` in the KV API by the `Arc<Vec<u8>>`, because in most cases, we use it only for deserialization, and the memory database can continue to store the type. It helps reduce the number of clones. - Used `BTreeMap` in the `MemoryStore`, because it allows to have a more optimized `iter_all` method. Without this change, a new e2e test fails with a timeout. - Dry run doesn't wait for relayer anymore.
crypto523
pushed a commit
to crypto523/fuel-core
that referenced
this issue
Oct 7, 2024
Closes FuelLabs/fuel-core#1137 The PR duplicates most of the changes from the FuelLabs/fuel-core#1157 but for the `master` branch. It adds two e2e tests, but they fail because of new changes in the `fuel-vm`(the change to the `PC` register). It is not critical, but replacing them with the correct scripts later would be nice. - One test uploads the contract with a huge large state and executes it. The test takes a lot of time to execute and highlights that we have performance problems with the state calculation: FuelLabs/fuel-core#1143 - Another test is more of a base for spamming the testnets. We can put any hex-encoded transaction and dry run it on the network. It contains a simple transfer transaction right now. The change improves the `fuel-core snapshot` command to accept the arguments, and you can either snap the whole state of the blockchain via the `everyone` command, or the state only of the contract. In the follow-up PRs we can snapping of the coins and messages too. Optimizations: - Use a lru cache to store the uncompressed data in the DB. The size is configurable via CLI `max-database-cache-size`. - Added a `DryRun` execution type that skips the coinbase `Mint` transaction. In the future, we can skip some non-actual steps too. - Updated the `PrometheusExtension` to not use the lock. - Removed usage of the tracing extension in the graph QL because we have `PrometheusExtension` to gather the statistic about requests. FuelLabs/fuel-core#1152 removes the honeycomb, so I didn't duplicate it here. - Replaced the `tokio::task::spawn_blocking` + `Semaphore` by `tokio_rayon::spawn_fifo`. - Replaced the concatenated `column_key` with the array for separate hash maps per column in the `MemoryStore` and `MemoryTransactionView`. - Replaced the `Vec<u8>` in the KV API by the `Arc<Vec<u8>>`, because in most cases, we use it only for deserialization, and the memory database can continue to store the type. It helps reduce the number of clones. - Used `BTreeMap` in the `MemoryStore`, because it allows to have a more optimized `iter_all` method. Without this change, a new e2e test fails with a timeout. - Dry run doesn't wait for relayer anymore.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: