Regenesis support #1693

MujkicA · 2024-02-21T13:33:02Z

The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files.

Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format.

The snapshot command now has the '--output-directory' for specifying where to save the snapshot.

Run command

The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group.

Contract States and Balances

Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees.

Closes #1450 Separates the loading of the initial chain state from loading `ChainConfig`. --------- Co-authored-by: Salka1988 <[email protected]> Co-authored-by: Brandon Vrooman <[email protected]> Co-authored-by: segfault_magnet <[email protected]>

Co-authored-by: Salka1988 <[email protected]> Co-authored-by: segfault_magnet <[email protected]> Co-authored-by: Ahmed Sagdati <[email protected]> Co-authored-by: Brandon Kite <[email protected]> Co-authored-by: Brandon Vrooman <[email protected]> Co-authored-by: Mitchell Turner <[email protected]> Co-authored-by: Green Baneling <[email protected]>

Imports chain state via streams and enables regenesis

…egenesis-support

CHANGELOG.md

Cargo.lock

bin/e2e-test-client/src/tests/script.rs

crates/chain-config/src/config/coin.rs

crates/chain-config/src/config/contract.rs

crates/chain-config/src/config/snapshot_metadata.rs

tests/tests/balances.rs

This reverts commit ca96107.

…egenesis-support

crates/chain-config/src/config/contract_state.rs

crates/fuel-core/src/service/genesis/workers.rs

…egenesis-support

xgreenx · 2024-03-05T22:46:49Z

crates/fuel-core/src/database/genesis_progress.rs

+    pub fn add_coin_root(&mut self, root: MerkleRoot) -> Result<()> {
+        StorageMutate::<GenesisCoinRoots>::insert(self, &root, &())?;
+        Ok(())
+    }
+
+    pub fn add_message_root(&mut self, root: MerkleRoot) -> Result<()> {
+        StorageMutate::<GenesisMessageRoots>::insert(self, &root, &())?;
+        Ok(())
+    }
+
+    pub fn add_contract_root(&mut self, root: MerkleRoot) -> Result<()> {
+        StorageMutate::<GenesisContractRoots>::insert(self, &root, &())?;
+        Ok(())
+    }


It seems we don't need a separate process step during regenesis, and we can simply iterate over the storage for coins, messages, and contracts, and call the root function. It will speed up the most critical part of interaction and slow down the last step that creates a genesis block.

It can be done in the follow up PR.

xgreenx · 2024-03-05T22:47:53Z

crates/fuel-core/src/database/genesis_progress.rs

+    ContractsRoot,
+}
+
+pub struct GenesisMetadata;


Based on the comment below, it seems we can remove all tables except this one=)

xgreenx

Thank you for the huge work, it is really nice! Sorry for so many conflicts=) I added several TODOs that we can address in the follow-up PRs=)

I'm waiting for performance results and e2e from https://github.com/FuelLabs/fuel-core-regenesis to see how fast is regenesis=) But it is really great that you implemented interrupt/resume functionality.

@Dentosal

## [Version 0.24.0] ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 **Full Changelog**: v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 * Release v0.24.0 by @xgreenx in #1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in #1792 **Full Changelog**: v0.23.0...v0.24.1

@Dentosal

## [Version 0.24.0] ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 * Release v0.24.0 by @xgreenx in FuelLabs/fuel-core#1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in FuelLabs/fuel-core#1792 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.1

MujkicA and others added 4 commits December 20, 2023 16:23

merge master

16817f2

Stream reader for chain state (#1519)

a4021f1

Imports chain state via streams and enables regenesis

MujkicA added the regenesis label Feb 21, 2024

MujkicA assigned MujkicA and segfault-magnet Feb 21, 2024

MujkicA added 9 commits February 21, 2024 18:22

merge master part 1

e9324c4

fmt

cfb7eab

merge

fd725ee

off chain genesis

f33eb56

off chain generated utxos

590793e

remove optional fields from configs

290c932

Merge branch 'master' of github.com:FuelLabs/fuel-core into feature/r…

503f177

…egenesis-support

merge]

c3818fa

put rand behind std flag

ca96107

xgreenx reviewed Feb 28, 2024

View reviewed changes

MujkicA added 4 commits February 28, 2024 13:03

Revert removing optional fields from config

81a0a98

This reverts commit ca96107.

revert

5c74cb0

Merge branch 'master' of github.com:FuelLabs/fuel-core into feature/r…

5d95ad9

…egenesis-support

update contract latest utxo insertion

edf72fa

MujkicA marked this pull request as ready for review February 28, 2024 12:24

Dentosal reviewed Feb 28, 2024

View reviewed changes

crates/chain-config/src/config/contract_state.rs Outdated Show resolved Hide resolved

Dentosal reviewed Feb 28, 2024

View reviewed changes

crates/fuel-core/src/service/genesis/workers.rs Outdated Show resolved Hide resolved

MujkicA added 6 commits February 29, 2024 22:01

review comments

8188f9e

fix chain config

650257d

merge

2bc8303

restore missing contract from dev config

10077ca

merge now, fix tests later

88938c8

Merge branch 'master' of github.com:FuelLabs/fuel-core into feature/r…

a4b98c6

…egenesis-support

MujkicA and others added 4 commits March 4, 2024 20:18

fix build

4f2f988

human readable storage slot value

caded52

fix p2p tests

2535d48

Merge branch 'master' of github.com:FuelLabs/fuel-core into feature/r…

fd14783

…egenesis-support

MujkicA requested a review from xgreenx March 5, 2024 11:50

MujkicA and others added 2 commits March 5, 2024 21:35

Merge branch 'master' into feature/regenesis-support

037686c

Added comments regarding the future work

4e12b37

xgreenx reviewed Mar 5, 2024

View reviewed changes

xgreenx mentioned this pull request Mar 5, 2024

Don't save roots of the coins, messages, contract into temporary tables #1734

Closed

Added mroe TODOS

b5d8e75

xgreenx changed the title ~~regenesis support~~ Regenesis support Mar 5, 2024

xgreenx approved these changes Mar 5, 2024

View reviewed changes

xgreenx merged commit 36e3363 into master Mar 5, 2024
30 checks passed

xgreenx deleted the feature/regenesis-support branch March 5, 2024 23:17

This was referenced Mar 7, 2024

Design and implement the flow for Messages in the genesis block #816

Closed

Re-genesis feature #1307

Closed

bvrooman mentioned this pull request Mar 18, 2024

Upgradability of the network #1511

Closed

8 tasks

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.0 #1791

Merged

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.1 #1793

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regenesis support #1693

Regenesis support #1693

MujkicA commented Feb 21, 2024 •

edited by xgreenx

Loading

xgreenx Mar 5, 2024

xgreenx Mar 5, 2024

xgreenx left a comment

Regenesis support #1693

Regenesis support #1693

Conversation

MujkicA commented Feb 21, 2024 • edited by xgreenx Loading

Snapshot command

Run command

Contract States and Balances

xgreenx Mar 5, 2024

Choose a reason for hiding this comment

xgreenx Mar 5, 2024

Choose a reason for hiding this comment

xgreenx left a comment

Choose a reason for hiding this comment

MujkicA commented Feb 21, 2024 •

edited by xgreenx

Loading