Prepare the codebase to use base gas price during block production #1642 #1752

MitchTurner · 2024-03-13T21:25:45Z

Introduce new ProducerGasPrice interface for getting gas price for a given block.

Create simple StaticGasPrice impl for parity with current behavior.

xgreenx · 2024-03-15T17:19:08Z

crates/fuel-core/src/schema/gas_price.rs

        let latest_block: Block<_> = query.latest_block()?;
        let block_height = u32::from(*latest_block.header().height());
+        let gas_price = gas_price_provider
+            .known_gas_price(block_height.into())


Is that gas price used by the block producer at the last block, or is that the gas price for the next block?

If it is gas price used for the last block, then maybe we can get it from the Mint transaction=)

xgreenx · 2024-03-15T17:20:04Z

crates/fuel-core/src/schema/gas_price.rs


-        let config = ctx.data_unchecked::<GraphQLConfig>();
-        let gas_price = config.min_gas_price.into();
+        let latest_block: Block<_> = query.latest_block()?;


Getting Block is super expensive here, maybe we can get the latest height?=D

xgreenx · 2024-03-15T17:23:24Z

crates/services/producer/src/block_producer/gas_price.rs

+    fn gas_price(&self, params: GasPriceParams) -> Option<u64>;
+}
+
+pub struct MockProducerGasPrice {


Maybe we can use mockall instead?

I dislike mockall.

We have a lot of tests in our codebase that test the internals of our code rather than the externals. We do that with mockall's expect_, with, times, and similar methods. It's true that mockall would give us a struct that is basically identical to what I added, but it also gives us a bunch of things that are used to write bad tests, so I avoid it.

If you don't like these functions, you can not use them=)

If you have a nice tool with some feature that you don't need, simply don't use them=) Auto-generation structures for testing simplify supporting of this code in the future.

If you worry that in the future someone will use expect and other methods, then usage of the custom structure will not solve that=D The person simply will use mockall and will use expect because he wants to=)

xgreenx · 2024-03-15T17:25:59Z

crates/services/producer/src/block_producer/tests.rs

+    // assert_eq!(mint_tx.gas_price, gas_price);
+}
+
+// TODO: Add test that checks the failure case for gas_price after `Executor` refactor


I didn't get why we can't implement this test now. Could you elaborate more on it, please?=)

You're right. I've implemented it.

xgreenx · 2024-03-15T17:26:37Z

crates/services/producer/src/block_producer/tests.rs

@@ -203,12 +210,44 @@ async fn production_fails_on_execution_error() {
    );
 }

+// TODO: Add test that checks the gas price on the mint tx after `Executor` refactor


We can test for now that we are passing the gas price from the GP to the executor.

xgreenx · 2024-03-15T17:34:29Z

crates/services/txpool/src/txpool.rs

+}
+
+#[derive(Debug, Clone)]
+pub struct MockTxPoolGasPrice {


Why not to use mockall?

xgreenx · 2024-03-15T17:35:15Z

crates/services/txpool/src/txpool.rs

+}
+
+#[derive(Debug, Clone)]
+pub struct MockTxPoolGasPrice {


Mocked types should be defined in the tests

xgreenx · 2024-03-15T17:37:41Z

crates/services/txpool/src/txpool/tests.rs


    let (_, gas_coin) = context.setup_coin();
    let tx = TransactionBuilder::script(vec![], vec![])
        .script_gas_limit(GAS_LIMIT)
        .add_input(gas_coin.clone())
        .finalize_as_transaction();
    let mut txpool = context.build();
-    let tx = check_unwrap_tx(tx, &txpool.config).await;
+    let tx = check_unwrap_tx(tx, &txpool.config, gas_price).await;


The check_unwrap_tx function is a test helper function. If you are using zero gas price everywhere, it is better to use zero inside of this function without modifying the signature.

In the tests where gas price matters, you can use another function=)

xgreenx · 2024-03-15T17:40:20Z

crates/services/txpool/src/txpool.rs

@@ -76,6 +76,33 @@ pub struct TxPool<ViewProvider> {
    database: ViewProvider,
 }

+pub trait TxPoolGasPrice {
+    fn gas_price(&self, block_height: BlockHeight) -> Option<GasPrice>;


Hmm, I think the port should only request either the price from the last block or for the next block. Overwise, it is harder to understand what your expectations are.

That's what I started with. I definitely think that would be a cleaner interface, but the problem is I don't know what "current block" really means. Like every time you call it it gives a new gas price? or does it have some kind of tracking behind the scenes to know when a block was committed and now we're on to the next block?

I actually used this:

pub struct GasPriceParams { block_height: BlockHeight, }

instead of BlockHeight on the ProducerGasPrice. Mostly because, honestly, I don't really know what the method should take yet, but BlockHeight seems like a good place to start, since it's absolute.

xgreenx · 2024-03-15T17:40:29Z

crates/services/txpool/src/txpool.rs

@@ -76,6 +76,33 @@ pub struct TxPool<ViewProvider> {
    database: ViewProvider,
 }

+pub trait TxPoolGasPrice {
+    fn gas_price(&self, block_height: BlockHeight) -> Option<GasPrice>;


BTW, comment would be helpful=D

xgreenx

I still think it is better to use the mockall for simple cases=)

@Dentosal

## [Version 0.24.0] ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 **Full Changelog**: v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 * Release v0.24.0 by @xgreenx in #1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in #1792 **Full Changelog**: v0.23.0...v0.24.1

@Dentosal

## [Version 0.24.0] ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 * Release v0.24.0 by @xgreenx in FuelLabs/fuel-core#1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in FuelLabs/fuel-core#1792 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.1

MitchTurner added 2 commits March 13, 2024 14:11

Add StaticGasPrice impl for ProducerGasPrice

f49484a

Use static gas price adapter in test harness

e0ae5d1

MitchTurner self-assigned this Mar 13, 2024

MAke gas_price fallible

e6f9253

MitchTurner changed the title ~~Prep for using base fee~~ Prep producer for using base fee Mar 13, 2024

MitchTurner changed the title ~~Prep producer for using base fee~~ Prepare the codebase to use base gas price during block production #1642 Mar 13, 2024

MitchTurner added 6 commits March 13, 2024 16:49

Remove TODO, update CHANGELOG

7beee61

Sort Cargo.toml

55ac2ee

Add unimplemented tests

7eadf7a

Refactor gas_price for configs

bd69316

Use static gas price impl in tx pool code

d4eb224

Add GraphQL stuff, fix tests

51e03b5

MitchTurner marked this pull request as ready for review March 15, 2024 00:35

MitchTurner requested a review from a team March 15, 2024 00:38

Fix tests with too low max fee limit

08bbd4a

xgreenx reviewed Mar 15, 2024

View reviewed changes

MitchTurner and others added 9 commits March 15, 2024 15:27

Use Mint for latest gas price, use block height directly, add test

ab027a5

Remove unused trait method

c3c2455

Add docs to traits

05e2f02

Remove Clone req and use Arc instead

1a8994b

Modify test helpers

bced681

Move Mock types closer to tests

f238c3e

Add placeholder test and implement failure mode test

56ebbdd

Appease Clippy-sama

e13dada

Small renaming and removed Clone from Components

9a4c9d8

xgreenx approved these changes Mar 16, 2024

View reviewed changes

xgreenx enabled auto-merge (squash) March 16, 2024 11:38

xgreenx merged commit 57722a0 into master Mar 16, 2024
27 checks passed

xgreenx deleted the prep-for-using-using-base-fee-1642-2 branch March 16, 2024 11:42

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.0 #1791

Merged

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.1 #1793

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prepare the codebase to use base gas price during block production #1642 #1752

Prepare the codebase to use base gas price during block production #1642 #1752

MitchTurner commented Mar 13, 2024 •

edited by xgreenx

Loading

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

MitchTurner Mar 15, 2024 •

edited

Loading

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

MitchTurner Mar 16, 2024

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

xgreenx Mar 15, 2024

MitchTurner Mar 15, 2024 •

edited

Loading

xgreenx Mar 15, 2024

xgreenx left a comment

Prepare the codebase to use base gas price during block production #1642 #1752

Prepare the codebase to use base gas price during block production #1642 #1752

Conversation

MitchTurner commented Mar 13, 2024 • edited by xgreenx Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MitchTurner Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MitchTurner Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xgreenx left a comment

Choose a reason for hiding this comment

MitchTurner commented Mar 13, 2024 •

edited by xgreenx

Loading

MitchTurner Mar 15, 2024 •

edited

Loading

MitchTurner Mar 15, 2024 •

edited

Loading