Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move the DatabaseTransaction logic into StorageTransaction #1589

Closed
xgreenx opened this issue Jan 8, 2024 · 0 comments · Fixed by #1694
Closed

Move the DatabaseTransaction logic into StorageTransaction #1589

xgreenx opened this issue Jan 8, 2024 · 0 comments · Fixed by #1694
Assignees

Comments

@xgreenx
Copy link
Collaborator

xgreenx commented Jan 8, 2024

Problem overview

The current logic of the executor works with the Transaction trait that hides the implementation and how database transaction works.

/// The type is storage transaction and holds uncommitted state.
pub trait Transaction<Storage: ?Sized>:
    AsRef<Storage> + AsMut<Storage> + Send + Sync
{
    /// Commits the pending state changes into the storage.
    fn commit(&mut self) -> StorageResult<()>;
}

/// The storage transaction for the `Storage` type.
pub struct StorageTransaction<Storage: ?Sized> {
    transaction: Box<dyn Transaction<Storage>>,
}

impl<Storage: ?Sized> Transaction<Storage> for StorageTransaction<Storage> {
    fn commit(&mut self) -> StorageResult<()> {
        self.transaction.commit()
    }
}

Because implementation is hidden, the executor can't be sure that StorageMutate methods don't update the underlying storage. It is crucial for the fraud proofs, and it will be useful to have control over that for forkless upgrades.

Solution

The solution is to move the logic of the DatabaseTransaction to the fuel-core-storage crate. But because it is still up to the database implementation of how the final commitment is done, we will add several requirements:

  • The database may be only modified through the commit_changes method.
    #[impl_tools::autoimpl(for<T: trait> &mut T, Box<T>)]
    pub trait Modifiable {
        fn commit_changes(&mut self, changes: Changes) -> StorageResult<()>;
    }
    
    pub type Changes = HashMap<(u32, Vec<u8>), WriteOperation>;
    
    pub struct StorageTransaction<Storage> {
        changes: Changes,
        storage: Storage,
    }
    
    impl<Storage> StorageTransaction<Storage>
    where
        Storage: Modifiable,
    {
        pub fn commit(&mut self) -> StorageResult<()> {
            self.storage
                .commit_changes(core::mem::take(&mut self.changes))
        }
    }
  • The StorageMutate is only implemented for the StorageTransaction, and the Storage should only implement read methods(and only one mutate commit_changes method).
  • The commit_changes requires &mut self, meaning that only one service will exclusively own the database. Consiquencly, all other services should only have read access to the database. We haven't trouble us with that before, but we need to guarantee a consistent view of the database during other service operations. The AtomicView from Extract off chain logic from the executor #1579 helps with doing that:
    /// Provides a view of the storage at the given height.
    /// It guarantees to be atomic, meaning the view is immutable to outside modifications.
    pub trait AtomicView: Send + Sync {
        /// The type of the storage view.
        type View;
    
        /// Returns the view of the storage at the given `height`.
        fn view_at(&self, height: BlockHeight) -> StorageResult<Self::View>;
    
        /// Returns the view of the storage for the latest block height.
        fn latest_view(&self) -> Self::View;
    }

That solution:

  • Moves the logic of the storage transaction to the fuel-core-storage, providing control over that to the executor for fraud proofs and allowing upgrades via forkless upgrades.
  • It introduces database ownership that guarantees that commitment is done only by one service on a type system level.
  • It adds a consistent state view for read-only services.
  • The decision of how commit_changes works is done on the database level, instead of the transaction level. It allows implementing the State Rewind #451 a much easier way because we have only one place where it may happen(The Transaction trait is too abstract, and we need to create different database transactions types for different underlying database).
xgreenx added a commit that referenced this issue Jan 19, 2024
The change is related to
#1589.

The idea of the change is to start using the `AtomicView` inside of the
`TxPool` to generate consistent database representation during the
insertion of the transactions.
@xgreenx xgreenx self-assigned this Jan 24, 2024
xgreenx added a commit that referenced this issue Jan 27, 2024
Related to #1589 and
preparation for #1583.

All services that use the database for read-only purposes use
`AtomicView` now instead of direct access to the database.

- Removed `Relayer` from the `Verifier` because if it is not used for
now, plus it may not be needed because of the shared sequencer and its
consensus rules.
- Added verification of the transactions root hash into `Verifier`.
- Removed requesting of the one block from p2p, it is possible to use
range request for that purposes.
- Removed not used `get_sealed_header` and `get_sealed_block` method.
- Added the `latest_height` method to `AtomicView` because the database
always knows its latest height.
- Added customisation of the `Height` used by the `AtomicView`. In the
case of the relayer, we want to use `DaBlockHeight` as a primary key to
create a snapshot, while in the case of the Fuel's databases, we want to
use `BockHeight` as a primary key.
- Renamed `Executor` into `ExecutionInstance` and changed it to be a
one-time usable instance. It consumes the `self` to perform the
execution of the block. The instance has a view of the database and
execution options.
xgreenx added a commit that referenced this issue Feb 2, 2024
Closes #1568

The change splits the `Database` into 3 databases: 
- `Database<OnChain>` - Stores only data required for normal work of the
blockchain.
- `Database<OffChain>` - Stores only data used by the off-chain services
like GraphQL.
- `Database<Relayer>` - Stores relayer-related data like events(messages
or transactions) from L1.

The `Database<Description>` type has a generic `Description` that
implements the `DatabaseDescription` trait:

```rust
/// The description of the database that makes it unique.
pub trait DatabaseDescription: 'static + Clone + Debug + Send + Sync {
    /// The type of the column used by the database.
    type Column: StorageColumn + strum::EnumCount + enum_iterator::Sequence;
    /// The type of the height of the database used to track commits.
    type Height: Copy;

    /// Returns the expected version of the database.
    fn version() -> u32;

    /// Returns the name of the database.
    fn name() -> &'static str;

    /// Returns the column used to store the metadata.
    fn metadata_column() -> Self::Column;

    /// Returns the prefix for the column.
    fn prefix(column: &Self::Column) -> Option<usize>;
}
```

Each database has its folder, defined by the
`DatabaseDescription::name`, where actual data is stored.

<img width="353" alt="image"
src="https://github.com/FuelLabs/fuel-core/assets/18346821/8b642384-0dd4-4668-a415-0748be3e88f0">


Each database has its own `Column` type that describes all columns,
avoiding overlaps with other tables. The change updates a little bit
`StrucutredStorage` implementation and `TableWithBlueprint` to be more
flexible and use the `Column` defined by the table, instead of hardcoded
`fuel_core_storage::column::Column`.


Other small changes:
- Unified the logic of storing the database's metadata. It will be
useful for #1589 to
implement a unified `commit_chagnes` function.
- The `latest_height` function now uses the height from the metadata
table.
- Removed relayers-related tables and columns from the
`fuel-core-storage` crate.
- Removed part of GraphQL tables and columns from the
`fuel-core-storage`. The last part will be removed during
#1583.
- Moved `tx_count` metrics from `BlockImporter` to GraphQL off-chain
worker. Any statistic that requires a persistent state in the database
may be done outside of the blockchain.
- Remove `chain_name` from the database. The `ConsensusParameters`
already contains this information.
- Removed the `checkpoint` function from the `RocksDB` since it is not
used. Later it will be added again back but with another implementation
during #1589.
- Removed `Column::ForeignColumn`, since each database has its own
`Column` type. Removed the macro rules added to handle `ForeignColumn`.
xgreenx added a commit that referenced this issue Mar 13, 2024
Closes #1589

## Overview

The change moves the database transaction logic from the `fuel-core` to
the `fuel-core-storage` level. The corresponding issue described the
reason behind it.

## Technical details of implementation

The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](7b1141a).

The `StorageTransaction` is a `StructuredStorage` that uses
`StorageTransactionInner` inside to accumulate modifications. Only
`StorageStorageInner` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).

The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).

As was planned in the issue, now the executor has full control over how
commits to the storage are done.

All mutation methods now require `&mut self` - exclusive ownership over
the object to be able to write into it. It almost negates the chance of
concurrent modification of the storage, but it is still possible since
the `Database` implements the `Clone` trait. To be sure that we don't
corrupt the state of the database, the `commit_changes` function
implements additional safety checks to be sure that we commit updates
per each height only once time.

Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
- The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dapp::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after #1664 .

---------

Co-authored-by: Voxelot <[email protected]>
xgreenx added a commit that referenced this issue Mar 30, 2024
## [Version 0.24.0]

### Added

- [#1786](#1786): Regenesis
now includes off-chain tables.
- [#1716](#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](#1770): Add the new
L1 event type for forced transactions.
- [#1767](#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](#1747): The DA block
height is now included in the genesis state.
- [#1740](#1740): Remove
optional fields from genesis configs
- [#1737](#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after #1664.

- [#1693](#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
#1731
* Regenesis support by @MujkicA in
#1693
* Remove genesis temp tables by @MujkicA in
#1737
* Remove optional fields from configs by @MujkicA in
#1740
* Weekly `cargo update` by @github-actions in
#1745
* Regenesis should also store da block height by @xgreenx in
#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in #1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in #1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in #1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
#1757
* Weekly `cargo update` by @github-actions in
#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
#1769
* Feature/new fti event by @Voxelot in
#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in #1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in #1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in #1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
#1786


**Full Changelog**:
v0.23.0...v0.24.0
xgreenx added a commit that referenced this issue Mar 30, 2024
## Version v0.24.0

### Added

- [#1786](#1786): Regenesis
now includes off-chain tables.
- [#1716](#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](#1770): Add the new
L1 event type for forced transactions.
- [#1767](#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](#1747): The DA block
height is now included in the genesis state.
- [#1740](#1740): Remove
optional fields from genesis configs
- [#1737](#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after #1664.

- [#1693](#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
#1731
* Regenesis support by @MujkicA in
#1693
* Remove genesis temp tables by @MujkicA in
#1737
* Remove optional fields from configs by @MujkicA in
#1740
* Weekly `cargo update` by @github-actions in
#1745
* Regenesis should also store da block height by @xgreenx in
#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in #1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in #1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in #1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
#1757
* Weekly `cargo update` by @github-actions in
#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
#1769
* Feature/new fti event by @Voxelot in
#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in #1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in #1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in #1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
#1786
* Release v0.24.0 by @xgreenx in
#1791
* Moved chain specification into `fuel-core-bin` crate by @xgreenx in
#1792


**Full Changelog**:
v0.23.0...v0.24.1
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
The change is related to
FuelLabs/fuel-core#1589.

The idea of the change is to start using the `AtomicView` inside of the
`TxPool` to generate consistent database representation during the
insertion of the transactions.
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
Related to FuelLabs/fuel-core#1589 and
preparation for FuelLabs/fuel-core#1583.

All services that use the database for read-only purposes use
`AtomicView` now instead of direct access to the database.

- Removed `Relayer` from the `Verifier` because if it is not used for
now, plus it may not be needed because of the shared sequencer and its
consensus rules.
- Added verification of the transactions root hash into `Verifier`.
- Removed requesting of the one block from p2p, it is possible to use
range request for that purposes.
- Removed not used `get_sealed_header` and `get_sealed_block` method.
- Added the `latest_height` method to `AtomicView` because the database
always knows its latest height.
- Added customisation of the `Height` used by the `AtomicView`. In the
case of the relayer, we want to use `DaBlockHeight` as a primary key to
create a snapshot, while in the case of the Fuel's databases, we want to
use `BockHeight` as a primary key.
- Renamed `Executor` into `ExecutionInstance` and changed it to be a
one-time usable instance. It consumes the `self` to perform the
execution of the block. The instance has a view of the database and
execution options.
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
Closes FuelLabs/fuel-core#1568

The change splits the `Database` into 3 databases: 
- `Database<OnChain>` - Stores only data required for normal work of the
blockchain.
- `Database<OffChain>` - Stores only data used by the off-chain services
like GraphQL.
- `Database<Relayer>` - Stores relayer-related data like events(messages
or transactions) from L1.

The `Database<Description>` type has a generic `Description` that
implements the `DatabaseDescription` trait:

```rust
/// The description of the database that makes it unique.
pub trait DatabaseDescription: 'static + Clone + Debug + Send + Sync {
    /// The type of the column used by the database.
    type Column: StorageColumn + strum::EnumCount + enum_iterator::Sequence;
    /// The type of the height of the database used to track commits.
    type Height: Copy;

    /// Returns the expected version of the database.
    fn version() -> u32;

    /// Returns the name of the database.
    fn name() -> &'static str;

    /// Returns the column used to store the metadata.
    fn metadata_column() -> Self::Column;

    /// Returns the prefix for the column.
    fn prefix(column: &Self::Column) -> Option<usize>;
}
```

Each database has its folder, defined by the
`DatabaseDescription::name`, where actual data is stored.

<img width="353" alt="image"
src="https://github.com/FuelLabs/fuel-core/assets/18346821/8b642384-0dd4-4668-a415-0748be3e88f0">


Each database has its own `Column` type that describes all columns,
avoiding overlaps with other tables. The change updates a little bit
`StrucutredStorage` implementation and `TableWithBlueprint` to be more
flexible and use the `Column` defined by the table, instead of hardcoded
`fuel_core_storage::column::Column`.


Other small changes:
- Unified the logic of storing the database's metadata. It will be
useful for FuelLabs/fuel-core#1589 to
implement a unified `commit_chagnes` function.
- The `latest_height` function now uses the height from the metadata
table.
- Removed relayers-related tables and columns from the
`fuel-core-storage` crate.
- Removed part of GraphQL tables and columns from the
`fuel-core-storage`. The last part will be removed during
FuelLabs/fuel-core#1583.
- Moved `tx_count` metrics from `BlockImporter` to GraphQL off-chain
worker. Any statistic that requires a persistent state in the database
may be done outside of the blockchain.
- Remove `chain_name` from the database. The `ConsensusParameters`
already contains this information.
- Removed the `checkpoint` function from the `RocksDB` since it is not
used. Later it will be added again back but with another implementation
during FuelLabs/fuel-core#1589.
- Removed `Column::ForeignColumn`, since each database has its own
`Column` type. Removed the macro rules added to handle `ForeignColumn`.
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
Closes FuelLabs/fuel-core#1589

## Overview

The change moves the database transaction logic from the `fuel-core` to
the `fuel-core-storage` level. The corresponding issue described the
reason behind it.

## Technical details of implementation

The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](FuelLabs/fuel-core@7b1141a).

The `StorageTransaction` is a `StructuredStorage` that uses
`StorageTransactionInner` inside to accumulate modifications. Only
`StorageStorageInner` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).

The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).

As was planned in the issue, now the executor has full control over how
commits to the storage are done.

All mutation methods now require `&mut self` - exclusive ownership over
the object to be able to write into it. It almost negates the chance of
concurrent modification of the storage, but it is still possible since
the `Database` implements the `Clone` trait. To be sure that we don't
corrupt the state of the database, the `commit_changes` function
implements additional safety checks to be sure that we commit updates
per each height only once time.

Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
- The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dapp::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](FuelLabs/fuel-core@5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after FuelLabs/fuel-core#1664 .

---------

Co-authored-by: Voxelot <[email protected]>
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
## [Version 0.24.0]

### Added

- [#1786](FuelLabs/fuel-core#1786): Regenesis
now includes off-chain tables.
- [#1716](FuelLabs/fuel-core#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](FuelLabs/fuel-core#1770): Add the new
L1 event type for forced transactions.
- [#1767](FuelLabs/fuel-core#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](FuelLabs/fuel-core#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](FuelLabs/fuel-core#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](FuelLabs/fuel-core#1747): The DA block
height is now included in the genesis state.
- [#1740](FuelLabs/fuel-core#1740): Remove
optional fields from genesis configs
- [#1737](FuelLabs/fuel-core#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](FuelLabs/fuel-core#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](FuelLabs/fuel-core#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](FuelLabs/fuel-core#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](FuelLabs/fuel-core#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](FuelLabs/fuel-core#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](FuelLabs/fuel-core#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](FuelLabs/fuel-core#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](FuelLabs/fuel-core#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](FuelLabs/fuel-core#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](FuelLabs/fuel-core@7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](FuelLabs/fuel-core@5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after FuelLabs/fuel-core#1664.

- [#1693](FuelLabs/fuel-core#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](FuelLabs/fuel-core#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
FuelLabs/fuel-core#1731
* Regenesis support by @MujkicA in
FuelLabs/fuel-core#1693
* Remove genesis temp tables by @MujkicA in
FuelLabs/fuel-core#1737
* Remove optional fields from configs by @MujkicA in
FuelLabs/fuel-core#1740
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1745
* Regenesis should also store da block height by @xgreenx in
FuelLabs/fuel-core#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in FuelLabs/fuel-core#1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in FuelLabs/fuel-core#1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in FuelLabs/fuel-core#1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
FuelLabs/fuel-core#1757
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
FuelLabs/fuel-core#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
FuelLabs/fuel-core#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
FuelLabs/fuel-core#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
FuelLabs/fuel-core#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
FuelLabs/fuel-core#1769
* Feature/new fti event by @Voxelot in
FuelLabs/fuel-core#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
FuelLabs/fuel-core#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in FuelLabs/fuel-core#1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
FuelLabs/fuel-core#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in FuelLabs/fuel-core#1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in FuelLabs/fuel-core#1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
FuelLabs/fuel-core#1786


**Full Changelog**:
FuelLabs/fuel-core@v0.23.0...v0.24.0
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
## Version v0.24.0

### Added

- [#1786](FuelLabs/fuel-core#1786): Regenesis
now includes off-chain tables.
- [#1716](FuelLabs/fuel-core#1716): Added
support of WASM state transition along with upgradable execution that
works with native(std) and WASM(non-std) executors. The `fuel-core` now
requires a `wasm32-unknown-unknown` target to build.
- [#1770](FuelLabs/fuel-core#1770): Add the new
L1 event type for forced transactions.
- [#1767](FuelLabs/fuel-core#1767): Added
consensus parameters version and state transition version to the
`ApplicationHeader` to describe what was used to produce this block.
- [#1760](FuelLabs/fuel-core#1760): Added tests
to verify that the network operates with a custom chain id and base
asset id.
- [#1752](FuelLabs/fuel-core#1752): Add
`ProducerGasPrice` trait that the `Producer` depends on to get the gas
price for the block.
- [#1747](FuelLabs/fuel-core#1747): The DA block
height is now included in the genesis state.
- [#1740](FuelLabs/fuel-core#1740): Remove
optional fields from genesis configs
- [#1737](FuelLabs/fuel-core#1737): Remove
temporary tables for calculating roots during genesis.
- [#1731](FuelLabs/fuel-core#1731): Expose
`schema.sdl` from `fuel-core-client`.

### Changed

#### Breaking

- [#1771](FuelLabs/fuel-core#1771): Contract
'states' and 'balances' brought back into `ContractConfig`. Parquet now
writes a file per table.
- [1779](FuelLabs/fuel-core#1779): Modify
Relayer service to order Events from L1 by block index
- [#1783](FuelLabs/fuel-core#1783): The PR
upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes,
we also adapted our codebase to follow them:
- Implementation of `Default` for configs was moved under the
`test-helpers` feature. The `fuel-core` binary uses testnet
configuration instead of `Default::default`(for cases when `ChainConfig`
was not provided by the user).
- All parameter types are enums now and require corresponding
modifications across the codebase(we need to use getters and setters).
The GraphQL API remains the same for simplicity, but each parameter now
has one more field - `version`, that can be used to decide how to
deserialize.
- The `UtxoId` type now is 34 bytes instead of 33. It affects hex
representation and requires adding `00`.
- The `block_gas_limit` was moved to `ConsensusParameters` from
`ChainConfig`. It means the block producer doesn't specify the block gas
limit anymore, and we don't need to propagate this information.
  - The `bytecodeLength` field is removed from the `Create` transaction.
- Removed `ConsensusParameters` from executor config because
`ConsensusParameters::default` is not available anymore. Instead,
executors fetch `ConsensusParameters` from the database.

- [#1769](FuelLabs/fuel-core#1769): Include new
field on header for the merkle root of imported events. Rename other
message root field.
- [#1768](FuelLabs/fuel-core#1768): Moved
`ContractsInfo` table to the off-chain database. Removed `salt` field
from the `ContractConfig`.
- [#1761](FuelLabs/fuel-core#1761): Adjustments
to the upcoming testnet configs:
  - Decreased the max size of the contract/predicate/script to be 100KB.
  - Decreased the max size of the transaction to be 110KB.
  - Decreased the max number of storage slots to be 1760(110KB / 64).
  - Removed fake coins from the genesis state.
  - Renamed folders to be "testnet" and "dev-testnet".
- The name of the networks are "Upgradable Testnet" and "Upgradable Dev
Testnet".

- [#1694](FuelLabs/fuel-core#1694): The change
moves the database transaction logic from the `fuel-core` to the
`fuel-core-storage` level. The corresponding
[issue](FuelLabs/fuel-core#1589) described the
reason behind it.

    ## Technical details of implementation

- The change splits the `KeyValueStore` into `KeyValueInspect` and
`KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and
`BlueprintMutate`. It allows requiring less restricted constraints for
any read-related operations.

- One of the main ideas of the change is to allow for the actual storage
only to implement `KeyValueInspect` and `Modifiable` without the
`KeyValueMutate`. It simplifies work with the databases and provides a
safe way of interacting with them (Modification into the database can
only go through the `Modifiable::commit_changes`). This feature is used
to [track the
height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898)
of each database during commits and even limit how commits are done,
providing additional safety. This part of the change was done as a
[separate
commit](FuelLabs/fuel-core@7b1141a).
    
- The `StorageTransaction` is a `StructuredStorage` that uses
`InMemoryTransaction` inside to accumulate modifications. Only
`InMemoryTransaction` has a real implementation of the
`KeyValueMutate`(Other types only implement it in tests).
    
- The implementation of the `Modifiable` for the `Database` contains a
business logic that provides additional safety but limits the usage of
the database. The `Database` now tracks its height and is responsible
for its updates. In the `commit_changes` function, it analyzes the
changes that were done and tries to find a new height(For example, in
the case of the `OnChain` database, we are looking for a new `Block` in
the `FuelBlocks` table).
    
- As was planned in the issue, now the executor has full control over
how commits to the storage are done.
    
- All mutation methods now require `&mut self` - exclusive ownership
over the object to be able to write into it. It almost negates the
chance of concurrent modification of the storage, but it is still
possible since the `Database` implements the `Clone` trait. To be sure
that we don't corrupt the state of the database, the `commit_changes`
function implements additional safety checks to be sure that we commit
updates per each height only once time.

    - Side changes:
- The `drop` function was moved from `Database` to `RocksDB` as a
preparation for the state rewind since the read view should also keep
the drop function until it is destroyed.
      - The `StatisticTable` table lives in the off-chain worker.
- Removed duplication of the `Database` from the `dap::ConcreteStorage`
since it is already available from the VM.
- The executor return only produced `Changes` instead of the storage
transaction, which simplifies the interaction between modules and port
definition.
- The logic related to the iteration over the storage is moved to the
`fuel-core-storage` crate and is now reusable. It provides an
`interator` method that duplicates the logic from `MemoryStore` on
iterating over the `BTreeMap` and methods like `iter_all`,
`iter_all_by_prefix`, etc. It was done in a separate revivable
[commit](FuelLabs/fuel-core@5b9bd78).
- The `MemoryTransactionView` is fully replaced by the
`StorageTransactionInner`.
- Removed `flush` method from the `Database` since it is not needed
after FuelLabs/fuel-core#1664.

- [#1693](FuelLabs/fuel-core#1693): The change
separates the initial chain state from the chain config and stores them
in separate files when generating a snapshot. The state snapshot can be
generated in a new format where parquet is used for compression and
indexing while postcard is used for encoding. This enables importing in
a stream like fashion which reduces memory requirements. Json encoding
is still supported to enable easy manual setup. However, parquet is
prefered for large state files.

  ### Snapshot command

The CLI was expanded to allow customizing the used encoding. Snapshots
are now generated along with a metadata file describing the encoding
used. The metadata file contains encoding details as well as the
location of additional files inside the snapshot directory containing
the actual data. The chain config is always generated in the JSON
format.

The snapshot command now has the '--output-directory' for specifying
where to save the snapshot.

  ### Run command

The run command now includes the 'db_prune' flag which when provided
will prune the existing db and start genesis from the provided snapshot
metadata file or the local testnet configuration.

The snapshot metadata file contains paths to the chain config file and
files containing chain state items (coins, messages, contracts, contract
states, and balances), which are loaded via streaming.

Each item group in the genesis process is handled by a separate worker,
allowing for parallel loading. Workers stream file contents in batches.

A database transaction is committed every time an item group is
succesfully loaded. Resumability is achieved by recording the last
loaded group index within the same db tx. If loading is aborted, the
remaining workers are shutdown. Upon restart, workers resume from the
last processed group.

  ### Contract States and Balances

Using uniform-sized batches may result in batches containing items from
multiple contracts. Optimal performance can presumably be achieved by
selecting a batch size that typically encompasses an entire contract's
state or balance, allowing for immediate initialization of relevant
Merkle trees.

### Removed

- [#1757](FuelLabs/fuel-core#1757): Removed
`protobuf` from everywhere since `libp2p` uses `quick-protobuf`.

## What's Changed
* Expose `schema.sdl` add some helper types and traits by @Dentosal in
FuelLabs/fuel-core#1731
* Regenesis support by @MujkicA in
FuelLabs/fuel-core#1693
* Remove genesis temp tables by @MujkicA in
FuelLabs/fuel-core#1737
* Remove optional fields from configs by @MujkicA in
FuelLabs/fuel-core#1740
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1745
* Regenesis should also store da block height by @xgreenx in
FuelLabs/fuel-core#1747
* Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx
in FuelLabs/fuel-core#1748
* Moved `StorageTransaction` to the `fuel-core-storage` crate by
@xgreenx in FuelLabs/fuel-core#1694
* Prepare the codebase to use base gas price during block production
#1642 by @MitchTurner in FuelLabs/fuel-core#1752
* Removed `protobuf` from everywhere since `libp2p` uses
`quick-protobuf` by @xgreenx in
FuelLabs/fuel-core#1757
* Weekly `cargo update` by @github-actions in
FuelLabs/fuel-core#1758
* Added tests to verify that the network operates with a custom chain id
and base asset id by @xgreenx in
FuelLabs/fuel-core#1760
* Adjustments to the upcoming testnet configs by @xgreenx in
FuelLabs/fuel-core#1761
* Added consensus parameters version and state transition version to the
`ApplicationHeader` by @xgreenx in
FuelLabs/fuel-core#1767
* Moved `ContractsInfo` table to the off-chain database by @xgreenx in
FuelLabs/fuel-core#1768
* Keep record of events from L1 in Block Header by @MitchTurner in
FuelLabs/fuel-core#1769
* Feature/new fti event by @Voxelot in
FuelLabs/fuel-core#1770
* Forkless state transition with upgradable WASM executor by @xgreenx in
FuelLabs/fuel-core#1716
* Removed the usage of the `lazy_static` from teh codebase by @xgreenx
in FuelLabs/fuel-core#1781
* Patch to use `fuel-vm 0.48.0` by @xgreenx in
FuelLabs/fuel-core#1783
* Modify Relayer service to order Events from L1 by block index by
@MitchTurner in FuelLabs/fuel-core#1779
* refactor: Prepare (re)genesis for off chain tables by @segfault-magnet
in FuelLabs/fuel-core#1771
* feat: Add some off chain tables to regenesis by @segfault-magnet in
FuelLabs/fuel-core#1786
* Release v0.24.0 by @xgreenx in
FuelLabs/fuel-core#1791
* Moved chain specification into `fuel-core-bin` crate by @xgreenx in
FuelLabs/fuel-core#1792


**Full Changelog**:
FuelLabs/fuel-core@v0.23.0...v0.24.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant