Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lazy vec and map #503

Merged
merged 83 commits into from
Oct 13, 2022
Merged
Changes from 1 commit
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
1a57908
ledger: add tx and VP traits with host env functions
tzemanovic Jul 20, 2022
5d8c61a
shared/vm: rename host_env TxEnv/VpEnv to TxVmEnv/VpVmEnv
tzemanovic Jul 20, 2022
fe365d9
shared/ledger/native_vp: implement VpEnv
tzemanovic Jul 20, 2022
4e46aca
shared: update native_vp implementations for VpEnv methods
tzemanovic Jul 20, 2022
e4b84a0
VM: move vm_env sub-mod into tx/vp_prelude and add Tx/VpEnv and Ctx
tzemanovic Jul 20, 2022
71820f6
tests: update for VM API changes
tzemanovic Jul 20, 2022
d7d0ef1
wasm: update for VM API changes
tzemanovic Jul 20, 2022
bf604a4
macros: add error handling to transaction and validity_predicate macros
tzemanovic Jul 20, 2022
d868424
wasm: improve error handling
tzemanovic Jul 21, 2022
221e9f2
changelog: add #1093
tzemanovic Jul 21, 2022
8bce915
Update shared/src/ledger/vp_env.rs
tzemanovic Jul 25, 2022
affa0b5
wasm_for_tests: make
tzemanovic Aug 11, 2022
14f638a
update wasm checksums
tzemanovic Aug 11, 2022
18526da
[ci skip] wasm checksums update
github-actions[bot] Aug 17, 2022
9a135eb
ledger: add StorageRead trait with extensible error type
tzemanovic Aug 12, 2022
97c6947
ledger/native_vp: implement StorageRead for pre and post state
tzemanovic Aug 12, 2022
7552c1e
ledger/vp: impl StorageRead for WASM VPs pre and post state
tzemanovic Aug 12, 2022
45f3f94
ledger/storage: impl StorageRead for Storage
tzemanovic Aug 12, 2022
5989412
ledger/tx: inherit StorageRead in TxEnv
tzemanovic Aug 12, 2022
10f94c2
ledger/pos: implement PosReadOnly using StorageRead trait
tzemanovic Aug 12, 2022
eeb1e8c
update wasm checksums
tzemanovic Aug 12, 2022
abd10c1
ledger: factor out TxEnv write methods into a new StorageWrite trait
tzemanovic Aug 12, 2022
08b0a8f
ledger: impl StorageWrite for Storage
tzemanovic Aug 12, 2022
773033d
add <'iter> lifetime to StorageRead trait to get around lack of GATs
tzemanovic Aug 16, 2022
28b4307
add more comments for StorageRead lifetime with an example usage
tzemanovic Aug 16, 2022
8bc5816
storage: remove unnecessary clone
tzemanovic Aug 16, 2022
7decfab
fix missing StorageWrite for Storage merkle tree update
tzemanovic Aug 16, 2022
b92edd4
update wasm checksums
tzemanovic Aug 20, 2022
ace6716
changelog: add #331
tzemanovic Aug 21, 2022
999df37
storage_api: add default borsh encoded read/write impl and handle errors
tzemanovic Aug 15, 2022
a547378
wasm checksums update
github-actions[bot] Aug 15, 2022
6f0be8f
changelog: add #334
tzemanovic Aug 22, 2022
d737acd
storage_api: build a nicer `iter_prefix` function on top of StorageRead
tzemanovic Aug 15, 2022
4558dfa
test/vm_host_env: refactor prefix iter tests with new `iter_prefix` fn
tzemanovic Aug 15, 2022
d32d7af
wasm checksums update
github-actions[bot] Aug 15, 2022
fec7220
changelog: add #335
tzemanovic Aug 22, 2022
2d177f8
shared: Add pre/post to VpEnv and use them to provide default impls
tzemanovic Aug 25, 2022
c86185c
changelog: add #380
tzemanovic Aug 25, 2022
b811466
update wasm checksums
tzemanovic Aug 25, 2022
b8eccf3
deps: replace hex with data-encoding
tzemanovic Sep 8, 2022
b47d1c1
shared/storage/key: add support for int/uint keys that maintain order
tzemanovic Sep 8, 2022
e648085
test/vm_host_env: check prefix iter order in tx and VP
tzemanovic Sep 8, 2022
1429b92
add support for rev_iter_prefix in storage and VP and tx envs
tzemanovic Sep 8, 2022
881c93c
tests: extend prefix iter tests for reverse order
tzemanovic Sep 8, 2022
d786a4a
changelog: add #458
tzemanovic Sep 8, 2022
1d2f1dd
[ci skip] wasm checksums update
github-actions[bot] Sep 9, 2022
a700e49
ledger: use storage_api::Error in VpEnv and TxEnv instead of generic
tzemanovic Sep 13, 2022
45dddbe
[ci skip] wasm checksums update
github-actions[bot] Sep 13, 2022
e85bf62
Merge branch 'namada/tomas/sorted-prefix-iter' (#458)
tzemanovic Sep 13, 2022
b9406a2
fixup! Merge branch 'namada/tomas/sorted-prefix-iter' (#458)
tzemanovic Sep 13, 2022
16d9aa6
changelog: add #465
tzemanovic Sep 13, 2022
4b14f6b
Merge branch 'tomas/vp-tx-env-concrete-error' (#465)
tzemanovic Sep 14, 2022
a0f193a
rustdoc: resolve ambiguous link
tzemanovic Aug 30, 2022
d7c30ac
create lazy data structures for storage access
brentstone Aug 12, 2022
b07b844
add lazy vector
brentstone Aug 12, 2022
13173b3
add lazy set (WIP), make LazyVec.get public
brentstone Aug 12, 2022
72e48e6
lazy hash map first commit
brentstone Aug 14, 2022
d964880
add fn get_elem_key_by_hash to LazyMap
brentstone Aug 14, 2022
6752132
fmt && clippy
brentstone Aug 14, 2022
68cd35a
refactored lazy collections, replaced hasher, added iter
tzemanovic Aug 15, 2022
bc5f615
add lazy map without hashing
brentstone Aug 15, 2022
50e2f25
Switch to use storage::KeySeg and add LazySet
tzemanovic Aug 16, 2022
12ce117
storage: add `Key::last` method, impl KeySeg for all ints
tzemanovic Aug 16, 2022
bb18f2b
update lazy for explicit lifetime in StorageRead trait
tzemanovic Aug 16, 2022
22c2ea5
cargo test test_lazy_vec_basics
tzemanovic Aug 16, 2022
fff7734
storage_api/collections/lazy: add basic tests and missing methods
tzemanovic Aug 16, 2022
ca50fb9
fix clippy
tzemanovic Aug 16, 2022
5819b23
add lazy_vec validation
tzemanovic Aug 23, 2022
a474c80
storage_api/collections/lazy: allow nested lazy collections in LazyMap
tzemanovic Aug 29, 2022
5103668
impl LazyCollection trait for all the collections + refactor
tzemanovic Sep 1, 2022
255b43b
storage/lazy_vec/validation: disallow unrecognized keys matching prefix
tzemanovic Sep 12, 2022
f854d82
storage/lazy_vec: add state machine test for lazy vec API
tzemanovic Sep 13, 2022
8c4e2ca
update after rebase on #458, #465
tzemanovic Sep 14, 2022
35b5f40
quick bug and documentation fix
brentstone Sep 14, 2022
5396df9
post-conditions for Transition::Update + some comments
brentstone Sep 14, 2022
dad616b
WIP: StateMachine tests for lazy_vec validation
brentstone Sep 14, 2022
4b60051
WIP: validation for lazy_map
brentstone Sep 16, 2022
8d2ad09
WIP: Nested LazyMap validation and testing
tzemanovic Sep 19, 2022
4adf301
ledger/storage/lazy: remove unused error cases
tzemanovic Sep 22, 2022
e7c868f
ledger/storage/lazy: update lazy_set for updated trait LazyCollection
tzemanovic Sep 22, 2022
a5d4a67
remove unfinished lazy_set, lazy_hashset and lazy_hashmap for now
tzemanovic Sep 22, 2022
5cedd45
[ci] wasm checksums update
github-actions[bot] Sep 22, 2022
707db38
changelog: #503
tzemanovic Sep 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 157 additions & 0 deletions shared/src/ledger/native_vp.rs
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
use std::cell::RefCell;
use std::collections::BTreeSet;

use super::storage_api::{self, ResultExt, StorageRead};
pub use super::vp_env::VpEnv;
use crate::ledger::gas::VpGasMeter;
use crate::ledger::storage::write_log::WriteLog;
@@ -72,6 +73,30 @@ where
pub cache_access: std::marker::PhantomData<CA>,
}

/// Read access to the prior storage (state before tx execution) via
/// [`trait@StorageRead`].
#[derive(Debug)]
pub struct CtxPreStorageRead<'b, 'a: 'b, DB, H, CA>
where
DB: storage::DB + for<'iter> storage::DBIter<'iter>,
H: StorageHasher,
CA: WasmCacheAccess,
{
ctx: &'b Ctx<'a, DB, H, CA>,
}

/// Read access to the posterior storage (state after tx execution) via
/// [`trait@StorageRead`].
#[derive(Debug)]
pub struct CtxPostStorageRead<'f, 'a: 'f, DB, H, CA>
where
DB: storage::DB + for<'iter> storage::DBIter<'iter>,
H: StorageHasher,
CA: WasmCacheAccess,
{
ctx: &'f Ctx<'a, DB, H, CA>,
}

impl<'a, DB, H, CA> Ctx<'a, DB, H, CA>
where
DB: 'static + storage::DB + for<'iter> storage::DBIter<'iter>,
@@ -111,6 +136,138 @@ where
pub fn add_gas(&self, used_gas: u64) -> Result<(), vp_env::RuntimeError> {
vp_env::add_gas(&mut *self.gas_meter.borrow_mut(), used_gas)
}

/// Read access to the prior storage (state before tx execution)
/// via [`trait@StorageRead`].
pub fn pre<'b>(&'b self) -> CtxPreStorageRead<'b, 'a, DB, H, CA> {
CtxPreStorageRead { ctx: self }
}

/// Read access to the posterior storage (state after tx execution)
/// via [`trait@StorageRead`].
pub fn post<'b>(&'b self) -> CtxPostStorageRead<'b, 'a, DB, H, CA> {
CtxPostStorageRead { ctx: self }
}
}

impl<'f, 'a, DB, H, CA> StorageRead for CtxPreStorageRead<'f, 'a, DB, H, CA>
where
DB: 'static + storage::DB + for<'iter> storage::DBIter<'iter>,
H: 'static + StorageHasher,
CA: 'static + WasmCacheAccess,
{
type PrefixIter = <DB as storage::DBIter<'a>>::PrefixIter;

fn read<T: borsh::BorshDeserialize>(
&self,
key: &crate::types::storage::Key,
) -> Result<Option<T>, storage_api::Error> {
self.ctx.read_pre(key).into_storage_result()
}

fn read_bytes(
&self,
key: &crate::types::storage::Key,
) -> Result<Option<Vec<u8>>, storage_api::Error> {
self.ctx.read_bytes_pre(key).into_storage_result()
}

fn has_key(
&self,
key: &crate::types::storage::Key,
) -> Result<bool, storage_api::Error> {
self.ctx.has_key_pre(key).into_storage_result()
}

fn iter_prefix(
&self,
prefix: &crate::types::storage::Key,
) -> Result<Self::PrefixIter, storage_api::Error> {
self.ctx.iter_prefix(prefix).into_storage_result()
}

fn iter_next(
&self,
iter: &mut Self::PrefixIter,
) -> Result<Option<(String, Vec<u8>)>, storage_api::Error> {
self.ctx.iter_pre_next(iter).into_storage_result()
}

fn get_chain_id(&self) -> Result<String, storage_api::Error> {
self.ctx.get_chain_id().into_storage_result()
}

fn get_block_height(&self) -> Result<BlockHeight, storage_api::Error> {
self.ctx.get_block_height().into_storage_result()
}

fn get_block_hash(&self) -> Result<BlockHash, storage_api::Error> {
self.ctx.get_block_hash().into_storage_result()
}

fn get_block_epoch(&self) -> Result<Epoch, storage_api::Error> {
self.ctx.get_block_epoch().into_storage_result()
}
}

impl<'f, 'a, DB, H, CA> StorageRead for CtxPostStorageRead<'f, 'a, DB, H, CA>
where
DB: 'static + storage::DB + for<'iter> storage::DBIter<'iter>,
H: 'static + StorageHasher,
CA: 'static + WasmCacheAccess,
{
type PrefixIter = <DB as storage::DBIter<'a>>::PrefixIter;

fn read<T: borsh::BorshDeserialize>(
&self,
key: &crate::types::storage::Key,
) -> Result<Option<T>, storage_api::Error> {
self.ctx.read_post(key).into_storage_result()
}

fn read_bytes(
&self,
key: &crate::types::storage::Key,
) -> Result<Option<Vec<u8>>, storage_api::Error> {
self.ctx.read_bytes_post(key).into_storage_result()
}

fn has_key(
&self,
key: &crate::types::storage::Key,
) -> Result<bool, storage_api::Error> {
self.ctx.has_key_post(key).into_storage_result()
}

fn iter_prefix(
&self,
prefix: &crate::types::storage::Key,
) -> Result<Self::PrefixIter, storage_api::Error> {
self.ctx.iter_prefix(prefix).into_storage_result()
}

fn iter_next(
&self,
iter: &mut Self::PrefixIter,
) -> Result<Option<(String, Vec<u8>)>, storage_api::Error> {
self.ctx.iter_post_next(iter).into_storage_result()
}

fn get_chain_id(&self) -> Result<String, storage_api::Error> {
self.ctx.get_chain_id().into_storage_result()
}

fn get_block_height(&self) -> Result<BlockHeight, storage_api::Error> {
self.ctx.get_block_height().into_storage_result()
}

fn get_block_hash(&self) -> Result<BlockHash, storage_api::Error> {
self.ctx.get_block_hash().into_storage_result()
}

fn get_block_epoch(&self) -> Result<Epoch, storage_api::Error> {
self.ctx.get_block_epoch().into_storage_result()
}
}

impl<'a, DB, H, CA> VpEnv for Ctx<'a, DB, H, CA>