Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lazy vec and map #503

Merged
merged 83 commits into from
Oct 13, 2022
Merged
Changes from 1 commit
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
1a57908
ledger: add tx and VP traits with host env functions
tzemanovic Jul 20, 2022
5d8c61a
shared/vm: rename host_env TxEnv/VpEnv to TxVmEnv/VpVmEnv
tzemanovic Jul 20, 2022
fe365d9
shared/ledger/native_vp: implement VpEnv
tzemanovic Jul 20, 2022
4e46aca
shared: update native_vp implementations for VpEnv methods
tzemanovic Jul 20, 2022
e4b84a0
VM: move vm_env sub-mod into tx/vp_prelude and add Tx/VpEnv and Ctx
tzemanovic Jul 20, 2022
71820f6
tests: update for VM API changes
tzemanovic Jul 20, 2022
d7d0ef1
wasm: update for VM API changes
tzemanovic Jul 20, 2022
bf604a4
macros: add error handling to transaction and validity_predicate macros
tzemanovic Jul 20, 2022
d868424
wasm: improve error handling
tzemanovic Jul 21, 2022
221e9f2
changelog: add #1093
tzemanovic Jul 21, 2022
8bce915
Update shared/src/ledger/vp_env.rs
tzemanovic Jul 25, 2022
affa0b5
wasm_for_tests: make
tzemanovic Aug 11, 2022
14f638a
update wasm checksums
tzemanovic Aug 11, 2022
18526da
[ci skip] wasm checksums update
github-actions[bot] Aug 17, 2022
9a135eb
ledger: add StorageRead trait with extensible error type
tzemanovic Aug 12, 2022
97c6947
ledger/native_vp: implement StorageRead for pre and post state
tzemanovic Aug 12, 2022
7552c1e
ledger/vp: impl StorageRead for WASM VPs pre and post state
tzemanovic Aug 12, 2022
45f3f94
ledger/storage: impl StorageRead for Storage
tzemanovic Aug 12, 2022
5989412
ledger/tx: inherit StorageRead in TxEnv
tzemanovic Aug 12, 2022
10f94c2
ledger/pos: implement PosReadOnly using StorageRead trait
tzemanovic Aug 12, 2022
eeb1e8c
update wasm checksums
tzemanovic Aug 12, 2022
abd10c1
ledger: factor out TxEnv write methods into a new StorageWrite trait
tzemanovic Aug 12, 2022
08b0a8f
ledger: impl StorageWrite for Storage
tzemanovic Aug 12, 2022
773033d
add <'iter> lifetime to StorageRead trait to get around lack of GATs
tzemanovic Aug 16, 2022
28b4307
add more comments for StorageRead lifetime with an example usage
tzemanovic Aug 16, 2022
8bc5816
storage: remove unnecessary clone
tzemanovic Aug 16, 2022
7decfab
fix missing StorageWrite for Storage merkle tree update
tzemanovic Aug 16, 2022
b92edd4
update wasm checksums
tzemanovic Aug 20, 2022
ace6716
changelog: add #331
tzemanovic Aug 21, 2022
999df37
storage_api: add default borsh encoded read/write impl and handle errors
tzemanovic Aug 15, 2022
a547378
wasm checksums update
github-actions[bot] Aug 15, 2022
6f0be8f
changelog: add #334
tzemanovic Aug 22, 2022
d737acd
storage_api: build a nicer `iter_prefix` function on top of StorageRead
tzemanovic Aug 15, 2022
4558dfa
test/vm_host_env: refactor prefix iter tests with new `iter_prefix` fn
tzemanovic Aug 15, 2022
d32d7af
wasm checksums update
github-actions[bot] Aug 15, 2022
fec7220
changelog: add #335
tzemanovic Aug 22, 2022
2d177f8
shared: Add pre/post to VpEnv and use them to provide default impls
tzemanovic Aug 25, 2022
c86185c
changelog: add #380
tzemanovic Aug 25, 2022
b811466
update wasm checksums
tzemanovic Aug 25, 2022
b8eccf3
deps: replace hex with data-encoding
tzemanovic Sep 8, 2022
b47d1c1
shared/storage/key: add support for int/uint keys that maintain order
tzemanovic Sep 8, 2022
e648085
test/vm_host_env: check prefix iter order in tx and VP
tzemanovic Sep 8, 2022
1429b92
add support for rev_iter_prefix in storage and VP and tx envs
tzemanovic Sep 8, 2022
881c93c
tests: extend prefix iter tests for reverse order
tzemanovic Sep 8, 2022
d786a4a
changelog: add #458
tzemanovic Sep 8, 2022
1d2f1dd
[ci skip] wasm checksums update
github-actions[bot] Sep 9, 2022
a700e49
ledger: use storage_api::Error in VpEnv and TxEnv instead of generic
tzemanovic Sep 13, 2022
45dddbe
[ci skip] wasm checksums update
github-actions[bot] Sep 13, 2022
e85bf62
Merge branch 'namada/tomas/sorted-prefix-iter' (#458)
tzemanovic Sep 13, 2022
b9406a2
fixup! Merge branch 'namada/tomas/sorted-prefix-iter' (#458)
tzemanovic Sep 13, 2022
16d9aa6
changelog: add #465
tzemanovic Sep 13, 2022
4b14f6b
Merge branch 'tomas/vp-tx-env-concrete-error' (#465)
tzemanovic Sep 14, 2022
a0f193a
rustdoc: resolve ambiguous link
tzemanovic Aug 30, 2022
d7c30ac
create lazy data structures for storage access
brentstone Aug 12, 2022
b07b844
add lazy vector
brentstone Aug 12, 2022
13173b3
add lazy set (WIP), make LazyVec.get public
brentstone Aug 12, 2022
72e48e6
lazy hash map first commit
brentstone Aug 14, 2022
d964880
add fn get_elem_key_by_hash to LazyMap
brentstone Aug 14, 2022
6752132
fmt && clippy
brentstone Aug 14, 2022
68cd35a
refactored lazy collections, replaced hasher, added iter
tzemanovic Aug 15, 2022
bc5f615
add lazy map without hashing
brentstone Aug 15, 2022
50e2f25
Switch to use storage::KeySeg and add LazySet
tzemanovic Aug 16, 2022
12ce117
storage: add `Key::last` method, impl KeySeg for all ints
tzemanovic Aug 16, 2022
bb18f2b
update lazy for explicit lifetime in StorageRead trait
tzemanovic Aug 16, 2022
22c2ea5
cargo test test_lazy_vec_basics
tzemanovic Aug 16, 2022
fff7734
storage_api/collections/lazy: add basic tests and missing methods
tzemanovic Aug 16, 2022
ca50fb9
fix clippy
tzemanovic Aug 16, 2022
5819b23
add lazy_vec validation
tzemanovic Aug 23, 2022
a474c80
storage_api/collections/lazy: allow nested lazy collections in LazyMap
tzemanovic Aug 29, 2022
5103668
impl LazyCollection trait for all the collections + refactor
tzemanovic Sep 1, 2022
255b43b
storage/lazy_vec/validation: disallow unrecognized keys matching prefix
tzemanovic Sep 12, 2022
f854d82
storage/lazy_vec: add state machine test for lazy vec API
tzemanovic Sep 13, 2022
8c4e2ca
update after rebase on #458, #465
tzemanovic Sep 14, 2022
35b5f40
quick bug and documentation fix
brentstone Sep 14, 2022
5396df9
post-conditions for Transition::Update + some comments
brentstone Sep 14, 2022
dad616b
WIP: StateMachine tests for lazy_vec validation
brentstone Sep 14, 2022
4b60051
WIP: validation for lazy_map
brentstone Sep 16, 2022
8d2ad09
WIP: Nested LazyMap validation and testing
tzemanovic Sep 19, 2022
4adf301
ledger/storage/lazy: remove unused error cases
tzemanovic Sep 22, 2022
e7c868f
ledger/storage/lazy: update lazy_set for updated trait LazyCollection
tzemanovic Sep 22, 2022
a5d4a67
remove unfinished lazy_set, lazy_hashset and lazy_hashmap for now
tzemanovic Sep 22, 2022
5cedd45
[ci] wasm checksums update
github-actions[bot] Sep 22, 2022
707db38
changelog: #503
tzemanovic Sep 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Switch to use storage::KeySeg and add LazySet
Also refactored the iterators implementation to take advantage of
changes from #335. Note that this requires `'static` lifetime bound on
the types of the collections' elements, which means we cannot use
non-static references, but we wouldn't do that anyway.
tzemanovic committed Sep 14, 2022

Verified

This commit was signed with the committer’s verified signature. The key has expired.
tzemanovic Tomas Zemanovic
commit 50e2f258838e4561d64b8421051950368eba394d
50 changes: 27 additions & 23 deletions shared/src/ledger/storage_api/collections/lazy_hashmap.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//! Lazy hash map
//! Lazy hash map.

use std::marker::PhantomData;

@@ -12,7 +12,22 @@ use crate::types::storage;
/// Subkey corresponding to the data elements of the LazyMap
pub const DATA_SUBKEY: &str = "data";

/// LazyHashmap ! fill in !
/// Lazy hash map.
///
/// This can be used as an alternative to `std::collections::HashMap` and
/// `BTreeMap`. In the lazy map, the elements do not reside in memory but are
/// instead read and written to storage sub-keys of the storage `key` given to
/// construct the map.
///
/// In the [`LazyHashMap`], the type of key `K` can be anything that
/// [`BorshSerialize`] and [`BorshDeserialize`] and a hex string of sha256 hash
/// over the borsh encoded keys are used as storage key segments.
///
/// This is different from [`super::LazyMap`], which uses [`storage::KeySeg`]
/// trait.
///
/// Additionally, [`LazyHashMap`] also writes the unhashed values into the
/// storage together with the values (using an internal `KeyVal` type).
pub struct LazyHashMap<K, V> {
key: storage::Key,
phantom_k: PhantomData<K>,
@@ -26,10 +41,10 @@ struct KeyVal<K, V> {
val: V,
}

impl<K, V> LazyMap<K, V>
impl<K, V> LazyHashMap<K, V>
where
K: BorshDeserialize + BorshSerialize,
V: BorshDeserialize + BorshSerialize,
K: BorshDeserialize + BorshSerialize + 'static,
V: BorshDeserialize + BorshSerialize + 'static,
{
/// Create or use an existing map with the given storage `key`.
pub fn new(key: storage::Key) -> Self {
@@ -85,7 +100,7 @@ where
key: &K,
) -> Result<Option<V>> {
let res = self.get_key_val(storage, key)?;
Ok(res.map(|elem| elem.1))
Ok(res.map(|(_key, val)| val))
}

/// Returns the key-value corresponding to the key, if any.
@@ -120,23 +135,12 @@ where
&self,
storage: &'a impl StorageRead,
) -> Result<impl Iterator<Item = Result<(K, V)>> + 'a> {
let iter = storage.iter_prefix(&self.get_data_prefix())?;
let iter = itertools::unfold(iter, |iter| {
match storage.iter_next(iter) {
Ok(Some((_key, value))) => {
match KeyVal::<K, V>::try_from_slice(&value[..]) {
Ok(KeyVal { key, val }) => Some(Ok((key, val))),
Err(err) => Some(Err(storage_api::Error::new(err))),
}
}
Ok(None) => None,
Err(err) => {
// Propagate errors into Iterator's Item
Some(Err(err))
}
}
});
Ok(iter)
let iter = storage_api::iter_prefix(storage, &self.get_data_prefix())?;
Ok(iter.map(|key_val_res| {
let (_key, val) = key_val_res?;
let KeyVal { key, val } = val;
Ok((key, val))
}))
}

/// Reads a key-value from storage
119 changes: 119 additions & 0 deletions shared/src/ledger/storage_api/collections/lazy_hashset.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
//! Lazy hash set.

use std::marker::PhantomData;

use borsh::{BorshDeserialize, BorshSerialize};

use super::super::Result;
use super::hasher::hash_for_storage_key;
use crate::ledger::storage_api::{self, StorageRead, StorageWrite};
use crate::types::storage;

/// Subkey corresponding to the data elements of the LazySet
pub const DATA_SUBKEY: &str = "data";

/// Lazy hash set.
///
/// This can be used as an alternative to `std::collections::HashSet` and
/// `BTreeSet`. In the lazy set, the elements do not reside in memory but are
/// instead read and written to storage sub-keys of the storage `key` given to
/// construct the set.
///
/// In the [`LazyHashSet`], the type of value `T` can be anything that
/// [`BorshSerialize`] and [`BorshDeserialize`] and a hex string of sha256 hash
/// over the borsh encoded values are used as storage key segments.
///
/// This is different from [`super::LazySet`], which uses [`storage::KeySeg`]
/// trait.
///
/// Additionally, [`LazyHashSet`] also writes the unhashed values into the
/// storage.
pub struct LazyHashSet<T> {
key: storage::Key,
phantom: PhantomData<T>,
}

impl<T> LazyHashSet<T>
where
T: BorshSerialize + BorshDeserialize + 'static,
{
/// Create or use an existing set with the given storage `key`.
pub fn new(key: storage::Key) -> Self {
Self {
key,
phantom: PhantomData,
}
}

/// Adds a value to the set. If the set did not have this value present,
/// `Ok(true)` is returned, `Ok(false)` otherwise.
pub fn insert<S>(&self, storage: &mut S, val: &T) -> Result<bool>
where
S: StorageWrite + StorageRead,
{
if self.contains(storage, val)? {
Ok(false)
} else {
let data_key = self.get_data_key(val);
storage.write(&data_key, &val)?;
Ok(true)
}
}

/// Removes a value from the set. Returns whether the value was present in
/// the set.
pub fn remove<S>(&self, storage: &mut S, val: &T) -> Result<bool>
where
S: StorageWrite + StorageRead,
{
let data_key = self.get_data_key(val);
let value: Option<T> = storage.read(&data_key)?;
storage.delete(&data_key)?;
Ok(value.is_some())
}

/// Returns whether the set contains a value.
pub fn contains(
&self,
storage: &impl StorageRead,
val: &T,
) -> Result<bool> {
let value: Option<T> = storage.read(&self.get_data_key(val))?;
Ok(value.is_some())
}

/// Returns whether the set contains no elements.
pub fn is_empty(&self, storage: &impl StorageRead) -> Result<bool> {
let mut iter = storage.iter_prefix(&self.get_data_prefix())?;
Ok(storage.iter_next(&mut iter)?.is_none())
}

/// An iterator visiting all elements. The iterator element type is
/// `Result<T>`, because iterator's call to `next` may fail with e.g. out of
/// gas or data decoding error.
///
/// Note that this function shouldn't be used in transactions and VPs code
/// on unbounded sets to avoid gas usage increasing with the length of the
/// set.
pub fn iter<'a>(
&self,
storage: &'a impl StorageRead,
) -> Result<impl Iterator<Item = Result<T>> + 'a> {
let iter = storage_api::iter_prefix(storage, &self.get_data_prefix())?;
Ok(iter.map(|key_val_res| {
let (_key, val) = key_val_res?;
Ok(val)
}))
}

/// Get the prefix of set's elements storage
fn get_data_prefix(&self) -> storage::Key {
self.key.push(&DATA_SUBKEY.to_owned()).unwrap()
}

/// Get the sub-key of a given element
fn get_data_key(&self, val: &T) -> storage::Key {
let hash_str = hash_for_storage_key(val);
self.get_data_prefix().push(&hash_str).unwrap()
}
}
58 changes: 32 additions & 26 deletions shared/src/ledger/storage_api/collections/lazy_map.rs
Original file line number Diff line number Diff line change
@@ -1,18 +1,30 @@
//! Lazy hash map
//! Lazy map.

use std::fmt::Display;
use std::marker::PhantomData;

use borsh::{BorshDeserialize, BorshSerialize};

use super::super::Result;
use crate::ledger::storage_api::{self, StorageRead, StorageWrite};
use crate::types::storage;
use super::ReadError;
use crate::ledger::storage_api::{self, ResultExt, StorageRead, StorageWrite};
use crate::types::storage::{self, KeySeg};

/// Subkey corresponding to the data elements of the LazyMap
pub const DATA_SUBKEY: &str = "data";

/// LazyMap ! fill in !
/// Lazy map.
///
/// This can be used as an alternative to `std::collections::HashMap` and
/// `BTreeMap`. In the lazy map, the elements do not reside in memory but are
/// instead read and written to storage sub-keys of the storage `key` used to
/// construct the map.
///
/// In the [`LazyMap`], the type of key `K` can be anything that implements
/// [`storage::KeySeg`] and this trait is used to turn the keys into key
/// segments.
///
/// This is different from [`super::LazyHashMap`], which hashes borsh encoded
/// key.
pub struct LazyMap<K, V> {
key: storage::Key,
phantom_k: PhantomData<K>,
@@ -21,8 +33,8 @@ pub struct LazyMap<K, V> {

impl<K, V> LazyMap<K, V>
where
K: BorshDeserialize + BorshSerialize + Display,
V: BorshDeserialize + BorshSerialize,
K: storage::KeySeg,
V: BorshDeserialize + BorshSerialize + 'static,
{
/// Create or use an existing map with the given storage `key`.
pub fn new(key: storage::Key) -> Self {
@@ -94,24 +106,17 @@ where
pub fn iter<'a>(
&self,
storage: &'a impl StorageRead,
) -> Result<impl Iterator<Item = Result<V>> + 'a> {
let iter = storage.iter_prefix(&self.get_data_prefix())?;
let iter = itertools::unfold(iter, |iter| {
match storage.iter_next(iter) {
Ok(Some((_key, value))) => {
match V::try_from_slice(&value[..]) {
Ok(decoded_value) => Some(Ok(decoded_value)),
Err(err) => Some(Err(storage_api::Error::new(err))),
}
}
Ok(None) => None,
Err(err) => {
// Propagate errors into Iterator's Item
Some(Err(err))
}
}
});
Ok(iter)
) -> Result<impl Iterator<Item = Result<(K, V)>> + 'a> {
let iter = storage_api::iter_prefix(storage, &self.get_data_prefix())?;
Ok(iter.map(|key_val_res| {
let (key, val) = key_val_res?;
let last_key_seg = key
.last()
.ok_or(ReadError::UnexpectedlyEmptyStorageKey)
.into_storage_result()?;
let key = K::parse(last_key_seg.raw()).into_storage_result()?;
Ok((key, val))
}))
}

/// Reads a value from storage
@@ -139,6 +144,7 @@ where

/// Get the sub-key of a given element
fn get_data_key(&self, key: &K) -> storage::Key {
self.get_data_prefix().push(&key.to_string()).unwrap()
let key_str = key.to_db_key();
self.get_data_prefix().push(&key_str).unwrap()
}
}
Loading