Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trie: use Uint8Array as value type in DB #3067

Merged
merged 18 commits into from
Oct 27, 2023
Merged

Conversation

jochem-brouwer
Copy link
Member

@jochem-brouwer jochem-brouwer commented Sep 28, 2023

In this PR the trie will use values as Uint8Array in the DB.

TODOs

  • Ensure client uses Uint8Array by default
  • Add client flag to use old version of DB (which used strings)
  • Ensure VM and other places where tries are used (block?) an option to also use Uint8Array values there
  • Mark blockchain/state tests + client to use Uint8Array DB
  • Write tests on Trie

Note: I tested that with the new --useStringValueTrieDB flag, client can still run the "old version" of the DB. If this flag is not provided and an old DB version is ran, it fails.

This PR also:

  • Ensures monorepo-wide we only create new Trie if we need it (we do not need a Trie when we want the transactions root of 0 transactions, for instance)
  • Optimizes block transactionsTrieIsValid: calculate the transactionsRoot once and caches this (does not cache trie, just the root)

@codecov
Copy link

codecov bot commented Sep 28, 2023

Codecov Report

Merging #3067 (d4675c9) into master (b848033) will decrease coverage by 0.01%.
The diff coverage is 95.94%.

Additional details and impacted files

Impacted file tree graph

Flag Coverage Δ
block 88.80% <100.00%> (+0.01%) ⬆️
blockchain 92.27% <ø> (ø)
client 87.56% <91.66%> (+0.01%) ⬆️
common 98.19% <ø> (ø)
ethash ?
evm 71.87% <ø> (ø)
rlp ?
statemanager 90.30% <ø> (ø)
trie 89.42% <94.28%> (-0.31%) ⬇️
tx 96.36% <ø> (ø)
util 87.57% <ø> (ø)
vm 76.13% <100.00%> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@holgerd77
Copy link
Member

Really important here to check as early on as possible on the performance side. I guess this is really only with it if it brings at least a 3-5 % jump in overall block execution.

@jochem-brouwer
Copy link
Member Author

I will report exact numbers, but I ran the entire blockchain test suite with and without this change (i.e. store values as Uint8Arrays) and as far as I remember the performance gain was above 10%. (I think 12%)

@jochem-brouwer jochem-brouwer marked this pull request as ready for review October 27, 2023 10:04
@holgerd77
Copy link
Member

Updated this via UI

Copy link
Member

@holgerd77 holgerd77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, synced the first 200.000 blocks with this, works like a charm! 🤩

Will merge.

}
result = equalsBytes(this.txTrie.root(), this.header.transactionsTrie)
result = equalsBytes(this.cache.txTrieRoot, this.header.transactionsTrie)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. 🙂👍

default: false,
deprecated:
'Usage of old DBs which uses string-values is temporary. Please sync new instances without this option.',
})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No big deal, but I think we should rather use StateDB as a terminoloy for something like this in the future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, good point, so it is not being confused with a blocks DB or anything 😄

@holgerd77 holgerd77 merged commit 0443f9b into master Oct 27, 2023
@holgerd77 holgerd77 deleted the db-store-uint8array branch October 27, 2023 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants