-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
segregate chain and state blockstores #5695
Conversation
This paves the way for better object lifetime management. Concretely, it makes it possible to: - have different stores backing chain and state data. - having the same datastore library, but using different parameters. - attach different caching layers/policies to each class of data, e.g. sizing caches differently. - specifying different retention policies for chain and state data. This separation is important because: - access patterns/frequency of chain and state data are different. - state is derivable from chain, so one could never expunge the chain store, and only retain state objects reachable from the last finality in the state store.
d63f67d
to
b34b4e0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGWM
The union blockstore takes a list of blockstores. It returns the first satisfying read, and broadcasts writes to all stores. It can be used for operations that require reading from any two blockstores, for example WalkSnapshot.
Is there a migration path for existing datastores to splitstore? |
It should happen automagically; on first run with the splistore, it will warm the hotstore by fetching all reachable objects from the old blockstore, which becomes the coldstore. The directory structure looks like this:
|
@magik6k ready again. |
Builds on #5484.
This paves the way for better object lifetime management.
Concretely, it makes it possible to:
This separation is important because:
I originally submitted this in #4771, but we @vyzo and I decided to do away with the caching for now. That was in the context of LMDB experiments (which uses the page cache aggressively). It may make sense to reintroduce Freecache on top of the hot badger store.