Splitstore discussion #5788
Replies: 11 comments 10 replies
-
|
So the first thing to do is just enable it! We'd love to hear back with your experience and help us improve it, as we want to make it the default in the future. We also recommend applying this patch: #5778, which should land to master (already approved) fairly soon. |
Beta Was this translation helpful? Give feedback.
-
|
We have also opened issue #5789 tracking next steps in development. |
Beta Was this translation helpful? Give feedback.
-
|
Another update: We have an open PR (#5808) that does a few optimizations:
|
Beta Was this translation helpful? Give feedback.
-
|
Dummy question: What is finality in this context? Does it mean an epoch that has been finalized? How many epoch does it take for one to finalize? Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
A finality is 900 epochs; it's the maximum reorg depth. |
Beta Was this translation helpful? Give feedback.
-
|
I have been using splitstore for about a month now on a full node synced from genesis, it has been very helpful allowing me to put the hot store on fast NVME and the cold store on slower sata SSDs. I've seen no issues since enabling it. |
Beta Was this translation helpful? Give feedback.
-
you cant or you’ll lose all the recent chain data and have to restart from a snapshot again |
Beta Was this translation helpful? Give feedback.
-
|
further improvement: #6577 |
Beta Was this translation helpful? Give feedback.
-
|
Just a heads up, #6474 brings great improvements -- in fact we call this splitstore v1 (what we currently have is v0). Stay tuned! |
Beta Was this translation helpful? Give feedback.
-
|
I've run into an issue. I have limited disk space on my hot store, but lots of storage on my cold store. But due to a different error my lotus lost sync for a while. Now that lotus is trying to catch up, it never compacts and fills my hot store disk, then stops syncing again. I tried starting lotus with --no-bootstrap in the hope that I could trigger a compaction manually, but there is no way to do that either. |
Beta Was this translation helpful? Give feedback.
-
|
Ouch, this is indeed annoying, I would classify it as a BUG. Can you open an issue for it so that we can track it and fix it? |
Beta Was this translation helpful? Give feedback.
-
|
As a (temporary workaround) solution for your current predicament, you can:
This should fix the system; I also recommend running latest master (soon to be in v1.11.1) as it brings in splitstore v1. Sidenote: this solution, while generally applicable, might need a small tweak to force the splitstore to warmup: you need to delete |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, this is part of my ongoing effort to maintain a full node synced from genesis, not snapshot. So I'd rather not wipe out my coldstore with a snapshot. In #6474
Does this mean that it will not do any compaction until lotus is in sync? |
Beta Was this translation helpful? Give feedback.
-
Yes
Aaah, yes, let's not do that. Basically the idea is:
In the meantime I'll try to fix the underlying issue by detecting when the node is way out of sync (or syncing from scratch) and redirecting writes to the coldstore for the sync, followed by a warmup |
Beta Was this translation helpful? Give feedback.
-
|
Created issue #6769 for the bug. |
Beta Was this translation helpful? Give feedback.
-
|
Some exciting news, as Splitstore.v1 is shipping with v1.11.1-rc1. |
Beta Was this translation helpful? Give feedback.
-
|
I have been running SplitStore for a couple of weeks now and everything is looking very stable. My hot store is hovering between 100GiB and 150GiB which is ideal. |
Beta Was this translation helpful? Give feedback.
-
|
yes, you can just move it and symlink!
…On Tue, Aug 31, 2021, 23:45 TippyFlits ***@***.***> wrote:
I have been running SplitStore for a couple of weeks now and everything is
looking very stable. My hot store is hovering between 100GiB and 150GiB
which is ideal.
I have one question... Is it possible to specify an alternative path for
my cold store. Both stores are currently located in my .lotus repo folder
of my NVME. I would like to move my cold store to my long-term HDD storage
cluster.
Many thanks!!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#5788 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAI4SQHNOOLFEVJKDQQ5KDT7U5NRANCNFSM4ZA6EJCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
Beta Was this translation helpful? Give feedback.
-
|
I added Current sizes
|
Beta Was this translation helpful? Give feedback.
-
|
no, you cant safely delete the coldstore yet -- you could use the discard
option however, and it wont gorw further.
…On Thu, Sep 9, 2021, 23:31 Stuart Berman ***@***.***> wrote:
I added [Chainstore] EnableSplitstore = true in my .lotus/config.toml a
few days ago.
I see the hot store growing, am I able to delete the
.lotus/datastore/chain directory or contents as its size is not changing?
Current sizes
365G .lotus/datastore/chain
60G .lotus/datastore/splitstore/hot.badger
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#5788 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAI4SVPIOZ6Y2ZMH3APU7LUBEKTNANCNFSM4ZA6EJCA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I have plenty of extra space (1TiB) for now, looking forward to a feature that will compact the coldstore or remove older pieces. I think we also need to understand how we would update the chain with a snapshot. I used to replace the chain every 3-4 weeks due to growth and performance. In the current model, would we still shut down the daemon, download and reimport a minimal snapshot to the .lotus/datastore/chain directory after deleting the old chain. I wish I had done that before enabling split store. |
Beta Was this translation helpful? Give feedback.
-
Lotus 1.5.1 introduced an experimental splitstore in #4992. This feature reduces the performance impact of the very large Filecoin datastore.
Design
This release also introduces the splitstore, a new optional blockstore that segregates the monolithic blockstore into cold and hot regions. The hot region contains objects from the last 4-5 finalities plus all reachable objects from two finalities away. All other objects are moved to the cold region using a compaction process that executes every finality, once 5 finalities have elapsed.
The splitstore allows us to separate the two regions quite effectively, using two separate badger blockstores. The separation means that the live working set is much smaller, which results in potentially significant performance improvements. In addition, it means that the coldstore can be moved to a separate (bigger, slower, cheaper) disk without loss of performance.
The design also allows us to use different implementations for the two blockstores; for example, an append-only blockstore could be used for coldstore and a faster memory mapped blockstore could be used for the hotstore (eg LMDB). We plan to experiment with these options in the future.
Once the splitstore has been enabled, the existing monolithic blockstore becomes the coldstore. On the first head change notification, the splitstore will warm up the hotstore by copying all reachable objects from the current tipset into the hotstore. All new writes go into the hotstore, with the splitstore tracking the write epoch. Once 5 finalities have elapsed, and every finality thereafter, the splitstore compacts by moving cold objects into the coldstore. There is also experimental support for garbage collection, whereby nunreachable objects are simply discarded.
Activation
To enable the splitstore, add the following to
config.toml:Experimentation
This feature is still experimental, so use at your own risk. Having said that, we would love if more adventurous users tried it out, and reported on their experience. In particular, we're curious to know about:
Beta Was this translation helpful? Give feedback.
All reactions