SPX Snapshot & SplitStoreV2 - Testing & Call Outs #9179

TippyFlitsUK · 2022-08-17T19:31:46Z

TippyFlitsUK
Aug 17, 2022
Maintainer

Hello SPX Fellows

Thank you to everyone who participated in the initial SpiltStoreV2 testing round!!

@ZenGround0 has been working tirelessly on the upcoming SplitStoreV2 rollout and we have a shiny new release for you to break 😉.

Thanks to @travisperson we also have the brand new Snapshot Service which was soft-launched on 4th August 2022! You can read more about the new service on this public Notion page.

It goes without saying that both of these new releases represent a huge improvement over the current implementations!

We would sincerely appreciate your help in testing both of these new enhancements and seeing as enabling splitstore ideally also involves a new snapshot import, we're going to test both at the same time. This assignment is open to all SPX fellows, please see the details below to get started,

This assignment is applicable to both mainnet and calibration instances but we are particularly interested in hearing about your mainnet experiences. In the event of any critical failures on your node, recovery is a very familiar procedure as detailed below.

Please reach out to @jennijuju or @TippyFlits on Slack if you have any questions! 🙏

Step 1: ⚠️ Backup ALL lotus wallets before doing anything! (`lotus wallet list`) ⚠️

lotus wallet export <wallet-address-1> > file-1
lotus wallet export <wallet-address-2> > file-2
lotus wallet export <wallet-address-3> > file-3
etc...

Step 2: Download a new lightweight snapshot & Update to testing release

You are all well used to downloading snapshots and have your own techniques already in place. Please note the new snapshot links as listed on the Notion page and Slack announcement:
Mainnet - https://snapshots.mainnet.filops.net/minimal/latest
Calibration - https://snapshots.calibrationnet.filops.net/minimal/latest
Upgrade both daemon and miner instances to: 1.17.1-rc3

Step 3: Enable SplitStore and configure ColdStore mode

SplitStore can be run in universal,discard and now AutoPrune modes and all three modes have been enhanced since the previous testing round. We are looking for a selection of SPX fellows to test a combination of three. Select 🎉 if you are testing universal mode and 🚀 if you are opting for discard mode and 👀 if your are testing AutoPrune. Please review the emoji count at the bottom of this post before selecting a mode, an even balance of the 3 options would be much appreciated. The selected mode can be set in your /.lotus/config.toml as follows:

Universal Mode	Discard Mode	AutoPrune Mode
[Chainstore] EnableSplitstore = true [Chainstore.Splitstore] ColdStoreType = "universal" EnableColdStoreAutoPrune = false	[Chainstore] EnableSplitstore = true [Chainstore.Splitstore] ColdStoreType = "discard" EnableColdStoreAutoPrune = false	[Chainstore] EnableSplitstore = true [Chainstore.Splitstore] ColdStoreType = "universal" EnableColdStoreAutoPrune = true

Step 4: Prepare your `/.lotus/datastore` and import snapshot

Stop all miner and daemon instances
If you are not yet running Splitstore - Delete all files in your /.lotus/datastore/chain folder.
If you are already running SplitStore - Delete all files in your /.lotus/datastore/chain folder and clear your /.lotus/datastore/splitstore folder by running ./lotus-shed splitstore clear --repo=/path/to/your/lotus/repo
Import your new lightweight snapshot in the normal manner
Please make a note of your /.lotus/datastore/chain and /.lotus/datastore/splitstore folder sizes immediately following snapshot import and successful chain sync completion and record your results in the reply card below.
Set lotus log set-level --system splitstore debug

Step 5: Reply card

Please provide the information below after completion of the above steps and immediately following the successful start-up of your SplitStore enabled daemon:

- If the new lightweight snapshot downloads correctly
- If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
	/.lotus/datastore/chain
	/.lotus/datastore/splitstore

Please complete the following after observing usual operations:

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your /.lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

We will be asking for updated sizes of your /.lotus/datastore/chain and /.lotus/datastore/splitstore folders on a weekly basis for a period of 4 weeks. We will publish weekly reminders in #spx-current-class but this is obviously dependent on your testing start date. Sizes should be recorded at the end of each week and in GiB format. Please consider setting weekly reminders so metrics remain consistent and accurate. 🙏

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
	/.lotus/datastore/splitstore
	/.lotus/datastore/chain

Reminders and Risk Mitigations

In the event of a critical failure or data corruption, you will need to re-import a lightweight snapshot with the following steps:

❗ Ensure that all lotus wallets are fully backed up.
Stop lotus and delete your /.lotus /datastore folder
Download a new lightweight snapshot
Disable SplitStoreV2 in your /.lotus/config.toml - EnableSplitstore = false

📓 In the event that you are unable to start your daemon after trying the above steps, please delete just the folders in your /.lotus directory and retry the lightweight snapshot import. ❗ ONLY DELETE THE FOLDERS - the api, token and config.toml files should not be deleted!

Questions are welcomed in this discussion, the team will be monitoring and will check in regularly.

Issues can be reported at https://github.com/filecoin-project/lotus/issues. Issues should be submitted with:

Daemon logs
GO routine dumps lotus pprof goroutines and lotus-miner pprof goroutines
The operation/step at which the problem occurred
The stack trace or logs as relevant. As always, for large log files, you may upload them to this Google Drive folder and link them in the issue.

📓 And thank you to Reiers Productions for the very cool banner!!

stuberman · 2022-08-18T22:43:06Z

stuberman
Aug 18, 2022

- If the new lightweight snapshot downloads correctly
- If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
AutoPrune	/.lotus/datastore/chain	84G on start - 84G after sync
AutoPrune	/.lotus/datastore/splitstore	1.2G on start - 23G after sync

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4	Week 5
AutoPrune	/.lotus/datastore/splitstore	159G	163G	153G	161G	178G
AutoPrune	/.lotus/datastore/chain	244G	403G	631G	831G	1.1T

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your /.lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

6 replies

TippyFlitsUK Aug 19, 2022
Maintainer Author

Well, that's new!! Thanks Stu, I will liaise with the team and get back to you.

TippyFlitsUK Aug 19, 2022
Maintainer Author

Just to confirm Stu, this is as a result of the new proofs release and nothing to be concerned about. Interesting spot, we don't often see new random new log entries in the wild these days!! 😉

stuberman Aug 19, 2022

I see the new path: /.lotus/datastore/splitstore/hot.badger
But is a symlink supposed to be created already? Mine is not symlinked.
** Update *** Now there is a symlink as of Aug 20

stuberman Aug 19, 2022

Just to confirm Stu, this is as a result of the new proofs release and nothing to be concerned about. Interesting spot, we don't often see new random new log entries in the wild these days!! 😉

I replaced my Bellman var with the RUST var, I am assuming that is the new var? Or should I return it to the Bellman var?

TippyFlitsUK Aug 24, 2022
Maintainer Author

No need to change back Stu, this will be the default going forward.

tmyuu · 2022-08-19T05:14:01Z

tmyuu
Aug 19, 2022

Step 4:

$ rm -rf /usr/local/filecoin/.node/datastore/chain/*
$ ./lotus-shed splitstore clear --repo=/usr/local/filecoin/.node
clearing splitstore directory...
deleting splitstore keys from metadata datastore...
2022-08-19T14:07:29.265+0900	INFO	badger	[email protected]/levels.go:183	All 0 tables opened in 0s

2022-08-19T14:07:29.268+0900	INFO	badger	[email protected]/value.go:1158	Replaying file id: 0 at offset: 0

2022-08-19T14:07:29.268+0900	INFO	badger	[email protected]/value.go:1178	Replay took: 4.028µs

2022-08-19T14:07:29.349+0900	INFO	badger	[email protected]/levels.go:183	All 0 tables opened in 0s

2022-08-19T14:07:29.351+0900	INFO	badger	[email protected]/value.go:1158	Replaying file id: 0 at offset: 0

2022-08-19T14:07:29.352+0900	INFO	badger	[email protected]/value.go:1178	Replay took: 4.789µs

deleting /splitstore/baseEpoch from datastore...
deleting /splitstore/compactionIndex from datastore...
deleting /splitstore/markSetSize from datastore...
deleting /splitstore/pruneEpoch from datastore...
deleting /splitstore/warmupEpoch from datastore...

Step5:

- If the new lightweight snapshot downloads correctly
- If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
AutoPrune Mode	/.lotus/datastore/chain	83G
AutoPrune Mode	/.lotus/datastore/splitstore	85G

Please complete the following after observing usual operations:

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your /.lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

Sizes of the folders on a weekly basis for a period of 4 weeks:

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
AutoPrune Mode	/.lotus/datastore/splitstore	144GB	321GB
AutoPrune Mode	/.lotus/datastore/chain	243GB	436GB

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
Discard Mode	/.lotus/datastore/splitstore
Discard Mode	/.lotus/datastore/chain

0 replies

benjaminh83 · 2022-08-22T13:44:32Z

benjaminh83
Aug 22, 2022

If the new lightweight snapshot downloads correctly
If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
AutoPrune	/.lotus/datastore/chain	85G
AutoPrune	/.lotus/datastore/splitstore	9G

If lotus daemon stays in sync
if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
if your /.lotus/datastore/chain folder is being successfully garbage collected
if normal mining tasks appear to be processed as expected
if normal daemon tasks appear to be processed as expected

Sizes of the folders on a weekly basis for a period of 4 weeks:

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
AutoPrune Mode	/.lotus/datastore/splitstore	86G	97G	90G	104G
AutoPrune Mode	/.lotus/datastore/chain	238G	442G	646G	793G

Round2 on v1.17.2-rc2

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
AutoPrune Mode	/.lotus/datastore/splitstore	173G
AutoPrune Mode	/.lotus/datastore/chain	89G

20 replies

benjaminh83 Sep 30, 2022

@TippyFlitsUK We should take another round on this using v.1.17.2.-rc2. So I have reverted over again to:

export LOTUS_CHAINSTORE_SPLITSTORE_COLDSTORETYPE="universal"
export LOTUS_CHAINSTORE_SPLITSTORE_ENABLECOLDSTOREAUTOPRUNE=true

Tracking it once again in here :))))

TippyFlitsUK Sep 30, 2022
Maintainer Author

Thanks Ben!! Really appreciate your efforts but I'm afraid that the fix for this is still pending 😢

benjaminh83 Sep 30, 2022

Hmm. @stuberman mentioned it worked for him now...

stuberman Sep 30, 2022

I upgraded to v1.17.2-rc2 24 hours ago (at release) and here are my stats (based on a new snapshot of 85G) running coldstore autoprune/universal

du -sch /.lotus/datastore/* | sort -rh

255G total
156G /.lotus/datastore/splitstore
100G /.lotus/datastore/chain

TippyFlitsUK Oct 1, 2022
Maintainer Author

Thanks guys!!

namoo1 · 2022-08-23T01:04:31Z

namoo1
Aug 23, 2022

Meest NAM

- If the new lightweight snapshot downloads correctly
- If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
AutoPrune	/.lotus/datastore/chain	5G
AutoPrune	/.lotus/datastore/splitstore	5.4G

If lotus daemon stays in sync
if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
if your /.lotus/datastore/chain folder is being successfully garbage collected
if normal mining tasks appear to be processed as expected
if normal daemon tasks appear to be processed as expected

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
AutoPrune Mode	/.lotus/datastore/splitstore	7.0G	6.5G	7.0G
AutoPrune Mode	/.lotus/datastore/chain	9.5G	12.0G	19.0G

After performing "Split store", wdpost failed, but let's see if there are any other causes

Daemon:  1.17.1-rc3+calibnet+git.079dab0d7+api1.5.0
Local: lotus-miner version 1.17.1-rc3+calibnet+git.079dab0d7

The capacity of the "splitstore" has decreased...
Is there any other reason?

1 reply

TippyFlitsUK Aug 23, 2022
Maintainer Author

Thanks, Meest!

We haven't had any other reports of this affecting wdpost but please keep us updated! 🙏

SBudo · 2022-08-23T03:19:20Z

SBudo
Aug 23, 2022

- If the new lightweight snapshot downloads correctly
- If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
Discard	/.lotus/datastore/chain	85GiB
Discard	/.lotus/datastore/splitstore	60GiB

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your /.lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
Discard	/.lotus/datastore/chain	85GiB	85GiB	85GiB	85GiB
Discard	/.lotus/datastore/splitstore	129GiB	148GiB	155GiB	287GiB

5 replies

SBudo Aug 23, 2022

Note: in my case, the /.lotus/datastore/splitstore/hot.badger is not a symlink, but just a folder

TippyFlitsUK Aug 23, 2022
Maintainer Author

Thanks, Stephane!! You should find this turns into a symlink following the first compaction. 🙏

SBudo Aug 25, 2022

Correct you are! :-)

SBudo Sep 1, 2022

Is it expected for the splitstore folder to continue to grow even though I'm using the "discard" mode?

TippyFlitsUK Sep 2, 2022
Maintainer Author

There will be some small increases over time Stephane but it should be fairly minimal.

cryptowhizzard · 2022-08-31T19:48:32Z

cryptowhizzard
Aug 31, 2022

SplitStore Mode Directory Size in GiBs immediately following daemon start
Autoprune /.lotus/datastore/chain 94GiB
Autoprune /.lotus/datastore/splitstore 20GiB

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your /.lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

SplitStore Mode Directory Week 1 Week 2 Week 3 Week 4
Autoprune /.lotus/datastore/chain 94GiB
Autoprune /.lotus/datastore/splitstore 20GiB

Edit: There is not a symlink for hot.badger. It is a normal directory?

0 replies

dgreuel2002 · 2022-09-09T16:37:04Z

dgreuel2002
Sep 9, 2022

Step 5: Reply card

Please provide the information below after completion of the above steps and immediately following the successful start-up of your SplitStore enabled daemon:

- If the new lightweight snapshot downloads correctly
- If the new lightweight snapshot imports correctly

SplitStore Mode	Directory	Size in GiBs immediately following daemon start
AutoPrune	/.lotus/datastore/chain	88G after sync
Auto Prune	/.lotus/datastore/splitstore	44G after sync

Please complete the following after observing usual operations:

- If lotus daemon stays in sync
- if you can see the following new symlink path - /.lotus/datastore/splitstore/hot.badger 🔥 🦡
- if your /.lotus/datastore/chain folder is being successfully garbage collected
- if normal mining tasks appear to be processed as expected
- if normal daemon tasks appear to be processed as expected

We will be asking for updated sizes of your /.lotus/datastore/chain and /.lotus/datastore/splitstore folders on a weekly basis for a period of 4 weeks. We will publish weekly reminders in #spx-current-class but this is obviously dependent on your testing start date. Sizes should be recorded at the end of each week and in GiB format. Please consider setting weekly reminders so metrics remain consistent and accurate. 🙏

SplitStore Mode	Directory	Week 1	Week 2	Week 3	Week 4
Autoprune	/.lotus/datastore/splitstore	277G
Autoprune	/.lotus/datastore/chain	106G

0 replies

SPX Snapshot & SplitStoreV2 - Testing & Call Outs #9179

TippyFlitsUK Aug 17, 2022 Maintainer

Hello SPX Fellows

Step 1: ⚠️ Backup ALL lotus wallets before doing anything! (lotus wallet list) ⚠️

Step 2: Download a new lightweight snapshot & Update to testing release

Step 3: Enable SplitStore and configure ColdStore mode

Step 4: Prepare your /.lotus/datastore and import snapshot

Step 5: Reply card

Reminders and Risk Mitigations

Replies: 7 comments · 32 replies

TippyFlitsUK Aug 19, 2022 Maintainer Author

TippyFlitsUK Aug 19, 2022 Maintainer Author

TippyFlitsUK Aug 24, 2022 Maintainer Author

Step 4:

Step5:

Round2 on v1.17.2-rc2

TippyFlitsUK Sep 30, 2022 Maintainer Author

TippyFlitsUK Oct 1, 2022 Maintainer Author

Meest NAM

TippyFlitsUK Aug 23, 2022 Maintainer Author

TippyFlitsUK Aug 23, 2022 Maintainer Author

TippyFlitsUK Sep 2, 2022 Maintainer Author

Step 5: Reply card

TippyFlitsUK
Aug 17, 2022
Maintainer

Step 1: ⚠️ Backup ALL lotus wallets before doing anything! (`lotus wallet list`) ⚠️

Step 4: Prepare your `/.lotus/datastore` and import snapshot

Replies: 7 comments 32 replies

TippyFlitsUK Aug 19, 2022
Maintainer Author

TippyFlitsUK Aug 19, 2022
Maintainer Author

TippyFlitsUK Aug 24, 2022
Maintainer Author

TippyFlitsUK Sep 30, 2022
Maintainer Author

TippyFlitsUK Oct 1, 2022
Maintainer Author

TippyFlitsUK Aug 23, 2022
Maintainer Author

TippyFlitsUK Aug 23, 2022
Maintainer Author

TippyFlitsUK Sep 2, 2022
Maintainer Author