Add prune command to lotus-shed (maybe lotus chain?)

[travisperson]

It would be nice to have the ability to prune all non linked keys (that do not exists under a certain tipset) from the data-store . This can be used as a recovery for when partial writes occur (I believe) in badger, or as an general purpose maintenance to reduce data-store size (eg prune everything from HEAD-1000)

lotus-shed prune <tipset>

Adding the help wanted tag because I'm interested in feedback on the idea.

[raulk]

There is a lot of nuance to this. We can't just retain the state referenced from a single tipset, because consensus uses lookbacks. So we'd need to retain at least one lookback (finality) period.

We not only need to retain the state trees from the finality range, but also the tipsets, blocks and messages themselves.

Most of the development that will ultimately enable us to prune/lighten the blockstore will be done in #5056.

This can be used as a recovery for when partial writes occur (I believe) in badger

@travisperson just to confirm, you are thinking of a case where an entry is written such that the key is fully represented, but the value is truncated? Therefore it errors permanently when reading it during the chain processing? In this case, the recovery you have in mind would be to truncate the chain up to the tipset before the borked block was first referenced?

[travisperson]

There is a lot of nuance to this. We can't just retain the state referenced from a single tipset, because consensus uses lookbacks. So we'd need to retain at least one lookback (finality) period.

Ya, the assumption in the original issue is that tipsets link to their parents so the entire chain would be preserved by default. The primary motivation for this was to remove all the objects that are not referenced by the chain. This would include all the small reorgs that occurred over the lifetime of the node.

@travisperson just to confirm, you are thinking of a case where an entry is written such that the key is fully represented, but the value is truncated? Therefore it errors permanently when reading it during the chain processing?

Ya, for example a failure during chain sync resulted in a partially written object that is part of the state.

In this case, the recovery you have in mind would be to truncate the chain up to the tipset before the borked block was first referenced?

In this particular case I would set the head back a few tipsets and then prune all non references objects from the datastore. The idea being that all objects created would then be removed, which would hopefully include the partially written object.

[travisperson]

This kind of exists, but doesn't work very well, the recent splitstore work makes this operation obsolete.