Manual finalization endpoint

[eserilev]

Introduce a new lighthouse endpoint that allows a user to force a manual finalization. This migrates data to the freezer db and prunes sidechains which may help reduce disk space issues on holesky at the moment.

[eserilev]

one quick and dirty thing we might be able to try

// add some for loop here so we only fetch at most one state per roots iterator
for {
    let state_roots = RootsIterator::new(&store, finalized_state)
        .take_while(|result| match result {
            Ok((_, _, slot)) => *slot >= current_split_slot,
            Err(_) => true,
        })
        .collect::<Result<Vec<_>, _>>()?;
    // do the hot db -> cold db migration + pruning
 }

[michaelsproul]

I think the RootsIterator is already inside a loop, so there should only be one at a time:

lighthouse/beacon_node/beacon_chain/src/migrate.rs

Lines 602 to 630 in 404e48d

	for (head_hash, head_slot) in heads {
	// Load head block. If it fails with a decode error, it's likely a reverted block,
	// so delete it from the head tracker but leave it and its states in the database
	// This is suboptimal as it wastes disk space, but it's difficult to fix. A re-sync
	// can be used to reclaim the space.
	let head_state_root = match store.get_blinded_block(&head_hash) {
	Ok(Some(block)) => block.state_root(),
	Ok(None) => {
	return Err(BeaconStateError::MissingBeaconBlock(head_hash.into()).into())
	}
	Err(Error::SszDecodeError(e)) => {
	warn!(
	log,
	"Forgetting invalid head block";
	"block_root" => ?head_hash,
	"error" => ?e,
	);
	abandoned_heads.insert(head_hash);
	continue;
	}
	Err(e) => return Err(e.into()),
	};
	let mut potentially_abandoned_head = Some(head_hash);
	let mut potentially_abandoned_blocks = vec![];
	// Iterate backwards from this head, staging blocks and states for deletion.
	let iter = std::iter::once(Ok((head_hash, head_state_root, head_slot)))
	.chain(RootsIterator::from_block(&store, head_hash)?);

Similarly the RootsIterator itself jumps back over states one at a time too.

The only thing I can think is that we're putting these states in the cache, and that's enough to cause an OOM with --state-cache-size 32

[jimmygchen]

I don't think we ever run store migrator in the foreground other than testing, but probably should support it given we do it for all 3 other notifications, and if we want to test this with BeaconChainHarness, it won't be immediately obvious why this doesn't get run.

[michaelsproul]

Good point. We should do this for the clean version of this PR