Flaky F3 itests

[rvagg]

Happening with some frequency, here's some recent examples:

• TestF3_PauseAndRebootstrap: https://github.com/filecoin-project/lotus/actions/runs/11046427121/job/30685760550?pr=12505
• TestF3_Rebootstrap: https://github.com/filecoin-project/lotus/actions/runs/11063981852/job/30741029746?pr=12518#step:9:25233

[rvagg]

🤞

[masih]

Reopening as it seem to continue to fail intermittently, though at a much lower failure rate. Example

[Stebalien]

Ok, I think this is caused by #12557 and should be "fixed" on the latest master because we now forcibly align "catchup" instances to 1/2 block time.

[masih]

Still flakes here

[Stebalien]

That one looks like a timeout that's just slightly too short.

[Stebalien]

Hm. No, it just never made progress after bootstrapping.

[Stebalien]

Ok, I did some digging and I'm not sure if F3 is to blame here (or the lotus integration side of it might be to blame?). See https://github.com/filecoin-project/lotus/actions/runs/11306342327/job/31446846079?pr=12591

I'm seeing "badger blockstore closed" errors 6 seconds into the test when the timeout is 20s. Then I see the test failures 15 seconds later.

I had originally assumed that the "badger blockstore closed" errors were due to shutting down the node because the test failed, but they're happening way too early for that.

Ideas:

1. The blockstore is closing itself (unlikely?).
2. Something is failing on start (e.g., some part of F3) causing the node to start shutting down.
3. The splitstore is doing something.