boostrapper: compact blocks before iterating them

[a1k0n]

Why this should be merged

Trying out @StephenButtolph's new interval tree based bootstrapping code, I found that it indeed fetched blocks much more quickly but that actually executing them was taking much longer than before. Ran a profile and found that most of the time was spent just iterating to the next block:

So as a quick experiment I hacked this in, and it dramatically reduced the ETA for execution.

How this works

Theory is that P chain blocks are stored in a pathological order on disk inside leveldb, despite the keys being contiguous, so spending a little time up front re-sorting the data blocks saves a lot of time later.

How this was tested

Seat of my pants, really. But this is the profile afterwards:

I had left it running overnight, and the ETA this morning was about 6 hours. When I made this change and compacted my DB, the ETA dropped quickly down to ~50 minutes.

[a1k0n]

I guess this should have a check that there are more than ~5000 blocks to execute before bothering with compaction?

[StephenButtolph]

I guess this should have a check that there are more than ~5000 blocks to execute before bothering with compaction?

I think it's fine... We're only compacting the block portion of the database... So if there are only a couple thousand blocks or so, I'd expect the compaction to be really fast.

[a1k0n]

I guess this should have a check that there are more than ~5000 blocks to execute before bothering with compaction?

I think it's fine... We're only compacting the block portion of the database... So if there are only a couple thousand blocks or so, I'd expect the compaction to be really fast.

Well, empirically it adds 2-3 seconds of startup time on every startup which I'd rather avoid, so I went ahead and added the check.

[a1k0n]

added the Compact fix to this PR; PTAL

[a1k0n]

Sorry, just saw your ask for a separate PR. Yeah, one sec.

[StephenButtolph]

super minor nits then lgtm

[StephenButtolph]

Great catch! 🚀

[StephenButtolph]

Going to run some perf tests on my end as well tonight - but code lgtm

[joshua-kim]

Minor nit then LGTM

[a1k0n]

I'm doing some tests too, and I think maybe this PR is a bad idea, and just tuning the leveldb writeBuffer and compactionTable* options would work better. We have ~15GB of P chain blocks on mainnet, being continuously re-compacted into 2MB files. Even just downloading the initial P chain blocks is mostly blocked on all the re-compaction leveldb is doing, at least on my laptop here.

I also tried pebbledb but its performance was even worse.

Did you guys do much tuning on the defaults here?

[StephenButtolph]

Did you guys do much tuning on the defaults here?

Not particularly. We do need to be careful with changing something like that to avoid any regressions in the (more important) normal operation of the node (rather than bootstrapping). Admittedly the verification logic is still happening though... So they might not be too different. Although bootstrapping is moving the blocks around whereas normally the blocks are written once by hash.

I'd be happy to merge this PR (once I can test the perf out on my node) and then we could look into optimizing level/pebble configs (we're hoping to move away from leveldb once we can get on a stable release of pebbledb).

[StephenButtolph]

I tested this on my laptop and didn't see the same performance improvement. Going to spin up some AWS boxes to get a more reproducible benchmark.

[a1k0n]

Yeah, might have been particular to my setup which is why I'm not particularly enthusiastic about this change...

[StephenButtolph]

Did a run on an AWS c5.2xlarge box with 1 TiB of storage on a GP3 SSD (all defaults). I ended up executing the first 1.5m mainnet P-chain blocks.

The current tip of master took:

• 29m 11s to execute
• ETA for completion was another 3h 51m 38s
• Total ETA 4h 20m 49s

This PR took:

• 4m 33s to run the compaction
• 27m 28s to execute
• ETA for completion was another 3h 37m 54s.
• Total ETA 4h 9m 55s

So, running the compaction did reduce the expected total time (after 1.5 M block executions) by 10m 54s or ~4.2%

@a1k0n what type of disk were you using?

[a1k0n]

@a1k0n what type of disk were you using?

This was NVMe on a Linux laptop, ext4 filesystem. I have no idea why the performance was so variable. Happy to abandon this PR.

[StephenButtolph]

We haven't seen a performance regression with this, and it seems like in some cases this can be a large performance improvement... So it makes sense to me to merge.