ApplyBucketWork at per-bucket level

[ThomasBrady]

Description

Refactors ApplyBucketsWork to apply buckets per-bucket rather than per-level. Deleted a lot of book keeping code referring to curr/snap apply ordering.

Issue: #4290

Checklist

• Reviewed the contributing document
• Rebased on top of master (no merge commits)
• Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
• Compiles
• Ran all tests
• If change impacts performance, include supporting evidence per the performance document

[SirTyson]

Currently, this doesn't work because advance is not being called properly. The way the work classes are designed, we do long running jobs in chunks to not hog resources. The expectation is that doWork() will be called periodically, do a little work, then return quickly.

BucketApplicator assumes this interface. If you look at BucketApplicator::advance, you'll see that it only applies LEDGER_ENTRY_BATCH_COMMIT_SIZE entries then exits. This means that, especially for large buckets, BucketApplicator::advance must be called many times before the Bucket finishes applying. The issue is this code only calls BucketApplicator::advance a single time before moving on to the next bucket.

If you look at the old ApplyBucketsWork, we maintained mFirstBucketApplicator and mSecondBucketApplicator pointers to persist Applicators across calls to doWork. We only move on to the next bucket when *mFirstBucketApplicator is false (note that this is pointer dereference, not checking if the pointer itself is null), indicating the applicator has reached EOF. You need to implement similar logic, where you maintain an applicator pointer and only advance to the next bucket when the applicator is false. Currently you check if applicator is true once then call apply, but you always advance to the next bucket unconditional, even if after the first call to advance applicator is still true.

For changes like this, in the future I'd recommend running a watcher node and catchup to the network in addition to the unit tests.

[SirTyson]

Overall looks good to me! Just a few nit pics. There's still some complexity involving mLevel and isCurr I'd like to see get factored out, but unfortunately it looks like this may be required given how we check invariants :(

[SirTyson]

r+ ba9f43f

[SirTyson]

Trying again

r+ 545b721

[graydon]

@latobarita: retry

[graydon]

r+ 545b721

[ThomasBrady]

@latobarita: retry