Enable faster sync

[antiochp]

Non-consensus related implementation for RFC mimblewimble/grin-rfcs#29

• refactor vec_backend so we can use it outside of tests
• introduce a "hash only" vec_backend for the accumulator
• introduce bitmap_accumulator -
  • MMR backed by a hash only vec backend
  • leaves are 1024 bit chunks of the leafset bitmap
  • it is relatively small, keep it in memory only
  • no need to store data in leaves (hash only) as data is maintained in the existing leafset
• implement "rewind" logic so we handle bitmap_accumulator correctly during a fork/reorg
  • it is not append-only as the bitmap updates over time
• build new bitmap accumulator root for each new block
  • log only (no consensus changes)

We will eventually be able to use this bitmap accumulator root to verify fragments of the leafset against subtrees of the existing output PMMR. This will allow us to verify "fast sync" data in chunks, without needing to download the full txhashset.zip which is increasingly becoming a bottleneck and single point of failure during initial fast sync.

---

TODO -

• test coverage for various bitmap accumulator scenarios (particularly rewind)
• test coverage for bitmap fragment serialization (endianness etc)

[antiochp]

We now log this for every new block -

20191115 11:48:41.272 DEBUG grin_chain::types - validate roots: 00000c3194d8 at 435707, 
output_root: 4856008b7c31, output pmmr: 4856008b7c31 (bitmap: 7a488eb915b8, merged: 454f61a2a6df)

It would be good to get this merged and have multiple nodes deployed with this code so we can verify the merged root looks acceptable across a wider sample of nodes on the network.

Note: We are not currently using the merged root for validation, simply logging it for visibility.
The RFC proposal is to move to validating this root at the HF.

[quentinlesceller]

@antiochp taking a look today this weekend.

[tromp]

will look over this weekend

…

On Fri, Nov 15, 2019 at 3:28 PM Quentin Le Sceller ***@***.***> wrote: @antiochp <https://github.com/antiochp> taking a look today. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#3108?email_source=notifications&email_token=AAR5BW4LUYNEZHSVQDG3HULQT2W2BA5CNFSM4JIVEHQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEFSSMQ#issuecomment-554379570>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR5BWZBBQODADE5YPXIWQ3QT2W2BANCNFSM4JIVEHQA> .

[tromp]

i added some comments and questions to the code diffs.

[antiochp]

I'm going to merge this now. It adds the accumulator without relying on it yet. And adds logging around the accumulator which I'd like to start seeing on multiple nodes on mainnet and floonet sooner rather than later.
There are no consensus changes in this PR - these are coming soon as part of the scheduled HF2.