Sloop was not able to reduce the size on disk after GC runs. This also plays a big role in memory consumption.

[sana-jawad]

PR includes following changes:

1. Upgraded Badger version.
2. Added flattening at start up time.
3. Fixed the event count spreading issue which resulted in uneven data distribution across partitions.
4. Moved to drop prefix as it yields better space claim.
5. Added feature flag for switching to delete prefix. Also changed the numberfversions to 0 so that delete prefix would reclaim space. GC doesn't work? (not cleaning up SST files properly) hypermodeinc/badger#1228
6. Fixed the issue of unclaimed !badger!move prefixes which are never cleaned up. Details: Does dropPrefix remove !badger!move keys? hypermodeinc/badger#1288
7. Added support in debugging pages to see internal keys.

Drop Prefix VS Delete Keys
In first 12 hours GC was run using Drop Prefix and rest of 12 hours were run using Delete Prefix

Debug Page showing internal keys info:

[duke-harlan]

passing on a few comments - still reviewing (sorry, keep getting distracted)

[duke-harlan]

some more comments - still trying to grok all of the delete key / drop prefix logic

[duke-harlan]

more comments

[mallow111]

left a few comments

[mallow111]

lgtm, make sure address comments from @duke-harlan, if there is a separate PR needed, please create the PR or issue so that we can track of it, thank you!