ledger-tool: read bigtable entries

[CriesofCarrots]

Problem

#34099 uploads entries to bigtable, but there is no plumbing to read this data.

Summary of Changes

Add solana-ledger-tool bigtable entries subcommand. This dumps the entry info for a single block, emulating the format of the entry lines in solana-ledger-tool slot -vv (output_entry()). However, the new command also supports --output json.

$ solana-ledger-tool bigtable entries 2
[2023-11-29T03:24:04.266284000Z INFO  solana_ledger_tool] solana-ledger-tool 1.18.0 (src:00000000; feat:1429699964, client:SolanaLabs)
[2023-11-29T03:24:04.268240000Z INFO  solana_storage_bigtable::bigtable] Connecting to bigtable emulator at localhost:8086
Slot 2
  Entry 0 - num_hashes: 1, hash: 9h44DYJRnb2CgPYVv17mLKMRw76W9Fh96C8UFMSCk2e6, transactions: 0, starting_transaction_index: 0
  Entry 1 - num_hashes: 1, hash: 3jxMrYfWKZ895nKwEthbRjwRCcLoChBpupdBu6xpUu3T, transactions: 0, starting_transaction_index: 0
  Entry 2 - num_hashes: 1, hash: FAUqexyGwZmN6SRRd28f14658pGd5jV6mz2Rv97HBY6D, transactions: 1, starting_transaction_index: 0
  Entry 3 - num_hashes: 1, hash: 4ArSCVJ8SA4JviV3bkztNJyc6pTFW9CWDEn9PPbKkpEC, transactions: 0, starting_transaction_index: 1
..

$ solana-ledger-tool bigtable entries 2 --output json
[2023-11-29T03:24:49.590422000Z INFO  solana_ledger_tool] solana-ledger-tool 1.18.0 (src:00000000; feat:1429699964, client:SolanaLabs)
[2023-11-29T03:24:49.592447000Z INFO  solana_storage_bigtable::bigtable] Connecting to bigtable emulator at localhost:8086
{
  "entries": [
    {
      "numHashes": 1,
      "hash": "9h44DYJRnb2CgPYVv17mLKMRw76W9Fh96C8UFMSCk2e6",
      "numTransactions": 0,
      "startingTransactionIndex": 0
    },
    {
      "numHashes": 1,
      "hash": "3jxMrYfWKZ895nKwEthbRjwRCcLoChBpupdBu6xpUu3T",
      "numTransactions": 0,
      "startingTransactionIndex": 0
    },
    {
      "numHashes": 1,
      "hash": "FAUqexyGwZmN6SRRd28f14658pGd5jV6mz2Rv97HBY6D",
      "numTransactions": 1,
      "startingTransactionIndex": 0
    },
    {
      "numHashes": 1,
      "hash": "4ArSCVJ8SA4JviV3bkztNJyc6pTFW9CWDEn9PPbKkpEC",
      "numTransactions": 0,
      "startingTransactionIndex": 1
    },
    ..
}

We could add a flag or option to solana-ledger-tool bigtable block that coalesces the block and entry data to match solana-ledger-tool slot -vv more exactly, if that seems useful in the future.

[codecov]

Codecov Report

Merging #34266 (2396b01) into master (4832b4e) will decrease coverage by 0.1%.
Report is 2 commits behind head on master.
The diff coverage is 0.0%.

❗ Current head 2396b01 differs from pull request most recent head a6e9ed3. Consider uploading reports for the commit a6e9ed3 to get more accurate results

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #34266     +/-   ##
=========================================
- Coverage    81.9%    81.9%   -0.1%     
=========================================
  Files         819      819             
  Lines      219817   219481    -336     
=========================================
- Hits       180111   179793    -318     
+ Misses      39706    39688     -18     

[steviez]

Code looks good, also ❤️ for json support. Certainly out of scope for this PR, but would be cool to bring json support to solana-ledger-tool slot -vv as well.

The above probably becomes easier if we end up doing the work to coalesce the blocks like you said.

We could add a flag or option to solana-ledger-tool bigtable block that coalesces the block and entry data to match solana-ledger-tool slot -vv more exactly, if that seems useful in the future.

I could definitely see value in this. One particular instance is if I want to replay a specific slot from 2 days ago (debugging consensus divergance, examining blocks when replay times spiked, etc). Given that most nodes only retain a day, the current workflow is to download the archive (100's of GBs) for the entire epoch from the Google bucket, even if I only need 100 slots out of those 432k.

With both transaction and entry info, we could download specific slots, and then re-shred the entries. This could greatly speed up some debugging workflows

[CriesofCarrots]

I rebased to pick up e425c1a locally. Otherwise, the only changes are the 2 new commits.

[steviez]

LGTM

[CriesofCarrots]

would be cool to bring json support to solana-ledger-tool slot -vv as well.

Agreed! I'll add it to the list.

With both transaction and entry info, we could download specific slots, and then re-shred the entries. This could greatly speed up some debugging workflows

Cool, yeah, I think it's very do-able. Maybe we can chat a little bit later on about what output format(s) would be most helpful. Mirroring solana-ledger-tool slot -vv as closely as possible is definitely one option.