Proposal: add basic subcommands to help query ledger data

[MonsieurNicolas]

As we're moving away from rich schema in SQL, we need some way to quickly search/query/analyze the current ledger state.

This is already a problem today even though the data is stored in SQL. For example, trying to understand what the order book contains involves:

• performing a query like select * from (select count(*) as c, sum(amount) as amount, sellingasset, buyingasset from offers group by sellingasset, buyingasset) AS b ORDER by c DESC;
• base64 and xdr convert some columns into human readable

In some cases, we can't even filter as what we want to filter on is inside the XDR (for example, there is no way to search for claimable balances that contain certain conditions).

The goal of this issue is to discuss a possible approach before implementing it.

Here are a few thoughts:

• I think we want to be able to compose commands, I think this probably means using "buckets" for low level operations
  • we have to be able to query/inspect data contained in buckets anyways (to take advantage of their temporal nature: if debugging a "write" that occurred in the last few minutes, there is no need to use the entire bucket list)
  • we can add a "flatten ledger" command (it's partially implemented) to generate a version of the ledger contained in a single bucket (without this, we'd have to query many buckets all the time)
• using the toolset should not require installing a lot of dependencies. We should be able to perform rich debugging queries on production servers.
• basic queries like filtering all ledger entries owned by an account for example should take ~ 1 minute or less
• arbitrary complicated queries should be doable, as a consequence we need to "export" the data so that we can perform those more advanced queries
  • if we can make the data set to export relatively small, export to json might work (we can already convert a bucket file into json, we'd probably want to tweak it to avoid the overhead of bucket entries themselves). Complex filtering/transforms with jq are easy to do
  • it would be nice if we can make this work in the future against smart contract state data
  • it would be nice if we can reuse this work when performing queries against historical data (various checkpoint files)

[MonsieurNicolas]

resolved by #3447