Add parquet processor

[yuunlimm]

This is to migrate some of tables in AlloyDB to BigQuery to fully deprecate those tables that are only used for analytics.

Components added:

1. generic_parquet_processor:
   • it store structs passed in from parquet_handler in in-memory buffer and whenever it reaches the max size, it writes to a parquet file and calls gcs_handler to upload the file to gcs bucket.
2. parquet_default_processor
   • basically replica of deafult_processor that is responsible for
     • spawning a task that receives parquet structs from the kanal channel and invoke parquet_manager.handle
     • converting TransactionPB into structs.
     • sending parquet structs to its channel, current structs being handled:
       • move_resources, table_items, transactions, write_set_changes.
3. parquet_handler:
   • util where it has a function that handles spawning a task for parquet_receiver to poll structs from the channel and calling parquet_manager.handle function.
4. gcs_handler:

• responsible for reading local parquet files and uploading to GCS bucket. it retries 3 times with expontential backoffs.

Test Plan
tested in testing environment and running locally.

[ying-w]

would be really nice to have but prob out of scope:

• having tx_version of previous state in move_resources and write_set_changes

[yuunlimm]

Gap Detector isn't working, but since I checked that all txns exist in BigQuery. this is a less concern for now. so I am prioritizing shipping this processor so that we can start backfilling mainnet
cc. @CapCap

[CapCap]

@yuunlimm please rebase this on main, will make the review easier for us :-)

[banool]

This is a refactor PR but it's my first time seeing lots of this code, so I left comments all over the place anyway. For any comment that's not relevant to the refactoring directly, please open an issue / task somewhere to track it for later!

I only reviewed the first bit for now. Since this is the refactoring PR I see there is now a mountain of new code hahah. I'm happy to do a full audit of the code later, but it's probably helpful for me to leave reviews in chunks rather than like 50 comments all at once.

[banool]

This method and convert_move_struct_tag don't have anything to do with MoveResource. Can you move them to the relevant struct or just make them freestanding?

As a general rule, if a method doesn't take self as a param or return Self there is no reason for it to a method on a struct.

[yuunlimm]

synced offline, will follow up in the next pr.

[banool]

How come we have to implement Default? There is no "default transaction" so I'm not sure it makes sense to implement Default for Transaction. Same comment for many of these. If some trait / library is forcing us to implement Default, we should at least use unreachable so we never use it, or if something does use it, for the fields we should use default, like txn_version: Default::default(), event_root_hash: Default::default(), etc.

[yuunlimm]

there is a usecase, where we have derive the parquet schema from the instance, and the we need to know this schema in advance. please take a look at hasParquetSchema trait in the generic_parquet_processor.rs. and we use this schema() to construct parquetHandler for each struct type. that's the only usecase of default but I am not too sure if there is any other workaround.

[rtso]

@banool for the non-Parquet processor, we use Default for transaction types where we can skip indexing most of its contents

[banool]

Not 100% sure the use case but if possible, it'd be good to just take in the owned object, it'd be more efficient. No worries if it doesn't make sense for this use case.

[yuunlimm]

synced offline, will follow up in the next pr.

will address in the follow-up pr

[banool]

Major items have been addressed. I'll leave a full audit here: #420.