[table info][2/4] add utils for table info backup and restore and redesign the db read

[jillxuu]

This PR

1. add gcs operator utils

Overview

[1/4] separate indexer async v2 db from aptosdb : #11799
[2/4] add gcs operational utils to set up for backup and restore to gcs: #11793
[3/4] add epoch based backup logic: #11794
[4/4] add db snapshot restore logic: #11795

Goal

https://www.notion.so/aptoslabs/Internal-Indexer-Deprecation-Next-Steps-824d5af7f16a4ff3aeccc4a4edee2763?pvs=4

1. Migrating internal indexer off of aptos db critical path, move it into its own standalone runtime service.
2. Still provide the table info mapping to both API and indexer services.
3. improve table info parsing performance to unblock indexer perf bottleneck

Context

This effort is broken down into two parts:

1. part 1: [indexer-grpc-table-info] add table info parsing logic to indexer grpc fn #10783. Add moving the table info service out of the critical path and convert it into multithread to concurrently process the request.
2. part 2 is this PR. Now we have the table info service, but this service will only be enabled by a handful FNs, so when a new FN wants to join the network but they don't want to sync from the genesis, they should be able to restore the db from a cloud service. In order to provide such a cloud service for others to download the db snapshot, this pr focuses on two things: backup and restore. backup is optional in the config, but restore logic is written in the code.

before	after

Detailed Changes

Tradeoffs

backup based on epoch or transaction version? what frequency?

Pros of backup using version:

1. We have more control over the backup frequency as frequency is tunable.
   Cons of backup using version:
2. Overhead of managing and comparing backed up versions and current processing versions

Pros of backup using epoch:

1. When to backup logic is much cleaner and less error prone
   Cons of backup using epoch:
2. not very configurable but still tunable by setting frequency on how many epochs behind

Decided to use epoch, because on testnet we have little over 10k epoch, divided by total txns, it gives us on average 70k txns per epoch, this sounds about the frequency of txns we'd like to backup.

when to restore

Decided to restore when both conditions are met:

1. the difference btw next versions to be processed and current version from ledger is greater than a version_diff.
2. and time difference btw last restored timestamp in db and current timestamp is greater than a RESTORE_TIME_DIFF_SECS.
   This is to prevent fullnode crashlooping and constantly try to restore without luck, and when version difference is not that big we don't need to spam the gcs service but rather directly state syncing from that close to head version.

structure of the gcs bucket

I followed the similar structure as of indexer's filestore, where we keep a metadata.json file in the bucket to keep track of the chain id and newest backed up epoch. and then a files folder to keep all the epoch based backup. Each db snapshot is first compressed into a tar file from folder, and then gzipped to compress to the best size possible. based on larry's point, alternative compression like bzip2 is less performant.

threads

Using a separate thread for backup only, base on the past experience, gcs upload could be as slow as minutes.

gcs prunning

Couple options we could pursue, since each backup file is a full db backup,

1. create another service to constantly clean up the backup files
2. use gcs own policy to delete files based on time, and other conditions
3. programmatically delete old files while upload
4. constantly writing to the same file

Decided to go with gcs own policy with the proper configuration setup in the gcs deployment. Reason behind is that deploying and maintaining another service is overhead and costs more money, especially this service's responsibility is very singular; writing code for gcs object deletion is not ideal, since we're writing and deleting, need to handle different multitude of edge cases; constantly writing to the same file is def not gonna work, since gcs has strict limitation on single object write limit, only once per second.

Test Plan

1. passing all the written unit tests on file system operation
2. locally tested and verified backup and restore, as well as table info read

Concerns

There's a bottleneck on the size of db snapshot. Currently this db on testnet is around 250mb, based on the nature of db, the compression could get it to be 50-150mb per db snapshot. It's still too big to upload to gcs as its too slow.

TODO

E2E test

from rustie:

1. set up a long-running fullnode in a new k8s project. We can use the same data-staging-us-central1 cluster, but use a new namespace, like indexer-fullnode-testnet-test or something. This is to isolate it from everything else, but we can use the same cluster for simplicity
2. Set up a job in the same namespace that does the backup to GCS.
3. We can set up a continuous job in aptos-core CI that:
   5.1. Spins up a fullnode based on the latest build. This would be the latest main nightly for instance
   3.2. We can quit immediately after verifying that the restore was successful
   3.3. The cost should be manageable, assuming that the restore process is quick.

integration test

load test

Couple things i want to verify with load testing

1. 10 fns boostrapping, can restore work for all of them
2. fns keep crashlooping, is gcs spammed based on egress & ingress
3. when file gets bigger, will backup still work and how long it could take

[trunk-io]

⏱️ 1h 25m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-unit-tests	33m	🟩
windows-build	26m	🟩
rust-lints	9m	🟩
run-tests-main-branch	7m	🟥 🟥
check	4m	🟩
general-lints	2m	🟩
check-dynamic-deps	2m	🟩
file_change_determinator	23s	🟩 🟩
semgrep/ci	19s	🟩
file_change_determinator	10s	🟩
permission-check	7s	🟩 🟩
permission-check	6s	🟩 🟩
permission-check	6s	🟩 🟩
permission-check	5s	🟩 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
windows-build	26m	19m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[bowenyang007]

I would say that clean up shouldn't happen right away. Let's onboard the ecosystem before we delete the "old way"

[jillxuu]

I would say that clean up shouldn't happen right away. Let's onboard the ecosystem before we delete the "old way"

old way was not deleted, and it remains exactly the same. we're just changing the way to read and write our new db. updated the PR description to make it more explicit. @bowenyang007

[larry-aptos]

note: std operations can block the main thread like std::thread::sleep if used with tokio tasks.

should be fine with std threads.

[jillxuu]

yep totally aware of that, in the next pr, creating a standalone spawn blocking thread to separately handle file operation and gcs upload, so that it won't block other concurrent async threads from running.

[larry-aptos]

oh my point is, you can use tokio::fs/tokio::io if IO operations needed in async context.

spawn_blocking is for CPU intensive operations.

[msmouse]

Are we comfortable operating the entire tarball in memory?

[jillxuu]

Are we comfortable operating the entire tarball in memory?

new commit addressed the memory concern to either write to disk directly or use bufread /bufwrite. also adding spawn_blocking when processing these sync io and fs operations to not block the async thread in general

[github-actions]

✅ Forge suite compat success on aptos-node-v1.8.3 ==> 7e7dfb10e4c50936306ad95d50b438c6fee98d56

Compatibility test results for aptos-node-v1.8.3 ==> 7e7dfb10e4c50936306ad95d50b438c6fee98d56 (PR)
1. Check liveness of validators at old version: aptos-node-v1.8.3
compatibility::simple-validator-upgrade::liveness-check : committed: 4749 txn/s, latency: 6800 ms, (p50: 6100 ms, p90: 9700 ms, p99: 24000 ms), latency samples: 175720
2. Upgrading first Validator to new version: 7e7dfb10e4c50936306ad95d50b438c6fee98d56
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1850 txn/s, latency: 15601 ms, (p50: 18700 ms, p90: 22000 ms, p99: 22300 ms), latency samples: 92500
3. Upgrading rest of first batch to new version: 7e7dfb10e4c50936306ad95d50b438c6fee98d56
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1809 txn/s, latency: 16188 ms, (p50: 18900 ms, p90: 22200 ms, p99: 22400 ms), latency samples: 94100
4. upgrading second batch to new version: 7e7dfb10e4c50936306ad95d50b438c6fee98d56
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3272 txn/s, latency: 9686 ms, (p50: 9900 ms, p90: 18500 ms, p99: 19800 ms), latency samples: 127640
5. check swarm health
Compatibility test for aptos-node-v1.8.3 ==> 7e7dfb10e4c50936306ad95d50b438c6fee98d56 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on 7e7dfb10e4c50936306ad95d50b438c6fee98d56

two traffics test: inner traffic : committed: 7101 txn/s, latency: 5353 ms, (p50: 5000 ms, p90: 6900 ms, p99: 12200 ms), latency samples: 3075100
two traffics test : committed: 100 txn/s, latency: 2236 ms, (p50: 2100 ms, p90: 2500 ms, p99: 6200 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.283, avg: 0.210", "QsPosToProposal: max: 0.250, avg: 0.191", "ConsensusProposalToOrdered: max: 0.612, avg: 0.578", "ConsensusOrderedToCommit: max: 0.486, avg: 0.459", "ConsensusProposalToCommit: max: 1.061, avg: 1.036"]
Max round gap was 1 [limit 4] at version 1316410. Max no progress secs was 4.614018 [limit 15] at version 1316410.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking