[GRPC] Enable data service ZSTD and update crate that uses old tonic #13621

CapCap · 2024-06-10T18:21:39Z

Description

As titled. Current deflate compression is slow and old. New compression is > 5x faster. This did require a fork of Tonic, for which I will submit a PR upstream.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Key Areas to Review

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-06-10T18:21:42Z

⏱️ 17h 15m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-targeted-unit-tests	4h 9m	🟩 🟩 🟩 🟥 🟥 (+7 more)
rust-move-tests	2h 22m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-images / rust-all	1h 38m	🟩 🟩 🟩 🟩 🟩 (+5 more)
execution-performance / single-node-performance	1h 20m	🟩 🟥 🟩
rust-smoke-tests	1h 15m	🟥 🟩 🟩
rust-lints	1h 11m	🟥 🟩 🟩 🟥 🟩 (+6 more)
run-tests-main-branch	49m	🟩 🟩 🟩 🟩 🟩 (+6 more)
forge-framework-upgrade-test / forge	45m	🟩 ⬜ 🟩
forge-e2e-test / forge	42m	🟩 ⬜ 🟩
test-target-determinator	41m	🟩 🟩 🟩 🟩 🟩 (+5 more)
forge-compat-test / forge	37m	🟩 🟩 🟩
general-lints	21m	🟩 🟩 🟩 🟩 🟩 (+6 more)
cli-e2e-tests / run-cli-tests	18m	🟩 🟩 🟩
check-dynamic-deps	17m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-build-cached-packages	13m	🟩 🟩 🟩
execution-performance / test-target-determinator	12m	🟩 🟩 🟥 🟩
check	12m	🟩 🟩 🟩
semgrep/ci	4m	🟩 🟩 🟩 🟩 🟩 (+6 more)
node-api-compatibility-tests / node-api-compatibility-tests	3m	🟩 🟩 🟩
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+5 more)
permission-check	54s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	32s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	32s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	29s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	26s	🟩 🟩 🟩 🟩 🟩 (+5 more)
determine-docker-build-metadata	23s	🟩 🟩 🟩 🟩 🟩 (+5 more)

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / test-target-determinator	3m	4m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

larry-aptos · 2024-06-15T01:14:41Z

need to fix linter.

also i didn't know why Cargo.lock complains in one test 🤦

CapCap · 2024-06-15T01:15:08Z

@larry-aptos I pushed up fix for cargo (rebased)

github-actions · 2024-06-15T02:10:47Z

✅ Forge suite `compat` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `f20b282439ee5a67f3830bccba26b3194c190889`

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> f20b282439ee5a67f3830bccba26b3194c190889 (PR)
1. Check liveness of validators at old version: a68e71c05caebf01504d4499110f3fba213fb53d
compatibility::simple-validator-upgrade::liveness-check : committed: 9821.710580129866 txn/s, latency: 3419.341649556948 ms, (p50: 2500 ms, p90: 6000 ms, p99: 24900 ms), latency samples: 370160
2. Upgrading first Validator to new version: f20b282439ee5a67f3830bccba26b3194c190889
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 3400.104430909868 txn/s, latency: 9133.663375796179 ms, (p50: 9500 ms, p90: 13800 ms, p99: 14200 ms), latency samples: 138160
3. Upgrading rest of first batch to new version: f20b282439ee5a67f3830bccba26b3194c190889
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 3290.2883526167557 txn/s, latency: 9436.072501110617 ms, (p50: 9200 ms, p90: 14100 ms, p99: 14500 ms), latency samples: 135060
4. upgrading second batch to new version: f20b282439ee5a67f3830bccba26b3194c190889
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 6407.365591517984 txn/s, latency: 5101.893518127271 ms, (p50: 4800 ms, p90: 8400 ms, p99: 9400 ms), latency samples: 236660
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> f20b282439ee5a67f3830bccba26b3194c190889 passed
Test Ok

github-actions · 2024-06-15T02:12:28Z

✅ Forge suite `realistic_env_max_load` success on `f20b282439ee5a67f3830bccba26b3194c190889`

two traffics test: inner traffic : committed: 8822.599966018566 txn/s, latency: 4440.70307230398 ms, (p50: 4400 ms, p90: 5400 ms, p99: 9300 ms), latency samples: 3820260
two traffics test : committed: 100.01397654159801 txn/s, latency: 2030.6331460674157 ms, (p50: 2000 ms, p90: 2200 ms, p99: 5600 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.214, avg: 0.207", "QsPosToProposal: max: 0.248, avg: 0.235", "ConsensusProposalToOrdered: max: 0.307, avg: 0.283", "ConsensusOrderedToCommit: max: 0.360, avg: 0.348", "ConsensusProposalToCommit: max: 0.641, avg: 0.631"]
Max round gap was 1 [limit 4] at version 1851036. Max no progress secs was 4.998856 [limit 15] at version 1851036.
Test Ok

github-actions · 2024-06-15T02:14:12Z

✅ Forge suite `framework_upgrade` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `f20b282439ee5a67f3830bccba26b3194c190889`

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> f20b282439ee5a67f3830bccba26b3194c190889 (PR)
Upgrade the nodes to version: f20b282439ee5a67f3830bccba26b3194c190889
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1455.196128280376 txn/s, submitted: 1456.7555454566007 txn/s, failed submission: 1.5594171762247286 txn/s, expired: 1.5594171762247286 txn/s, latency: 2572.243230934095 ms, (p50: 2000 ms, p90: 4800 ms, p99: 7500 ms), latency samples: 111980
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1130.626133854293 txn/s, submitted: 1132.669434096198 txn/s, failed submission: 2.0433002419053485 txn/s, expired: 2.0433002419053485 txn/s, latency: 2693.4751305220884 ms, (p50: 2100 ms, p90: 4400 ms, p99: 8600 ms), latency samples: 99600
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> f20b282439ee5a67f3830bccba26b3194c190889 passed
Upgrade the remaining nodes to version: f20b282439ee5a67f3830bccba26b3194c190889
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1269.4782304694158 txn/s, submitted: 1271.322397137975 txn/s, failed submission: 1.844166668559166 txn/s, expired: 1.844166668559166 txn/s, latency: 2644.375349555112 ms, (p50: 2100 ms, p90: 4500 ms, p99: 8500 ms), latency samples: 110140
Test Ok

…13621)

@CapCap

* [GRPC] Enable data service ZSTD and update crate that uses old tonic (#13621) * replace println with tracing * [GRPC] Simple Transaction Filtering * Improve transaction filter comments, exports, README, fix lz4 in tests * [Data Service] Implement simple upstream transaction filtering (#13699) There are two types of transaction filtering we will support in the future: 1. Per stream configuration: The downstream declares what txns they want to receive. 2. Global configuration: At the data service level we refuse to include full txns for all streams. This PR implements the second of these, using @CapCap's work here: aptos-labs/aptos-indexer-processors#398. Rather than not sending txns at all if they match the blocklist filters, we just omit the writesets and events. Not sending the txns entirely would cause issues with processors, which today assume that they will receive all txns. --------- Co-authored-by: Max Kaplan <[email protected]> Co-authored-by: yuunlimm <[email protected]> Co-authored-by: CapCap <[email protected]> Co-authored-by: Daniel Porteous <[email protected]>

grao1991 approved these changes Jun 10, 2024

View reviewed changes

CapCap marked this pull request as ready for review June 11, 2024 00:42

CapCap added the CICD:build-images when this label is present github actions will start build+push rust images from the PR. label Jun 11, 2024

CapCap force-pushed the max/grpc_zstd branch 4 times, most recently from 0d2b621 to 50ddd29 Compare June 13, 2024 01:05

grao1991 approved these changes Jun 13, 2024

View reviewed changes

CapCap force-pushed the max/grpc_zstd branch from 50ddd29 to a4a493d Compare June 14, 2024 20:08

larry-aptos approved these changes Jun 14, 2024

View reviewed changes

larry-aptos enabled auto-merge (squash) June 15, 2024 00:23

This comment has been minimized.

Sign in to view

CapCap force-pushed the max/grpc_zstd branch from f15c409 to 5c63cab Compare June 15, 2024 01:04

This comment has been minimized.

Sign in to view

Enable data service ZSTD and update crate that uses old tonic

f20b282

This comment has been minimized.

Sign in to view

CapCap force-pushed the max/grpc_zstd branch from 5c63cab to f20b282 Compare June 15, 2024 01:44

This comment has been minimized.

Sign in to view

CapCap merged commit 2c3de68 into main Jun 15, 2024
42 of 43 checks passed

CapCap deleted the max/grpc_zstd branch June 15, 2024 05:23

larry-aptos pushed a commit that referenced this pull request Jun 20, 2024

[GRPC] Enable data service ZSTD and update crate that uses old tonic (#…

ce2de7f

…13621)

larry-aptos pushed a commit that referenced this pull request Jun 20, 2024

[GRPC] Enable data service ZSTD and update crate that uses old tonic (#…

7ba9171

…13621)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GRPC] Enable data service ZSTD and update crate that uses old tonic #13621

[GRPC] Enable data service ZSTD and update crate that uses old tonic #13621

CapCap commented Jun 10, 2024 •

edited

Loading

trunk-io bot commented Jun 10, 2024 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

larry-aptos commented Jun 15, 2024

CapCap commented Jun 15, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jun 15, 2024

github-actions bot commented Jun 15, 2024

github-actions bot commented Jun 15, 2024

[GRPC] Enable data service ZSTD and update crate that uses old tonic #13621

[GRPC] Enable data service ZSTD and update crate that uses old tonic #13621

Conversation

CapCap commented Jun 10, 2024 • edited Loading

Description

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Key Areas to Review

Checklist

trunk-io bot commented Jun 10, 2024 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

larry-aptos commented Jun 15, 2024

CapCap commented Jun 15, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jun 15, 2024

✅ Forge suite compat success on a68e71c05caebf01504d4499110f3fba213fb53d ==> f20b282439ee5a67f3830bccba26b3194c190889

github-actions bot commented Jun 15, 2024

✅ Forge suite realistic_env_max_load success on f20b282439ee5a67f3830bccba26b3194c190889

github-actions bot commented Jun 15, 2024

✅ Forge suite framework_upgrade success on a68e71c05caebf01504d4499110f3fba213fb53d ==> f20b282439ee5a67f3830bccba26b3194c190889

CapCap commented Jun 10, 2024 •

edited

Loading

trunk-io bot commented Jun 10, 2024 •

edited

Loading

✅ Forge suite `compat` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `f20b282439ee5a67f3830bccba26b3194c190889`

✅ Forge suite `realistic_env_max_load` success on `f20b282439ee5a67f3830bccba26b3194c190889`

✅ Forge suite `framework_upgrade` success on `a68e71c05caebf01504d4499110f3fba213fb53d` ==> `f20b282439ee5a67f3830bccba26b3194c190889`