Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a benchmark for transaction signing #14174

Merged
merged 2 commits into from
Oct 11, 2023

Conversation

halfprice
Copy link
Contributor

@halfprice halfprice commented Oct 9, 2023

Description

Adding a benchmark to measure validator signing transaction performace in the single node benchmark suite.

Sample output:

cargo run --release --bin sui-single-node-benchmark -- move --component txn-signing

2023-10-10T16:08:05.023632Z  INFO sui_single_node_benchmark::benchmark_context: Creating 500001 accounts and 1000002 gas objects
2023-10-10T16:08:06.430838Z  INFO sui_single_node_benchmark::benchmark_context: Initializing validator
2023-10-10T16:08:10.306098Z  INFO sui_single_node_benchmark::benchmark_context: Programmable Move Transaction Generator: Creating 500001 transactions
2023-10-10T16:08:11.236391Z  INFO sui_single_node_benchmark::execution: Sample transaction: SenderSignedData([SenderSignedTransaction { intent_message: IntentMessage { intent: Intent { scope: TransactionData, version: V0, app_id: Sui }, value: V1(TransactionDataV1 { kind: ProgrammableTransaction(ProgrammableTransaction { inputs: [Object(ImmOrOwnedObject((0xff01000000000000000000000000000000000000000000000000000000000000, SequenceNumber(1), o#3Wy9EiapZQKhT8rAyQMyrNNYY16hUSJdqhwetB92C4pu))), Pure([184, 159, 82, 182, 144, 220, 83, 194, 8, 51, 15, 138, 44, 214, 190, 240, 85, 19, 16, 201, 173, 213, 246, 216, 58, 70, 65, 252, 110, 209, 66, 161]), Pure([0, 0, 0, 0, 0, 0, 0, 0])], commands: [MakeMoveVec(None, [Input(0)]), MoveCall(ProgrammableMoveCall { package: 0x22b99c7c24f1fe4e0ba3a3b65b120f85fc3923fcf9111ddc75caa54150604920, module: Identifier("benchmark"), function: Identifier("merge_input_coins"), type_arguments: [], arguments: [Result(0)] }), TransferObjects([Result(1)], Input(1)), MoveCall(ProgrammableMoveCall { package: 0x22b99c7c24f1fe4e0ba3a3b65b120f85fc3923fcf9111ddc75caa54150604920, module: Identifier("benchmark"), function: Identifier("run_computation"), type_arguments: [], arguments: [Input(2)] })] }), sender: 0xb89f52b690dc53c208330f8a2cd6bef0551310c9add5f6d83a4641fc6ed142a1, gas_data: GasData { payment: [(0xff00000000000000000000000000000000000000000000000000000000000000, SequenceNumber(1), o#9tCQKCfifCQzaicSPdBpWyPKVD6fJFcKatKoHn9TyBEn)], owner: 0xb89f52b690dc53c208330f8a2cd6bef0551310c9add5f6d83a4641fc6ed142a1, price: 1000, budget: 5000000000 }, expiration: None }) }, tx_signatures: [Signature(Ed25519SuiSignature(Ed25519SuiSignature([0, 124, 197, 41, 122, 155, 114, 67, 77, 13, 124, 161, 104, 88, 199, 110, 40, 181, 128, 92, 5, 144, 52, 28, 230, 159, 225, 27, 88, 94, 74, 10, 49, 189, 110, 178, 7, 46, 169, 154, 72, 167, 128, 132, 104, 166, 96, 142, 46, 92, 11, 217, 238, 201, 121, 206, 109, 215, 229, 219, 52, 157, 237, 154, 4, 98, 34, 97, 21, 172, 104, 110, 152, 8, 19, 17, 185, 26, 203, 149, 57, 52, 64, 167, 160, 214, 216, 121, 224, 126, 86, 237, 26, 8, 57, 116, 90])))] }])
2023-10-10T16:08:11.236431Z  INFO sui_single_node_benchmark::benchmark_context: Started signing 500001 transactions. You can now attach a profiler
2023-10-10T16:08:23.669227Z  INFO sui_single_node_benchmark::execution: Transaction signing finished in 12.432s, TPS=40218.87065637066

Test Plan

How did you test the new or updated feature?


If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.

Type of Change (Check all that apply)

  • protocol change
  • user-visible impact
  • breaking change for a client SDKs
  • breaking change for FNs (FN binary must upgrade)
  • breaking change for validators or node operators (must upgrade binaries)
  • breaking change for on-chain data layout
  • necessitate either a data wipe or data migration

Release notes

@vercel
Copy link

vercel bot commented Oct 9, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
mysten-ui ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 11, 2023 2:09am
sui-typescript-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 11, 2023 2:09am
3 Ignored Deployments
Name Status Preview Comments Updated (UTC)
explorer ⬜️ Ignored (Inspect) Visit Preview Oct 11, 2023 2:09am
multisig-toolkit ⬜️ Ignored (Inspect) Visit Preview Oct 11, 2023 2:09am
sui-kiosk ⬜️ Ignored (Inspect) Visit Preview Oct 11, 2023 2:09am

@vercel vercel bot temporarily deployed to Preview – mysten-ui October 9, 2023 22:59 Inactive
@halfprice halfprice force-pushed the zhewu/signing-benchmark branch from 69fab0f to 365fdd2 Compare October 10, 2023 16:06
@vercel vercel bot temporarily deployed to Preview – mysten-ui October 10, 2023 16:07 Inactive
@halfprice halfprice changed the title A draft of adding benchmark for transaction signing Adding a benchmark for transaction signing Oct 10, 2023
crates/sui-single-node-benchmark/src/benchmark_context.rs Outdated Show resolved Hide resolved
.get_validator()
.handle_transaction(
&self.epoch_store,
VerifiedTransaction::new_unchecked(transaction),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not 100% sure this is the right way to make a transaction verified in the benchmark.

@halfprice halfprice marked this pull request as ready for review October 10, 2023 16:16
@mwtian
Copy link
Member

mwtian commented Oct 10, 2023

Please wait for @lxfind to take a look on code organizations.

.get_validator()
.handle_transaction(
&self.epoch_store,
VerifiedTransaction::new_unchecked(transaction),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that verification is an important part of tx signing flow, which we probably want to include in the benchmarking. You may want to expose the handle_transaction function in ValidatorService instead. (you can call it by passing a Tonic request)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. PTAL. I tried to use the Transaction function, which follows Component::ValidatorService code path.

crates/sui-single-node-benchmark/src/benchmark_context.rs Outdated Show resolved Hide resolved
Copy link
Contributor Author

@halfprice halfprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments are addressed. PTAL!

Copy link
Contributor

@lxfind lxfind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@halfprice halfprice force-pushed the zhewu/signing-benchmark branch from 1f6d845 to c597dcb Compare October 11, 2023 02:08
@vercel vercel bot temporarily deployed to Preview – mysten-ui October 11, 2023 02:09 Inactive
@halfprice halfprice merged commit fe2bd7f into MystenLabs:main Oct 11, 2023
31 checks passed
@halfprice halfprice deleted the zhewu/signing-benchmark branch October 11, 2023 02:30
.into_iter()
.map(|tx| {
let validator = self.validator();
tokio::spawn(async move { validator.sign_transaction(tx).await })
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no need to spawn a task here, since you do tasks.collect().await

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. That's on me (I wrote it that way in all other places by copy paste).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually Zhe points out that this may not run in parallel, so we may want to leave the spawn here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least for tx signing benchmark, removing the spawn actually makes it slightly faster. It could be because signing is too cheap that parallelism doesn't help

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I thought about the spawn() pattern for the initial PRs. Maybe only for executions using a separate task makes sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants