Add `benchmark extrinsic` command #11456

ggwpez · 2022-05-18T15:21:16Z

Makes it possible to benchmark the execution speed of an extrinsic. It does so by filling a block with these extrinsics and measures its execution time.
The results is then divided by the number of extrinsics and printed to console.

Example invocation:

$ substrate benchmark extrinsic --pallet system --extrinsic remark --dev

Creating empty BABE epoch changes on what appears to be first startup.    
Building block, this takes some time...    
Extrinsics per block: 12000    
Running 10 warmups...    
Executing block 100 times    
Executing a SYSTEM::REMARK extrinsic takes[ns]:
Total: 6897382
Min: 68685, Max: 69797
Average: 68973, Median: 68899, Stddev: 188.99
Percentiles 99th, 95th, 75th: 69293, 69230, 69122

Changes:

Add a benchmark extrinsic <pallet> <extrinsic> command
Implement the command for System::Remark and Balances::TransferKeepAlive
Modify benchmark overhead to re-use the logic
Update node+template to integrate the command

WIP:

~~Document that it is non-endowing, non-reaping and currently always the same sender and receiver account~~
Print all available pallets and extrinsics in the help menu
~~Do some more lenient input transformation (eg. lower-case, trim - and _)~~

Polkadot companion: paritytech/polkadot#5620
Cumulus companion: paritytech/cumulus#1350

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Running this 1000 times with a full block takes ~33 minutes 🙈. Reducing it to ~3 minutes per default. Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

gilescope

Nice addition. Looking forward to trying it out.

bin/node-template/node/src/benchmarking.rs

bin/node-template/runtime/src/lib.rs

utils/frame/benchmarking-cli/src/extrinsic/bench.rs

utils/frame/benchmarking-cli/src/extrinsic/extrinsic_factory.rs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

The .to_lowercase() on the builder is actually not needes since its already documented to only return lower case. Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

bkchr · 2022-06-17T15:22:59Z

utils/frame/benchmarking-cli/src/extrinsic/bench.rs

-		let record = self.measure_block(&block, num_ext, bench_type)?;
+	pub fn bench(&self, ext_builder: Option<&dyn ExtrinsicBuilder>) -> Result<Stats> {
+		let (block, num_ext) = self.build_block(ext_builder)?;
+		let record = self.measure_block(&block, num_ext)?;


You don't remove the block execution time of an empty block. Meaning you don't really measure the extrinsic execution time.

Yes this is a simplification in the code. An empty block executes in about 5.3ms, which is only 0.265% of a 2s block and therefore negligible.
I could still change it to measure an empty block first, do you think its worth it?

How do you know for every runtime what it takes to execute an empty block? :P

Yes, please do this.

I updated it to first measure an empty block and then subtract the average of that from the full block to account for the inherent timings and execution overhead. @bkchr

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

This should already be the case since --dev is passed but somehow not... Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

ggwpez · 2022-07-19T06:10:07Z

bot merge

* Benchmark extrinsic Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Reduce warmup and repeat Running this 1000 times with a full block takes ~33 minutes 🙈. Reducing it to ~3 minutes per default. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Make ExistentialDeposit public Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add 'bechmark extrinsic' command Signed-off-by: Oliver Tale-Yazdi <[email protected]> * fmt Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add --list and cleanup Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Unrelated Clippy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Clippy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix tests and doc Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Move implementations up + fmt Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Dont use parameter_types macro Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Cache to_lowercase() call The .to_lowercase() on the builder is actually not needes since its already documented to only return lower case. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Spelling Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Use correct nightly for fmt... Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Rename ED Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix compile Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Subtract block base weight Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fixes Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Use tmp folder for test This should already be the case since --dev is passed but somehow not... Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix test Signed-off-by: Oliver Tale-Yazdi <[email protected]>

ggwpez added 5 commits May 18, 2022 14:17

Benchmark extrinsic

051fa6e

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Reduce warmup and repeat

43cedac

Running this 1000 times with a full block takes ~33 minutes 🙈. Reducing it to ~3 minutes per default. Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Make ExistentialDeposit public

5af2d41

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Add 'bechmark extrinsic' command

6d25d10

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

fmt

b1d07ee

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

github-actions bot added the A0-please_review Pull request needs code review. label May 18, 2022

ggwpez added A3-in_progress Pull request is in progress. No review needed at this stage. and removed A0-please_review Pull request needs code review. labels May 18, 2022

shawntabrizi approved these changes May 31, 2022

View reviewed changes

ggwpez added 2 commits May 31, 2022 20:14

Add --list and cleanup

d8e2cdc

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Merge branch 'master' into oty-bench-extrinsic

10361dd

ggwpez mentioned this pull request Jun 1, 2022

Co #11456: Expose benchmark extrinsic command paritytech/polkadot#5620

Merged

ggwpez added 3 commits June 1, 2022 14:53

Merge remote-tracking branch 'origin/master' into oty-bench-extrinsic

9a850ea

Unrelated Clippy

4ecc0d3

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Clippy

603d0ae

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

ggwpez added B5-clientnoteworthy C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit labels Jun 1, 2022

ggwpez added 2 commits June 1, 2022 15:53

Fix tests and doc

a336902

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Merge remote-tracking branch 'origin/master' into oty-bench-extrinsic

b6d1f25

ggwpez added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Jun 1, 2022

Merge remote-tracking branch 'origin/master' into oty-bench-extrinsic

abf1326

ggwpez mentioned this pull request Jun 12, 2022

Co #11456: Match statement fallthrough for benchmark commands paritytech/cumulus#1350

Merged

gilescope approved these changes Jun 17, 2022

View reviewed changes

ggwpez added 4 commits June 17, 2022 15:08

Move implementations up + fmt

5efa7c1

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Dont use parameter_types macro

fba7975

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Cache to_lowercase() call

83f8fcf

The .to_lowercase() on the builder is actually not needes since its already documented to only return lower case. Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Spelling

3688b13

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

ggwpez added 2 commits June 17, 2022 15:46

Use correct nightly for fmt...

58d7d3f

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Rename ED

7aea64b

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

bkchr reviewed Jun 17, 2022

View reviewed changes

ggwpez added 2 commits June 17, 2022 17:24

Fix compile

b5a193e

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Merge remote-tracking branch 'origin/master' into oty-bench-extrinsic

87136b1

github-actions bot added the A7-needspolkadotpr label Jun 17, 2022

ggwpez added 3 commits July 14, 2022 16:26

Subtract block base weight

a6eaec5

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Fixes

d94db3d

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Merge remote-tracking branch 'origin/master' into oty-bench-extrinsic

e51a14f

ggwpez requested a review from bkchr July 15, 2022 07:28

bkchr approved these changes Jul 16, 2022

View reviewed changes

ggwpez added 3 commits July 18, 2022 15:49

Merge remote-tracking branch 'origin/master' into oty-bench-extrinsic

8bc5762

Use tmp folder for test

929eaa0

This should already be the case since --dev is passed but somehow not... Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Fix test

f603a06

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

paritytech-processbot bot merged commit ced4169 into master Jul 19, 2022

paritytech-processbot bot deleted the oty-bench-extrinsic branch July 19, 2022 06:10

wischli mentioned this pull request Aug 5, 2022

feat: upgrade to polkadot v0.9.27 KILTprotocol/kilt-node#394

Merged

8 tasks

realnimish mentioned this pull request Aug 23, 2022

Update to polkadot-v0.9.27 Laguna-Chain/laguna-chain#51

Merged

librelois mentioned this pull request Sep 15, 2022

Update substrate/polkadot/cumulus from v0.9.26 to v0.9.29 moonbeam-foundation/moonbeam#1814

Closed

2 tasks

ggwpez mentioned this pull request Sep 28, 2022

Member Request polkadot-fellows/seeding#21

Merged

theodorebugnet mentioned this pull request Oct 13, 2022

Theo/polkadot v0.9.29 interlay/interbtc#726

Merged

gregdhill mentioned this pull request Oct 26, 2022

upgrade to polkadot-v0.9.29 zenlinkpro/Zenlink-DEX-Module#45

Merged

Dengjianping mentioned this pull request Jan 16, 2023

Bump Polkadot Dependencies to v0.9.28 Manta-Network/Manta#967

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `benchmark extrinsic` command #11456

Add `benchmark extrinsic` command #11456

ggwpez commented May 18, 2022 •

edited

Loading

gilescope left a comment

bkchr Jun 17, 2022

ggwpez Jun 17, 2022

bkchr Jun 17, 2022

ggwpez Jul 15, 2022

ggwpez commented Jul 19, 2022

Add benchmark extrinsic command #11456

Add benchmark extrinsic command #11456

Conversation

ggwpez commented May 18, 2022 • edited Loading

gilescope left a comment

Choose a reason for hiding this comment

bkchr Jun 17, 2022

Choose a reason for hiding this comment

ggwpez Jun 17, 2022

Choose a reason for hiding this comment

bkchr Jun 17, 2022

Choose a reason for hiding this comment

ggwpez Jul 15, 2022

Choose a reason for hiding this comment

ggwpez commented Jul 19, 2022

Add `benchmark extrinsic` command #11456

Add `benchmark extrinsic` command #11456

ggwpez commented May 18, 2022 •

edited

Loading