Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchstat GH action #1227

Merged
merged 1 commit into from
Nov 9, 2021
Merged

Add benchstat GH action #1227

merged 1 commit into from
Nov 9, 2021

Conversation

janezpodhostnik
Copy link
Contributor

@janezpodhostnik janezpodhostnik commented Nov 9, 2021

Description

Add a github action to output performance changes with the base branch of a PR using benchstat.

See https://github.com/janezpodhostnik/cadence/pull/1#issuecomment-964450162 for sample output.


  • Targeted PR against master branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work
  • Code follows the standards mentioned here
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

@janezpodhostnik janezpodhostnik self-assigned this Nov 9, 2021
@codecov-commenter
Copy link

Codecov Report

Merging #1227 (2178eb7) into master (e8500a0) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1227   +/-   ##
=======================================
  Coverage   77.40%   77.40%           
=======================================
  Files         273      273           
  Lines       34821    34821           
=======================================
  Hits        26953    26953           
  Misses       6793     6793           
  Partials     1075     1075           
Flag Coverage Δ
unittests 77.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8500a0...2178eb7. Read the comment docs.

@github-actions
Copy link

github-actions bot commented Nov 9, 2021

Cadence Benchstat comparison

This branch with compared with the base branch onflow:master commit e8500a0
The command go test ./... -run=XXX -bench=. -shuffle=on -count N was used.
Bench tests were run a total of 7 times on each branch.

Results

old.txtnew.txt
time/opdelta
RuntimeStorageWriteCached-2169µs ± 2%228µs ±26%+34.91%(p=0.011 n=7+7)
ParseFungibleToken-2559µs ± 3%583µs ± 2%+4.24%(p=0.001 n=7+7)
ParseInfix-228.9µs ± 1%29.7µs ± 2%+2.76%(p=0.002 n=7+7)
InterpretRecursionFib-23.25ms ± 2%3.34ms ± 4%+2.75%(p=0.002 n=7+7)
RuntimeResourceDictionaryValues-220.3ms ± 3%20.5ms ± 2%~(p=0.620 n=7+7)
RuntimeFungibleTokenTransfer-21.52ms ± 3%1.56ms ± 7%~(p=0.259 n=7+7)
ParseDeploy/byte_array-241.6ms ± 1%42.4ms ± 4%~(p=0.181 n=6+7)
ParseDeploy/decode_hex-21.71ms ± 1%1.72ms ± 3%~(p=0.259 n=7+7)
ParseArray-227.8ms ± 5%27.8ms ± 3%~(p=0.805 n=7+7)
QualifiedIdentifierCreation/Three_levels-2203ns ± 3%202ns ± 2%~(p=0.535 n=7+7)
ContractInterfaceFungibleToken-258.5µs ± 2%59.9µs ± 3%~(p=0.101 n=6+7)
CheckContractInterfaceFungibleTokenConformance-2209µs ± 5%209µs ± 2%~(p=0.902 n=7+7)
NewInterpreter/new_interpreter-21.42µs ± 1%1.43µs ± 2%~(p=0.511 n=6+6)
NewInterpreter/new_sub-interpreter-22.54µs ± 2%2.50µs ± 2%−1.51%(p=0.026 n=7+7)
QualifiedIdentifierCreation/One_level-23.86ns ± 2%3.78ns ± 0%−1.99%(p=0.020 n=7+6)
 
alloc/opdelta
RuntimeResourceDictionaryValues-24.33MB ± 0%4.33MB ± 0%~(p=0.383 n=7+7)
RuntimeStorageWriteCached-283.7kB ± 0%83.7kB ± 0%~(p=0.152 n=6+6)
QualifiedIdentifierCreation/One_level-20.00B 0.00B ~(all equal)
QualifiedIdentifierCreation/Three_levels-264.0B ± 0%64.0B ± 0%~(all equal)
ContractInterfaceFungibleToken-226.7kB ± 0%26.7kB ± 0%~(all equal)
InterpretRecursionFib-21.21MB ± 0%1.21MB ± 0%~(all equal)
NewInterpreter/new_interpreter-2680B ± 0%680B ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-21.02kB ± 0%1.02kB ± 0%~(all equal)
CheckContractInterfaceFungibleTokenConformance-266.4kB ± 0%66.4kB ± 0%−0.00%(p=0.009 n=7+6)
RuntimeFungibleTokenTransfer-2225kB ± 0%225kB ± 0%−0.01%(p=0.030 n=7+7)
 
allocs/opdelta
RuntimeResourceDictionaryValues-2108k ± 0%108k ± 0%~(p=0.421 n=7+6)
RuntimeStorageWriteCached-21.42k ± 0%1.42k ± 0%~(all equal)
RuntimeFungibleTokenTransfer-24.38k ± 0%4.38k ± 0%~(p=0.694 n=7+7)
QualifiedIdentifierCreation/One_level-20.00 0.00 ~(all equal)
QualifiedIdentifierCreation/Three_levels-22.00 ± 0%2.00 ± 0%~(all equal)
ContractInterfaceFungibleToken-2458 ± 0%458 ± 0%~(all equal)
CheckContractInterfaceFungibleTokenConformance-21.07k ± 0%1.07k ± 0%~(all equal)
InterpretRecursionFib-225.0k ± 0%25.0k ± 0%~(all equal)
NewInterpreter/new_interpreter-211.0 ± 0%11.0 ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-230.0 ± 0%30.0 ± 0%~(all equal)
 

Copy link
Member

@SupunS SupunS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding this! Looks good!

Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thank you for adding this 👏

@janezpodhostnik janezpodhostnik merged commit 82e2962 into master Nov 9, 2021
@janezpodhostnik janezpodhostnik deleted the janez/bench branch May 11, 2022 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants