Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a package to run a script against all accounts #1434

Merged
merged 5 commits into from
Apr 13, 2022

Conversation

turbolent
Copy link
Member

@turbolent turbolent commented Feb 18, 2022

Description

Add a package that allows running a script against all accounts on a network.

Includes a tool to fetch all contracts and write them into a CSV file.

Code is taken from internal tool authored by @tehranifar, @farbodg, and @janezpodhostnik 🙏


  • Targeted PR against master branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work
  • Code follows the standards mentioned here
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

@codecov-commenter
Copy link

codecov-commenter commented Feb 18, 2022

Codecov Report

Merging #1434 (d8ea5e4) into master (1ef8460) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1434      +/-   ##
==========================================
- Coverage   74.41%   74.40%   -0.01%     
==========================================
  Files         289      289              
  Lines       55671    55671              
==========================================
- Hits        41426    41423       -3     
- Misses      12751    12752       +1     
- Partials     1494     1496       +2     
Flag Coverage Δ
unittests 74.40% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
runtime/sema/checker.go 89.47% <0.00%> (-0.12%) ⬇️
runtime/sema/type.go 88.58% <0.00%> (-0.09%) ⬇️
runtime/sema/simple_type.go 96.15% <0.00%> (+3.84%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ef8460...d8ea5e4. Read the comment docs.

@github-actions
Copy link

github-actions bot commented Feb 18, 2022

Cadence Benchstat comparison

This branch with compared with the base branch onflow:master commit 1ef8460
The command for i in {1..N}; do go test ./... -run=XXX -bench=. -shuffle=on; done was used.
Bench tests were run a total of 7 times on each branch.

Results

old.txtnew.txt
time/opdelta
RuntimeFungibleTokenTransfer-21.21ms ± 3%1.61ms ±26%+33.07%(p=0.035 n=6+7)
RuntimeResourceDictionaryValues-27.85ms ± 6%7.76ms ± 4%~(p=0.710 n=7+7)
Transfer-2101ns ± 1%102ns ± 4%~(p=0.376 n=6+7)
ParseFungibleToken-2226µs ± 2%234µs ± 7%~(p=0.209 n=7+7)
ParseArray-219.3ms ± 2%19.1ms ± 3%~(p=0.366 n=7+6)
ParseInfix-210.3µs ± 2%10.2µs ± 3%~(p=0.445 n=7+6)
ParseDeploy/byte_array-231.9ms ± 3%33.4ms ± 7%~(p=0.097 n=7+7)
ParseDeploy/decode_hex-21.42ms ± 3%1.43ms ± 2%~(p=0.383 n=7+7)
QualifiedIdentifierCreation/One_level-22.82ns ± 0%2.82ns ± 1%~(p=0.835 n=6+6)
QualifiedIdentifierCreation/Three_levels-2173ns ± 3%173ns ± 2%~(p=0.648 n=7+7)
CheckContractInterfaceFungibleTokenConformance-2165µs ± 2%165µs ± 5%~(p=0.456 n=7+7)
ContractInterfaceFungibleToken-250.1µs ± 5%49.6µs ± 2%~(p=0.628 n=7+6)
InterpretRecursionFib-23.15ms ± 4%3.21ms ± 3%~(p=0.259 n=7+7)
NewInterpreter/new_interpreter-21.40µs ± 3%1.42µs ± 2%~(p=0.234 n=7+6)
NewInterpreter/new_sub-interpreter-22.77µs ± 2%2.78µs ± 2%~(p=0.628 n=6+7)
 
alloc/opdelta
RuntimeFungibleTokenTransfer-2274kB ± 0%274kB ± 0%~(p=0.301 n=7+7)
RuntimeResourceDictionaryValues-22.25MB ± 0%2.25MB ± 0%~(p=0.165 n=7+7)
Transfer-248.0B ± 0%48.0B ± 0%~(all equal)
QualifiedIdentifierCreation/One_level-20.00B 0.00B ~(all equal)
QualifiedIdentifierCreation/Three_levels-264.0B ± 0%64.0B ± 0%~(all equal)
CheckContractInterfaceFungibleTokenConformance-266.2kB ± 0%66.2kB ± 0%~(p=0.538 n=6+7)
ContractInterfaceFungibleToken-226.6kB ± 0%26.6kB ± 0%~(p=0.462 n=7+7)
InterpretRecursionFib-21.14MB ± 0%1.14MB ± 0%~(p=0.636 n=5+7)
NewInterpreter/new_interpreter-2848B ± 0%848B ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-21.34kB ± 0%1.34kB ± 0%~(all equal)
 
allocs/opdelta
RuntimeFungibleTokenTransfer-24.58k ± 0%4.58k ± 0%~(p=0.193 n=7+7)
Transfer-21.00 ± 0%1.00 ± 0%~(all equal)
QualifiedIdentifierCreation/One_level-20.00 0.00 ~(all equal)
QualifiedIdentifierCreation/Three_levels-22.00 ± 0%2.00 ± 0%~(all equal)
CheckContractInterfaceFungibleTokenConformance-21.07k ± 0%1.07k ± 0%~(all equal)
ContractInterfaceFungibleToken-2458 ± 0%458 ± 0%~(all equal)
InterpretRecursionFib-223.8k ± 0%23.8k ± 0%~(all equal)
NewInterpreter/new_interpreter-213.0 ± 0%13.0 ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-240.0 ± 0%40.0 ± 0%~(all equal)
RuntimeResourceDictionaryValues-237.6k ± 0%37.6k ± 0%−0.00%(p=0.038 n=5+7)
 

Copy link
Contributor

@janezpodhostnik janezpodhostnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

currentIndex uint
}

const endOfAccountsError = "storage used is not initialized or not initialized correctly"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note: The address provider relies on a predictable error that happens when you to run getAccount(address).storageUsed for an address that dos not exist yet (is not initialised yet). There should probably be a better way to find out that the address does not exist within cadence.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, when defining getAccount we didn't define the behaviour for accounts that do not exist. We should maybe define this better, e.g. abort execution, or allow it and add a field indicating if the account exists

Comment on lines +95 to +96
// We assume address #2 exists
lastAddressIndex, err := ap.getLastAddress(1, 2, true, addressExistsAtIndex)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe short-circuit the check if address #2 doesn't exist?

Copy link
Member Author

@turbolent turbolent Mar 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the internal code as is and didn't notice this.

@janezpodhostnik Do you know why address 2 is assumed to exist?

Maybe you @tehranifar?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the code was refactored by @janezpodhostnik and @farbodg
The original version was meant to drive a script that operates on a range of accounts, e.g. 30, 31, 32. If one of those accounts doesn't exist the script panics so the code had to try with 30, 31 and then 30 to make sure it's capturing the end range correctly. The new version is probably achieving the same.

Copy link
Contributor

@janezpodhostnik janezpodhostnik Mar 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added that assumption (that account 2 exists) because all the networks the program would be run on had account 2.

During bootstrapping we create at least 4 accounts.

If this is a problematic assumption, the code just needs to test that address no. 2 exists, and if it doesn't address no. 1 is the last address.

// 4. (6,8): check address 7 address exists so next pair is (7,8)
// 5. (7,8): check address (8 - 7) / 2 = 7 ... ok already checked so this is the last existing address
//
func (p *AddressProvider) getLastAddress(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go really needs to get fast generics so this kind of thing can be generalized :/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, how would generics help here, the types are always constant, no?

Copy link
Member

@SupunS SupunS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a very little understanding of the logic; mostly reviewed the general Go code. Looks fine to me!


module github.com/onflow/cadence/tools/batch-script

go 1.17
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other modules still seem to use 1.16

@turbolent turbolent merged commit 2aaae59 into master Apr 13, 2022
@turbolent turbolent deleted the bastian/batch-script branch April 13, 2022 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants