Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixture Format: Add descriptive name to every fixture test. #69

Merged
merged 11 commits into from
Mar 7, 2023

Conversation

spencer-tb
Copy link
Collaborator

@spencer-tb spencer-tb commented Feb 28, 2023

Currently a lot of tests have a duplicate name: 000_shanghai, among others depending on the fork of the test fixture. When running tests in hive this makes it difficult to determine what tests are throwing errors.

This PR makes sure every test has its own distinguishable name. It will help aid using --sim-limit within the hive simulation: ethereum/hive#732.

@marioevz
Copy link
Member

marioevz commented Mar 1, 2023

In reality, the hive name of the test case should not be 000_shanghai but

<folder>_<subfolder>_<fixture_file_name_without_extension>_000_shanghai

E.g. a test in fixture:

fixtures/eips/eip3651/warm_coinbase_call_out_of_gas.json

should be included as test case:

eips_eip3651_warm_coinbase_call_out_of_gas_000_shanghai

in hive.
Is there any way we could include the full path in the test case name when crawling the fixutres?

@spencer-tb
Copy link
Collaborator Author

spencer-tb commented Mar 1, 2023

The above commits achieve the following:

  1. yield test names updated to match above comment.

  2. test function names in withdrawals.py changed to match more structured naming convention.

  3. name field in BaseTest, StateTest & BlockchainTest has been changed to tag.

  4. Logging of tf slightly more descriptive:

╰─>$ tf --output="fixtures"
DEBUG:ethereum_test_filling_tool.main:searching withdrawals.withdrawals for fillers
DEBUG:ethereum_test_filling_tool.main:searching eips.eip3651 for fillers
DEBUG:ethereum_test_filling_tool.main:searching eips.eip3855 for fillers
DEBUG:ethereum_test_filling_tool.main:searching eips.eip3860 for fillers
DEBUG:ethereum_test_filling_tool.main:searching vm.chain_id for fillers
DEBUG:ethereum_test_filling_tool.main:searching vm.dup for fillers
DEBUG:ethereum_test_filling_tool.main:searching example.example for fillers
INFO:ethereum_test_filling_tool.main:collected 20 fillers
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.use_value_in_tx
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.use_value_in_contract
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.balance_within_block
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.multiple_withdrawals_same_address
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.many_withdrawals
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.self_destructing_account
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.newly_created_contract
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.no_evm_execution
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.zero_amount
DEBUG:ethereum_test_filling_tool.main:filling - withdrawals.withdrawals.large_amount
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3651.warm_coinbase_call_out_of_gas
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3651.warm_coinbase_gas_usage
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3855.push0
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3860.initcode_limit_contract_creating_tx
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3860.initcode_limit_contract_creating_tx_gas_usage
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3860.initcode_limit_create_opcode
DEBUG:ethereum_test_filling_tool.main:filling - eips.eip3860.initcode_limit_create2_opcode
DEBUG:ethereum_test_filling_tool.main:filling - vm.chain_id.chain_id
DEBUG:ethereum_test_filling_tool.main:filling - vm.dup.dup
DEBUG:ethereum_test_filling_tool.main:filling - example.example.yul

@spencer-tb
Copy link
Collaborator Author

Here is a list of every hive test name using the current state of this PR:

eip3651 & eip3855:

[5b86fce3]  eips_eip3651_warm_coinbase_call_out_of_gas_003_opcode_staticcall_shanghai                                  
[5b86fce3]  eips_eip3651_warm_coinbase_call_out_of_gas_000_opcode_call_shanghai                                        
[5b86fce3]  eips_eip3651_warm_coinbase_call_out_of_gas_001_opcode_callcode_shanghai                                    
[5b86fce3]  eips_eip3651_warm_coinbase_call_out_of_gas_002_opcode_delegatecall_shanghai                                
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_001_opcode_extcodecopy_shanghai                                       
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_002_opcode_extcodehash_shanghai                                       
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_003_opcode_balance_shanghai                                           
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_004_opcode_call_shanghai                                              
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_005_opcode_callcode_shanghai                                          
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_006_opcode_delegatecall_shanghai                                      
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_007_opcode_staticcall_shanghai                                        
[5b86fce3]  eips_eip3651_warm_coinbase_gas_usage_000_opcode_extcodesize_shanghai                                       
[5b86fce3]  eips_eip3855_push0_006_gas_cost_shanghai                                                                   
[5b86fce3]  eips_eip3855_push0_000_key_sstore_shanghai                                                                 
[5b86fce3]  eips_eip3855_push0_001_fill_stack_shanghai                                                                 
[5b86fce3]  eips_eip3855_push0_002_stack_overflow_shanghai                                                             
[5b86fce3]  eips_eip3855_push0_003_storage_overwrite_shanghai                                                          
[5b86fce3]  eips_eip3855_push0_004_during_staticcall_shanghai                                                          
[5b86fce3]  eips_eip3855_push0_005_before_jumpdest_shanghai                      

eip3860:

[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_000_max_size_zeros_shanghai                               
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_001_max_size_ones_shanghai                                
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_002_over_limit_zeros_shanghai                             
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_003_over_limit_ones_shanghai                              
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_012_single_byte_under_intrinsic_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_014_32_bytes_exact_intrinsic_gas_shanghai       
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_024_49120_bytes_under_execution_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_003_max_size_zeros_exact_execution_gas_shanghai 
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_002_max_size_zeros_under_execution_gas_shanghai 
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_015_32_bytes_under_intrinsic_gas_shanghai       
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_005_max_size_ones_under_intrinsic_gas_shanghai  
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_018_33_bytes_exact_intrinsic_gas_shanghai       
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_022_49120_bytes_exact_intrinsic_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_023_49120_bytes_under_intrinsic_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_028_49121_bytes_under_execution_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_011_single_byte_exact_intrinsic_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_006_max_size_ones_under_execution_gas_shanghai  
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_007_max_size_ones_exact_execution_gas_shanghai  
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_009_empty_under_intrinsic_gas_shanghai          
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_010_empty_exact_execution_gas_shanghai          
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_013_single_byte_exact_execution_gas_shanghai    
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_020_33_bytes_under_execution_gas_shanghai       
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_021_33_bytes_exact_execution_gas_shanghai       
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_000_max_size_zeros_exact_intrinsic_gas_shanghai 
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_027_49121_bytes_under_intrinsic_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_026_49121_bytes_exact_intrinsic_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_008_empty_exact_intrinsic_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_017_32_bytes_exact_execution_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_025_49120_bytes_exact_execution_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_001_max_size_zeros_under_intrinsic_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_016_32_bytes_under_execution_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_019_33_bytes_under_intrinsic_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_029_49121_bytes_exact_execution_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_contract_creating_tx_gas_usage_004_max_size_ones_exact_intrinsic_gas_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_003_over_limit_ones_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_005_single_byte_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_006_32_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_007_33_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_008_49120_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_000_max_size_zeros_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_002_over_limit_zeros_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_004_empty_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_009_49121_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create2_opcode_001_max_size_ones_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_009_49121_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_000_max_size_zeros_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_004_empty_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_005_single_byte_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_008_49120_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_007_33_bytes_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_001_max_size_ones_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_002_over_limit_zeros_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_003_over_limit_ones_shanghai
[5b86fce3]  eips_eip3860_initcode_limit_create_opcode_006_32_bytes_shanghai

vm/chain_id & vm/dup:

[5b86fce3]  vm_chain_id_chain_id_000_merge
[5b86fce3]  vm_chain_id_chain_id_000_shanghai
[5b86fce3]  vm_dup_dup_000_merge
[5b86fce3]  vm_dup_dup_000_shanghai

withdrawals:

[5b86fce3]  withdrawals_withdrawals_balance_within_block_000_shanghai
[5b86fce3]  withdrawals_withdrawals_large_amount_000_shanghai
[5b86fce3]  withdrawals_withdrawals_many_withdrawals_000_shanghai
[5b86fce3]  withdrawals_withdrawals_multiple_withdrawals_same_address_000_shanghai
[5b86fce3]  withdrawals_withdrawals_multiple_withdrawals_same_address_001_multiple_blocks_shanghai
[5b86fce3]  withdrawals_withdrawals_newly_created_contract_000_shanghai
[5b86fce3]  withdrawals_withdrawals_newly_created_contract_001_with_tx_value_shanghai
[5b86fce3]  withdrawals_withdrawals_no_evm_execution_000_shanghai
[5b86fce3]  withdrawals_withdrawals_self_destructing_account_000_shanghai
[5b86fce3]  withdrawals_withdrawals_use_value_in_contract_000_shanghai
[5b86fce3]  withdrawals_withdrawals_use_value_in_tx_000_shanghai
[5b86fce3]  withdrawals_withdrawals_use_value_in_tx_001_shanghai
[5b86fce3]  withdrawals_withdrawals_zero_amount_001_with_extra_positive_amount_shanghai
[5b86fce3]  withdrawals_withdrawals_zero_amount_002_with_extra_max_amount_shanghai
[5b86fce3]  withdrawals_withdrawals_zero_amount_003_reverse_withdrawal_order_shanghai
[5b86fce3]  withdrawals_withdrawals_zero_amount_000_shanghai

Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making these changes!

Regarding the hive test case names, I think it would look a bit more readable to change the underscores between folder names, filename, tag and network for a slash /, but that is a change for the hive PR.

@spencer-tb
Copy link
Collaborator Author

Here is the new list:

eip3651 & eip3855:

[730ba4fb]  eips/eip3651/warm_coinbase_call_out_of_gas/002/opcode_delegatecall/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_call_out_of_gas/003/opcode_staticcall/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_call_out_of_gas/000/opcode_call/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_call_out_of_gas/001/opcode_callcode/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/007/opcode_staticcall/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/000/opcode_extcodesize/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/001/opcode_extcodecopy/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/002/opcode_extcodehash/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/003/opcode_balance/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/004/opcode_call/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/005/opcode_callcode/shanghai
[730ba4fb]  eips/eip3651/warm_coinbase_gas_usage/006/opcode_delegatecall/shanghai
[730ba4fb]  eips/eip3855/push0/001/fill_stack/shanghai
[730ba4fb]  eips/eip3855/push0/002/stack_overflow/shanghai
[730ba4fb]  eips/eip3855/push0/003/storage_overwrite/shanghai
[730ba4fb]  eips/eip3855/push0/004/during_staticcall/shanghai
[730ba4fb]  eips/eip3855/push0/005/before_jumpdest/shanghai
[730ba4fb]  eips/eip3855/push0/006/gas_cost/shanghai
[730ba4fb]  eips/eip3855/push0/000/key_sstore/shanghai

eip3860:

[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx/000/max_size_zeros/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx/001/max_size_ones/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx/002/over_limit_zeros/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx/003/over_limit_ones/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/009/empty_under_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/012/single_byte_under_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/015/32_bytes_under_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/022/49120_bytes_exact_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/027/49121_bytes_under_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/001/max_size_zeros_under_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/004/max_size_ones_exact_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/005/max_size_ones_under_intrinsic_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/003/max_size_zeros_exact_execution_gas/shanghai
[730ba4fb]  eips/eip3860/initcode_limit_contract_creating_tx_gas_usage/010/empty_exact_execution_gas/shanghai
...

vm/chain_id & vm/dup:

[730ba4fb]  vm/chain_id/chain_id/000/merge
[730ba4fb]  vm/chain_id/chain_id/000/shanghai
[730ba4fb]  vm/dup/dup/000/merge
[730ba4fb]  vm/dup/dup/000/shanghai

withdrawals:

[730ba4fb]  withdrawals/withdrawals/balance_within_block/000/shanghai
[730ba4fb]  withdrawals/withdrawals/large_amount/000/shanghai
[730ba4fb]  withdrawals/withdrawals/large_withdrawals/000_shanghai
[730ba4fb]  withdrawals/withdrawals/many_withdrawals/000/shanghai
[730ba4fb]  withdrawals/withdrawals/multiple_withdrawals_same_address/000/shanghai
[730ba4fb]  withdrawals/withdrawals/multiple_withdrawals_same_address/001/multiple_blocks/shanghai
[730ba4fb]  withdrawals/withdrawals/newly_created_contract/000/shanghai
[730ba4fb]  withdrawals/withdrawals/newly_created_contract/001/with_tx_value/shanghai
[730ba4fb]  withdrawals/withdrawals/no_evm_execution/000/shanghai
[730ba4fb]  withdrawals/withdrawals/self_destructing_account/000/shanghai
[730ba4fb]  withdrawals/withdrawals/use_value_in_contract/000/shanghai
[730ba4fb]  withdrawals/withdrawals/use_value_in_tx/000/shanghai
[730ba4fb]  withdrawals/withdrawals/use_value_in_tx/001/shanghai
...

@spencer-tb spencer-tb merged commit 6636828 into ethereum:main Mar 7, 2023
@spencer-tb spencer-tb deleted the name-every-fixtures branch March 7, 2023 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants