Blake2f test vectors #619

shemnon · 2019-08-05T21:37:30Z

Add CALL and CALLCODE tests for the standard unit test vectors for Blake2F, targeting Istanbul.

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLCODEBlake2fFiller.json

winsvega · 2019-08-06T14:40:24Z

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLBlake2fFiller.json

+    "expect": [
+      {
+        "indexes": {
+          "data": [0],


also a comment about what is the case on data 0

described the outputs in the "// data" section, they are all hash values so they are quite opaque.

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLBlake2fFiller.json

shemnon · 2019-08-06T16:53:38Z

Is there some sort fo a standard published as to what is expected from a community contributed test case? These objections seem arbitrary when I can point to other tests cases that don't have these features. While it is valid to say "past bad behavior does not set a precedent" without a clear document describing what is expected all I have are the past behaviors and I don't know which are bad, which are acceptable, and which are good.

winsvega · 2019-08-06T17:31:27Z

hm. no such standard created yet. good point this notes is well to be published in test creation wiki.
we have a lot of tests for the past 4 years, it would be good to have more comments in tests, but not enough hands to write both tests, manuals and tools

and also non of the external contributors mostly ever wrote any .json tests to the repo. so thats why no manuals. now when other clients could write their own tests the manuals are to be created from precedents

#620

* add input data descriptions * add new tests for input length and round extremes (0, max)

shemnon · 2019-08-07T16:03:35Z

Also added a few more test cases.

Also, we should have cross comparison against another before committing. Some details are not settled yet that are covered by these test cases.

winsvega · 2019-08-07T16:42:13Z

Stay in touch. Once geth have the implementation you could run this test against two clients

holiman · 2019-08-07T17:29:56Z

This may not be the best discussion forum, but one thing that struck me is that it would be neat if all[*] tests use the sender account as coinbase - in this case 0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b. If we do that, then the postStates (and poststate dumps) become smaller, and we get less questions about "how did account X (coinbase) get into the poststate?"

[*] all meaning all cases where there isn't some explicit test involving the coinbase address

shemnon · 2019-08-07T19:03:52Z

It totally demolishes gas calculation comparison because all the fees go right back where they started. Perhaps the contract becomes the coinbase? But then you would have to explain why the contract's balance goes up in the same way the coinbase would go up. "gasPrice": "0" may be more what we are looking for.

holiman · 2019-08-07T19:18:58Z

Ah, that's true. So I guess the trick is not really that nifty, unless we explicitly want to avoid letting gas calc be part of the test.

winsvega · 2019-08-10T02:29:17Z

more docs: https://github.com/ethereum/retesteth/wiki/Retesteth-commands

holiman · 2019-08-23T08:23:54Z

Geth doesn't pass these tests...

[user@work go-ethereum]$ build/bin/evm statetest /home/user/workspace/tests/GeneralStateTests/stPreCompiledContracts2/CALLBlake2f.json 
[
  {
    "name": "CALLBlake2f",
    "pass": false,
    "fork": "Istanbul",
    "error": "post state root mismatch: got 56c0c7bd91cff99f993ede0733fa3a38e194d01f5c87a9b882305f61c3a4c338, want b46f92edbed07baacf55c64dc3c9169e1f04966f9ce87e81cc107eebd7ce94af"
  },
...

Question: is this PR up to date?

shemnon · 2019-08-23T14:48:01Z

Should be up to date. What does a retesteth run at --verbosity 6 say? Given that this is a state root mismatch and not off the filler it could be many things.

winsvega · 2019-08-23T15:01:39Z

Also to see the post state
'--fullstate --poststate'

shemnon · 2019-08-23T17:13:43Z

Is this off of develop/head? I don't think the contract is firing as there is nothing stored in storage - https://gist.github.com/shemnon/adf5b5d24f05aadbbcd2272405b1c3e0

winsvega · 2019-08-23T17:31:30Z

The precompiled must be supported by the client implementation. You use Pantheon ?

shemnon · 2019-08-23T17:39:11Z

Yes, Pantheon (as it currently is named). it's an internal build off of head.

shemnon · 2019-08-23T18:31:33Z

The failing vectors are d = {0,1,2,6}, which are all the bad data failure cases. All the "good data" cases are passing. Geth has ending balance of 999999999999901559711 for 0,1, and 2 whereas pantheon has multiple balances (caused by intrinsic tx cost changes) of around 999999999999999954832 (at gas 0). Pantheon charges 45,168 wei for this bad call and Geth 98,440,289 at gas 0 and the full limit at gas 1, 2, and 3.

When the precompile fails due to validation is the correct failure mode to fail the call or to fail the whole tx and charge the gas limit?

winsvega · 2019-08-24T06:56:08Z

I am afraid geth didn't implement the contract yet. If so then its just a call to empty address. If the precompile is implemented but fails then it just eat the whole gas. If the precompile works need to check the results againt other clients. (By putting the results into storage)

winsvega

please add expect section for uncovered transactions
and refill with latest retesteth (ethereum/retesteth#48)
and with latest lllc version

rebase on tests:develop to fetch the retesteth config first

Run with the new LLLC Also, add Istanbul genesis config

shemnon · 2019-09-16T19:24:25Z

I think the no TXes complaint is from the lack of the Istanbul config. I pulled over the one I had in place for Martin's changes.

winsvega · 2019-09-17T12:36:03Z

hmm. how you created this tests? did you get any errors?
I see this errors on that retesteth version:

Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 0, g: 1, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 0, g: 2, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 0, g: 3, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 1, g: 1, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 1, g: 2, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 1, g: 3, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 2, g: 1, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 2, g: 2, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 2, g: 3, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 3, g: 1, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 3, g: 2, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 3, g: 3, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 9, g: 1, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 9, g: 2, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 9, g: 3, v: 0 (CALLBlake2f)
Error: Test has transaction uncovered with expect sections!  (CALLBlake2f)

meaning there are some transactions that does not have post condition checks

# Conflicts: # Retesteth/default/genesis/Istanbul.json

shemnon · 2019-09-17T18:57:24Z

dannos-mbp:build dannoferrin$ PATH+=":../../../ethereum/solidity/build/lllc/" ./retesteth/retesteth -t  GeneralStateTests/stPreCompiledContracts2 -- --testpath ../../../shemnon/ethereum-tests/ --clients besu --singletest CALLBlake2f --filltests --fillblockchain
Unknown option: --fillblockchain
dannos-mbp:build dannoferrin$ PATH+=":../../../ethereum/solidity/build/lllc/" ./retesteth/retesteth -t  GeneralStateTests/stPreCompiledContracts2 -- --testpath ../../../shemnon/ethereum-tests/ --clients besu --singletest CALLBlake2f --filltests --fillchain
Running tests using path: "../../../shemnon/ethereum-tests/"
Running 1 test case...
Active client configurations: 'besu '
Filter: 'CALLBlake2f'
Running tests for config 'Hyperledger Besu on TCP' 2
Test Case "stPreCompiledContracts2": 
100%

*** No errors detected
dannos-mbp:build dannoferrin$ PATH+=":../../../ethereum/solidity/build/lllc/" ./retesteth/retesteth -t  GeneralStateTests/stPreCompiledContracts2 -- --testpath ../../../shemnon/ethereum-tests/ --clients besu --singletest CALLCODEBlake2f --filltests --fillchain
Running tests using path: "../../../shemnon/ethereum-tests/"
Running 1 test case...
Active client configurations: 'besu '
Filter: 'CALLCODEBlake2f'
Running tests for config 'Hyperledger Besu on TCP' 2
Test Case "stPreCompiledContracts2": 
100%

*** No errors detected
dannos-mbp:build dannoferrin$ PATH+=":../../../ethereum/solidity/build/lllc/" ./retesteth/retesteth -t  GeneralStateTests/stTimeConsuming -- --testpath ../../../shemnon/ethereum-tests/ --clients besu --singletest CALLBlake2f_MaxRounds --filltests --fillchain
Running tests using path: "../../../shemnon/ethereum-tests/"
Running 1 test case...
Skipping stTimeConsuming because --all option is not specified.

*** No errors detected
dannos-mbp:build dannoferrin$ PATH+=":../../../ethereum/solidity/build/lllc/" ./retesteth/retesteth -t  GeneralStateTests/stTimeConsuming -- --testpath ../../../shemnon/ethereum-tests/ --clients besu --singletest CALLBlake2f_MaxRounds --filltests --fillchain --all
Running tests using path: "../../../shemnon/ethereum-tests/"
Running 1 test case...
Active client configurations: 'besu '
Filter: 'CALLBlake2f_MaxRounds'
Running tests for config 'Hyperledger Besu on TCP' 2
Test Case "stTimeConsuming": 
100%

winsvega · 2019-09-17T21:02:37Z

No. I don't understand why. the tests state that retesteth used commit is from master branch. but the behaviour of your retesteth is way outdated. It is filling the blockchain tests into a wrong folder name. and it does not perform the state test validity check.

could you double check that you running the tool on latest commit from the master branch?
ethereum/retesteth@eeb4805

shemnon · 2019-09-18T00:01:36Z

at @eeb4805 I get a whole lot of WARNING: Error getting LLLC Version

shemnon · 2019-09-18T05:02:21Z

lllc version is hand fed because the lllc version logic is broken on retesteth on a mac.

winsvega · 2019-09-19T08:41:59Z

this is very strange. you must at least see this errors:

Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 0, g: 1, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 0, g: 2, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 0, g: 3, v: 0 (CALLBlake2f)
Error: A transaction was specified, but there is no execution results in a test! CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0 (Errored TR: d: 1, g: 1, v: 0 (CALLBlake2f)

how you deal with lllc version problem ?

shemnon · 2019-09-19T15:28:41Z

I don't see those errors, never have.

I hand edited the LLLC versions into the JSON. I think there is some problems with the way you detect if LLLC is present on mac.

winsvega · 2019-09-19T16:23:11Z

I've just compiled lllc and retesteth on mac and created a test for it:
https://circleci.com/gh/ethereum/retesteth/665?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

the build went through just fine. smth is wrong on your end.

I hand edited the LLLC versions into the JSON. I think there is some problems with the way you detect if LLLC is present on mac.

You can't just edit those fields. You provide misinformation about the test. If smth went wrong with lllc then the test is not filled correctly. lllc version is only a warning if when you fill the tests there is no lllc code in the test.

shemnon · 2019-09-20T03:13:51Z

LLLC version - when I put the LLLC setting in the exported environment instead of as a shell parameter the error stops occuring. You will note the version is exactly the same as was copied in by hand.

ethereum/retesteth#52 version includes the v,s,r values that were complained about in another thread. So I re-ran the tests to get those in the blockchain tests.

The transaction errors are flakey, so I think it's something inside of retesteth. I think it is a timing issue as I didn't see them until today and I see them less than 10% or the time and only under load (running a compile in another thread or some ad-filled chrome webpage). The JSON-RPC traces are mostly the same, except that a success will continue with more JSON-RPCs to do the fill. https://gist.github.com/shemnon/72cd75520148dc546e995e63a5ecd938

Have you tried to local test against geth or besu and do you see the same errors consistantly?

winsvega · 2019-09-20T10:23:30Z

yes the process is fully deterministic. how can I build pantheon/besu on ubuntu?

shemnon · 2019-09-20T16:18:16Z

The full version is here.

The short version for this case is

Install Java JDK 11+ (make sure javac and java are on the path)
clone the repo
./gradlew installdist
./build/install/besu/bin/besu retesteth

gumb0 · 2019-09-23T15:45:12Z

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLCODEBlake2fFiller.json

+        "9 - RFC 7693 Appendix-A, 8 million rounds"
+      ],
+      "data": [
+        "0x",


Another test suggestion: non-empty data, but fewer than 4 bytes, to catch possible errors of trying to calculate the precompile cost out of it

added as data 10

shemnon · 2019-09-23T16:24:15Z

Two things

(a) the docs and errors were not particularly clear that all permutations must be covered. The error message could be re-tooled to communicate this better

Missing transaction execution results. All permutations for a test transaction must be covered. Please add the validation data for CALLBlake2f, fork: Istanbul, TrInfo: d: 9, g: 0, v: 0

(b) This error must be produced all the time on all platforms. I only rarely see it on my macbook pro, and then only when I put the system under load. This feels like a memory fencing or threading issues, like data is getting overwritten before it is moved to a more permanent location.

winsvega · 2019-09-23T16:30:30Z

(b)
how is this possible?

the error is added to error list upon test execution. then all errors from the list are stdeerored once threads are merged. so the output is done at the main thread.

shemnon · 2019-09-24T00:34:07Z

I don't know how it's possible, but I have observed that it does occur. Perhaps a memory fence needs to be raised prior to the data check?

Add 20 case blake2b test vector

ceadc70

shemnon force-pushed the blake2f branch from 918ca62 to ceadc70 Compare August 5, 2019 21:42

Add Istanbul to validation spec

afec120

winsvega reviewed Aug 6, 2019

View reviewed changes

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLCODEBlake2fFiller.json Outdated Show resolved Hide resolved

winsvega reviewed Aug 6, 2019

View reviewed changes

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLBlake2fFiller.json Outdated Show resolved Hide resolved

winsvega reviewed Aug 6, 2019

View reviewed changes

src/GeneralStateTestsFiller/stPreCompiledContracts2/CALLBlake2fFiller.json Outdated Show resolved Hide resolved

review responses

7da1adb

* add input data descriptions * add new tests for input length and round extremes (0, max)

s1na mentioned this pull request Aug 8, 2019

[Istanbul] EIP-2129/152: Add Blake2b F precompile ethereumjs/ethereumjs-monorepo#568

Closed

shemnon added 5 commits August 16, 2019 10:17

Merge branch 'develop' of github.com:ethereum/tests into blake2f

d742494

update to f in {0,1} and bit endian packing

8591252

align test cases with those in the EIP

3c9a951

All zero input data test case

95194bd

update max rounds to EIP vector

6a3e985

winsvega requested changes Sep 15, 2019

View reviewed changes

shemnon added 2 commits September 16, 2019 12:50

Merge branch 'develop' of github.com:ethereum/tests into blake2f

fb40c07

update with new LLLC

9740ab2

Run with the new LLLC Also, add Istanbul genesis config

Merge branch 'develop' of github.com:ethereum/tests into blake2f

adac6f5

# Conflicts: # Retesteth/default/genesis/Istanbul.json

blockchain tests

a53001e

run retesteth with latest maser

b60077b

voith mentioned this pull request Sep 18, 2019

[WIP] Update upstream tests for Istanbul ethereum/py-evm#1852

Closed

7 tasks

from develop branch with v,s,r values in chaintest

8405633

gumb0 mentioned this pull request Sep 23, 2019

Implement EIP-152: Add Blake2 compression function F precompile ethereum/aleth#5751

Merged

gumb0 reviewed Sep 23, 2019

View reviewed changes

shemnon added 2 commits September 23, 2019 10:08

cover all permutations

4a96382

Add two byte data validaiton case

ac056b3

winsvega merged commit 86b8a27 into ethereum:develop Sep 23, 2019

holgerd77 mentioned this pull request Sep 21, 2021

VM: Blake2 precompiled contract gives incorrect result on some inputs ethereumjs/ethereumjs-monorepo#1482

Closed

shemnon deleted the blake2f branch November 5, 2022 19:15

Blake2f test vectors #619

Blake2f test vectors #619

Conversation

shemnon commented Aug 5, 2019

winsvega Aug 6, 2019

Choose a reason for hiding this comment

shemnon Aug 7, 2019

Choose a reason for hiding this comment

shemnon commented Aug 6, 2019

winsvega commented Aug 6, 2019 • edited Loading

shemnon commented Aug 7, 2019

winsvega commented Aug 7, 2019

holiman commented Aug 7, 2019 • edited Loading

shemnon commented Aug 7, 2019

holiman commented Aug 7, 2019 via email

winsvega commented Aug 10, 2019

holiman commented Aug 23, 2019

shemnon commented Aug 23, 2019

winsvega commented Aug 23, 2019

shemnon commented Aug 23, 2019

winsvega commented Aug 23, 2019

shemnon commented Aug 23, 2019

shemnon commented Aug 23, 2019 • edited Loading

winsvega commented Aug 24, 2019

winsvega left a comment

Choose a reason for hiding this comment

shemnon commented Sep 16, 2019

winsvega commented Sep 17, 2019

shemnon commented Sep 17, 2019

winsvega commented Sep 17, 2019 • edited Loading

shemnon commented Sep 18, 2019 • edited Loading

shemnon commented Sep 18, 2019 • edited Loading

winsvega commented Sep 19, 2019

shemnon commented Sep 19, 2019

winsvega commented Sep 19, 2019 • edited Loading

shemnon commented Sep 20, 2019 • edited Loading

winsvega commented Sep 20, 2019

shemnon commented Sep 20, 2019

gumb0 Sep 23, 2019

Choose a reason for hiding this comment

shemnon Sep 23, 2019

Choose a reason for hiding this comment

shemnon commented Sep 23, 2019 • edited Loading

winsvega commented Sep 23, 2019 • edited Loading

shemnon commented Sep 24, 2019

winsvega commented Aug 6, 2019 •

edited

Loading

holiman commented Aug 7, 2019 •

edited

Loading

shemnon commented Aug 23, 2019 •

edited

Loading

winsvega commented Sep 17, 2019 •

edited

Loading

shemnon commented Sep 18, 2019 •

edited

Loading

shemnon commented Sep 18, 2019 •

edited

Loading

winsvega commented Sep 19, 2019 •

edited

Loading

shemnon commented Sep 20, 2019 •

edited

Loading

shemnon commented Sep 23, 2019 •

edited

Loading

winsvega commented Sep 23, 2019 •

edited

Loading