Almost all cross_shard_tx* tests are flaky #3140
Assignees
Labels
A-chain
Area: Chain, client & related
Comments
SkidanovAlex
added a commit
that referenced
this issue
Aug 15, 2020
This change fine-tunes the block production times in cross_shard_tx tests to be appropriate for nayduck workers. The tests still have some level of flakiness (see Test Plan below), I will be debugging if those are real issues, so not closing #3140 yet. Test plan: --------- The tests maintain statistics that ensure that they are producing sufficient number of forks. Thus a passing test not only indicates that the block prod is sufficiently high for blockchain to make progress, but is also sufficiently low to test complex situations. Here are 12 runs of each test. Vanilla `cross_shard_tx` failures are timeouts: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/103 And 12 reruns of `cross_shard_tx` after increasing the timeout: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/106
SkidanovAlex
added a commit
that referenced
this issue
Aug 15, 2020
This change fine-tunes the block production times in cross_shard_tx tests to be appropriate for nayduck workers. The tests still have some level of flakiness (see Test Plan below), I will be debugging if those are real issues, so not closing #3140 yet. Test plan: --------- The tests maintain statistics that ensure that they are producing sufficient number of forks. Thus a passing test not only indicates that the block prod is sufficiently high for blockchain to make progress, but is also sufficiently low to test complex situations. Here are 12 runs of each test. Vanilla `cross_shard_tx` failures are timeouts: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/103 And 12 reruns of `cross_shard_tx` after increasing the timeout: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/106
SkidanovAlex
added a commit
that referenced
this issue
Aug 15, 2020
This change fine-tunes the block production times in cross_shard_tx tests to be appropriate for nayduck workers. The tests still have some level of flakiness (see Test Plan below), I will be debugging if those are real issues, so not closing #3140 yet. Test plan: --------- The tests maintain statistics that ensure that they are producing sufficient number of forks. Thus a passing test not only indicates that the block prod is sufficiently high for blockchain to make progress, but is also sufficiently low to test complex situations. Here are 12 runs of each test. Vanilla `cross_shard_tx` failures are timeouts: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/103 And 12 reruns of `cross_shard_tx` after increasing the timeout: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/106
nearprotocol-bulldozer bot
pushed a commit
that referenced
this issue
Aug 15, 2020
This change fine-tunes the block production times in cross_shard_tx tests to be appropriate for nayduck workers. The tests still have some level of flakiness (see Test Plan below), I will be debugging if those are real issues, so not closing #3140 yet. Test plan: --------- The tests maintain statistics that ensure that they are producing sufficient number of forks. Thus a passing test not only indicates that the block prod is sufficiently high for blockchain to make progress, but is also sufficiently low to test complex situations. Here are 12 runs of each test. Vanilla `cross_shard_tx` failures are timeouts: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/103 And 12 reruns of `cross_shard_tx` after increasing the timeout: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/106
SkidanovAlex
added a commit
that referenced
this issue
Aug 19, 2020
After disabling seals, the minimum block time that `cross_shard_tx` can tolerate dropped drastically, to 80ms. We recently introduced a check at the end of the test that makes sure that the block prod time is not too high, so that `cross_shard_tx` actually ends up producing lots of forks. That check was failing recently. With 80ms block time the test passes very consistently. Fixes #3140 Test plan: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/154
nearprotocol-bulldozer bot
pushed a commit
that referenced
this issue
Aug 19, 2020
) After disabling seals, the minimum block time that `cross_shard_tx` can tolerate dropped drastically, to 80ms. We recently introduced a check at the end of the test that makes sure that the block prod time is not too high, so that `cross_shard_tx` actually ends up producing lots of forks. That check was failing recently. With 80ms block time the test passes very consistently. Fixes #3140 Test plan: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/154
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
E.g. consider this run: http://52.149.162.182:3000/#/run/64
The text was updated successfully, but these errors were encountered: