Race condition with concurrent executions

[toote]

From a PR in the Buildkite test-collector plugin.

When a script being tested has concurrent executions of a stubbed command there is a race condition. In particular, concurrent executions only consume a single stub execution plan but if one of the concurrent calls starts after one of the others has finished it will fail the stub due to a missing or unmatched plan. What's worse, if the failure path finishes before one othe other concurrent executions already in progress, it will not even be properly recorded and the stub will record as being called just fine :(

In the example, the stub is of a single command called 3 times and found that testing that functionality results in a flaky test that can fail either on the unstubbing or due to other checks in the same tests due to the failed stub execution

Original test with correct execution order

pre-exit-success.bats
 bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-2.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-1.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-2.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-1.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-3.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-3.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): unstubbing
Uploading '3' files matching '**/*/junit-*.xml'
Uploading './tests/fixtures/junit-1.xml'...
Uploading './tests/fixtures/junit-2.xml'...
Uploading './tests/fixtures/junit-3.xml'...
curl success data=@"./tests/fixtures/junit-2.xml"
curl success data=@"./tests/fixtures/junit-1.xml"
curl success data=@"./tests/fixtures/junit-3.xml"
 ✓ Uploads multiple files concurrently does not break basic functionality

1 test, 0 failures

Race condition that causes the unstub call to fail:

pre-exit-success.bats
 bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-2.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-2.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-1.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-1.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-3.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): no plan row found
bats-mock(curl): result 1
bats-mock(curl): unstubbing
 ✗ Uploads multiple files concurrently does not break basic functionality
   (from function `unstub' in file /usr/lib/bats/bats-mock/stub.bash, line 70,
    in test file tests/pre-exit-success.bats, line 69)
     `unstub curl' failed

1 test, 1 failure

Race condition that does not make unstub fail

pre-exit-success.bats
 bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-2.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-2.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-1.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): arguments [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' 'data=@\"./tests/fixtures/junit-1.xml\"' '--form' 'run_env\[CI\]=buildkite' '--form' 'run_env\[key\]=an-id' '--form' 'run_env\[url\]=https://url.com/' '--form' 'run_env\[branch\]=a-branch' '--form' 'run_env\[commit_sha\]=a-commit' '--form' 'run_env\[number\]=123' '--form' 'run_env\[job_id\]=321' '--form' 'run_env\[message\]=A\ message' '--form' 'run_env\[collector\]=test-collector-buildkite-plugin' 'https://analytics-api.buildkite.com/v1/uploads' '-H' 'Authorization:\ Token\ token=\"a-secret-analytics-token\"' 
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): patterns  [31] = '-X' 'POST' '--silent' '--show-error' '--max-time' '30' '--form' 'format=junit' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '--form' '\*' '\*' '-H' '\*' 
bats-mock(curl): running echo "curl success ${10}"
bats-mock(curl): command result was 0
bats-mock(curl): result 0
bats-mock(curl): got curl -X POST --silent --show-error --max-time 30 --form format=junit --form data=@"./tests/fixtures/junit-3.xml" --form run_env[CI]=buildkite --form run_env[key]=an-id --form run_env[url]=https://url.com/ --form run_env[branch]=a-branch --form run_env[commit_sha]=a-commit --form run_env[number]=123 --form run_env[job_id]=321 --form run_env[message]=A message --form run_env[collector]=test-collector-buildkite-plugin https://analytics-api.buildkite.com/v1/uploads -H Authorization: Token token="a-secret-analytics-token"
bats-mock(curl): no plan row found
bats-mock(curl): result 1
bats-mock(curl): unstubbing
 ✗ Uploads multiple files concurrently does not break basic functionality
   (from function `unstub' in file /usr/lib/bats/bats-mock/stub.bash, line 70,
    in test file tests/pre-exit-success.bats, line 69)
     `unstub curl' failed

1 test, 1 failure