Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional clarity around test retry setup #7989

Merged
merged 1 commit into from
Oct 1, 2021
Merged

Additional clarity around test retry setup #7989

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions Documentation/Projects/Build Analysis/Test Retry Documentation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ It's important to know that all the reruns happen on the same machine and that t
Ex. if you are uploading to any fixed URLs, those need to check if that process was completed, and if copying files on the hard drive, it should be resilient to the same copy command having previously ran.

## How to get on board
As a first step you should [configurate the test-configuration.json](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Helix/Sdk#test-retry) file. This file should be allocated at `eng/test-configuration.json` and it will be [automatically picked up](https://github.com/dotnet/arcade/blob/b4fd1cc3817e0e85213dcc219ff7f7252761659f/src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets#L8), no further changes are needed.
As a first step you should [configure the test-configuration.json](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Helix/Sdk#test-retry) file. This file should be allocated at `eng/test-configuration.json` and it will be [automatically picked up](https://github.com/dotnet/arcade/blob/b4fd1cc3817e0e85213dcc219ff7f7252761659f/src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets#L8), no further changes are needed.

There are a couple of rules that your test should meet:
1. Using Helix
1. Using Azure Reporter - if your tests are already reported on azure this means that is all good
1. Tests should be idempotent on the same machine
2. Using Azure Reporter - if your tests are already reported on azure this means that is all good
3. Tests should be idempotent on the same machine

**NOTE**: If your tests are not using the Helix SDK, you will need to replicate parts of Helix to ensure that the `test-configuration.json` file is uploaded with the payload.

## How to know if a test was retried
There are a couple of ways in which you can confirm if a test was rerun, we are going to divide then in failed and succeeded tests:
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.DotNet.Helix/Sdk/Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ If a test passes or fails in all attempts, only a single report is made represen

To opt-in and configure test retries when using helix, create file in the reporitory at "eng/test-configuration.json"

### test-configuraion.json format
### test-configuration.json format
```json
{
"version" : 1,
Expand Down