From ad2ec2af88875e047582082d73a61cc22174eb2c Mon Sep 17 00:00:00 2001 From: Missy Messa Date: Fri, 1 Oct 2021 14:27:17 -0700 Subject: [PATCH] Additional clarity around test retry setup --- .../Projects/Build Analysis/Test Retry Documentation.md | 8 +++++--- src/Microsoft.DotNet.Helix/Sdk/Readme.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Documentation/Projects/Build Analysis/Test Retry Documentation.md b/Documentation/Projects/Build Analysis/Test Retry Documentation.md index fbc83dd2290..6a877f10062 100644 --- a/Documentation/Projects/Build Analysis/Test Retry Documentation.md +++ b/Documentation/Projects/Build Analysis/Test Retry Documentation.md @@ -7,12 +7,14 @@ It's important to know that all the reruns happen on the same machine and that t Ex. if you are uploading to any fixed URLs, those need to check if that process was completed, and if copying files on the hard drive, it should be resilient to the same copy command having previously ran. ## How to get on board -As a first step you should [configurate the test-configuration.json](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Helix/Sdk#test-retry) file. This file should be allocated at `eng/test-configuration.json` and it will be [automatically picked up](https://github.com/dotnet/arcade/blob/b4fd1cc3817e0e85213dcc219ff7f7252761659f/src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets#L8), no further changes are needed. +As a first step you should [configure the test-configuration.json](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Helix/Sdk#test-retry) file. This file should be allocated at `eng/test-configuration.json` and it will be [automatically picked up](https://github.com/dotnet/arcade/blob/b4fd1cc3817e0e85213dcc219ff7f7252761659f/src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets#L8), no further changes are needed. There are a couple of rules that your test should meet: 1. Using Helix -1. Using Azure Reporter - if your tests are already reported on azure this means that is all good -1. Tests should be idempotent on the same machine +2. Using Azure Reporter - if your tests are already reported on azure this means that is all good +3. Tests should be idempotent on the same machine + +**NOTE**: If your tests are not using the Helix SDK, you will need to replicate parts of Helix to ensure that the `test-configuration.json` file is uploaded with the payload. ## How to know if a test was retried There are a couple of ways in which you can confirm if a test was rerun, we are going to divide then in failed and succeeded tests: diff --git a/src/Microsoft.DotNet.Helix/Sdk/Readme.md b/src/Microsoft.DotNet.Helix/Sdk/Readme.md index 8b4a7d3a406..eb8ca7fe33f 100644 --- a/src/Microsoft.DotNet.Helix/Sdk/Readme.md +++ b/src/Microsoft.DotNet.Helix/Sdk/Readme.md @@ -339,7 +339,7 @@ If a test passes or fails in all attempts, only a single report is made represen To opt-in and configure test retries when using helix, create file in the reporitory at "eng/test-configuration.json" -### test-configuraion.json format +### test-configuration.json format ```json { "version" : 1,