Skip to content

Commit

Permalink
Linting doc
Browse files Browse the repository at this point in the history
  • Loading branch information
pawelpasterz committed Jul 31, 2020
1 parent f448bd3 commit 2f003e3
Showing 1 changed file with 11 additions and 11 deletions.
22 changes: 11 additions & 11 deletions docs/bugs/914_falsy_positive_outcome_for_flaky_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,22 @@
[task 2020-07-23T20:29:15.153Z] │ flaky │ matrix-1vj2pt9wih182 │ 1 test cases failed, 106 passed, 2 flaky │
[task 2020-07-23T20:29:15.153Z] └─────────┴──────────────────────┴──────────────────────────────────────────┘
```
FTL confirmed the matrix outcome is actually failure. Flank reported flaky.
FTL confirmed the matrix outcome is a failure. Flank reported flaky.

### Steps to reproduce
So far no reliable way to reproduce it (with 10/10 rate) found. One can simulate error with test:
So far no reliable way to reproduce it (with 10/10 rate) found. One can simulate error with the test:
```
SavedMatrixTest#`savedMatrix should have failed outcome when at least one test is failed and the last one is flaky`
```

Due to incorrectly implemented logic regarding status update there were cases when flaky tests obscure failed results.
Following must occur:
1. Matrix needs to have both flaky and failed tests (at least one of each)
2. TestExecutions need to be returned in specific order:
1. Failed execution must be consumed by flank prior to flaky one
2. Flaky execution must be actually failed but outcome of the all re-runs is success
3. Then flank does what is following:
1. Outcome is set to `failure` when failed test/shards/executions are processed
Due to the incorrectly implemented logic regarding the status update, there were cases when flaky tests obscure failed results.
The following must occur:
1. The matrix needs to have both flaky and failed tests (at least one of each)
2. TestExecutions need to be returned in a specific order:
1. Failed execution must be consumed by flank before flaky one
2. The following flaky execution must be actually failed but the outcome of all re-runs is success
3. Then flank does what is the following:
1. The outcome is set to `failure` when failed test/shards/executions are processed
```kotlin
private fun Outcome?.updateOutcome(
when {
Expand Down Expand Up @@ -71,4 +71,4 @@ Following must occur:
} ?: outcome
}
```
2. Add unit test to cover this case.
2. Add unit tests to cover this case.

0 comments on commit 2f003e3

Please sign in to comment.