Skip to content

Commit

Permalink
Update 914_falsy_positive_outcome_for_flaky_tests.md (#1181)
Browse files Browse the repository at this point in the history
  • Loading branch information
adamfilipow92 authored Oct 1, 2020
1 parent 5d129af commit 7ab3946
Showing 1 changed file with 34 additions and 1 deletion.
35 changes: 34 additions & 1 deletion docs/bugs/914_falsy_positive_outcome_for_flaky_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
|Date|Who?|Action|
|---|---|---|
|31th July 2020|[pawelpasterz](https://github.com/pawelpasterz)|created|
|1st October 2020|[adamfilipow92](https://github.com/adamfilipow92)|update|
| | | |

### Description
Expand Down Expand Up @@ -71,4 +72,36 @@ The following must occur:
} ?: outcome
}
```
2. Add unit tests to cover this case.
2. The test below reflects potential but rare behavior.

```kotlin
@Test
fun `savedMatrix should have flaky outcome when at least one test is flaky`() {
val expectedOutcome = "flaky"
val successStepExecution = createStepExecution(1) // success
// https://github.com/Flank/flank/issues/918
// This test covers edge case where summary for both step and execution is null and outcome of
// saved matrix was not changed and is set to success
val malformed = createStepExecution(stepId = -666, executionId = -666) // flaky

// below order in the list matters!
val executions = listOf(
successStepExecution,
successStepExecution,
malformed
)

val testMatrix = testMatrix().apply {
testMatrixId = "123"
state = FINISHED
resultStorage = createResultsStorage()
testExecutions = executions
}

val savedMatrix = createSavedMatrix(testMatrix)

assertEquals(expectedOutcome, savedMatrix.outcome)
}
```

3. This issue is not deterministic and really difficult to reproduce by manual tests. The community not reporting this issue for some time.

0 comments on commit 7ab3946

Please sign in to comment.