Added printing outcome details

docs/gcloud/firebase/test/results_summary.puml

@@ -0,0 +1,57 @@
+@startuml
+start
+:CreateMatrixOutcomeSummaryUsingSteps;
+:_GetStepOutcomeDetails;
+if (outcome success) then (yes)
+    :_GetSuccessCountDetails;
+    if (successDetail and otherNativeCrash) then (yes)
+        :format details;
+    else (no)
+        :details;
+    endif
+else if (failure) then (yes)
+    if (failureDetail) then (yes)
+        :_GetFailureDetail;
+    elseif (not testExecutionStep) then (yes)
+        :Unknown failure;
+    else (no)
+        :_GetFailureOrFlakyCountDetails;
+    endif
+else if (inconclusive) then (yes)
+    :_GetInconclusiveDetail;
+elseif (skipped) then (yes)
+    :_GetSkippedDetail;
+else (no)
+    :Unknown outcome;
+endif
+end
+
+start
+:CreateMatrixOutcomeSummaryUsingEnvironments;
+if (environments and all environmentResult outcome) then (yes)
+:_GetEnvironmentOutcomeDetails;
+    if (outcome success) then (yes)
+        :_GetSuccessCountDetails;
+        if (successDetail and otherNativeCrash) then (yes)
+            :format details;
+        else (no)
+            :details;
+        endif
+    else if (failure or flaky) then (yes)
+        if (failureDetail) then (yes)
+            :_GetFailureDetail;
+        else (no)
+            :_GetFailureOrFlakyCountDetails;
+        endif
+    else if (inconclusive) then (yes)
+        :_GetInconclusiveDetail;
+    elseif (skipped) then (yes)
+        :_GetSkippedDetail;
+    else (no)
+        :Unknown outcome;
+    endif
+else (no)
+    :_GetStepOutcomeDetails;
+endif
+end
+@enduml

docs/summary_output.md

@@ -0,0 +1,64 @@
+# Formatted summary output 
+```
+┌─────────┬──────────────────────┬─────────────────────┐
+│ OUTCOME │      MATRIX ID       │    TEST DETAILS     │
+├─────────┼──────────────────────┼─────────────────────┤
+│ success │ matrix-1z85qtvdnvb0l │ 4 test cases passed │
+└─────────┴──────────────────────┴─────────────────────┘
+```
+
+
+## User scenario
+As a user I want to see finely formatted summary result at the end of execution output.
+
+## Motivation
+Gcloud prints summary output in the table. It looks nice and is readable. Why we wouldn't have same in flank?
+
+## Possible outputs
+Numbers are representing `OUTCOME` column, points are representing `TEST DETAILS` column.
+1. `success | flaky`
+    * `${1} test cases passed | ${2} skipped | ${3} flakes | (Native crash) | ---`
+2. `failure`
+    * `${1} test cases failed | ${2} errors | ${3} passed | ${4} skipped |  ${4} flakes  | (Native crash)`
+    * `Application crashed | (Native crash)`
+    * `Test timed out | (Native crash)`
+    * `App failed to install | (Native crash)`
+    * `Unknown failure | (Native crash)`
+3. `inconclusive`
+    * `Infrastructure failure`
+    * `Test run aborted by user`
+    * `Unknown reason`
+4. `skipped`
+    * `Incompatible device/OS combination`
+    * `App does not support the device architecture`
+    * `App does not support the OS version`
+    * `Unknown reason`
+
+## Implementation details
+
+### Outcome calculation
+It should be mentioned there are some crucial differences how flank and gcloud calculates outcome value.
+
+Gcloud is using following API calls
+1. `self._client.projects_histories_executions_environments.List(request)`
+2. `self._client.projects_histories_executions_steps.List(request)`
+
+The first one is default, but if returns any `environment` without `environmentResult.outcome`, the second one will be used to obtain `steps`. 
+Both `environemnts` and `steps` can provide `outcome`. The difference between them is the `steps` returns `success` event if tests are `flaky`.
+Currently, we don't know why `self._client.projects_histories_executions_environments.List(request)` may return empty `environmentResult.outcome`.
+
+In difference to gcloud flank uses 3 api call to obtain necessary data
+1. `TestMatrix` - `GcTesting.get.projects().testMatrices().get(projectId, testMatrixId)`
+2. `Step` - `toolsResults.projects().histories().executions().steps().get(projectId, historyId, executionId, stepId)`
+3. `ListTestCasesResponse` - `toolsResults.projects().histories().executions().steps().testCases().get(projectId, historyId, executionId, stepId)`
+
+`TestMatrix` from first call provides `ToolResultsStep` through `TestExecution` which is used to obtain arguments for next two calls.  
+
+This is part of flank legacy. Those api calls provides data for `JUnitResult`. 
+As `JUnitResult` contains all data required to generate `table` output, we can reuse it.
+In result, we are forced to calculate `flaky` outcomes on flank site because of `step`.
+Probably it is place for little improvement in the future.
+
+### Test details calculation
+When flank and gcloud implementations can be slightly different because of programming languages,
+the logic behind is the mainly same.

release_notes.md

@@ -5,7 +5,7 @@
 - [#828](https://github.com/Flank/flank/pull/828) Store test results in gcloud bucket. ([adamfilipow92](https://github.com/adamfilipow92))
 - [#865](https://github.com/Flank/flank/pull/865) Flank needs to respect the timeout value as that's a cap for billing purposes. ([adamfilipow92](https://github.com/adamfilipow92), [pawelpasterz](https://github.com/pawelpasterz))
 - [#866](https://github.com/Flank/flank/pull/866) Fix printing all matrix links. ([piotradamczyk5](https://github.com/piotradamczyk5))
--
+- [#862](https://github.com/Flank/flank/pull/862) Added printing outcome details. ([piotradamczyk5](https://github.com/piotradamczyk5), [jan-gogo](https://github.com/jan-gogo))
 -
 -
 

test_runner/src/main/kotlin/ftl/json/OutcomeDetailsFormatter.kt

@@ -0,0 +1,70 @@
+package ftl.json
+
+import com.google.api.services.toolresults.model.FailureDetail
+import com.google.api.services.toolresults.model.InconclusiveDetail
+import com.google.api.services.toolresults.model.Outcome
+import com.google.api.services.toolresults.model.SkippedDetail
+import ftl.reports.api.data.TestSuiteOverviewData
+import ftl.util.StepOutcome.failure
+import ftl.util.StepOutcome.flaky
+import ftl.util.StepOutcome.inconclusive
+import ftl.util.StepOutcome.skipped
+import ftl.util.StepOutcome.success
+import ftl.util.StepOutcome.unset
+
+fun Outcome.getDetails(testSuiteOverviewData: TestSuiteOverviewData?) = when (summary) {
+    success, flaky -> testSuiteOverviewData?.getSuccessOutcomeDetails(successDetail?.otherNativeCrash ?: false)
+    failure -> failureDetail.getFailureOutcomeDetails(testSuiteOverviewData)
+    inconclusive -> inconclusiveDetail.formatOutcomeDetails()
+    skipped -> skippedDetail.formatOutcomeDetails()
+    unset -> "Unset outcome"
+    else -> "Unknown outcome"
+}
+
+private fun TestSuiteOverviewData.getSuccessOutcomeDetails(
+    otherNativeCrash: Boolean
+) = StringBuilder("$successCount test cases passed").apply {
+    if (skipped > 0) append(skippedMessage(skipped))
+    if (flakes > 0) append(flakesMessage(flakes))
+    if (otherNativeCrash) append(NATIVE_CRASH_MESSAGE)
+}.toString()
+
+private val TestSuiteOverviewData.successCount
+    get() = total - errors - failures - skipped - flakes
+
+private fun FailureDetail?.getFailureOutcomeDetails(testSuiteOverviewData: TestSuiteOverviewData?) = when {
+    this == null -> testSuiteOverviewData?.buildFailureOutcomeDetailsSummary() ?: "Unknown failure"
+    crashed == true -> "Application crashed"
+    timedOut == true -> "Test timed out"
+    notInstalled == true -> "App failed to install"
+    else -> testSuiteOverviewData?.buildFailureOutcomeDetailsSummary() ?: "Unknown failure"
+} + this?.takeIf { it.otherNativeCrash }?.let { NATIVE_CRASH_MESSAGE }.orEmpty()
+
+private fun TestSuiteOverviewData.buildFailureOutcomeDetailsSummary() =
+    StringBuilder("$failures test cases failed").apply {
+        if (errors > 0) append(errorMessage(errors))
+        successCount.takeIf { it > 0 }?.let { append(successMessage(it)) }
+        if (skipped > 0) append(skippedMessage(skipped))
+        if (flakes > 0) append(flakesMessage(flakes))
+    }.toString()
+
+private fun InconclusiveDetail?.formatOutcomeDetails() = when {
+    this == null -> "Unknown reason"
+    infrastructureFailure == true -> "Infrastructure failure"
+    abortedByUser == true -> "Test run aborted by user"
+    else -> "Unknown reason"
+}
+
+private fun SkippedDetail?.formatOutcomeDetails(): String = when {
+    this == null -> "Unknown reason"
+    incompatibleDevice == true -> "Incompatible device/OS combination"
+    incompatibleArchitecture == true -> "App does not support the device architecture"
+    incompatibleAppVersion == true -> "App does not support the OS version"
+    else -> "Unknown reason"
+}
+
+private const val NATIVE_CRASH_MESSAGE = " (Native crash)"
+private val flakesMessage: (Int) -> String = { ", $it flaky" }
+private val skippedMessage: (Int) -> String = { ", $it skipped" }
+private val successMessage: (Int) -> String = { ", $it passed" }
+private val errorMessage: (Int) -> String = { ", $it errors" }

test_runner/src/main/kotlin/ftl/json/SavedMatrix.kt

@@ -1,18 +1,21 @@
 package ftl.json
 
-import com.google.api.client.json.GenericJson
+import com.google.api.services.testing.model.TestExecution
 import com.google.api.services.testing.model.TestMatrix
 import com.google.api.services.toolresults.model.Outcome
-import ftl.android.AndroidCatalog
+import ftl.android.AndroidCatalog.isVirtualDevice
 import ftl.gc.GcToolResults
+import ftl.reports.api.createTestExecutionDataListAsync
+import ftl.reports.api.createTestSuitOverviewData
+import ftl.reports.api.data.TestSuiteOverviewData
+import ftl.reports.api.prepareForJUnitResult
 import ftl.util.MatrixState.ERROR
 import ftl.util.MatrixState.FINISHED
 import ftl.util.StepOutcome.failure
 import ftl.util.StepOutcome.flaky
 import ftl.util.StepOutcome.inconclusive
 import ftl.util.StepOutcome.skipped
 import ftl.util.StepOutcome.success
-import ftl.util.StepOutcome.unset
 import ftl.util.billableMinutes
 import ftl.util.timeoutToSeconds
 import ftl.util.webLink
@@ -82,57 +85,81 @@ class SavedMatrix(matrix: TestMatrix) {
         billableVirtualMinutes = 0
         billablePhysicalMinutes = 0
         outcome = success
-        if (matrix.testExecutions == null) return
 
-        matrix.testExecutions.forEach {
-            val executionResult = GcToolResults.getExecutionResult(it)
-
-            updateOutcome(executionResult.outcome)
-
-            // flank should not calculate billable minutes if an infrastructure error occurred
-            if (it.state == ERROR) return
-
-            // testExecutionStep, testTiming, etc. can all be null.
-            // sometimes testExecutionStep is present and testTiming is null
-            val stepResult = GcToolResults.getStepResult(it.toolResultsStep)
-            val testTimeSeconds = stepResult.testExecutionStep?.testTiming?.testProcessDuration?.seconds ?: return
-            val testTimeout = timeoutToSeconds(it.testSpecification?.testTimeout ?: "0s")
-
-            // if overall test duration time is higher then testTimeout flank should calculate billable minutes for testTimeout
-            val timeToBill = min(testTimeSeconds, testTimeout)
-            val billableMinutes = billableMinutes(timeToBill)
+        updateFinishedMatrixData(matrix)
+    }
 
-            if (AndroidCatalog.isVirtualDevice(it.environment?.androidDevice, matrix.projectId ?: "")) {
-                billableVirtualMinutes += billableMinutes
-            } else {
-                billablePhysicalMinutes += billableMinutes
+    private fun updateFinishedMatrixData(matrix: TestMatrix) {
+        val testExecutionsData = matrix.testExecutions?.createTestExecutionDataListAsync() ?: return
+        val summedTestSuiteOverviewData =
+            testExecutionsData
+                .prepareForJUnitResult()
+                .fold(TestSuiteOverviewData(0, 0, 0, 0, 0, 0.0, 0.0)) { sum, test ->
+                    sum + test.createTestSuitOverviewData()
+                }
+
+        testExecutionsData
+            .forEach {
+                with(GcToolResults.getExecutionResult(it.testExecution).outcome) {
+                    updateOutcome(it.step.outcome?.summary != this?.summary)
+                    updateOutcomeDetails(
+                        testSuiteOverviewData = summedTestSuiteOverviewData,
+                        isRoboTests = it.testExecution.testSpecification?.androidRoboTest != null
+                    )
+                }
+                it.testExecution.getBillableMinutes()?.let { billableMinutes ->
+                    updateBillableMinutes(
+                        billableMinutes = billableMinutes,
+                        isVirtualDevice = isVirtualDevice(
+                            it.testExecution.environment.androidDevice,
+                            matrix.projectId.orEmpty()
+                        )
+                    )
+                }
             }
-        }
     }
 
-    private fun updateOutcome(stepOutcome: Outcome) {
-        // the matrix outcome is failure if any step fails
-        // if the matrix outcome is already set to failure then we can ignore the other step outcomes.
-        // inconclusive is treated as a failure
-        if (outcome == failure || outcome == inconclusive) return
-
-        outcome = stepOutcome.summary
+    private fun TestExecution.getBillableMinutes() =
+        takeIf { testExecution -> testExecution.state != ERROR }
+            ?.run {
+                // testExecutionStep, testTiming, etc. can all be null.
+                // sometimes testExecutionStep is present and testTiming is null
+                val testTimeSeconds =
+                    GcToolResults.getStepResult(toolResultsStep).testExecutionStep?.testTiming?.testProcessDuration?.seconds
+                        ?: return null
+                val testTimeout = timeoutToSeconds(testSpecification?.testTimeout ?: "0s")
+
+                // if overall test duration time is higher then testTimeout flank should calculate billable minutes for testTimeout
+                billableMinutes(min(testTimeSeconds, testTimeout))
+            }
 
-        // Treat flaky outcome as a success
-        if (outcome == flaky) outcome = success
+    private fun Outcome?.updateOutcome(
+        flakyOutcome: Boolean
+    ) {
+        outcome = when {
+            flakyOutcome -> flaky
+            // the matrix outcome is failure if any step fails
+            // if the matrix outcome is already set to failure then we can ignore the other step outcomes.
+            // inconclusive is treated as a failure
+            outcome == failure || outcome == inconclusive -> return
+            outcome == flaky -> this?.summary?.takeIf { it == failure || it == inconclusive }
+            else -> this?.summary
+        } ?: outcome
+    }
 
-        outcomeDetails = when (outcome) {
-            failure -> stepOutcome.failureDetail.keysToString()
-            success -> stepOutcome.successDetail.keysToString()
-            inconclusive -> stepOutcome.inconclusiveDetail.keysToString()
-            skipped -> stepOutcome.skippedDetail.keysToString()
-            unset -> "unset"
-            else -> "unknown"
-        }
+    private fun Outcome?.updateOutcomeDetails(
+        testSuiteOverviewData: TestSuiteOverviewData?,
+        isRoboTests: Boolean
+    ) {
+        outcomeDetails = if (isRoboTests) "Robo test" else (this?.getDetails(testSuiteOverviewData) ?: "---")
     }
 
-    private fun GenericJson?.keysToString(): String {
-        return this?.keys?.joinToString(",") ?: ""
+    private fun updateBillableMinutes(billableMinutes: Long, isVirtualDevice: Boolean) {
+        if (isVirtualDevice) {
+            billableVirtualMinutes += billableMinutes
+        } else {
+            billablePhysicalMinutes += billableMinutes
+        }
     }
 
     val gcsPathWithoutRootBucket get() = gcsPath.substringAfter('/')

test_runner/src/main/kotlin/ftl/mock/MockServer.kt

@@ -79,12 +79,15 @@ object MockServer {
                 outcome.summary = failure
                 val failureDetail = FailureDetail()
                 failureDetail.timedOut = true
+                failureDetail.crashed = false
+                failureDetail.otherNativeCrash = false
                 outcome.failureDetail = failureDetail
             }
             "-2" -> {
                 outcome.summary = inconclusive
                 val inconclusiveDetail = InconclusiveDetail()
                 inconclusiveDetail.abortedByUser = true
+                inconclusiveDetail.infrastructureFailure = false
                 outcome.inconclusiveDetail = inconclusiveDetail
             }
             "-3" -> {
@@ -194,6 +197,13 @@ object MockServer {
                     val executionId = call.parameters["executionId"] ?: ""
                     call.respond(fakeStep(executionId))
                 }
+                // GcToolResults.listTestCases(toolResultsStep)
+                // GET /toolresults/v1beta3/projects/flank-open-source/histories/1/executions/1/steps/1/testCases
+                get("/toolresults/v1beta3/projects/{project}/histories/{historyId}/executions/{executionId}/steps/{stepId}/testCases") {
+                    println("Responding to GET ${call.request.uri}")
+                    val executionId = call.parameters["executionId"] ?: ""
+                    call.respond(fakeStep(executionId))
+                }
 
                 // GcToolResults.getDefaultBucket(project)
                 post("/toolresults/v1beta3/projects/{project}:initializeSettings") {

test_runner/src/main/kotlin/ftl/reports/api/CreateTestSuitOverviewData.kt

@@ -12,7 +12,7 @@ internal fun TestExecutionData.createTestSuitOverviewData(): TestSuiteOverviewDa
     ?.let { overview ->
         val skipped: Int = overview.skippedCount ?: 0
         TestSuiteOverviewData(
-            total = testCases.size + skipped,
+            total = testCases.size,
             errors = testCases.countErrors(),
             failures = testCases.countFailures(),
             flakes = testCases.countFlakes(),