feat: Add storing output as json #1559

piotradamczyk5 · 2021-02-03T12:12:45Z

Fixes #1437

Test Plan

How do we know the code works?

Enable output report using option ( cli: --output-report=json, yaml: output-report: json under flank root)
Run Flank and see that the output reported is stored ( file outputReport.json)
Compare file content with console output

Checklist

SDD
Documented
Unit tested
Integration tests updated

github-actions · 2021-02-03T12:12:59Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

github-actions · 2021-02-03T12:17:58Z

Timestamp: 2021-02-04 17:01:25
Buildscan url for ubuntu-workflow run 537794707

adamfilipow92 · 2021-02-03T16:12:32Z

test_runner/src/main/kotlin/ftl/reports/output/OutputReportConfiguration.kt

+    )
+)
+
+private val IArgs.localStorageDirectory


It could be reusable, wdyt about moving this property to IArgs.kt?

Thanks! I will use it in my pr 😄

piotradamczyk5 · 2021-02-04T16:49:44Z

@Mergifyio rebase

mergify · 2021-02-04T16:50:28Z

Command rebase: success

Branch has been successfully rebased

jan-goral

Test Report

Run

From repository root

cd test_runner
flank run android

Result

outputReport.json:

{
  "args": {
    "commonArgs": {
      "data": "---\ngcloud:\n  app: \"../test_projects/android/apks/app-debug.apk\"\n  test: \"../test_projects/android/apks/app-debug-androidTest.apk\"\nflank: {}\n",
      "devices": [
        {
          "model": "NexusLowRes",
          "version": "28",
          "locale": "en",
          "orientation": "portrait",
          "isVirtual": true
        }
      ],
      "resultsBucket": "test-lab-v9cn46bb990nx-kz69ymd4nm9aq",
      "resultsDir": "2021-02-04_16-48-31.143000_OhrI",
      "recordVideo": false,
      "testTimeout": "15m",
      "async": false,
      "flakyTestAttempts": 0,
      "otherFiles": {},
      "directoriesToPull": [],
      "scenarioNumbers": [],
      "failFast": false,
      "project": "flank-open-source",
      "maxTestShards": 1,
      "shardTime": -1,
      "repeatTests": 1,
      "smartFlankGcsPath": "",
      "smartFlankDisableUpload": false,
      "testTargetsAlwaysRun": [],
      "filesToDownload": [],
      "disableSharding": false,
      "localResultDir": "results",
      "runTimeout": "-1",
      "fullJUnitResult": false,
      "ignoreFailedTests": false,
      "keepFilePath": false,
      "outputStyle": "Verbose",
      "disableResultsUpload": false,
      "defaultTestTime": 120.0,
      "defaultClassTestTime": 240.0,
      "useAverageTestTimeForNewTests": false,
      "disableUsageStatistics": false,
      "outputReportType": "JSON"
    },
    "appApk": "/Users/jan-gogo/projects/flank-project/flank/test_projects/android/apks/app-debug.apk",
    "testApk": "/Users/jan-gogo/projects/flank-project/flank/test_projects/android/apks/app-debug-androidTest.apk",
    "additionalApks": [],
    "autoGoogleLogin": false,
    "useOrchestrator": true,
    "roboDirectives": [],
    "environmentVariables": {},
    "grantPermissions": "all",
    "scenarioLabels": [],
    "obbFiles": [],
    "obbNames": [],
    "performanceMetrics": false,
    "testTargets": [],
    "additionalAppTestApks": [],
    "useLegacyJUnitResult": false,
    "obfuscateDumpShards": false,
    "testTargetsForShard": []
  },
  "cost": {
    "physical": 0.00,
    "virtual": 0.02,
    "total": 0.02
  },
  "test_results": {
    "matrix-1g0ipqet4iz4m": [
      {
        "device": "NexusLowRes-28-en-portrait",
        "outcome": "success",
        "details": "1 test cases passed",
        "testSuiteOverview": {
          "total": 1,
          "errors": 0,
          "failures": 0,
          "flakes": 0,
          "skipped": 0,
          "elapsedTime": 1.075,
          "overheadTime": 0.0
        }
      }
    ]
  },
  "weblinks": [
    "https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7143386027735431817/executions/bs.6aa4fb040cccf3bb"
  ]
}

Summary

The implementation looks good 👍
outputResult.json looks as expected

Good Job!

jan-goral · 2021-02-04T17:06:12Z

We need one review more and test different config

adamfilipow92 · 2021-02-04T17:31:05Z

Please update pr description

adamfilipow92

Code looks good
Output works - tested for ios simple-ios-flank with output-report flag
Integration tests works, tested locally (macos)

Here is my outputReport.json

{
  "args": {
    "useLegacyJUnitResult": true,
    "xcTestRunData$delegate": {
      "_value": {
        "rootDir": "/Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/",
        "nsDict": {
          "EarlGreyExampleSwiftTests": {
            "BlueprintName": {
              "content": "EarlGreyExampleSwiftTests"
            },
            "BundleIdentifiersForCrashReportEmphasis": {
              "array": [
                {
                  "content": "io.gogoapps.earlgrey.samples.EarlGreySwiftTests"
                },
                {
                  "content": "io.gogoapps.earlgrey.samples.EarlGrey"
                }
              ]
            },
            "CommandLineArguments": {
              "array": []
            },
            "DefaultTestExecutionTimeAllowance": {
              "type": 0,
              "longValue": 600,
              "doubleValue": 600.0,
              "boolValue": false
            },
            "DependentProductPaths": {
              "array": [
                {
                  "content": "__TESTROOT__/Debug-iphoneos/EarlGreyExampleSwift.app"
                },
                {
                  "content": "__TESTROOT__/Debug-iphoneos/EarlGreyExampleSwift.app/PlugIns/EarlGreyExampleSwiftTests.xctest"
                }
              ]
            },
            "EnvironmentVariables": {
              "DYLD_INSERT_LIBRARIES": {
                "content": "@executable_path/EarlGrey.framework/EarlGrey"
              },
              "OS_ACTIVITY_DT_MODE": {
                "content": "YES"
              },
              "SQLITE_ENABLE_THREAD_ASSERTIONS": {
                "content": "1"
              }
            },
            "IsAppHostedTestBundle": {
              "type": 2,
              "longValue": 1,
              "doubleValue": 1.0,
              "boolValue": true
            },
            "ProductModuleName": {
              "content": "EarlGreyExampleTestsSwift"
            },
            "RunOrder": {
              "type": 0,
              "longValue": 0,
              "doubleValue": 0.0,
              "boolValue": false
            },
            "SystemAttachmentLifetime": {
              "content": "deleteOnSuccess"
            },
            "TestBundlePath": {
              "content": "__TESTHOST__/PlugIns/EarlGreyExampleSwiftTests.xctest"
            },
            "TestHostBundleIdentifier": {
              "content": "io.gogoapps.earlgrey.samples.EarlGrey"
            },
            "TestHostPath": {
              "content": "__TESTROOT__/Debug-iphoneos/EarlGreyExampleSwift.app"
            },
            "TestLanguage": {
              "content": ""
            },
            "TestRegion": {
              "content": ""
            },
            "TestTimeoutsEnabled": {
              "type": 2,
              "longValue": 0,
              "doubleValue": 0.0,
              "boolValue": false
            },
            "TestingEnvironmentVariables": {
              "DYLD_INSERT_LIBRARIES": {
                "content": "__TESTHOST__/Frameworks/libXCTestBundleInject.dylib"
              },
              "XCInjectBundleInto": {
                "content": "unused"
              }
            },
            "ToolchainsSettingValue": {
              "array": []
            },
            "UITargetAppCommandLineArguments": {
              "array": []
            },
            "UserAttachmentLifetime": {
              "content": "deleteOnSuccess"
            }
          },
          "__xctestrun_metadata__": {
            "FormatVersion": {
              "type": 0,
              "longValue": 1,
              "doubleValue": 1.0,
              "boolValue": false
            }
          }
        },
        "version": "V1",
        "shardTargets": {
          "": [
            {
              "EarlGreyExampleSwiftTests": [
                "EarlGreyExampleSwiftTests/testWithGreyAssertions",
                "EarlGreyExampleSwiftTests/testLayout",
                "EarlGreyExampleSwiftTests/testThatThrows",
                "EarlGreyExampleSwiftTests/testWithInRoot",
                "EarlGreyExampleSwiftTests/testCustomAction",
                "EarlGreyExampleSwiftTests/testWithCondition",
                "EarlGreyExampleSwiftTests/testBasicSelection",
                "EarlGreyExampleSwiftTests/testWithCustomMatcher",
                "EarlGreyExampleSwiftTests/testCollectionMatchers",
                "EarlGreyExampleSwiftTests/testCatchErrorOnFailure",
                "EarlGreyExampleSwiftTests/testWithCustomAssertion",
                "EarlGreyExampleSwiftTests/testTableCellOutOfScreen",
                "EarlGreyExampleSwiftTests/testBasicSelectionAndAction",
                "EarlGreyExampleSwiftTests/testBasicSelectionAndAssert",
                "EarlGreyExampleSwiftTests/testWithCustomFailureHandler",
                "EarlGreyExampleSwiftTests/testBasicSelectionActionAssert",
                "EarlGreyExampleSwiftTests/testSelectionOnMultipleElements"
              ]
            }
          ]
        },
        "shardChunks": {
          "": [
            {
              "testMethodNames": [
                "EarlGreyExampleSwiftTests/testWithGreyAssertions",
                "EarlGreyExampleSwiftTests/testLayout",
                "EarlGreyExampleSwiftTests/testThatThrows",
                "EarlGreyExampleSwiftTests/testWithInRoot",
                "EarlGreyExampleSwiftTests/testCustomAction",
                "EarlGreyExampleSwiftTests/testWithCondition",
                "EarlGreyExampleSwiftTests/testBasicSelection",
                "EarlGreyExampleSwiftTests/testWithCustomMatcher",
                "EarlGreyExampleSwiftTests/testCollectionMatchers",
                "EarlGreyExampleSwiftTests/testCatchErrorOnFailure",
                "EarlGreyExampleSwiftTests/testWithCustomAssertion",
                "EarlGreyExampleSwiftTests/testTableCellOutOfScreen",
                "EarlGreyExampleSwiftTests/testBasicSelectionAndAction",
                "EarlGreyExampleSwiftTests/testBasicSelectionAndAssert",
                "EarlGreyExampleSwiftTests/testWithCustomFailureHandler",
                "EarlGreyExampleSwiftTests/testBasicSelectionActionAssert",
                "EarlGreyExampleSwiftTests/testSelectionOnMultipleElements"
              ],
              "testMethods": [
                {
                  "name": "EarlGreyExampleSwiftTests/testWithGreyAssertions",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testLayout",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testThatThrows",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testWithInRoot",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testCustomAction",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testWithCondition",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testBasicSelection",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testWithCustomMatcher",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testCollectionMatchers",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testCatchErrorOnFailure",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testWithCustomAssertion",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testTableCellOutOfScreen",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testBasicSelectionAndAction",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testBasicSelectionAndAssert",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testWithCustomFailureHandler",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testBasicSelectionActionAssert",
                  "time": 120.0,
                  "isParameterized": false
                },
                {
                  "name": "EarlGreyExampleSwiftTests/testSelectionOnMultipleElements",
                  "time": 120.0,
                  "isParameterized": false
                }
              ]
            }
          ]
        }
      }
    },
    "commonArgs": {
      "data": "---\ngcloud:\n  test: \"./test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExample.zip\"\n  xctestrun-file: \"./test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExampleSwiftTests.xctestrun\"\n  results-dir: \"test_dir\"\nflank:\n  output-report: \"json\"\n",
      "devices": [
        {
          "model": "iphone8",
          "version": "13.6",
          "locale": "en",
          "orientation": "portrait",
          "isVirtual": false
        }
      ],
      "resultsBucket": "test-lab-v9cn46bb990nx-kz69ymd4nm9aq",
      "resultsDir": "test_dir",
      "recordVideo": false,
      "testTimeout": "15m",
      "async": false,
      "flakyTestAttempts": 0,
      "otherFiles": {},
      "type": "XCTEST",
      "directoriesToPull": [],
      "scenarioNumbers": [],
      "failFast": false,
      "project": "flank-open-source",
      "maxTestShards": 1,
      "shardTime": -1,
      "repeatTests": 1,
      "smartFlankGcsPath": "",
      "smartFlankDisableUpload": false,
      "testTargetsAlwaysRun": [],
      "filesToDownload": [],
      "disableSharding": false,
      "localResultDir": "results",
      "runTimeout": "-1",
      "fullJUnitResult": false,
      "ignoreFailedTests": false,
      "keepFilePath": false,
      "outputStyle": "Single",
      "disableResultsUpload": false,
      "defaultTestTime": 120.0,
      "defaultClassTestTime": 240.0,
      "useAverageTestTimeForNewTests": false,
      "disableUsageStatistics": false,
      "outputReportType": "JSON"
    },
    "xctestrunZip": "/Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExample.zip",
    "xctestrunFile": "/Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExampleSwiftTests.xctestrun",
    "testTargets": [],
    "obfuscateDumpShards": false,
    "additionalIpas": [],
    "app": "",
    "testSpecialEntitlements": false,
    "onlyTestConfiguration": "",
    "skipTestConfiguration": ""
  },
  "cost": {
    "physical": 0.08,
    "virtual": 0.00,
    "total": 0.08
  },
  "test_results": {
    "matrix-1iimg89acphkj": [
      {
        "device": "iphone8-13.6-en-portrait",
        "outcome": "success",
        "details": "17 test cases passed",
        "testSuiteOverview": {
          "total": 17,
          "errors": 0,
          "failures": 0,
          "flakes": 0,
          "skipped": 0,
          "elapsedTime": 22.419,
          "overheadTime": 0.0
        }
      }
    ]
  },
  "weblinks": [
    "https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.a3b607c9bb6d0088/matrices/7683345571758085610/executions/bs.9a78518a6501c50e"
  ]
}

piotradamczyk5 requested review from bootstraponline, Sloox, zuziaka, pawelpasterz, jan-goral and adamfilipow92 February 3, 2021 12:12

github-actions bot added the Feature label Feb 3, 2021

github-actions bot assigned adamfilipow92 Feb 3, 2021

github-actions bot added the Tiger 🐯 label Feb 3, 2021

github-actions bot assigned piotradamczyk5 Feb 3, 2021

piotradamczyk5 force-pushed the #1437_add_storing_output_as_json branch from a06b9de to 507205a Compare February 3, 2021 15:13

piotradamczyk5 marked this pull request as ready for review February 3, 2021 16:06

adamfilipow92 reviewed Feb 3, 2021

View reviewed changes

piotradamczyk5 force-pushed the #1437_add_storing_output_as_json branch from 57c4f6e to 9baad05 Compare February 4, 2021 15:33

piotradamczyk5 requested a review from adamfilipow92 February 4, 2021 16:18

Piotr Adamczyk and others added 7 commits February 4, 2021 16:50

feat: Add storing output as json

94b6ba2

added enable-output-report to printing

e1586f8

added unit tests

4934cd8

added documentation

9963a1c

Updated integration tests

b98b1ff

moved extension to IArgs

a7d67b3

change enable-output-report to output-report(type)

918a5f7

bootstraponline force-pushed the #1437_add_storing_output_as_json branch from 9baad05 to 918a5f7 Compare February 4, 2021 16:50

jan-goral approved these changes Feb 4, 2021

View reviewed changes

adamfilipow92 approved these changes Feb 4, 2021

View reviewed changes

piotradamczyk5 merged commit cf1b4ba into master Feb 4, 2021

piotradamczyk5 deleted the #1437_add_storing_output_as_json branch February 4, 2021 18:22

github-actions bot locked and limited conversation to collaborators Feb 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add storing output as json #1559

feat: Add storing output as json #1559

piotradamczyk5 commented Feb 3, 2021 •

edited

Loading

github-actions bot commented Feb 3, 2021 •

edited

Loading

github-actions bot commented Feb 3, 2021 •

edited

Loading

adamfilipow92 Feb 3, 2021

piotradamczyk5 Feb 3, 2021

adamfilipow92 Feb 3, 2021

piotradamczyk5 commented Feb 4, 2021

mergify bot commented Feb 4, 2021

jan-goral left a comment

jan-goral commented Feb 4, 2021

adamfilipow92 commented Feb 4, 2021

adamfilipow92 left a comment

feat: Add storing output as json #1559

feat: Add storing output as json #1559

Conversation

piotradamczyk5 commented Feb 3, 2021 • edited Loading

Test Plan

Checklist

github-actions bot commented Feb 3, 2021 • edited Loading

github-actions bot commented Feb 3, 2021 • edited Loading

adamfilipow92 Feb 3, 2021

Choose a reason for hiding this comment

piotradamczyk5 Feb 3, 2021

Choose a reason for hiding this comment

adamfilipow92 Feb 3, 2021

Choose a reason for hiding this comment

piotradamczyk5 commented Feb 4, 2021

mergify bot commented Feb 4, 2021

jan-goral left a comment

Choose a reason for hiding this comment

Test Report

Run

Result

Summary

jan-goral commented Feb 4, 2021

adamfilipow92 commented Feb 4, 2021

adamfilipow92 left a comment

Choose a reason for hiding this comment

piotradamczyk5 commented Feb 3, 2021 •

edited

Loading

github-actions bot commented Feb 3, 2021 •

edited

Loading

github-actions bot commented Feb 3, 2021 •

edited

Loading