Merge branch 'master' into #899-test-artifacts-scripts

.github/workflows/macos_workflow.yml

@@ -18,9 +18,11 @@ jobs:
 
     steps:
     - uses: actions/checkout@v2
+      with: 
+        submodules: true
     - uses: actions/setup-java@v1
       with:
-        java-version: 8
+        java-version: 11
     - uses: actions/cache@v2
       with:
         path: ~/.gradle/caches

.github/workflows/release.yml

@@ -12,10 +12,12 @@ on:
 
 jobs:
   release:
-    runs-on: macos-latest
+    runs-on: ubuntu-latest
 
     steps:
     - uses: actions/checkout@v2
+      with: 
+        submodules: true
 
     - name: Get all git tags
       run: |
@@ -108,11 +110,12 @@ jobs:
         gradle-executable: "./test_runner/gradlew"
         arguments: "-p test_runner publish -PGITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}"
 
-    - name: Post Message of Flank Release TEST ONLY
-      uses: archive/github-actions-slack@master
+    - name: Post Message of Flank Release
+      uses: Flank/[email protected]
       if: startsWith(github.ref, 'refs/tags/v')
       with:
-          slack-bot-user-oauth-access-token: ${{ secrets.SLACK_BOT_USER_OAUTH_ACCESS_TOKEN }}
-          slack-channel: flank_test_channel
-          slack-optional-icon_emoji: ":fire:"  
-          slack-text: Flank ${{ env.RELEASE_TAG }} has been released. View more information <http://github.com/Flank/flank/releases/tag/${{ env.RELEASE_TAG }}| here!>
+          xoxctoken: ${{ secrets.SLACK_XOXCTOKEN }}
+          message: Flank ${{ env.RELEASE_TAG }} has been released. View more information <http://github.com/Flank/flank/releases/tag/${{ env.RELEASE_TAG }}| here!>
+          channel: ${{ secrets.SLACK_CHANNEL}}
+          cookie: ${{ secrets.SLACK_COOKIE}}
+

.github/workflows/ubuntu-workflow.yml

@@ -16,6 +16,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
+        with: 
+          submodules: true
       - name: Display the java version
         run: java -version 
         shell: bash

README.md

@@ -16,8 +16,9 @@ See [docs/error_monitoring.md](./docs/error_monitoring.md) to disable Bugsnag er
 
 ### Contributing
 
-- Install [Oracle JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
-  - JDK 9 or later will not work
+- Install JDK 11 (it works also correctly on previous version, newer version are not guaranteed to works properly):
+  - [Oracle](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html)
+  - [OpenJDK](https://jdk.java.net/11/)
 - Use [JetBrains Toolbox](https://www.jetbrains.com/toolbox/app/) to install `IntelliJ IDEA Community`
 - Clone the repo `git clone --recursive https://github.com/Flank/flank.git`
   - `git submodule update --init --recursive` updates the submodules
@@ -729,11 +730,7 @@ and flank's example [gradle-export-api](https://github.com/Flank/flank/tree/mast
 
     With the [update_flank.sh](https://github.com/Flank/flank/blob/master/test_runner/bash/update_flank.sh) shell script, you can rebuild `flank.jar`.
 
-3)  > Symbol is declared in module 'java.xml' which does not export package 'com.sun.org.apache.xerces.internal.dom'
-
-    Make sure you're using JDK 8 to compile Flank.
-
-4)  > Test run failed to complete. Expected 786 tests, received 660
+3)  > Test run failed to complete. Expected 786 tests, received 660
 
     Try setting `use-orchestrator: false`. Parameterized tests [are not compatible with orchestrator](https://stackoverflow.com/questions/48735268/unable-to-run-parameterized-tests-with-android-test-orchestrator). Flank uses [orchestrator by default on Android.](https://developer.android.com/training/testing/junit-runner)
 

action.yml

@@ -0,0 +1,26 @@
+name: slack-user-bot
+
+description: Make a user act like a bot in slack
+author: Flank
+
+branding:
+  icon: file-plus
+  color: blue
+
+runs:
+  using: 'docker'
+  image: './flank-actions/Dockerfile'
+
+inputs:
+  xoxctoken:
+    description: A slack token that gives permission to act as a user.
+    required: true
+  message: 
+    description: The message to send
+    required: true
+  channel:
+    description: Channel to send to
+    required: true
+  cookie: 
+    description: Required cookie for auth
+    required: true

docs/bugs/986_test_count_and_smart_sharding_ count_dont_match.md

@@ -0,0 +1,36 @@
+# Test count and smart sharding count don't match
+
+Bug reported in [this issue](https://github.com/Flank/flank/issues/986)
+
+## The problem
+
+Flank does not support parameterized tests sharding. Every class with parameterized is considered as one test during shard calculation.
+
+Flank is using [DEX parser](https://github.com/linkedin/dex-test-parser) to decompile apks and gather info about all the tests inside. As for now, Flank is unable to determine how many times a test in a parameterized class is invoked. Due to this fact scans apks for any class with an annotation that contains `JUnitParamsRunner` or `Parameterized`:
+
+```kotlin
+
+@RunWith(JUnitParamsRunner::class)
+...
+@RunWith(Parameterized::class)
+
+```
+
+## Solution
+
+1. Flank knows how many tests and classes are being sent to Firebase. So we can inform the user of how many classes we have. Example:
+
+```txt
+
+ Smart Flank cache hit: 0% (0 / 9)
+  Shard times: 240s, 240s, 240s, 360s
+
+  Uploading app-debug.apk .
+  Uploading app-multiple-flaky-debug-androidTest.apk .
+  5 tests + 4 parameterized classes / 4 shards
+
+```
+
+1. Default test time for classes should be different from the default time for test
+   1. You can set default test time for class with ```--default-class-test-time``` command
+   2. If you did not set this time, the default value is 240s

flank-actions/Dockerfile

@@ -0,0 +1,26 @@
+FROM openjdk:8-alpine
+
+COPY . /
+
+RUN apk add bash curl git zip && \
+  curl -s "https://get.sdkman.io" | bash
+
+RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && sdk version"
+
+RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && sdk install kotlin"
+
+RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && sdk install kscript"
+
+RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && sdk install gradle"
+
+RUN chmod +x /compile.sh
+
+RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && /compile.sh"
+
+RUN chmod +x /entrypoint.sh
+
+RUN chmod +x /sendMessage
+
+RUN bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && /test.sh"
+
+ENTRYPOINT /bin/bash /entrypoint.sh

flank-actions/common.kt

@@ -0,0 +1,9 @@
+object Common {
+    const val EXIT_CODE_SUCCESS = 0
+    const val EXIT_CODE_FAILURE = -1
+    const val URL_SLACK ="https://slack.com/api/chat.postMessage"
+    const val ARGS_TOKEN = 0
+    const val ARGS_CHANNEL = 1
+    const val ARGS_MESSAGE = 2
+    const val ARGS_COOKIE = 3
+}

flank-actions/compile.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+kscript --package sendMessage.kts

flank-actions/entrypoint.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+echo 'Sending message'
+./sendMessage $INPUT_XOXCTOKEN $INPUT_MESSAGE $INPUT_CHANNEL $INPUT_COOKIE
+echo 'Sending message done!'

flank-actions/sendMessage.kts

@@ -0,0 +1,15 @@
+import kotlin.system.exitProcess
+
+
+//DEPS com.github.kittinunf.fuel:fuel:2.2.3
+
+//INCLUDE slackService.kt
+//INCLUDE common.kt
+
+val result = sendMessage(args)
+
+println("Message has been sent with result $result")
+
+if (result != 0){
+    exitProcess(result)
+}

flank-actions/slackService.kt

@@ -0,0 +1,32 @@
+
+import com.github.kittinunf.fuel.Fuel
+import com.github.kittinunf.fuel.core.Headers
+
+fun sendMessage(args: Array<String>): Int{
+    val token = args[Common.ARGS_TOKEN]
+    val channel = args[Common.ARGS_CHANNEL]
+    val message = args[Common.ARGS_MESSAGE]
+    val cookie = args.copyOfRange(Common.ARGS_COOKIE, args.lastIndex).joinToString(" ")
+
+    return try {
+        val (req, rep, res) = Fuel.get(Common.URL_SLACK, 
+            listOf("token" to token,"channel" to channel,"text" to message))
+            .header(Headers.COOKIE to cookie)
+            .responseString()
+
+        debug("Result: $res")
+        Common.EXIT_CODE_SUCCESS
+    }
+    catch(e: Exception) {
+        error("Error has occured: $e")
+        Common.EXIT_CODE_FAILURE
+    }
+}
+
+fun debug(message: String){
+    println("::debug::$message")
+}
+
+fun error(message: String) {
+    println("::error::$message")
+}

release_notes.md

@@ -1,3 +1,21 @@
+## v20.09.2
+### Features
+- [#1111](https://github.com/Flank/flank/pull/1111) Check if gcs path exist before run ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1110](https://github.com/Flank/flank/pull/1110) Finialize the slack sending integration with actions ([Sloox](https://github.com/Sloox))
+- [#1108](https://github.com/Flank/flank/pull/1108) Support for robo tests without robo script ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1104](https://github.com/Flank/flank/pull/1104) Send slack message for releases ([Sloox](https://github.com/Sloox))
+- [#1097](https://github.com/Flank/flank/pull/1097) Print the matrices web link at the end of a run ([adamfilipow92](https://github.com/adamfilipow92))
+### Bug Fixes
+- [#1107](https://github.com/Flank/flank/pull/1107) Running iOS test on Linux ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1059](https://github.com/Flank/flank/pull/1059) Shards and Tests count do not match ([pawelpasterz](https://github.com/pawelpasterz), [adamfilipow92](https://github.com/adamfilipow92))
+- [#1067](https://github.com/Flank/flank/pull/1067) Matrix path not found in json ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+
+## v20.09.1
+### Bug Fixes
+- [#1100](https://github.com/Flank/flank/pull/1100) Fix gcs files validation ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1093](https://github.com/Flank/flank/pull/1093) Flank release bot formatting ([Sloox](https://github.com/Sloox))
+- [#1091](https://github.com/Flank/flank/pull/1091) Add missing authors to release notes ([adamfilipow92](https://github.com/adamfilipow92))
+
 ## v20.09.0
 ### Bug Fixes
 - [#1087](https://github.com/Flank/flank/pull/1087) Fix release action ([adamfilipow92](https://github.com/adamfilipow92))

test_runner/docs/ascii/flank.jar_-android-run.adoc

@@ -87,6 +87,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -102,9 +105,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -117,12 +117,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -236,12 +236,12 @@ You can guide the Robo test to perform specific actions by recording a Robo Scri
 +
 Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting. 
 
-*--legacy-junit-result*::
-  Fallback for legacy xml junit results parsing.
-
 *--additional-app-test-apks*=_<String=String>_[,_<String=String>_...]::
   A list of app & test apks to include in the run. Useful for running multiple module tests within a single Flank run.
 
+*--legacy-junit-result*::
+  Fallback for legacy xml junit results parsing.
+
 *--dump-shards*::
   Measures test shards from given test apks and writes them into android_shards.json file instead of executing.
 

test_runner/docs/ascii/flank.jar_-firebase-test-android-run.adoc

@@ -99,6 +99,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -114,9 +117,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -129,12 +129,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -248,12 +248,12 @@ You can guide the Robo test to perform specific actions by recording a Robo Scri
 +
 Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting. 
 
-*--legacy-junit-result*::
-  Fallback for legacy xml junit results parsing.
-
 *--additional-app-test-apks*=_<String=String>_[,_<String=String>_...]::
   A list of app & test apks to include in the run. Useful for running multiple module tests within a single Flank run.
 
+*--legacy-junit-result*::
+  Fallback for legacy xml junit results parsing.
+
 *--dump-shards*::
   Measures test shards from given test apks and writes them into android_shards.json file instead of executing.