From d72fe5e082304af0915998492e9942dc7f1a5ca8 Mon Sep 17 00:00:00 2001 From: Michael Wright Date: Tue, 11 Aug 2020 16:44:35 +0200 Subject: [PATCH 01/13] Added inital analytics template for team review --- README.md | 4 +++- docs/readme.md | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4c613826ba..8099fb2a4c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # Flank [![Build Status](https://app.bitrise.io/app/9767f3e19047d4db/status.svg?token=uDM3wCumR2xTd0axh4bjDQ&branch=master)](https://app.bitrise.io/app/9767f3e19047d4db) [![codecov](https://codecov.io/gh/Flank/flank/branch/master/graph/badge.svg)](https://codecov.io/gh/Flank/flank) [![pullreminders](https://pullreminders.com/badge.svg)](https://pullreminders.com?ref=badge) -Flank is a [massively parallel Android and iOS test runner](https://docs.google.com/presentation/d/1goan9cXpimSJsS3L60WjljnFA_seUyaWb2e-bezm084/edit#slide=id.p1) for [Firebase Test Lab](https://firebase.google.com/docs/test-lab/). +Flank is a [massively parallel Android and iOS test runner](https://docs.google.com/presentation/d/1goan9cXpimSJsS3L60WjljnFA_seUyaWb2e-bezm084/edit#slide=id.p1) for [Firebase Test Lab](https://firebase.google.com/docs/test-lab/). + +Flank is YAML compatible with the gcloud CLI and has extra features for performance and stability. It provides testing features to accelerate velocity and quality. ## Download diff --git a/docs/readme.md b/docs/readme.md index 452b4f8ab9..e71905c2ea 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -75,6 +75,10 @@ Client may be generated manually using `testing_v1.json` and the master branch o - `gradle build` +## Analytics + +Flank makes use of various analytics. For more information on what they are and how to control them please see [analyitics.md](analytics.md) + ## Mock Servers API Discovery JSON may be converted to OpenAPI 3 using [apimatic.io/transformer](https://apimatic.io/transformer). See [mock_server.md](mock_server.md) for details. From 2e6767e8c8e8787ca55a0c93d21d9ece94331b9c Mon Sep 17 00:00:00 2001 From: Michael Wright Date: Tue, 11 Aug 2020 16:45:00 +0200 Subject: [PATCH 02/13] Added analytics.md file --- docs/analytics.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 docs/analytics.md diff --git a/docs/analytics.md b/docs/analytics.md new file mode 100644 index 0000000000..d8508cff00 --- /dev/null +++ b/docs/analytics.md @@ -0,0 +1,10 @@ +# Flank Analytics + +Flank makes use of various analytics, below contains a list of the analytics and how to enable/disable them from being sent to FTL. + +## Flank + + - BugSnag + - **Disable** - ADD "DISABLE" to the analytics_uuid for example: echo "DISABLED" > ~/.gsutil/analytics-uuid + - **Enable** - Remove "DISABLE" from the analytics_uuid file for example: echo "$(grep -v "DISABLED" ~/.gsutil/analytics-uuid)" > ~/.gsutil/analytics-uuid +- From 5dd96918abf0556897abef5d35f039d569334adf Mon Sep 17 00:00:00 2001 From: Michael Wright Date: Wed, 12 Aug 2020 13:21:17 +0200 Subject: [PATCH 03/13] Re-worked the wording as per discussion --- docs/analytics.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/analytics.md b/docs/analytics.md index d8508cff00..ce342e9f1e 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -1,10 +1,22 @@ # Flank Analytics -Flank makes use of various analytics, below contains a list of the analytics and how to enable/disable them from being sent to FTL. +Flank makes use of Bugsnag as its only analytics platform. It uses Bugsnag as an error monitor and project stability tool. + +For more information about Bugsnag please visit its website [here](https://www.bugsnag.com/) and documentation [here](https://docs.bugsnag.com/) + +## Control + +It is possible to disable the Bugsnag analytics integration used by flank or to make use of custom Bugsnag key so that any analytics can be displayed for custom use. + + - To disable, place the word ```DISABLED``` in the file ````~/.gsutil/analytics-uuid```` and Flank will no longer send any analytics data + - Or simply running the following code (make sure the file exists!) will disable: ```echo "DISABLED" > ~/.gsutil/analytics-uuid``` + + - To enable or reenable bugsnag simply remove the file ```~/.gsutil/analytics-uuid``` and the project will default to make use of the flanks Bugsnag key + + - To make use of custom Bugsnag integration, place your bugsnag key/UUID in ```~/.gsutil/analytics-uuid``` and the flank project will make use of it instead of the default key. + + ## More information + + To see how Bugsnag integration is used within the project please see the Flank Bugsnag testcase [here](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) -## Flank - - BugSnag - - **Disable** - ADD "DISABLE" to the analytics_uuid for example: echo "DISABLED" > ~/.gsutil/analytics-uuid - - **Enable** - Remove "DISABLE" from the analytics_uuid file for example: echo "$(grep -v "DISABLED" ~/.gsutil/analytics-uuid)" > ~/.gsutil/analytics-uuid -- From f49a5a5763a624baabd754917c86d7e317df3953 Mon Sep 17 00:00:00 2001 From: Michael Wright Date: Wed, 12 Aug 2020 13:25:24 +0200 Subject: [PATCH 04/13] Release notes updated --- release_notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release_notes.md b/release_notes.md index 5701400b84..a93e541ed2 100644 --- a/release_notes.md +++ b/release_notes.md @@ -1,5 +1,5 @@ ## next (unreleased) -- +- [#987](https://github.com/Flank/flank/pull/987) Analytics readme addition ([sloox](https://github.com/Sloox)) - ## v20.08.1 From 918c89298cbc4f97e0b2ae8605b7dfb0863142a0 Mon Sep 17 00:00:00 2001 From: Michael Wright Date: Wed, 12 Aug 2020 17:47:30 +0200 Subject: [PATCH 05/13] Fixes to wording, typos and recommendations as per pull request --- docs/analytics.md | 27 ++++++++++++++++++++------- docs/readme.md | 2 +- 2 files changed, 21 insertions(+), 8 deletions(-) diff --git a/docs/analytics.md b/docs/analytics.md index ce342e9f1e..5fbc588e4d 100644 --- a/docs/analytics.md +++ b/docs/analytics.md @@ -1,22 +1,35 @@ # Flank Analytics -Flank makes use of Bugsnag as its only analytics platform. It uses Bugsnag as an error monitor and project stability tool. +Flank makes use of Bugsnag as an Error monitoring and project stability tool. For more information about Bugsnag please visit its website [here](https://www.bugsnag.com/) and documentation [here](https://docs.bugsnag.com/) ## Control -It is possible to disable the Bugsnag analytics integration used by flank or to make use of custom Bugsnag key so that any analytics can be displayed for custom use. +It is possible to disable the Bugsnag analytics integration used by Flank. Flank makes use of the same OPT-OUT approach as Gcloud CLI and can be disabled the same way. - - To disable, place the word ```DISABLED``` in the file ````~/.gsutil/analytics-uuid```` and Flank will no longer send any analytics data - - Or simply running the following code (make sure the file exists!) will disable: ```echo "DISABLED" > ~/.gsutil/analytics-uuid``` + - To disable, place ONLY the word ```DISABLED``` in the file ````~/.gsutil/analytics-uuid```` and Flank will no longer make use of Bugsnag integration. + - It is recommended to backup the file ````~/.gsutil/analytics-uuid```` if it exists as it may contain your project specific UUID. + - For example running the following code will disable Bugsnag: ```echo "DISABLED" > ~/.gsutil/analytics-uuid``` (make sure the file exists!) - - To enable or reenable bugsnag simply remove the file ```~/.gsutil/analytics-uuid``` and the project will default to make use of the flanks Bugsnag key + - To enable or reenable bugsnag simply remove the file ```~/.gsutil/analytics-uuid``` or replace it with your backed-up file and the project will again begin to make use Bugsnag. - - To make use of custom Bugsnag integration, place your bugsnag key/UUID in ```~/.gsutil/analytics-uuid``` and the flank project will make use of it instead of the default key. ## More information - To see how Bugsnag integration is used within the project please see the Flank Bugsnag testcase [here](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) +To see how Bugsnag is integrated within the Flank project please see the Flank Bugsnag testcase [here](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) and the actual Bugsnag implementation found [here](../test_runner/src/main/kotlin/ftl/util/BugsnagInitHelper.kt) + +Flank makes use of Bugsnag to monitor test runner stability and to capture errors. It provides diagnostic data that is used to make data driven decisions when prioritizing fixing bugs or adding new features. + +The goal is to use language that demonstrates value to Flank's customers. + +For example American express wants a stable and reliable test runner. Bugsnag is helping us deliver this too them. + + +BugSnag sumarizes the following about how it makes use of the data it collects and stores: + + - Bugsnag monitors web, mobile and server applications and programs under the direction of its Customers, to provide error Event Data about the Customer’s applications and programs to the Customer. Event Data may contain Personal Information of the individuals who use the Customer’s applications. Bugsnag itself does not collect this end user Personal Information and Bugsnag has no direct relationship with these individuals. For example, a Customer may direct Bugsnag to log certain software application’s end user information associated with Event Data, such as the end user’s device ID, email and name. Bugsnag is a data processor and follows the instructions of the Customer in these cases. + +More information about Bugsnag data policy can be found [here](https://docs.bugsnag.com/legal/privacy-policy/#:~:text=Services%20via%20Mobile%20Devices.&text=Bugsnag%20will%20collect%20certain%20information,operating%20system%20of%20your%20device.) diff --git a/docs/readme.md b/docs/readme.md index e71905c2ea..1a21e5ab94 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -77,7 +77,7 @@ Client may be generated manually using `testing_v1.json` and the master branch o ## Analytics -Flank makes use of various analytics. For more information on what they are and how to control them please see [analyitics.md](analytics.md) +Flank makes use of Bugsnag for analytics. For more information about analytics and how to control them please see [analyitics.md](analytics.md) ## Mock Servers From 10c749771ea5bf33e170db48f9df505477e00547 Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 12:00:32 -0400 Subject: [PATCH 06/13] Update and rename analytics.md to error_monitoring.md --- docs/{analytics.md => error_monitoring.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename docs/{analytics.md => error_monitoring.md} (93%) diff --git a/docs/analytics.md b/docs/error_monitoring.md similarity index 93% rename from docs/analytics.md rename to docs/error_monitoring.md index 5fbc588e4d..20ec7075da 100644 --- a/docs/analytics.md +++ b/docs/error_monitoring.md @@ -1,4 +1,4 @@ -# Flank Analytics +# Flank Error Monitoring Flank makes use of Bugsnag as an Error monitoring and project stability tool. @@ -6,7 +6,7 @@ For more information about Bugsnag please visit its website [here](https://www.b ## Control -It is possible to disable the Bugsnag analytics integration used by Flank. Flank makes use of the same OPT-OUT approach as Gcloud CLI and can be disabled the same way. +It is possible to disable the Bugsnag integration used by Flank. Flank makes use of the same OPT-OUT approach as Gcloud CLI and can be disabled the same way. - To disable, place ONLY the word ```DISABLED``` in the file ````~/.gsutil/analytics-uuid```` and Flank will no longer make use of Bugsnag integration. - It is recommended to backup the file ````~/.gsutil/analytics-uuid```` if it exists as it may contain your project specific UUID. From 42a931648176873e620809d9ed8e3fb8cf7993af Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:20:52 -0400 Subject: [PATCH 07/13] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8099fb2a4c..2742d74101 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ -# Flank [![Build Status](https://app.bitrise.io/app/9767f3e19047d4db/status.svg?token=uDM3wCumR2xTd0axh4bjDQ&branch=master)](https://app.bitrise.io/app/9767f3e19047d4db) [![codecov](https://codecov.io/gh/Flank/flank/branch/master/graph/badge.svg)](https://codecov.io/gh/Flank/flank) [![pullreminders](https://pullreminders.com/badge.svg)](https://pullreminders.com?ref=badge) +# Flank [![codecov](https://codecov.io/gh/Flank/flank/branch/master/graph/badge.svg)](https://codecov.io/gh/Flank/flank) Flank is a [massively parallel Android and iOS test runner](https://docs.google.com/presentation/d/1goan9cXpimSJsS3L60WjljnFA_seUyaWb2e-bezm084/edit#slide=id.p1) for [Firebase Test Lab](https://firebase.google.com/docs/test-lab/). -Flank is YAML compatible with the gcloud CLI and has extra features for performance and stability. It provides testing features to accelerate velocity and quality. +Flank is YAML compatible with the gcloud CLI and has extra features for performance and stability. Flank provides testing features to accelerate velocity and quality. ## Download From 6c158bf856b3af6d62de1c6734c0dea6e52616ed Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:30:08 -0400 Subject: [PATCH 08/13] Update error_monitoring.md --- docs/error_monitoring.md | 42 ++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/error_monitoring.md b/docs/error_monitoring.md index 20ec7075da..f656384405 100644 --- a/docs/error_monitoring.md +++ b/docs/error_monitoring.md @@ -1,35 +1,35 @@ # Flank Error Monitoring -Flank makes use of Bugsnag as an Error monitoring and project stability tool. +Flank uses Bugsnag to monitor test runner stability. Bugsnag provides diagnostic data that is used to make data driven decisions when prioritizing fixing bugs or adding new features. -For more information about Bugsnag please visit its website [here](https://www.bugsnag.com/) and documentation [here](https://docs.bugsnag.com/) +- https://www.bugsnag.com/ +- https://docs.bugsnag.com/ -## Control +## Data Captured -It is possible to disable the Bugsnag integration used by Flank. Flank makes use of the same OPT-OUT approach as Gcloud CLI and can be disabled the same way. +Bugsnag captures the following error data: - - To disable, place ONLY the word ```DISABLED``` in the file ````~/.gsutil/analytics-uuid```` and Flank will no longer make use of Bugsnag integration. - - It is recommended to backup the file ````~/.gsutil/analytics-uuid```` if it exists as it may contain your project specific UUID. - - For example running the following code will disable Bugsnag: ```echo "DISABLED" > ~/.gsutil/analytics-uuid``` (make sure the file exists!) +Flank + - Stacktrace + - releaseStage of Flank (production or snapshot) + - version of Flank (git commit Flank was built from) - - To enable or reenable bugsnag simply remove the file ```~/.gsutil/analytics-uuid``` or replace it with your backed-up file and the project will again begin to make use Bugsnag. +Device + - hostname + - locale + - osArch + - osName + - osVersion + - runtimeVersions of Java +## Disable Bugsnag - ## More information - -To see how Bugsnag is integrated within the Flank project please see the Flank Bugsnag testcase [here](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) and the actual Bugsnag implementation found [here](../test_runner/src/main/kotlin/ftl/util/BugsnagInitHelper.kt) - -Flank makes use of Bugsnag to monitor test runner stability and to capture errors. It provides diagnostic data that is used to make data driven decisions when prioritizing fixing bugs or adding new features. - -The goal is to use language that demonstrates value to Flank's customers. +Flank respects the same analytics opt out as gcloud CLI. -For example American express wants a stable and reliable test runner. Bugsnag is helping us deliver this too them. +`echo "DISABLED" > ~/.gsutil/analytics-uuid` + ## More information -BugSnag sumarizes the following about how it makes use of the data it collects and stores: - - - Bugsnag monitors web, mobile and server applications and programs under the direction of its Customers, to provide error Event Data about the Customer’s applications and programs to the Customer. Event Data may contain Personal Information of the individuals who use the Customer’s applications. Bugsnag itself does not collect this end user Personal Information and Bugsnag has no direct relationship with these individuals. For example, a Customer may direct Bugsnag to log certain software application’s end user information associated with Event Data, such as the end user’s device ID, email and name. Bugsnag is a data processor and follows the instructions of the Customer in these cases. +To see how Bugsnag is integrated within the Flank project please see the Flank Bugsnag testcase [here](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) and the actual Bugsnag implementation found [here](../test_runner/src/main/kotlin/ftl/util/BugsnagInitHelper.kt) More information about Bugsnag data policy can be found [here](https://docs.bugsnag.com/legal/privacy-policy/#:~:text=Services%20via%20Mobile%20Devices.&text=Bugsnag%20will%20collect%20certain%20information,operating%20system%20of%20your%20device.) - - From 50e9dba8876381fe6c03c5bad5ea6b4e3a757f37 Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:32:05 -0400 Subject: [PATCH 09/13] Update README.md --- README.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 2742d74101..b15ac38e23 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Flank is a [massively parallel Android and iOS test runner](https://docs.google.com/presentation/d/1goan9cXpimSJsS3L60WjljnFA_seUyaWb2e-bezm084/edit#slide=id.p1) for [Firebase Test Lab](https://firebase.google.com/docs/test-lab/). -Flank is YAML compatible with the gcloud CLI and has extra features for performance and stability. Flank provides testing features to accelerate velocity and quality. +Flank is YAML compatible with [the gcloud CLI](https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test). Flank provides extra features to accelerate velocity and increase quality. ## Download @@ -23,14 +23,12 @@ https://github.com/Flank/flank/releases/latest/download/flank.jar ### Features -Available now | 2019 - -- | -- -Test sharding | Client/Server refactor -Cost reporting -Stability testing -HTML report -JUnit XML report -Smart Flank +- Test sharding +- Cost reporting +- Stability testing +- HTML report +- JUnit XML report +- Smart Flank ### Exit Codes From 9bc6263b2c9c7734b5d69ac68f4ed981a540a4ed Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:34:30 -0400 Subject: [PATCH 10/13] Update error_monitoring.md --- docs/error_monitoring.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/error_monitoring.md b/docs/error_monitoring.md index f656384405..5cb1241efb 100644 --- a/docs/error_monitoring.md +++ b/docs/error_monitoring.md @@ -1,6 +1,6 @@ # Flank Error Monitoring -Flank uses Bugsnag to monitor test runner stability. Bugsnag provides diagnostic data that is used to make data driven decisions when prioritizing fixing bugs or adding new features. +Flank uses Bugsnag to monitor test runner stability. Bugsnag enables data driven decisions when prioritizing bug fixes. - https://www.bugsnag.com/ - https://docs.bugsnag.com/ @@ -30,6 +30,6 @@ Flank respects the same analytics opt out as gcloud CLI. ## More information -To see how Bugsnag is integrated within the Flank project please see the Flank Bugsnag testcase [here](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) and the actual Bugsnag implementation found [here](../test_runner/src/main/kotlin/ftl/util/BugsnagInitHelper.kt) +To see how Bugsnag is integrated within the Flank project please see the [Flank Bugsnag testcase](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) and [the actual Bugsnag implementation](../test_runner/src/main/kotlin/ftl/util/BugsnagInitHelper.kt) -More information about Bugsnag data policy can be found [here](https://docs.bugsnag.com/legal/privacy-policy/#:~:text=Services%20via%20Mobile%20Devices.&text=Bugsnag%20will%20collect%20certain%20information,operating%20system%20of%20your%20device.) +- [Bugsnag data policy](https://docs.bugsnag.com/legal/privacy-policy/#:~:text=Services%20via%20Mobile%20Devices.&text=Bugsnag%20will%20collect%20certain%20information,operating%20system%20of%20your%20device.) From 6bd7dbb3a03a58f93945fe750fbdf7b234d25715 Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:35:55 -0400 Subject: [PATCH 11/13] Update error_monitoring.md --- docs/error_monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/error_monitoring.md b/docs/error_monitoring.md index 5cb1241efb..e491052a5f 100644 --- a/docs/error_monitoring.md +++ b/docs/error_monitoring.md @@ -28,7 +28,7 @@ Flank respects the same analytics opt out as gcloud CLI. `echo "DISABLED" > ~/.gsutil/analytics-uuid` - ## More information +## More information To see how Bugsnag is integrated within the Flank project please see the [Flank Bugsnag testcase](../test_runner/src/test/kotlin/ftl/util/FlankBugsnagInitHelperTest.kt) and [the actual Bugsnag implementation](../test_runner/src/main/kotlin/ftl/util/BugsnagInitHelper.kt) From a1c6b43544610e78fa7e17b12e9e56566b581488 Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:50:20 -0400 Subject: [PATCH 12/13] Update release_notes.md --- release_notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release_notes.md b/release_notes.md index a93e541ed2..70582ee3e1 100644 --- a/release_notes.md +++ b/release_notes.md @@ -1,5 +1,5 @@ ## next (unreleased) -- [#987](https://github.com/Flank/flank/pull/987) Analytics readme addition ([sloox](https://github.com/Sloox)) +- [#987](https://github.com/Flank/flank/pull/987) Flank Error Monitoring readme addition ([sloox](https://github.com/Sloox)) - ## v20.08.1 From 8d9bc1a43980b96e4e7e20a3606cc169dfa6a791 Mon Sep 17 00:00:00 2001 From: bootstraponline Date: Wed, 12 Aug 2020 19:51:21 -0400 Subject: [PATCH 13/13] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index b15ac38e23..9419758cee 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ https://github.com/Flank/flank/releases/latest/download/flank.jar [](https://www.bugsnag.com/) +See [docs/error_monitoring.md](./docs/error_monitoring.md) to disable Bugsnag error monitoring. + ### Contributing - Install [Oracle JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)