Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Remove Sentry analytics #2283

Merged
merged 1 commit into from
Oct 12, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion buildSrc/src/main/kotlin/Dependencies.kt
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
object Dependencies {
const val SENTRY = "io.sentry:sentry:${Versions.SENTRY}"
const val MIXPANEL = "com.mixpanel:mixpanel-java:${Versions.MIXPANEL}"
const val DD_PLIST = "com.googlecode.plist:dd-plist:${Versions.DD_PLIST}"
const val DEX_TEST_PARSER = "com.linkedin.dextestparser:parser:${Versions.DEX_TEST_PARSER}"
Expand Down
3 changes: 0 additions & 3 deletions buildSrc/src/main/kotlin/Versions.kt
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
object Versions {
// https://github.com/getsentry/sentry-java/releases
const val SENTRY = "6.1.4"

// https://github.com/mixpanel/mixpanel-java/releases
const val MIXPANEL = "1.5.0"

Expand Down
37 changes: 0 additions & 37 deletions docs/error_monitoring.md

This file was deleted.

74 changes: 35 additions & 39 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,13 @@
# Flank [![codecov](https://codecov.io/gh/Flank/flank/branch/master/graph/badge.svg)](https://codecov.io/gh/Flank/flank)

Flank is a [massively parallel Android and iOS test runner](https://docs.google.com/presentation/d/1goan9cXpimSJsS3L60WjljnFA_seUyaWb2e-bezm084/edit#slide=id.p1) for [Firebase Test Lab](https://firebase.google.com/docs/test-lab/).
Flank is a [massively parallel Android and iOS test runner](https://docs.google.com/presentation/d/1goan9cXpimSJsS3L60WjljnFA_seUyaWb2e-bezm084/edit#slide=id.p1) for [Firebase Test Lab](https://firebase.google.com/docs/test-lab/).

Flank is YAML compatible with [the gcloud CLI](https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test). Flank provides extra features to accelerate velocity and increase quality.

## Download

https://github.com/Flank/flank/releases/latest/download/flank.jar

## Sponsors

See [error monitoring docs](./error_monitoring.md) to disable Sentry error monitoring.

## Contributing

- Install JDK 15 (it works also correctly on the previous version, a newer version is not guaranteed to work properly):
Expand All @@ -37,14 +33,14 @@ See [error monitoring docs](./error_monitoring.md) to disable Sentry error monit

| Exit code | Description |
| ---: | :--- |
| 0 | All tests passed
| 1 | A general failure occurred. Possible causes include: a filename that does not exist or an HTTP/network error.
| 2 | Usually indicates missing or wrong usage of flags, incorrect parameters, errors in config files.
| 10 | At least one matrix not finished (usually a FTL internal error) or unexpected error occurred.
| 15 | Firebase Test Lab could not determine if the test matrix passed or failed, because of an unexpected error.
| 18 | The test environment for this test execution is not supported because of incompatible test dimensions. This error might occur if the selected Android API level is not supported by the selected device type.
| 19 | The test matrix was canceled by the user.
| 20 | A test infrastructure error occurred.
| 0 | All tests passed
| 1 | A general failure occurred. Possible causes include: a filename that does not exist or an HTTP/network error.
| 2 | Usually indicates missing or wrong usage of flags, incorrect parameters, errors in config files.
| 10 | At least one matrix not finished (usually a FTL internal error) or unexpected error occurred.
| 15 | Firebase Test Lab could not determine if the test matrix passed or failed, because of an unexpected error.
| 18 | The test environment for this test execution is not supported because of incompatible test dimensions. This error might occur if the selected Android API level is not supported by the selected device type.
| 19 | The test matrix was canceled by the user.
| 20 | A test infrastructure error occurred.

## CLI

Expand All @@ -70,11 +66,11 @@ Run `test_runner/flank.ios.yml` with flank to verify iOS execution is working.
# https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test/ios/run
gcloud:
# -- GcloudYml --
### Results Bucket

### Results Bucket
## The name of a Google Cloud Storage bucket where raw test results will be stored
# results-bucket: tmp_flank

### Results Directory
## The name of a unique Google Cloud Storage object within the results bucket where raw test results will be stored
## (default: a timestamp with a random suffix).
Expand Down Expand Up @@ -164,7 +160,7 @@ gcloud:
# version: 12.0
# locale: es_ES
# orientation: landscape

### Directories to Pull
## A list of paths that will be copied from the device's storage to the designated results bucket after the test
## is complete. These must be absolute paths under /private/var/mobile/Media or /Documents
Expand All @@ -180,9 +176,9 @@ gcloud:
# other-files
# com.my.app:/Documents/file.txt: local/file.txt
# /private/var/mobile/Media/file.jpg: gs://bucket/file.jpg

### Additional IPA's
## List of up to 100 additional IPAs to install, in addition to the one being directly tested.
## List of up to 100 additional IPAs to install, in addition to the one being directly tested.
## The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
# additional-ipas:
# - gs://bucket/additional.ipa
Expand All @@ -199,10 +195,10 @@ gcloud:
### Test type
## The type of iOS test to run. TYPE must be one of: xctest, game-loop. Default: xctest
# type: xctest

### Application Path
## The path to the application archive (.ipa file) for game-loop testing.
## The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
## The path to the application archive (.ipa file) for game-loop testing.
## The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
## This flag is only valid when --type=game-loop is also set
# app:
# - gs://bucket/additional.ipa OR path/to/local/ipa/file.ipa
Expand Down Expand Up @@ -328,14 +324,14 @@ flank:

### Only Test Configuration
## Constrains a test action to only test a specified test configuration within a test plan and exclude all other test configurations.
## Flank can combine multiple constraint options, but -only-test-configuration has precedence over -skip-test-configuration.
## Flank can combine multiple constraint options, but -only-test-configuration has precedence over -skip-test-configuration.
## Each test configuration name must match the name of a configuration specified in a test plan and is case-sensitive.
## Default: null (run all test configurations)
# only-test-configuration: en

### Skip Test Configuration
## Constrains a test action to skip a specified test configuration and include all other test configurations.
## Flank can combine multiple constraint options, but -only-test-configuration has precedence over -skip-test-configuration.
## Flank can combine multiple constraint options, but -only-test-configuration has precedence over -skip-test-configuration.
## Each test configuration name must match the name of a configuration specified in a test plan and is case-sensitive.
## Default: null (run all test configurations)
# skip-test-configuration: en
Expand Down Expand Up @@ -384,7 +380,7 @@ gcloud:
## Enable video recording during the test. Disabled by default. Use --record-video to enable.
# record-video: true

### Timeout
### Timeout
## The max time this test execution can run before it is cancelled (default: 15m).
## It does not include any time necessary to prepare and clean up the target device.
## The maximum possible testing time is 45m on physical devices and 60m on virtual devices.
Expand Down Expand Up @@ -514,7 +510,7 @@ gcloud:
# - label2

### OBB filenames
## A list of OBB required filenames. OBB file name must conform to the format as specified by Android e.g.
## A list of OBB required filenames. OBB file name must conform to the format as specified by Android e.g.
## [main|patch].0300110.com.example.android.obb which will be installed into <shared-storage>/Android/obb/<package-name>/ on the device.
# obb-names:
# - [main|patch].<VERSION>.com.example.android.obb
Expand Down Expand Up @@ -579,8 +575,8 @@ gcloud:
# version: 27

### test-targets-for-shard
## Specifies a group of packages, classes, and/or test cases to run in each shard (a group of test cases).
## The shards are run in parallel on separate devices. You can repeat this flag up to 50 times to specify multiple shards when one or more physical devices are selected,
## Specifies a group of packages, classes, and/or test cases to run in each shard (a group of test cases).
## The shards are run in parallel on separate devices. You can repeat this flag up to 50 times to specify multiple shards when one or more physical devices are selected,
## or up to 500 times when no physical devices are selected.
## Note: If you include the flags environment-variable or test-targets when running test-targets-for-shard, the flags are applied to all the shards you create.
# test-target-for-shard:
Expand Down Expand Up @@ -635,7 +631,7 @@ flank:
## Default: false
# use-average-test-time-for-new-tests: true

### Default Test Time
### Default Test Time
## Set default test time used for calculating shards.
## Default: 120.0
# default-test-time: 15
Expand Down Expand Up @@ -981,7 +977,7 @@ v3.0-SNAPSHOT

## Maven

You can consume Flank via maven.
You can consume Flank via maven.
See the [maven repo](https://bintray.com/flank/maven/flank) for all supported versions.

```
Expand Down Expand Up @@ -1041,30 +1037,30 @@ and flank's example [gradle-export-api](https://github.com/Flank/flank/tree/mast
3) > Test run failed to complete. Expected 786 tests, received 660

Try setting `use-orchestrator: false`. Parameterized tests [are not compatible with orchestrator](https://stackoverflow.com/questions/48735268/unable-to-run-parameterized-tests-with-android-test-orchestrator). Flank uses [orchestrator by default on Android.](https://developer.android.com/training/testing/junit-runner)

4) > I have an issue when attempting to sync the Flank Gradle project
> Task 'prepareKotlinBuildScriptModel' not found in project ':test_runner'.
> Task 'prepareKotlinBuildScriptModel' not found in project ':test_runner'.
> or similar

- Make sure you do not change any module specific settings for Gradle
- Clear IDE cache using `File > Invalidate Caches / Restart`
- Re-import project using root `build.gradle.kts`
- Sync project again

5) > Does Flank support Cucumber?

Please check [document](cucumber_support.md) for more info

6) > How can I find project id?

Please check the [firebase documentation](https://firebase.google.com/docs/projects/learn-more?hl=en#find_the_project_id) about finding the project id

7) > How do I run Flank with a proxy?

`java -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8080 -Dhttp.proxyUser=user -Dhttp.proxyPassword=pass -jar ./test_runner/build/libs/flank.jar firebase test android run`

See [google-auth-library-java](https://github.com/googleapis/google-auth-library-java#configuring-a-proxy) for details.


# Resources

Expand Down
12 changes: 5 additions & 7 deletions docs/refactor/flank_run/flank_run_refactor_sdd.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@

# References

* `https://github.com/Flank/flank/issues/1317`
* `https://github.com/Flank/flank/issues/1317`

# Motivation

Flank run is getting bigger over the time.
Currently, the amount of code related to test run is large,
Flank run is getting bigger over the time.
Currently, the amount of code related to test run is large,
so it could be hard to understand and keep in mind the whole process.
To make the work more convenient, faster and scalable, we should re-think
and reorganize the code between packages and files.

# Goals

* `All logic related to flank run grouped in one root package`
* `All logic related to flank run grouped in one root package`
* `Package structure flat as possible.`
* `Flank run process simplified to sequence of synchronous atomic steps.`
* `Packages and code are easy to identify as steps.`
Expand All @@ -32,7 +32,7 @@ and reorganize the code between packages and files.
# API

> What will the proposed API look like?

# Results

> What was the outcome of the project?
Expand Down Expand Up @@ -60,7 +60,6 @@ ftl/run/
│ ├── GetLastMatrices.kt
│ ├── PollMatrices.kt
│ ├── PrettyPrint.kt
│ ├── SaveSessionId.kt
│ └── UpdateMatrixFile.kt
├── exception
│ ├── ExceptionHandler.kt
Expand Down Expand Up @@ -142,7 +141,6 @@ ftl/run/
* `ftl/shard/Shard.kt`
* `ftl/shard/TestCasesCreator.kt`
* `ftl/util/Artifacts.kt`
* `ftl/util/CrashReporter.kt`
* `ftl/util/FileReference.kt`
* `ftl/util/FlankTestMethod.kt`
* `ftl/util/FlowExt.kt`
Expand Down
12 changes: 5 additions & 7 deletions docs/refactor/investigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,6 @@ command that:
* `com.google.cloud:google-cloud-nio`
* `com.google.cloud:google-cloud-storage`
* `com.google.apis:google-api-services-toolresults`
* `io.sentry:sentry`
* `com.mixpanel:mixpanel-java`

#### List of external API usages in files
Expand Down Expand Up @@ -446,7 +445,7 @@ where
* `NetworkProfilesListCommand`
* `IosOrientationsListCommand`
* `IPBlocksListCommand`

* `AndroidRunCommand`
* `AndroidArgs/validate`
* `AndroidArgs/assertDevicesSupported`
Expand All @@ -471,7 +470,6 @@ where
* `GcAndroidTestMatrix/build`
* `AbstractGoogleJsonClientRequest<T>/executeWithRetry`
* `IArgs/afterRunTests`
* `IArgs/uploadSessionId` -> `GcStorage/upload`
* `MatrixMap/printMatricesWebLinks` -> `getOrUpdateWebLink` -> `GcTestMatrix/refresh`
* `pollMatrices` -> `matrixChangesFlow` -> `GcTestMatrix/refresh`
* `Iterable<TestMatrix>/updateMatrixMap` -> `SavedMatrix/updateWithMatrix` -> `TestMatrix/fetchTestOutcomeContext`
Expand All @@ -494,7 +492,7 @@ where
* `GcToolResults/listTestCases`
* `GcToolResults/getStepResult`
* `FullJUnitReport.run` -> `GcStorage.uploadReportResult`
* `ReportManager/createAndUploadPerformanceMetricsForAndroid` -> `List<Pair<TestExecution, String>>.getAndUploadPerformanceMetrics` ->
* `ReportManager/createAndUploadPerformanceMetricsForAndroid` -> `List<Pair<TestExecution, String>>.getAndUploadPerformanceMetrics` ->
* `TestExecution.getPerformanceMetric` -> `GcToolResults.getPerformanceMetric`
* `PerfMetricsSummary.upload` -> `GcStorage.uploadPerformanceMetrics`
* `GcStorage/uploadMatricesId`
Expand All @@ -504,7 +502,7 @@ where
* `GcStorage.storage.list`
* `Blob.downloadTo`
* `MatrixMap/printMatricesWebLinks` -> `getOrUpdateWebLink` -> `GcTestMatrix/refresh`

* `IosRunCommand`
* `IosArgs/validate`
* `IosArgs/assertDevicesSupported`
Expand All @@ -517,12 +515,12 @@ where
* `IArgs/uploadOtherFiles` -> `GcStorage/upload`
* `IosArgs.uploadAdditionalIpas`
* `IosArgs.dumpShardsIfXcTest` -> `GcStorage/upload`
* `IosArgs/createIosTestContexts`
* `IosArgs/createIosTestContexts`
* `IosArgs.createXcTestContexts`
* `IArgs.uploadIfNeeded` -> `FileReference.uploadIfNeeded` -> `GcStorage.upload`
* `GcStorage.uploadXCTestFile`
* `IosArgs.createGameloopTestContexts`
* `IArgs.uploadIfNeeded` -> `FileReference.uploadIfNeeded` -> `GcStorage.upload`
* `IArgs.uploadIfNeeded` -> `FileReference.uploadIfNeeded` -> `GcStorage.upload`
* `GcIosTestMatrix/build`
* `AbstractGoogleJsonClientRequest<T>/executeWithRetry`
* `IArgs/afterRunTests` - the rest of steps are same as for android
7 changes: 0 additions & 7 deletions docs/refactor/uml/flank_run_relations.puml
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,6 @@ run_common_pollMatrices --> gc_GcTestMatrix
run_common_pollMatrices --> run_status_TestMatrixStatusPrinter
run_common_pollMatrices --> util_MatrixState

run_common_saveSessionId --> args_IArgs
run_common_saveSessionId --> util_sessionId
run_common_saveSessionId --> util_sessionId

run_common_updateMatrixFile --> args_IArgs
run_common_updateMatrixFile --> json_MatrixMap

Expand All @@ -96,7 +92,6 @@ run_common_getMatrixFilePath --> json_MatrixMap

run_exception_withGlobalExceptionHandling --> json_SavedMatrix
run_exception_withGlobalExceptionHandling --> run_cancelMatrices
run_exception_withGlobalExceptionHandling --> util_closeCrashReporter
run_exception_withGlobalExceptionHandling --> util_report

run_platform_android_createAndroidLoopConfig --> args_AndroidArgs
Expand Down Expand Up @@ -169,11 +164,9 @@ run_platform_android_uploadObbFiles --> gc_GcStorage
run_platform_common_afterRunTests --> args_IArgs
run_platform_common_afterRunTests --> config_FtlConstants
run_platform_common_afterRunTests --> config_FtlConstants_GCS_STORAGE_LINK
run_platform_common_afterRunTests --> gc_GcStorage_uploadSessionId
run_platform_common_afterRunTests --> gc_GcTestMatrix
run_platform_common_afterRunTests --> json_MatrixMap
run_platform_common_afterRunTests --> json_createSavedMatrix
run_platform_common_afterRunTests --> run_common_saveSessionId
run_platform_common_afterRunTests --> run_common_updateMatrixFile
run_platform_common_afterRunTests --> util_StopWatch
run_platform_common_afterRunTests --> util_isInvalid
Expand Down
1 change: 0 additions & 1 deletion flank_wrapper/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ publishing {
dependencies {
implementation(project(Modules.MIXPANEL_ANALYTICS))
implementation(project(Modules.COMMON))
implementation(Dependencies.SENTRY)
implementation(Dependencies.Fuel.CORE)
testImplementation(Dependencies.JUNIT)
testImplementation(Dependencies.TRUTH)
Expand Down
Loading