From 567eee32fe2e79ea274f661059bbd99b3c00a7ad Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 07:03:38 +0100 Subject: [PATCH 01/35] Use v8 steps --- features/auto_detect_errors.feature | 4 ++-- features/delivery.feature | 12 ++++++------ features/recrash_reports.feature | 2 +- features/release_stage_errors.feature | 6 +++--- features/telemetry.feature | 6 +++--- 5 files changed, 15 insertions(+), 15 deletions(-) diff --git a/features/auto_detect_errors.feature b/features/auto_detect_errors.feature index 4dac9c972..36ac2db53 100644 --- a/features/auto_detect_errors.feature +++ b/features/auto_detect_errors.feature @@ -15,7 +15,7 @@ Feature: autoDetectErrors flag controls whether errors are captured automaticall And I kill and relaunch the app When I run "AutoDetectFalseNSExceptionScenario" and relaunch the crashed app And I configure Bugsnag for "AutoDetectFalseHandledScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Signal not reported when autoDetectErrors is false When I run "AutoDetectFalseHandledScenario" @@ -27,4 +27,4 @@ Feature: autoDetectErrors flag controls whether errors are captured automaticall And I kill and relaunch the app When I run "AutoDetectFalseAbortScenario" and relaunch the crashed app And I configure Bugsnag for "AutoDetectFalseHandledScenario" - Then I should receive no requests + Then I should receive no errors diff --git a/features/delivery.feature b/features/delivery.feature index 4ce317685..6855ffa90 100644 --- a/features/delivery.feature +++ b/features/delivery.feature @@ -22,7 +22,7 @@ Feature: Delivery of errors And I kill and relaunch the app And I clear the error queue And I configure Bugsnag for "HandledExceptionScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Delivery is not retried for oversized handled payloads Given I set the HTTP status code to 500 @@ -33,7 +33,7 @@ Feature: Delivery of errors And I kill and relaunch the app And I clear the error queue And I configure Bugsnag for "OversizedHandledErrorScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Delivery is not retried for old handled payloads Given I set the HTTP status code to 500 @@ -50,7 +50,7 @@ Feature: Delivery of errors And I kill and relaunch the app And I clear the error queue And I configure Bugsnag for "OldHandledErrorScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Delivery is not retried for oversized crash payloads Given I set the HTTP status code to 500 @@ -62,7 +62,7 @@ Feature: Delivery of errors And I kill and relaunch the app And I clear the error queue And I configure Bugsnag for "OversizedCrashReportScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Delivery is not retried for old crash payloads Given I set the HTTP status code to 500 @@ -74,7 +74,7 @@ Feature: Delivery of errors And I kill and relaunch the app And I clear the error queue And I configure Bugsnag for "OldCrashReportScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Bugsnag.start() should block for 2 seconds after a launch crash When I run "SendLaunchCrashesSynchronouslyScenario" and relaunch the crashed app @@ -125,7 +125,7 @@ Feature: Delivery of errors And I wait to receive a session Then the session "user.id" equals "new" And I discard the oldest session - And I should receive no requests + And I should receive no sessions Scenario: The oldest sessions should be deleted to comply with maxPersistedSessions Given I set the HTTP status code to 500 diff --git a/features/recrash_reports.feature b/features/recrash_reports.feature index df28c2ac7..8b66b9964 100644 --- a/features/recrash_reports.feature +++ b/features/recrash_reports.feature @@ -11,7 +11,7 @@ Feature: Detection of crashes during crash handling Given I run "" and relaunch the crashed app And I configure Bugsnag for "" And I wait to receive an error - And the error "Bugsnag-Api-Key" header is null + And the error "Bugsnag-Api-Key" header is not present And the error "Bugsnag-Internal-Error" header equals "bugsnag-cocoa" And the error payload field "events.0.threads" is an array with 0 elements And the event "metaData.BugsnagDiagnostics.apiKey" equals "12312312312312312312312312312312" diff --git a/features/release_stage_errors.feature b/features/release_stage_errors.feature index 3320aa17f..3451904f8 100644 --- a/features/release_stage_errors.feature +++ b/features/release_stage_errors.feature @@ -6,7 +6,7 @@ Feature: Discarding reports based on release stage Scenario: Unhandled error ignored when release stage is not present in enabledReleaseStages When I run "UnhandledErrorInvalidReleaseStageScenario" and relaunch the crashed app And I configure Bugsnag for "UnhandledErrorInvalidReleaseStageScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Unhandled error captured when release stage is present in enabledReleaseStages When I run "UnhandledErrorValidReleaseStageScenario" and relaunch the crashed app @@ -26,7 +26,7 @@ Feature: Discarding reports based on release stage When I run "UnhandledErrorChangeInvalidReleaseStageScenario" and relaunch the crashed app And I configure Bugsnag for "UnhandledErrorChangeInvalidReleaseStageScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Crash when release stage is changed to be present in enabledReleaseStages before the event When I run "UnhandledErrorChangeValidReleaseStageScenario" and relaunch the crashed app @@ -39,7 +39,7 @@ Feature: Discarding reports based on release stage Scenario: Handled error when release stage is not present in enabledReleaseStages When I run "HandledErrorInvalidReleaseStageScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Handled error when release stage is present in enabledReleaseStages When I run "HandledErrorValidReleaseStageScenario" diff --git a/features/telemetry.feature b/features/telemetry.feature index 5c7c6f306..dfbaf2af7 100644 --- a/features/telemetry.feature +++ b/features/telemetry.feature @@ -7,7 +7,7 @@ Feature: Telemetry When I run "InvalidCrashReportScenario" and relaunch the crashed app And I configure Bugsnag for "InvalidCrashReportScenario" And I wait to receive an error - And the error "Bugsnag-Api-Key" header is null + And the error "Bugsnag-Api-Key" header is not present And the error "Bugsnag-Internal-Error" header equals "bugsnag-cocoa" And the error payload field "events.0.exceptions.0.stacktrace" is an array with 0 elements And the error payload field "events.0.threads" is an array with 0 elements @@ -25,7 +25,7 @@ Feature: Telemetry Scenario: An internal error report is sent if directories cannot be created When I run "CouldNotCreateDirectoryScenario" And I wait to receive an error - And the error "Bugsnag-Api-Key" header is null + And the error "Bugsnag-Api-Key" header is not present And the error "Bugsnag-Internal-Error" header equals "bugsnag-cocoa" And the error payload field "events.0.exceptions.0.stacktrace" is an array with 0 elements And the error payload field "events.0.threads" is an array with 0 elements @@ -41,7 +41,7 @@ Feature: Telemetry When I run "InvalidCrashReportScenario" and relaunch the crashed app And I set the app to "internalErrorsDisabled" mode And I configure Bugsnag for "InvalidCrashReportScenario" - Then I should receive no requests + Then I should receive no errors Scenario: Usage telemetry is not send if disabled When I run "TelemetryUsageDisabledScenario" From be6f0e8023ea1b69aca042f1b23e43ffe556b957 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 07:03:46 +0100 Subject: [PATCH 02/35] Remove unused steps --- features/steps/app_steps.rb | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/features/steps/app_steps.rb b/features/steps/app_steps.rb index 3d3ee6c50..da6c5a48a 100644 --- a/features/steps/app_steps.rb +++ b/features/steps/app_steps.rb @@ -65,18 +65,6 @@ # 3: The application is running in the background and is not suspended # 4: The application is running in the foreground -Then('the app is running in the foreground') do - wait_for_true do - Maze.driver.app_state('com.bugsnag.iOSTestApp') == :running_in_foreground - end -end - -Then('the app is running in the background') do - wait_for_true do - Maze.driver.app_state('com.bugsnag.iOSTestApp') == :running_in_background - end -end - Then('the app is not running') do wait_for_true do case Maze::Helper.get_current_platform From c1f65b2af814da6d4aeeae60f9496114c38730c6 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 07:16:11 +0100 Subject: [PATCH 03/35] Test iOS 10/11 --- .buildkite/pipeline.full.yml | 4 +- .buildkite/pipeline.yml | 80 +++++++++++++++++++++++++++++++++++- features/support/env.rb | 4 ++ 3 files changed, 84 insertions(+), 4 deletions(-) diff --git a/.buildkite/pipeline.full.yml b/.buildkite/pipeline.full.yml index 5e21ac793..a51b28104 100644 --- a/.buildkite/pipeline.full.yml +++ b/.buildkite/pipeline.full.yml @@ -266,7 +266,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.8.0" + - "--appium-version=1.16.0" - "--fail-fast" - "--exclude=features/[e-z].*.feature$" - "--order=random" @@ -297,7 +297,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.8.0" + - "--appium-version=1.16.0" - "--fail-fast" - "--exclude=features/[a-d].*.feature$" - "--order=random" diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 2c417a50e..f3baa115d 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -488,7 +488,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.8.0" + - "--appium-version=1.16.0" - "features/barebone_tests.feature" concurrency: 5 concurrency_group: browserstack-app @@ -515,7 +515,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_10" - - "--appium-version=1.8.0" + - "--appium-version=1.15.0" - "--fail-fast" - "features/barebone_tests.feature" concurrency: 5 @@ -525,6 +525,82 @@ steps: automatic: - exit_status: -1 # Agent was lost limit: 2 + manual: + permit_on_passed: true + + + + # Temporary additions + - label: ':browserstack: iOS 11 E2E tests batch 1' + depends_on: + - cocoa_fixture + # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and + # sessions need resetting frequently, taking a minute or more each time. + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.5.0: + download: "features/fixtures/ios/output/ipa_url.txt" + upload: "maze_output/failed/**/*" + docker-compose#v3.7.0: + pull: cocoa-maze-runner-legacy + run: cocoa-maze-runner-legacy + command: + - "--app=@build/ipa_url.txt" + - "--farm=bs" + - "--device=IOS_11_0_IPHONE_8_PLUS" + - "--appium-version=1.16.0" + - "--fail-fast" + - "--exclude=features/[e-z].*.feature$" + - "--order=random" + concurrency: 5 + concurrency_group: browserstack-app + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + manual: + permit_on_passed: true + + - label: ':browserstack: iOS 11 E2E tests batch 2' + depends_on: + - cocoa_fixture + # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and + # sessions need resetting frequently, taking a minute or more each time. + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.5.0: + download: "features/fixtures/ios/output/ipa_url.txt" + upload: "maze_output/failed/**/*" + docker-compose#v3.7.0: + pull: cocoa-maze-runner-legacy + run: cocoa-maze-runner-legacy + command: + - "--app=@build/ipa_url.txt" + - "--farm=bs" + - "--device=IOS_11_0_IPHONE_8_PLUS" + - "--appium-version=1.16.0" + - "--fail-fast" + - "--exclude=features/[a-d].*.feature$" + - "--order=random" + concurrency: 5 + concurrency_group: browserstack-app + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + manual: + permit_on_passed: true + + + + + ############################################################################## # diff --git a/features/support/env.rb b/features/support/env.rb index 8aa7ae11d..7a3426c20 100644 --- a/features/support/env.rb +++ b/features/support/env.rb @@ -32,6 +32,10 @@ disable_unexpectedly_quit_dialog end + + # Experiment + Maze.config.enable_retries + end # Disables the "macOSTestApp quit unexpectedly" dialog to prevent focus being stolen from the fixture. From ae5eafb597963fa5f1221d0392bf58c6dbb5759b Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 07:21:59 +0100 Subject: [PATCH 04/35] Maze v8 --- Gemfile | 2 +- Gemfile.lock | 28 +++++++++++++++++----------- docker-compose.yml | 6 +++--- 3 files changed, 21 insertions(+), 15 deletions(-) diff --git a/Gemfile b/Gemfile index 27731722f..427043d09 100644 --- a/Gemfile +++ b/Gemfile @@ -4,7 +4,7 @@ gem 'cocoapods' # A reference to Maze Runner is only needed for running tests locally and if committed it must be # portable for CI, e.g. a specific release. However, leaving it commented out would mean quicker CI. -gem 'bugsnag-maze-runner', '~> 7.0' +gem 'bugsnag-maze-runner', '~> 8.0' # Use a specific branch #gem 'bugsnag-maze-runner', git: 'https://github.com/bugsnag/maze-runner', branch: 'master' diff --git a/Gemfile.lock b/Gemfile.lock index d1c99bec8..a6a23dfc9 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -3,7 +3,7 @@ GEM specs: CFPropertyList (3.0.6) rexml - activesupport (7.0.4.3) + activesupport (7.0.5) concurrent-ruby (~> 1.0, >= 1.0.2) i18n (>= 1.6, < 2) minitest (>= 5.1) @@ -23,7 +23,7 @@ GEM atomos (0.1.3) bugsnag (6.25.2) concurrent-ruby (~> 1.0) - bugsnag-maze-runner (7.26.1) + bugsnag-maze-runner (8.0.2) appium_lib (~> 12.0.0) appium_lib_core (~> 5.4.0) bugsnag (~> 6.24) @@ -131,33 +131,34 @@ GEM gh_inspector (1.1.3) hana (1.3.7) httpclient (2.8.3) - i18n (1.12.0) + i18n (1.14.1) concurrent-ruby (~> 1.0) json (2.6.3) - json_schemer (0.2.24) + json_schemer (0.2.25) ecma-re-validator (~> 0.3) hana (~> 1.3) regexp_parser (~> 2.0) + simpleidn (~> 0.2) uri_template (~> 0.7) mime-types (3.4.1) mime-types-data (~> 3.2015) mime-types-data (3.2023.0218.1) - minitest (5.18.0) + minitest (5.18.1) molinillo (0.8.0) multi_test (0.1.2) nanaimo (0.3.0) nap (1.1.0) netrc (0.11.0) - nokogiri (1.14.3-x86_64-darwin) + nokogiri (1.15.2-x86_64-darwin) racc (~> 1.4) optimist (3.0.1) os (1.0.1) power_assert (2.0.3) public_suffix (4.0.7) - racc (1.6.2) - rack (2.2.6.4) + racc (1.7.1) + rack (2.2.7) rake (12.3.3) - regexp_parser (2.8.0) + regexp_parser (2.8.1) rexml (3.2.5) ruby-macho (2.5.1) rubyzip (2.3.2) @@ -166,15 +167,20 @@ GEM rexml (~> 3.2, >= 3.2.5) rubyzip (>= 1.2.2, < 3.0) websocket (~> 1.0) + simpleidn (0.2.1) + unf (~> 0.1.4) sys-uname (1.2.3) ffi (~> 1.1) - test-unit (3.5.7) + test-unit (3.5.9) power_assert tomlrb (2.0.3) typhoeus (1.4.0) ethon (>= 0.9.0) tzinfo (2.0.6) concurrent-ruby (~> 1.0) + unf (0.1.4) + unf_ext + unf_ext (0.0.8.2) uri_template (0.7.0) webrick (1.7.0) websocket (1.2.9) @@ -193,7 +199,7 @@ PLATFORMS x86_64-darwin-20 DEPENDENCIES - bugsnag-maze-runner (~> 7.0) + bugsnag-maze-runner (~> 8.0) cocoapods BUNDLED WITH diff --git a/docker-compose.yml b/docker-compose.yml index ea95ac572..539321250 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,7 +20,7 @@ x-common-environment: &common-environment services: cocoa-maze-runner: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v7-cli + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli environment: <<: *common-environment BROWSER_STACK_USERNAME: @@ -31,7 +31,7 @@ services: - ./maze_output:/app/maze_output cocoa-maze-runner-bitbar: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v7-cli + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli environment: <<: *common-environment BITBAR_USERNAME: @@ -45,7 +45,7 @@ services: - /var/run/docker.sock:/var/run/docker.sock cocoa-maze-runner-legacy: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v7-cli-legacy + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli-legacy environment: <<: *common-environment BROWSER_STACK_USERNAME: From a1a993d1cb76fd53721d6311645d1da5472eff03 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 07:27:38 +0100 Subject: [PATCH 05/35] Temp debug --- .buildkite/pipeline.yml | 370 ++++++++++++++++++++-------------------- 1 file changed, 185 insertions(+), 185 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index f3baa115d..dcbe0239e 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -282,191 +282,191 @@ steps: # # Basic build E2E tests # - - # - # BitBar - # - - label: ':bitbar: iOS 16 E2E tests batch 1' - depends_on: - - cocoa_fixture - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_16" - - "--no-tunnel" - - "--aws-public-ip" - - "--fail-fast" - - "--exclude=features/[e-z].*.feature$" - - "--order=random" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - - - label: ':bitbar: iOS 16 E2E tests batch 2' - depends_on: - - cocoa_fixture - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_16" - - "--no-tunnel" - - "--aws-public-ip" - - "--fail-fast" - - "--exclude=features/[a-d].*.feature$" - - "--order=random" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - - - label: ':bitbar: iOS 15 barebone tests' - depends_on: - - cocoa_fixture - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_15" - - "--no-tunnel" - - "--aws-public-ip" - - "--fail-fast" - - "features/barebone_tests.feature" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - - - label: ':bitbar: iOS 14 barebone tests' - depends_on: - - cocoa_fixture - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_14" - - "--no-tunnel" - - "--aws-public-ip" - - "--fail-fast" - - "features/barebone_tests.feature" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - - - label: ':bitbar: iOS 13 barebone tests' - depends_on: - - cocoa_fixture - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_13" - - "--no-tunnel" - - "--aws-public-ip" - - "--fail-fast" - - "features/barebone_tests.feature" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - - - label: ':bitbar: iOS 12 barebone tests' - depends_on: - - cocoa_fixture - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_12" - - "--no-tunnel" - - "--aws-public-ip" - - "--fail-fast" - - "features/barebone_tests.feature" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 +# +# # +# # BitBar +# # +# - label: ':bitbar: iOS 16 E2E tests batch 1' +# depends_on: +# - cocoa_fixture +# timeout_in_minutes: 60 +# agents: +# queue: opensource +# plugins: +# artifacts#v1.9.0: +# download: "features/fixtures/ios/output/iOSTestApp.ipa" +# upload: "maze_output/failed/**/*" +# docker-compose#v4.7.0: +# pull: cocoa-maze-runner-bitbar +# run: cocoa-maze-runner-bitbar +# service-ports: true +# command: +# - "--app=/app/build/iOSTestApp.ipa" +# - "--farm=bb" +# - "--device=IOS_16" +# - "--no-tunnel" +# - "--aws-public-ip" +# - "--fail-fast" +# - "--exclude=features/[e-z].*.feature$" +# - "--order=random" +# concurrency: 25 +# concurrency_group: 'bitbar-app' +# concurrency_method: eager +# retry: +# automatic: +# - exit_status: -1 # Agent was lost +# limit: 2 +# +# - label: ':bitbar: iOS 16 E2E tests batch 2' +# depends_on: +# - cocoa_fixture +# timeout_in_minutes: 60 +# agents: +# queue: opensource +# plugins: +# artifacts#v1.9.0: +# download: "features/fixtures/ios/output/iOSTestApp.ipa" +# upload: "maze_output/failed/**/*" +# docker-compose#v4.7.0: +# pull: cocoa-maze-runner-bitbar +# run: cocoa-maze-runner-bitbar +# service-ports: true +# command: +# - "--app=/app/build/iOSTestApp.ipa" +# - "--farm=bb" +# - "--device=IOS_16" +# - "--no-tunnel" +# - "--aws-public-ip" +# - "--fail-fast" +# - "--exclude=features/[a-d].*.feature$" +# - "--order=random" +# concurrency: 25 +# concurrency_group: 'bitbar-app' +# concurrency_method: eager +# retry: +# automatic: +# - exit_status: -1 # Agent was lost +# limit: 2 +# +# - label: ':bitbar: iOS 15 barebone tests' +# depends_on: +# - cocoa_fixture +# timeout_in_minutes: 60 +# agents: +# queue: opensource +# plugins: +# artifacts#v1.9.0: +# download: "features/fixtures/ios/output/iOSTestApp.ipa" +# upload: "maze_output/failed/**/*" +# docker-compose#v4.7.0: +# pull: cocoa-maze-runner-bitbar +# run: cocoa-maze-runner-bitbar +# service-ports: true +# command: +# - "--app=/app/build/iOSTestApp.ipa" +# - "--farm=bb" +# - "--device=IOS_15" +# - "--no-tunnel" +# - "--aws-public-ip" +# - "--fail-fast" +# - "features/barebone_tests.feature" +# concurrency: 25 +# concurrency_group: 'bitbar-app' +# concurrency_method: eager +# retry: +# automatic: +# - exit_status: -1 # Agent was lost +# limit: 2 +# +# - label: ':bitbar: iOS 14 barebone tests' +# depends_on: +# - cocoa_fixture +# timeout_in_minutes: 60 +# agents: +# queue: opensource +# plugins: +# artifacts#v1.9.0: +# download: "features/fixtures/ios/output/iOSTestApp.ipa" +# upload: "maze_output/failed/**/*" +# docker-compose#v4.7.0: +# pull: cocoa-maze-runner-bitbar +# run: cocoa-maze-runner-bitbar +# service-ports: true +# command: +# - "--app=/app/build/iOSTestApp.ipa" +# - "--farm=bb" +# - "--device=IOS_14" +# - "--no-tunnel" +# - "--aws-public-ip" +# - "--fail-fast" +# - "features/barebone_tests.feature" +# concurrency: 25 +# concurrency_group: 'bitbar-app' +# concurrency_method: eager +# retry: +# automatic: +# - exit_status: -1 # Agent was lost +# limit: 2 +# +# - label: ':bitbar: iOS 13 barebone tests' +# depends_on: +# - cocoa_fixture +# timeout_in_minutes: 60 +# agents: +# queue: opensource +# plugins: +# artifacts#v1.9.0: +# download: "features/fixtures/ios/output/iOSTestApp.ipa" +# upload: "maze_output/failed/**/*" +# docker-compose#v4.7.0: +# pull: cocoa-maze-runner-bitbar +# run: cocoa-maze-runner-bitbar +# service-ports: true +# command: +# - "--app=/app/build/iOSTestApp.ipa" +# - "--farm=bb" +# - "--device=IOS_13" +# - "--no-tunnel" +# - "--aws-public-ip" +# - "--fail-fast" +# - "features/barebone_tests.feature" +# concurrency: 25 +# concurrency_group: 'bitbar-app' +# concurrency_method: eager +# retry: +# automatic: +# - exit_status: -1 # Agent was lost +# limit: 2 +# +# - label: ':bitbar: iOS 12 barebone tests' +# depends_on: +# - cocoa_fixture +# timeout_in_minutes: 60 +# agents: +# queue: opensource +# plugins: +# artifacts#v1.9.0: +# download: "features/fixtures/ios/output/iOSTestApp.ipa" +# upload: "maze_output/failed/**/*" +# docker-compose#v4.7.0: +# pull: cocoa-maze-runner-bitbar +# run: cocoa-maze-runner-bitbar +# service-ports: true +# command: +# - "--app=/app/build/iOSTestApp.ipa" +# - "--farm=bb" +# - "--device=IOS_12" +# - "--no-tunnel" +# - "--aws-public-ip" +# - "--fail-fast" +# - "features/barebone_tests.feature" +# concurrency: 25 +# concurrency_group: 'bitbar-app' +# concurrency_method: eager +# retry: +# automatic: +# - exit_status: -1 # Agent was lost +# limit: 2 # # BrowserStack From d3fe5e163ce91393cf62aafc6de5451d5748d4ab Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 07:28:15 +0100 Subject: [PATCH 06/35] Use w3c image --- .buildkite/pipeline.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index dcbe0239e..20f277afa 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -482,8 +482,8 @@ steps: download: "features/fixtures/ios/output/ipa_url.txt" upload: "maze_output/failed/**/*" docker-compose#v3.7.0: - pull: cocoa-maze-runner-legacy - run: cocoa-maze-runner-legacy + pull: cocoa-maze-runner + run: cocoa-maze-runner command: - "--app=@build/ipa_url.txt" - "--farm=bs" @@ -509,8 +509,8 @@ steps: download: "features/fixtures/ios/output/ipa_url.txt" upload: "maze_output/failed/**/*" docker-compose#v3.7.0: - pull: cocoa-maze-runner-legacy - run: cocoa-maze-runner-legacy + pull: cocoa-maze-runner + run: cocoa-maze-runner command: - "--app=@build/ipa_url.txt" - "--farm=bs" @@ -544,8 +544,8 @@ steps: download: "features/fixtures/ios/output/ipa_url.txt" upload: "maze_output/failed/**/*" docker-compose#v3.7.0: - pull: cocoa-maze-runner-legacy - run: cocoa-maze-runner-legacy + pull: cocoa-maze-runner + run: cocoa-maze-runner command: - "--app=@build/ipa_url.txt" - "--farm=bs" @@ -577,8 +577,8 @@ steps: download: "features/fixtures/ios/output/ipa_url.txt" upload: "maze_output/failed/**/*" docker-compose#v3.7.0: - pull: cocoa-maze-runner-legacy - run: cocoa-maze-runner-legacy + pull: cocoa-maze-runner + run: cocoa-maze-runner command: - "--app=@build/ipa_url.txt" - "--farm=bs" From 237665cd3dcaf9ae872f0d54de502fe35108f70a Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 08:51:33 +0100 Subject: [PATCH 07/35] Try 1.15 --- .buildkite/pipeline.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 20f277afa..87aa2eed1 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -550,7 +550,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.16.0" + - "--appium-version=1.15.0" - "--fail-fast" - "--exclude=features/[e-z].*.feature$" - "--order=random" @@ -583,7 +583,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.16.0" + - "--appium-version=1.15.0" - "--fail-fast" - "--exclude=features/[a-d].*.feature$" - "--order=random" From 947f72f244852caf2cfe5ac20ccb9c8adb8f7b30 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 10:05:56 +0100 Subject: [PATCH 08/35] Try more phones --- .buildkite/pipeline.yml | 140 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 138 insertions(+), 2 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 87aa2eed1..9594ed1a3 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -550,7 +550,7 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.15.0" + - "--appium-version=1.16.0" - "--fail-fast" - "--exclude=features/[e-z].*.feature$" - "--order=random" @@ -583,7 +583,77 @@ steps: - "--app=@build/ipa_url.txt" - "--farm=bs" - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.15.0" + - "--appium-version=1.16.0" + - "--fail-fast" + - "--exclude=features/[a-d].*.feature$" + - "--order=random" + concurrency: 5 + concurrency_group: browserstack-app + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + manual: + permit_on_passed: true + + + + + # Temporary additions + - label: ':browserstack: iOS 11 E2E tests batch 1' + depends_on: + - cocoa_fixture + # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and + # sessions need resetting frequently, taking a minute or more each time. + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.5.0: + download: "features/fixtures/ios/output/ipa_url.txt" + upload: "maze_output/failed/**/*" + docker-compose#v3.7.0: + pull: cocoa-maze-runner + run: cocoa-maze-runner + command: + - "--app=@build/ipa_url.txt" + - "--farm=bs" + - "--device=IOS_11_0_IPHONE_X" + - "--appium-version=1.16.0" + - "--fail-fast" + - "--exclude=features/[e-z].*.feature$" + - "--order=random" + concurrency: 5 + concurrency_group: browserstack-app + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + manual: + permit_on_passed: true + + - label: ':browserstack: iOS 11 E2E tests batch 2' + depends_on: + - cocoa_fixture + # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and + # sessions need resetting frequently, taking a minute or more each time. + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.5.0: + download: "features/fixtures/ios/output/ipa_url.txt" + upload: "maze_output/failed/**/*" + docker-compose#v3.7.0: + pull: cocoa-maze-runner + run: cocoa-maze-runner + command: + - "--app=@build/ipa_url.txt" + - "--farm=bs" + - "--device=IOS_11_0_IPHONE_X" + - "--appium-version=1.16.0" - "--fail-fast" - "--exclude=features/[a-d].*.feature$" - "--order=random" @@ -598,6 +668,72 @@ steps: permit_on_passed: true + # Temporary additions + - label: ':browserstack: iOS 11 E2E tests batch 1' + depends_on: + - cocoa_fixture + # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and + # sessions need resetting frequently, taking a minute or more each time. + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.5.0: + download: "features/fixtures/ios/output/ipa_url.txt" + upload: "maze_output/failed/**/*" + docker-compose#v3.7.0: + pull: cocoa-maze-runner + run: cocoa-maze-runner + command: + - "--app=@build/ipa_url.txt" + - "--farm=bs" + - "--device=IOS_11_0_IPHONE_6S_PLUS" + - "--appium-version=1.16.0" + - "--fail-fast" + - "--exclude=features/[e-z].*.feature$" + - "--order=random" + concurrency: 5 + concurrency_group: browserstack-app + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + manual: + permit_on_passed: true + + - label: ':browserstack: iOS 11 E2E tests batch 2' + depends_on: + - cocoa_fixture + # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and + # sessions need resetting frequently, taking a minute or more each time. + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.5.0: + download: "features/fixtures/ios/output/ipa_url.txt" + upload: "maze_output/failed/**/*" + docker-compose#v3.7.0: + pull: cocoa-maze-runner + run: cocoa-maze-runner + command: + - "--app=@build/ipa_url.txt" + - "--farm=bs" + - "--device=IOS_11_0_IPHONE_6S_PLUS" + - "--appium-version=1.16.0" + - "--fail-fast" + - "--exclude=features/[a-d].*.feature$" + - "--order=random" + concurrency: 5 + concurrency_group: browserstack-app + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + manual: + permit_on_passed: true From eb38da0807892093494b35374bb9a2d9f1b687ab Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Sat, 17 Jun 2023 10:54:02 +0100 Subject: [PATCH 09/35] No not pull file --- features/support/env.rb | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/features/support/env.rb b/features/support/env.rb index 7a3426c20..d32874f53 100644 --- a/features/support/env.rb +++ b/features/support/env.rb @@ -136,12 +136,12 @@ def skip_between(os, version_lo, version_hi) Process.wait log FileUtils.mv '/tmp/kscrash.log', path end - when 'ios' - begin - data = Maze.driver.pull_file '@com.bugsnag.iOSTestApp/Documents/kscrash.log' - File.open(File.join(path, 'kscrash.log'), 'wb') { |file| file << data } - rescue StandardError - puts "Maze.driver.pull_file failed: #{$ERROR_INFO}" - end + # when 'ios' + # begin + # data = Maze.driver.pull_file '@com.bugsnag.iOSTestApp/Documents/kscrash.log' + # File.open(File.join(path, 'kscrash.log'), 'wb') { |file| file << data } + # rescue StandardError + # puts "Maze.driver.pull_file failed: #{$ERROR_INFO}" + # end end end From 3ee722d3b47f17d2523a42389052986ff6e09c2b Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Mon, 19 Jun 2023 14:14:46 +0100 Subject: [PATCH 10/35] Poll for MR commands --- .../ios/iOSTestApp/Base.lproj/Main.storyboard | 25 +++---------------- .../ios/iOSTestApp/ViewController.swift | 19 ++++++++------ features/fixtures/shared/scenarios/Scenario.m | 5 ++++ features/steps/app_steps.rb | 2 +- 4 files changed, 21 insertions(+), 30 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/Base.lproj/Main.storyboard b/features/fixtures/ios/iOSTestApp/Base.lproj/Main.storyboard index 5e37b3c22..07c690f53 100644 --- a/features/fixtures/ios/iOSTestApp/Base.lproj/Main.storyboard +++ b/features/fixtures/ios/iOSTestApp/Base.lproj/Main.storyboard @@ -1,9 +1,9 @@ - + - + @@ -17,7 +17,7 @@ - + @@ -112,7 +95,7 @@ - + diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index b2ed6df4e..46680b08c 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -25,6 +25,17 @@ class ViewController: UIViewController { self.view.addGestureRecognizer(UITapGestureRecognizer(target: self.view, action: #selector(UIView.endEditing(_:)))) NotificationCenter.default.addObserver(self, selector: #selector(didEnterBackgroundNotification), name: UIApplication.didEnterBackgroundNotification, object: nil) apiKeyField.text = UserDefaults.standard.string(forKey: "apiKey") + + // Poll for commands to run + Scenario.baseMazeAddress = loadMazeRunnerAddress() + if #available(iOS 10.0, *) { + Timer.scheduledTimer(withTimeInterval: 2, repeats: true) { timer in + Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in + self.scenarioNameField.text = scenarioName + self.scenarioMetaDataField.text = eventMode + } + } + } } @IBAction func runTestScenario() { @@ -103,14 +114,6 @@ class ViewController: UIViewController { Scenario.current!.eventMode = scenarioMetaDataField.text } - @IBAction func executeCommand(_ sender: Any) { - Scenario.baseMazeAddress = loadMazeRunnerAddress() - Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in - self.scenarioNameField.text = scenarioName - self.scenarioMetaDataField.text = eventMode - } - } - @objc func didEnterBackgroundNotification() { Scenario.current?.didEnterBackgroundNotification() } diff --git a/features/fixtures/shared/scenarios/Scenario.m b/features/fixtures/shared/scenarios/Scenario.m index 32af23ba5..2bffe0de0 100644 --- a/features/fixtures/shared/scenarios/Scenario.m +++ b/features/fixtures/shared/scenarios/Scenario.m @@ -231,6 +231,11 @@ + (void)executeMazeRunnerCommand:(void (^)(NSString *action, NSString *scenarioN NSDictionary *command = [NSJSONSerialization JSONObjectWithData:data options:0 error:&error]; NSString *action = [command objectForKey:@"action"]; + if ([action isEqualToString:@"noop"]) { + NSLog(@"No Maze Runner command queued at present"); + return; + } + NSParameterAssert([action isKindOfClass:[NSString class]]); NSString *scenarioName = [command objectForKey:@"scenario_name"]; diff --git a/features/steps/app_steps.rb b/features/steps/app_steps.rb index da6c5a48a..69f5e933c 100644 --- a/features/steps/app_steps.rb +++ b/features/steps/app_steps.rb @@ -102,7 +102,7 @@ def execute_command(action, scenario_name) def trigger_app_command case Maze::Helper.get_current_platform when 'ios' - Maze.driver.click_element :execute_command + # Do nothing when 'macos' run_macos_app when 'watchos' From 62472aec14af935f179e80f1a2cc3a547a7cd304 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Tue, 20 Jun 2023 15:14:55 +0100 Subject: [PATCH 11/35] Reduce polling interval --- features/fixtures/ios/iOSTestApp/ViewController.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 46680b08c..1b607756f 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -29,7 +29,7 @@ class ViewController: UIViewController { // Poll for commands to run Scenario.baseMazeAddress = loadMazeRunnerAddress() if #available(iOS 10.0, *) { - Timer.scheduledTimer(withTimeInterval: 2, repeats: true) { timer in + Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in self.scenarioNameField.text = scenarioName self.scenarioMetaDataField.text = eventMode From f8ffefeee7a3e3c9b7ab1afbb30fb177fdb4b5ca Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Tue, 20 Jun 2023 15:15:09 +0100 Subject: [PATCH 12/35] Use dev maze --- docker-compose.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 539321250..7cd74c794 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,7 +20,7 @@ x-common-environment: &common-environment services: cocoa-maze-runner: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner:tms-old-appium-cli environment: <<: *common-environment BROWSER_STACK_USERNAME: @@ -31,7 +31,7 @@ services: - ./maze_output:/app/maze_output cocoa-maze-runner-bitbar: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner:tms-old-appium-cli environment: <<: *common-environment BITBAR_USERNAME: @@ -45,7 +45,7 @@ services: - /var/run/docker.sock:/var/run/docker.sock cocoa-maze-runner-legacy: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli-legacy + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner:tms-old-appium-cli-legacy environment: <<: *common-environment BROWSER_STACK_USERNAME: From bca642c223860456540a4f8309e3d1d481d7c888 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Tue, 20 Jun 2023 15:18:25 +0100 Subject: [PATCH 13/35] Reinstate pull_file --- features/support/env.rb | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/features/support/env.rb b/features/support/env.rb index d32874f53..7a3426c20 100644 --- a/features/support/env.rb +++ b/features/support/env.rb @@ -136,12 +136,12 @@ def skip_between(os, version_lo, version_hi) Process.wait log FileUtils.mv '/tmp/kscrash.log', path end - # when 'ios' - # begin - # data = Maze.driver.pull_file '@com.bugsnag.iOSTestApp/Documents/kscrash.log' - # File.open(File.join(path, 'kscrash.log'), 'wb') { |file| file << data } - # rescue StandardError - # puts "Maze.driver.pull_file failed: #{$ERROR_INFO}" - # end + when 'ios' + begin + data = Maze.driver.pull_file '@com.bugsnag.iOSTestApp/Documents/kscrash.log' + File.open(File.join(path, 'kscrash.log'), 'wb') { |file| file << data } + rescue StandardError + puts "Maze.driver.pull_file failed: #{$ERROR_INFO}" + end end end From 5e20f7a6fec14af8fb0f7fecebd70f505b4be7cd Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Tue, 20 Jun 2023 19:53:53 +0100 Subject: [PATCH 14/35] Remove debug --- .buildkite/pipeline.yml | 211 ---------------------------------------- 1 file changed, 211 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 9594ed1a3..f9523309a 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -525,217 +525,6 @@ steps: automatic: - exit_status: -1 # Agent was lost limit: 2 - manual: - permit_on_passed: true - - - - # Temporary additions - - label: ':browserstack: iOS 11 E2E tests batch 1' - depends_on: - - cocoa_fixture - # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and - # sessions need resetting frequently, taking a minute or more each time. - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.5.0: - download: "features/fixtures/ios/output/ipa_url.txt" - upload: "maze_output/failed/**/*" - docker-compose#v3.7.0: - pull: cocoa-maze-runner - run: cocoa-maze-runner - command: - - "--app=@build/ipa_url.txt" - - "--farm=bs" - - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.16.0" - - "--fail-fast" - - "--exclude=features/[e-z].*.feature$" - - "--order=random" - concurrency: 5 - concurrency_group: browserstack-app - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - manual: - permit_on_passed: true - - - label: ':browserstack: iOS 11 E2E tests batch 2' - depends_on: - - cocoa_fixture - # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and - # sessions need resetting frequently, taking a minute or more each time. - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.5.0: - download: "features/fixtures/ios/output/ipa_url.txt" - upload: "maze_output/failed/**/*" - docker-compose#v3.7.0: - pull: cocoa-maze-runner - run: cocoa-maze-runner - command: - - "--app=@build/ipa_url.txt" - - "--farm=bs" - - "--device=IOS_11_0_IPHONE_8_PLUS" - - "--appium-version=1.16.0" - - "--fail-fast" - - "--exclude=features/[a-d].*.feature$" - - "--order=random" - concurrency: 5 - concurrency_group: browserstack-app - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - manual: - permit_on_passed: true - - - - - # Temporary additions - - label: ':browserstack: iOS 11 E2E tests batch 1' - depends_on: - - cocoa_fixture - # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and - # sessions need resetting frequently, taking a minute or more each time. - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.5.0: - download: "features/fixtures/ios/output/ipa_url.txt" - upload: "maze_output/failed/**/*" - docker-compose#v3.7.0: - pull: cocoa-maze-runner - run: cocoa-maze-runner - command: - - "--app=@build/ipa_url.txt" - - "--farm=bs" - - "--device=IOS_11_0_IPHONE_X" - - "--appium-version=1.16.0" - - "--fail-fast" - - "--exclude=features/[e-z].*.feature$" - - "--order=random" - concurrency: 5 - concurrency_group: browserstack-app - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - manual: - permit_on_passed: true - - - label: ':browserstack: iOS 11 E2E tests batch 2' - depends_on: - - cocoa_fixture - # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and - # sessions need resetting frequently, taking a minute or more each time. - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.5.0: - download: "features/fixtures/ios/output/ipa_url.txt" - upload: "maze_output/failed/**/*" - docker-compose#v3.7.0: - pull: cocoa-maze-runner - run: cocoa-maze-runner - command: - - "--app=@build/ipa_url.txt" - - "--farm=bs" - - "--device=IOS_11_0_IPHONE_X" - - "--appium-version=1.16.0" - - "--fail-fast" - - "--exclude=features/[a-d].*.feature$" - - "--order=random" - concurrency: 5 - concurrency_group: browserstack-app - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - manual: - permit_on_passed: true - - - # Temporary additions - - label: ':browserstack: iOS 11 E2E tests batch 1' - depends_on: - - cocoa_fixture - # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and - # sessions need resetting frequently, taking a minute or more each time. - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.5.0: - download: "features/fixtures/ios/output/ipa_url.txt" - upload: "maze_output/failed/**/*" - docker-compose#v3.7.0: - pull: cocoa-maze-runner - run: cocoa-maze-runner - command: - - "--app=@build/ipa_url.txt" - - "--farm=bs" - - "--device=IOS_11_0_IPHONE_6S_PLUS" - - "--appium-version=1.16.0" - - "--fail-fast" - - "--exclude=features/[e-z].*.feature$" - - "--order=random" - concurrency: 5 - concurrency_group: browserstack-app - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - manual: - permit_on_passed: true - - - label: ':browserstack: iOS 11 E2E tests batch 2' - depends_on: - - cocoa_fixture - # More time than other steps as the BrowserStack iOS 11 devices seem particularly unstable and - # sessions need resetting frequently, taking a minute or more each time. - timeout_in_minutes: 60 - agents: - queue: opensource - plugins: - artifacts#v1.5.0: - download: "features/fixtures/ios/output/ipa_url.txt" - upload: "maze_output/failed/**/*" - docker-compose#v3.7.0: - pull: cocoa-maze-runner - run: cocoa-maze-runner - command: - - "--app=@build/ipa_url.txt" - - "--farm=bs" - - "--device=IOS_11_0_IPHONE_6S_PLUS" - - "--appium-version=1.16.0" - - "--fail-fast" - - "--exclude=features/[a-d].*.feature$" - - "--order=random" - concurrency: 5 - concurrency_group: browserstack-app - concurrency_method: eager - retry: - automatic: - - exit_status: -1 # Agent was lost - limit: 2 - manual: - permit_on_passed: true - - ############################################################################## From e5c98a3de9d8a7d3acf94dba5d459378f4153117 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 21 Jun 2023 16:21:43 +0100 Subject: [PATCH 15/35] Use the latest release --- docker-compose.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 7cd74c794..539321250 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,7 +20,7 @@ x-common-environment: &common-environment services: cocoa-maze-runner: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner:tms-old-appium-cli + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli environment: <<: *common-environment BROWSER_STACK_USERNAME: @@ -31,7 +31,7 @@ services: - ./maze_output:/app/maze_output cocoa-maze-runner-bitbar: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner:tms-old-appium-cli + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli environment: <<: *common-environment BITBAR_USERNAME: @@ -45,7 +45,7 @@ services: - /var/run/docker.sock:/var/run/docker.sock cocoa-maze-runner-legacy: - image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner:tms-old-appium-cli-legacy + image: 855461928731.dkr.ecr.us-west-1.amazonaws.com/maze-runner-releases:latest-v8-cli-legacy environment: <<: *common-environment BROWSER_STACK_USERNAME: From a65a828418fc7e29fe19166897398276e305bb8d Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 00:02:30 +0100 Subject: [PATCH 16/35] Reduce timeout --- features/fixtures/ios/iOSTestApp/ViewController.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 1b607756f..4cf7cb87d 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -71,7 +71,7 @@ class ViewController: UIViewController { return bsAddress; } - for _ in 1...60 { + for _ in 1...15 { let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] log("Reading Maze Runner address from fixture_config.json") From a47fe6c020f30588cd21a3c6615be8cb6f9367fd Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 07:52:23 +0100 Subject: [PATCH 17/35] Extra logging --- features/fixtures/ios/iOSTestApp/ViewController.swift | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 4cf7cb87d..7eae0d10e 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -31,8 +31,10 @@ class ViewController: UIViewController { if #available(iOS 10.0, *) { Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in + log("Setting field values") self.scenarioNameField.text = scenarioName self.scenarioMetaDataField.text = eventMode + log("Done setting field values") } } } @@ -71,7 +73,7 @@ class ViewController: UIViewController { return bsAddress; } - for _ in 1...15 { + for n in 1...20 { let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] log("Reading Maze Runner address from fixture_config.json") @@ -80,6 +82,7 @@ class ViewController: UIViewController { relativeTo: documentsUrl).appendingPathExtension("json") let savedData = try Data(contentsOf: fileUrl) if let contents = String(data: savedData, encoding: .utf8) { + NSLog("Found file after %@ seconds", n) let decoder = JSONDecoder() let jsonData = contents.data(using: .utf8) let config = try decoder.decode(FixtureConfig.self, from: jsonData!) From 763a2a124a6ab0a99d7f9aceecc658c639459e6b Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 07:52:30 +0100 Subject: [PATCH 18/35] Temp --- .buildkite/pipeline.yml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index f9523309a..6d1449613 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -526,6 +526,29 @@ steps: - exit_status: -1 # Agent was lost limit: 2 + - label: ':bitbar: Temp test' + depends_on: "cocoa_fixture" + timeout_in_minutes: 20 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_15" + - "--no-tunnel" + - "--aws-public-ip" + - "features/app_and_device_attributes.feature:6" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager ############################################################################## # From c3a962b461502de20d55bdcdd9944ab002297de9 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 09:29:22 +0100 Subject: [PATCH 19/35] debug --- .buildkite/pipeline.yml | 24 +++++++++++++++++++ .../ios/iOSTestApp/ViewController.swift | 14 ++++++++--- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 6d1449613..1e90f6983 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -550,6 +550,30 @@ steps: concurrency_group: 'bitbar-app' concurrency_method: eager + - label: ':bitbar: Temp test' + depends_on: "cocoa_fixture" + timeout_in_minutes: 20 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_15" + - "--no-tunnel" + - "--aws-public-ip" + - "features/app_and_device_attributes.feature:6" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + ############################################################################## # # Trigger more tests diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 7eae0d10e..a961b7f5a 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -18,13 +18,21 @@ class ViewController: UIViewController { @IBOutlet var scenarioNameField : UITextField! @IBOutlet var scenarioMetaDataField : UITextField! @IBOutlet var apiKeyField: UITextField! - var mazeRunnerAddress: String = ""; + var mazeRunnerAddress: String = "" + var timerStarted: Bool = false override func viewDidLoad() { super.viewDidLoad() self.view.addGestureRecognizer(UITapGestureRecognizer(target: self.view, action: #selector(UIView.endEditing(_:)))) NotificationCenter.default.addObserver(self, selector: #selector(didEnterBackgroundNotification), name: UIApplication.didEnterBackgroundNotification, object: nil) apiKeyField.text = UserDefaults.standard.string(forKey: "apiKey") + } + + override func viewDidAppear(_ animated: Bool) { + super.viewDidAppear(animated) + + if timerStarted { return } + timerStarted = true // Poll for commands to run Scenario.baseMazeAddress = loadMazeRunnerAddress() @@ -73,7 +81,7 @@ class ViewController: UIViewController { return bsAddress; } - for n in 1...20 { + for n in 1...60 { let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] log("Reading Maze Runner address from fixture_config.json") @@ -82,7 +90,7 @@ class ViewController: UIViewController { relativeTo: documentsUrl).appendingPathExtension("json") let savedData = try Data(contentsOf: fileUrl) if let contents = String(data: savedData, encoding: .utf8) { - NSLog("Found file after %@ seconds", n) + NSLog("Found fixture_config.json after %@ seconds", n) let decoder = JSONDecoder() let jsonData = contents.data(using: .utf8) let config = try decoder.decode(FixtureConfig.self, from: jsonData!) From 9737f7f0a1a5272fdedf369f8d09f826f0e51e1b Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 17:05:50 +0100 Subject: [PATCH 20/35] Refactor --- features/fixtures/ios/iOSTestApp/ViewController.swift | 8 +++++--- features/fixtures/shared/scenarios/Scenario.m | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index a961b7f5a..954421598 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -35,9 +35,11 @@ class ViewController: UIViewController { timerStarted = true // Poll for commands to run - Scenario.baseMazeAddress = loadMazeRunnerAddress() if #available(iOS 10.0, *) { - Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in + if Scenario.baseMazeAddress.isEmpty { + Scenario.baseMazeAddress = loadMazeRunnerAddress() + } + Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in log("Setting field values") self.scenarioNameField.text = scenarioName @@ -81,7 +83,7 @@ class ViewController: UIViewController { return bsAddress; } - for n in 1...60 { + for n in 1...30 { let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] log("Reading Maze Runner address from fixture_config.json") diff --git a/features/fixtures/shared/scenarios/Scenario.m b/features/fixtures/shared/scenarios/Scenario.m index 2bffe0de0..9bfc27101 100644 --- a/features/fixtures/shared/scenarios/Scenario.m +++ b/features/fixtures/shared/scenarios/Scenario.m @@ -32,7 +32,7 @@ void markErrorHandledCallback(const BSG_KSCrashReportWriter *writer) { // MARK: - static Scenario *theScenario; -static NSString *theBaseMazeAddress; +static NSString *theBaseMazeAddress = @""; #if !TARGET_OS_WATCH static char ksLogPath[PATH_MAX]; From 617577910191e80ecbd8cd3e13b39be371ceca88 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 18:11:50 +0100 Subject: [PATCH 21/35] Fixes --- .buildkite/pipeline.yml | 24 ------------------- .../ios/iOSTestApp/ViewController.swift | 8 +++---- 2 files changed, 4 insertions(+), 28 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 1e90f6983..6d1449613 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -550,30 +550,6 @@ steps: concurrency_group: 'bitbar-app' concurrency_method: eager - - label: ':bitbar: Temp test' - depends_on: "cocoa_fixture" - timeout_in_minutes: 20 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_15" - - "--no-tunnel" - - "--aws-public-ip" - - "features/app_and_device_attributes.feature:6" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - ############################################################################## # # Trigger more tests diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 954421598..5620f94e6 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -36,10 +36,10 @@ class ViewController: UIViewController { // Poll for commands to run if #available(iOS 10.0, *) { - if Scenario.baseMazeAddress.isEmpty { - Scenario.baseMazeAddress = loadMazeRunnerAddress() - } - Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in + Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in + if Scenario.baseMazeAddress.isEmpty { + Scenario.baseMazeAddress = self.loadMazeRunnerAddress() + } Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in log("Setting field values") self.scenarioNameField.text = scenarioName From a40135a122ada41744988d0c5fc6277ab0964c63 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Wed, 28 Jun 2023 18:48:24 +0100 Subject: [PATCH 22/35] Run timer in the background --- .../ios/iOSTestApp/ViewController.swift | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 5620f94e6..c23a877a5 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -36,15 +36,17 @@ class ViewController: UIViewController { // Poll for commands to run if #available(iOS 10.0, *) { - Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in - if Scenario.baseMazeAddress.isEmpty { - Scenario.baseMazeAddress = self.loadMazeRunnerAddress() - } - Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in - log("Setting field values") - self.scenarioNameField.text = scenarioName - self.scenarioMetaDataField.text = eventMode - log("Done setting field values") + DispatchQueue.global(qos: .background).async { + Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in + if Scenario.baseMazeAddress.isEmpty { + Scenario.baseMazeAddress = self.loadMazeRunnerAddress() + } + Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in + log("Setting field values") + self.scenarioNameField.text = scenarioName + self.scenarioMetaDataField.text = eventMode + log("Done setting field values") + } } } } From 5ea301eac825a5e7b9f084b071a0a793758dd948 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Thu, 29 Jun 2023 07:16:46 +0100 Subject: [PATCH 23/35] Add debug --- .ruby-version | 1 + .../ios/iOSTestApp/ViewController.swift | 27 ++++++++++++++----- 2 files changed, 21 insertions(+), 7 deletions(-) create mode 100644 .ruby-version diff --git a/.ruby-version b/.ruby-version new file mode 100644 index 000000000..2c9b4ef42 --- /dev/null +++ b/.ruby-version @@ -0,0 +1 @@ +2.7.3 diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index c23a877a5..482950a64 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -36,17 +36,18 @@ class ViewController: UIViewController { // Poll for commands to run if #available(iOS 10.0, *) { - DispatchQueue.global(qos: .background).async { - Timer.scheduledTimer(withTimeInterval: 1, repeats: true) { timer in - if Scenario.baseMazeAddress.isEmpty { - Scenario.baseMazeAddress = self.loadMazeRunnerAddress() - } + + DispatchQueue.global().async { + Scenario.baseMazeAddress = self.loadMazeRunnerAddress() + + while true { Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in log("Setting field values") self.scenarioNameField.text = scenarioName self.scenarioMetaDataField.text = eventMode log("Done setting field values") } + sleep(1) } } } @@ -78,21 +79,31 @@ class ViewController: UIViewController { func loadMazeRunnerAddress() -> String { - let bsAddress = "http://bs-local.com:9339" - + // TODO Debug - default to nonsense for now so it also fails on BS + //let bsAddress = "http://bs-local.com:9339" + let bsAddress = "http://sdsdfcsdcfewcw:1234" + // Only iOS 12 and above will run on BitBar for now if #available(iOS 12.0, *) {} else { return bsAddress; } for n in 1...30 { + log("SKW0") let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] log("Reading Maze Runner address from fixture_config.json") do { + + log("SKW1") + let fileUrl = URL(fileURLWithPath: "fixture_config", relativeTo: documentsUrl).appendingPathExtension("json") + + log("SKW2") let savedData = try Data(contentsOf: fileUrl) + + log("SKW3") if let contents = String(data: savedData, encoding: .utf8) { NSLog("Found fixture_config.json after %@ seconds", n) let decoder = JSONDecoder() @@ -102,6 +113,8 @@ class ViewController: UIViewController { log("Using Maze Runner address: " + address) return address } + + log("SKW4") } catch let error as NSError { log("Failed to read fixture_config.json: \(error)") From 853a1ef4dd7e14a977d56d4c44860077523c676f Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Thu, 29 Jun 2023 07:58:52 +0100 Subject: [PATCH 24/35] Refactor --- features/fixtures/ios/iOSTestApp/ViewController.swift | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 482950a64..c6d3ecaa6 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -26,19 +26,14 @@ class ViewController: UIViewController { self.view.addGestureRecognizer(UITapGestureRecognizer(target: self.view, action: #selector(UIView.endEditing(_:)))) NotificationCenter.default.addObserver(self, selector: #selector(didEnterBackgroundNotification), name: UIApplication.didEnterBackgroundNotification, object: nil) apiKeyField.text = UserDefaults.standard.string(forKey: "apiKey") - } - - override func viewDidAppear(_ animated: Bool) { - super.viewDidAppear(animated) - - if timerStarted { return } - timerStarted = true // Poll for commands to run if #available(iOS 10.0, *) { DispatchQueue.global().async { - Scenario.baseMazeAddress = self.loadMazeRunnerAddress() + if Scenario.baseMazeAddress.isEmpty { + Scenario.baseMazeAddress = self.loadMazeRunnerAddress() + } while true { Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in From 9527e7ad4d735ff6e61ae18c20241f7ac132b173 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Thu, 29 Jun 2023 19:26:44 +0100 Subject: [PATCH 25/35] Refactor into separate Thread class --- .../ios/iOSTestApp.xcodeproj/project.pbxproj | 4 + .../ios/iOSTestApp/CommandReaderThread.swift | 86 +++++++++++++++++++ .../ios/iOSTestApp/ViewController.swift | 77 +++-------------- .../macos/macOSTestApp/MainWindowController.m | 2 +- features/fixtures/shared/scenarios/Scenario.h | 2 +- features/fixtures/shared/scenarios/Scenario.m | 5 +- 6 files changed, 107 insertions(+), 69 deletions(-) create mode 100644 features/fixtures/ios/iOSTestApp/CommandReaderThread.swift diff --git a/features/fixtures/ios/iOSTestApp.xcodeproj/project.pbxproj b/features/fixtures/ios/iOSTestApp.xcodeproj/project.pbxproj index 118ada868..41cc658ab 100644 --- a/features/fixtures/ios/iOSTestApp.xcodeproj/project.pbxproj +++ b/features/fixtures/ios/iOSTestApp.xcodeproj/project.pbxproj @@ -123,6 +123,7 @@ AA4C7F1529AEA0C4009B09A9 /* BugsnagWrapper.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA4C7F1429AEA0C4009B09A9 /* BugsnagWrapper.swift */; }; AA6ACD1C2773E0B3006464C4 /* UserFromConfigScenario.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA6ACD1B2773E0B3006464C4 /* UserFromConfigScenario.swift */; }; AA6ACD202773E3F0006464C4 /* UserInfoScenario.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA6ACD1F2773E3F0006464C4 /* UserInfoScenario.swift */; }; + AA8BAEE32A4DD90E00A8BEA7 /* CommandReaderThread.swift in Sources */ = {isa = PBXBuildFile; fileRef = AA8BAEE22A4DD90E00A8BEA7 /* CommandReaderThread.swift */; }; AAFEF9EA26EB533800980A10 /* NetworkBreadcrumbsScenario.swift in Sources */ = {isa = PBXBuildFile; fileRef = AAFEF9E926EB533800980A10 /* NetworkBreadcrumbsScenario.swift */; }; CB0AE1F1287D89C90079B28E /* OnSendErrorCallbackFeatureFlagsScenario.swift in Sources */ = {isa = PBXBuildFile; fileRef = CB0AE1F0287D89C90079B28E /* OnSendErrorCallbackFeatureFlagsScenario.swift */; }; CB42FF9026F1EDB500E8D5D2 /* libBugsnagNetworkRequestPluginStatic.a in Frameworks */ = {isa = PBXBuildFile; fileRef = CB42FF8F26F1EDB500E8D5D2 /* libBugsnagNetworkRequestPluginStatic.a */; }; @@ -320,6 +321,7 @@ AA4C7F1429AEA0C4009B09A9 /* BugsnagWrapper.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = BugsnagWrapper.swift; sourceTree = ""; }; AA6ACD1B2773E0B3006464C4 /* UserFromConfigScenario.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = UserFromConfigScenario.swift; sourceTree = ""; }; AA6ACD1F2773E3F0006464C4 /* UserInfoScenario.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = UserInfoScenario.swift; sourceTree = ""; }; + AA8BAEE22A4DD90E00A8BEA7 /* CommandReaderThread.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CommandReaderThread.swift; sourceTree = ""; }; AAFEF9E926EB533800980A10 /* NetworkBreadcrumbsScenario.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NetworkBreadcrumbsScenario.swift; sourceTree = ""; }; AAFEFA0226EB6B5A00980A10 /* BugsnagNetworkRequestPlugin.framework */ = {isa = PBXFileReference; explicitFileType = wrapper.framework; path = BugsnagNetworkRequestPlugin.framework; sourceTree = BUILT_PRODUCTS_DIR; }; CB0AE1F0287D89C90079B28E /* OnSendErrorCallbackFeatureFlagsScenario.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = OnSendErrorCallbackFeatureFlagsScenario.swift; sourceTree = ""; }; @@ -443,6 +445,7 @@ 8AB8866C20404DD30003E444 /* LaunchScreen.storyboard */, 8AB8866520404DD30003E444 /* ViewController.swift */, 8AB8866720404DD30003E444 /* Main.storyboard */, + AA8BAEE22A4DD90E00A8BEA7 /* CommandReaderThread.swift */, ); path = iOSTestApp; sourceTree = ""; @@ -778,6 +781,7 @@ E75040A2247801A8005D33BD /* AutoDetectFalseNSExceptionScenario.swift in Sources */, E700EE7E247D7A61008CFFB6 /* SIGSYSScenario.m in Sources */, 010BAB272833D0160003FF36 /* AppHangDefaultConfigScenario.swift in Sources */, + AA8BAEE32A4DD90E00A8BEA7 /* CommandReaderThread.swift in Sources */, 010BAB212833CFE60003FF36 /* AppHangFatalDisabledScenario.swift in Sources */, E75040A42478052D005D33BD /* AutoDetectFalseAbortScenario.swift in Sources */, 8A14F0F62282D4AE00337B05 /* (null) in Sources */, diff --git a/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift b/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift new file mode 100644 index 000000000..96d9ff9f9 --- /dev/null +++ b/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift @@ -0,0 +1,86 @@ +// +// CommandReaderThread.swift +// iOSTestApp +// +// Created by Steve Kirkland-Walton on 29/06/2023. +// Copyright © 2023 Bugsnag. All rights reserved. +// + +import UIKit +import os + +class CommandReaderThread: Thread { + + var action: (String, String) -> Void + init(action: @escaping (String, String) -> Void) { + self.action = action + } + + override func main() { + if Scenario.baseMazeAddress.isEmpty { + Scenario.baseMazeAddress = self.loadMazeRunnerAddress() + } + + while true { + Scenario.executeMazeRunnerCommand() { scenarioName, eventMode in + self.action(scenarioName, eventMode) + } + Thread.sleep(forTimeInterval: 1) + } + } + + func loadMazeRunnerAddress() -> String { + + // TODO Debug - default to nonsense for now so it also fails on BS + //let bsAddress = "http://bs-local.com:9339" + let bsAddress = "http://sdsdfcsdcfewcw:1234" + + // Only iOS 12 and above will run on BitBar for now + if #available(iOS 12.0, *) {} else { + return bsAddress; + } + + for n in 1...30 { + log("SKW0") + let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] + + log("Reading Maze Runner address from fixture_config.json") + do { + + log("SKW1") + + let fileUrl = URL(fileURLWithPath: "fixture_config", + relativeTo: documentsUrl).appendingPathExtension("json") + + log("SKW2") + let savedData = try Data(contentsOf: fileUrl) + + log("SKW3") + if let contents = String(data: savedData, encoding: .utf8) { + NSLog("Found fixture_config.json after %d seconds", n) + let decoder = JSONDecoder() + let jsonData = contents.data(using: .utf8) + let config = try decoder.decode(FixtureConfig.self, from: jsonData!) + let address = "http://" + config.maze_address + log("Using Maze Runner address: " + address) + return address + } + + log("SKW4") + } + catch let error as NSError { + log("Failed to read fixture_config.json: \(error)") + } + log("Waiting for fixture_config.json to appear") + Thread.sleep(forTimeInterval: 1) + } + + log("Unable to read from fixture_config.json, defaulting to BrowserStack environment") + return bsAddress; + } +} + +private func log(_ message: String) { + NSLog("%@", message) + kslog("\(Date()) \(message)") +} diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index c6d3ecaa6..020acbbdf 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -11,6 +11,11 @@ import os class FixtureConfig: Codable { var maze_address: String + + + func startBugsnag() { + sleep(1) + } } class ViewController: UIViewController { @@ -27,24 +32,16 @@ class ViewController: UIViewController { NotificationCenter.default.addObserver(self, selector: #selector(didEnterBackgroundNotification), name: UIApplication.didEnterBackgroundNotification, object: nil) apiKeyField.text = UserDefaults.standard.string(forKey: "apiKey") + // Poll for commands to run if #available(iOS 10.0, *) { - - DispatchQueue.global().async { - if Scenario.baseMazeAddress.isEmpty { - Scenario.baseMazeAddress = self.loadMazeRunnerAddress() - } - - while true { - Scenario.executeMazeRunnerCommand { _, scenarioName, eventMode in - log("Setting field values") - self.scenarioNameField.text = scenarioName - self.scenarioMetaDataField.text = eventMode - log("Done setting field values") - } - sleep(1) - } + let uiUpdater = { (scenarioName: String, eventMode: String) in + self.scenarioNameField.text = scenarioName + self.scenarioMetaDataField.text = eventMode } + + let thread = CommandReaderThread(action: uiUpdater) + thread.start() } } @@ -72,56 +69,6 @@ class ViewController: UIViewController { Scenario.clearPersistentData() } - func loadMazeRunnerAddress() -> String { - - // TODO Debug - default to nonsense for now so it also fails on BS - //let bsAddress = "http://bs-local.com:9339" - let bsAddress = "http://sdsdfcsdcfewcw:1234" - - // Only iOS 12 and above will run on BitBar for now - if #available(iOS 12.0, *) {} else { - return bsAddress; - } - - for n in 1...30 { - log("SKW0") - let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] - - log("Reading Maze Runner address from fixture_config.json") - do { - - log("SKW1") - - let fileUrl = URL(fileURLWithPath: "fixture_config", - relativeTo: documentsUrl).appendingPathExtension("json") - - log("SKW2") - let savedData = try Data(contentsOf: fileUrl) - - log("SKW3") - if let contents = String(data: savedData, encoding: .utf8) { - NSLog("Found fixture_config.json after %@ seconds", n) - let decoder = JSONDecoder() - let jsonData = contents.data(using: .utf8) - let config = try decoder.decode(FixtureConfig.self, from: jsonData!) - let address = "http://" + config.maze_address - log("Using Maze Runner address: " + address) - return address - } - - log("SKW4") - } - catch let error as NSError { - log("Failed to read fixture_config.json: \(error)") - } - log("Waiting for fixture_config.json to appear") - sleep(1) - } - - log("Unable to read from fixture_config.json, defaulting to BrowserStack environment") - return bsAddress; - } - internal func prepareScenario() { var config: BugsnagConfiguration? if (apiKeyField.text!.count > 0) { diff --git a/features/fixtures/macos/macOSTestApp/MainWindowController.m b/features/fixtures/macos/macOSTestApp/MainWindowController.m index 5b88b35e8..479d24ee4 100644 --- a/features/fixtures/macos/macOSTestApp/MainWindowController.m +++ b/features/fixtures/macos/macOSTestApp/MainWindowController.m @@ -93,7 +93,7 @@ - (IBAction)useDashboardEndpoints:(id)sender { - (IBAction)executeMazeRunnerCommand:(id)sender { Scenario.baseMazeAddress = @"http://localhost:9339"; - [Scenario executeMazeRunnerCommand:^(NSString *action, NSString *scenarioName, NSString *eventMode){ + [Scenario executeMazeRunnerCommand:^(NSString *scenarioName, NSString *eventMode){ self.scenarioName = scenarioName; self.scenarioMetadata = eventMode; }]; diff --git a/features/fixtures/shared/scenarios/Scenario.h b/features/fixtures/shared/scenarios/Scenario.h index 387df60ef..a419f8247 100644 --- a/features/fixtures/shared/scenarios/Scenario.h +++ b/features/fixtures/shared/scenarios/Scenario.h @@ -42,7 +42,7 @@ void markErrorHandledCallback(const BSG_KSCrashReportWriter *writer); + (void)clearPersistentData; -+ (void)executeMazeRunnerCommand:(nullable void (^)(NSString *action, NSString *scenarioName, NSString *scenarioMode))preHandler; ++ (void)executeMazeRunnerCommand:(nullable void (^)(NSString *scenarioName, NSString *scenarioMode))preHandler; @end diff --git a/features/fixtures/shared/scenarios/Scenario.m b/features/fixtures/shared/scenarios/Scenario.m index 9bfc27101..b80a1570a 100644 --- a/features/fixtures/shared/scenarios/Scenario.m +++ b/features/fixtures/shared/scenarios/Scenario.m @@ -217,7 +217,7 @@ + (void)clearPersistentData { #endif } -+ (void)executeMazeRunnerCommand:(void (^)(NSString *action, NSString *scenarioName, NSString *scenarioMode))preHandler { ++ (void)executeMazeRunnerCommand:(void (^)(NSString *scenarioName, NSString *scenarioMode))preHandler { NSLog(@"%s", __PRETTY_FUNCTION__); NSURLSession *session = [NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration ephemeralSessionConfiguration]]; NSString *commandEndpoint = [NSString stringWithFormat:@"%@/command", theBaseMazeAddress]; @@ -250,9 +250,10 @@ + (void)executeMazeRunnerCommand:(void (^)(NSString *action, NSString *scenarioN [self clearPersistentData]; } + // Display the scenario name/data on the UI if (preHandler) { dispatch_async(dispatch_get_main_queue(), ^{ - preHandler(action, scenarioName, eventMode); + preHandler(scenarioName, eventMode); }); } From 009d719ce80f5c4fa81e464d8d06468302cedf25 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Thu, 29 Jun 2023 19:30:16 +0100 Subject: [PATCH 26/35] Not wanted --- .ruby-version | 1 - 1 file changed, 1 deletion(-) delete mode 100644 .ruby-version diff --git a/.ruby-version b/.ruby-version deleted file mode 100644 index 2c9b4ef42..000000000 --- a/.ruby-version +++ /dev/null @@ -1 +0,0 @@ -2.7.3 From 4861074ebc1da25fc14906aadc00ac2764314bf0 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Thu, 29 Jun 2023 20:24:47 +0100 Subject: [PATCH 27/35] Tidy [full ci] --- .../ios/iOSTestApp/CommandReaderThread.swift | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift b/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift index 96d9ff9f9..c410abb8f 100644 --- a/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift +++ b/features/fixtures/ios/iOSTestApp/CommandReaderThread.swift @@ -31,9 +31,7 @@ class CommandReaderThread: Thread { func loadMazeRunnerAddress() -> String { - // TODO Debug - default to nonsense for now so it also fails on BS - //let bsAddress = "http://bs-local.com:9339" - let bsAddress = "http://sdsdfcsdcfewcw:1234" + let bsAddress = "http://bs-local.com:9339" // Only iOS 12 and above will run on BitBar for now if #available(iOS 12.0, *) {} else { @@ -41,21 +39,15 @@ class CommandReaderThread: Thread { } for n in 1...30 { - log("SKW0") let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] log("Reading Maze Runner address from fixture_config.json") do { - - log("SKW1") - let fileUrl = URL(fileURLWithPath: "fixture_config", relativeTo: documentsUrl).appendingPathExtension("json") - log("SKW2") let savedData = try Data(contentsOf: fileUrl) - log("SKW3") if let contents = String(data: savedData, encoding: .utf8) { NSLog("Found fixture_config.json after %d seconds", n) let decoder = JSONDecoder() @@ -65,8 +57,6 @@ class CommandReaderThread: Thread { log("Using Maze Runner address: " + address) return address } - - log("SKW4") } catch let error as NSError { log("Failed to read fixture_config.json: \(error)") From 75501ac809d8f361b2966598cedf47935c7e9770 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 30 Jun 2023 09:35:33 +0100 Subject: [PATCH 28/35] Only include /reflect urls in breadcrumbs --- .../fixtures/shared/scenarios/NetworkBreadcrumbsScenario.swift | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/features/fixtures/shared/scenarios/NetworkBreadcrumbsScenario.swift b/features/fixtures/shared/scenarios/NetworkBreadcrumbsScenario.swift index 62c4b05a0..099b9eba6 100644 --- a/features/fixtures/shared/scenarios/NetworkBreadcrumbsScenario.swift +++ b/features/fixtures/shared/scenarios/NetworkBreadcrumbsScenario.swift @@ -15,7 +15,8 @@ class NetworkBreadcrumbsScenario : Scenario { config.autoTrackSessions = false; config.add(BugsnagNetworkRequestPlugin()) config.addOnBreadcrumb { - ($0.metadata["url"] as? String ?? "").hasPrefix(Scenario.baseMazeAddress) + let url = $0.metadata["url"] as? String ?? "" + return url.hasPrefix(Scenario.baseMazeAddress) && url.contains("/reflect") } super.startBugsnag() From 4c8def0dadd1459ddb297c9ed4b1739200c54922 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 30 Jun 2023 09:36:04 +0100 Subject: [PATCH 29/35] Lock file update --- Gemfile.lock | 1 + 1 file changed, 1 insertion(+) diff --git a/Gemfile.lock b/Gemfile.lock index a6a23dfc9..89a5f2cfa 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -196,6 +196,7 @@ GEM rexml (~> 3.2.4) PLATFORMS + x86_64-darwin-19 x86_64-darwin-20 DEPENDENCIES From a0a4506025d6c6f16269946d00e18ed05a3c3dbc Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 30 Jun 2023 09:37:44 +0100 Subject: [PATCH 30/35] Reinstate --- .buildkite/pipeline.yml | 370 ++++++++++++++++++++-------------------- 1 file changed, 185 insertions(+), 185 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 6d1449613..227cf6b7f 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -282,191 +282,191 @@ steps: # # Basic build E2E tests # -# -# # -# # BitBar -# # -# - label: ':bitbar: iOS 16 E2E tests batch 1' -# depends_on: -# - cocoa_fixture -# timeout_in_minutes: 60 -# agents: -# queue: opensource -# plugins: -# artifacts#v1.9.0: -# download: "features/fixtures/ios/output/iOSTestApp.ipa" -# upload: "maze_output/failed/**/*" -# docker-compose#v4.7.0: -# pull: cocoa-maze-runner-bitbar -# run: cocoa-maze-runner-bitbar -# service-ports: true -# command: -# - "--app=/app/build/iOSTestApp.ipa" -# - "--farm=bb" -# - "--device=IOS_16" -# - "--no-tunnel" -# - "--aws-public-ip" -# - "--fail-fast" -# - "--exclude=features/[e-z].*.feature$" -# - "--order=random" -# concurrency: 25 -# concurrency_group: 'bitbar-app' -# concurrency_method: eager -# retry: -# automatic: -# - exit_status: -1 # Agent was lost -# limit: 2 -# -# - label: ':bitbar: iOS 16 E2E tests batch 2' -# depends_on: -# - cocoa_fixture -# timeout_in_minutes: 60 -# agents: -# queue: opensource -# plugins: -# artifacts#v1.9.0: -# download: "features/fixtures/ios/output/iOSTestApp.ipa" -# upload: "maze_output/failed/**/*" -# docker-compose#v4.7.0: -# pull: cocoa-maze-runner-bitbar -# run: cocoa-maze-runner-bitbar -# service-ports: true -# command: -# - "--app=/app/build/iOSTestApp.ipa" -# - "--farm=bb" -# - "--device=IOS_16" -# - "--no-tunnel" -# - "--aws-public-ip" -# - "--fail-fast" -# - "--exclude=features/[a-d].*.feature$" -# - "--order=random" -# concurrency: 25 -# concurrency_group: 'bitbar-app' -# concurrency_method: eager -# retry: -# automatic: -# - exit_status: -1 # Agent was lost -# limit: 2 -# -# - label: ':bitbar: iOS 15 barebone tests' -# depends_on: -# - cocoa_fixture -# timeout_in_minutes: 60 -# agents: -# queue: opensource -# plugins: -# artifacts#v1.9.0: -# download: "features/fixtures/ios/output/iOSTestApp.ipa" -# upload: "maze_output/failed/**/*" -# docker-compose#v4.7.0: -# pull: cocoa-maze-runner-bitbar -# run: cocoa-maze-runner-bitbar -# service-ports: true -# command: -# - "--app=/app/build/iOSTestApp.ipa" -# - "--farm=bb" -# - "--device=IOS_15" -# - "--no-tunnel" -# - "--aws-public-ip" -# - "--fail-fast" -# - "features/barebone_tests.feature" -# concurrency: 25 -# concurrency_group: 'bitbar-app' -# concurrency_method: eager -# retry: -# automatic: -# - exit_status: -1 # Agent was lost -# limit: 2 -# -# - label: ':bitbar: iOS 14 barebone tests' -# depends_on: -# - cocoa_fixture -# timeout_in_minutes: 60 -# agents: -# queue: opensource -# plugins: -# artifacts#v1.9.0: -# download: "features/fixtures/ios/output/iOSTestApp.ipa" -# upload: "maze_output/failed/**/*" -# docker-compose#v4.7.0: -# pull: cocoa-maze-runner-bitbar -# run: cocoa-maze-runner-bitbar -# service-ports: true -# command: -# - "--app=/app/build/iOSTestApp.ipa" -# - "--farm=bb" -# - "--device=IOS_14" -# - "--no-tunnel" -# - "--aws-public-ip" -# - "--fail-fast" -# - "features/barebone_tests.feature" -# concurrency: 25 -# concurrency_group: 'bitbar-app' -# concurrency_method: eager -# retry: -# automatic: -# - exit_status: -1 # Agent was lost -# limit: 2 -# -# - label: ':bitbar: iOS 13 barebone tests' -# depends_on: -# - cocoa_fixture -# timeout_in_minutes: 60 -# agents: -# queue: opensource -# plugins: -# artifacts#v1.9.0: -# download: "features/fixtures/ios/output/iOSTestApp.ipa" -# upload: "maze_output/failed/**/*" -# docker-compose#v4.7.0: -# pull: cocoa-maze-runner-bitbar -# run: cocoa-maze-runner-bitbar -# service-ports: true -# command: -# - "--app=/app/build/iOSTestApp.ipa" -# - "--farm=bb" -# - "--device=IOS_13" -# - "--no-tunnel" -# - "--aws-public-ip" -# - "--fail-fast" -# - "features/barebone_tests.feature" -# concurrency: 25 -# concurrency_group: 'bitbar-app' -# concurrency_method: eager -# retry: -# automatic: -# - exit_status: -1 # Agent was lost -# limit: 2 -# -# - label: ':bitbar: iOS 12 barebone tests' -# depends_on: -# - cocoa_fixture -# timeout_in_minutes: 60 -# agents: -# queue: opensource -# plugins: -# artifacts#v1.9.0: -# download: "features/fixtures/ios/output/iOSTestApp.ipa" -# upload: "maze_output/failed/**/*" -# docker-compose#v4.7.0: -# pull: cocoa-maze-runner-bitbar -# run: cocoa-maze-runner-bitbar -# service-ports: true -# command: -# - "--app=/app/build/iOSTestApp.ipa" -# - "--farm=bb" -# - "--device=IOS_12" -# - "--no-tunnel" -# - "--aws-public-ip" -# - "--fail-fast" -# - "features/barebone_tests.feature" -# concurrency: 25 -# concurrency_group: 'bitbar-app' -# concurrency_method: eager -# retry: -# automatic: -# - exit_status: -1 # Agent was lost -# limit: 2 + + # + # BitBar + # + - label: ':bitbar: iOS 16 E2E tests batch 1' + depends_on: + - cocoa_fixture + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_16" + - "--no-tunnel" + - "--aws-public-ip" + - "--fail-fast" + - "--exclude=features/[e-z].*.feature$" + - "--order=random" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + + - label: ':bitbar: iOS 16 E2E tests batch 2' + depends_on: + - cocoa_fixture + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_16" + - "--no-tunnel" + - "--aws-public-ip" + - "--fail-fast" + - "--exclude=features/[a-d].*.feature$" + - "--order=random" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + + - label: ':bitbar: iOS 15 barebone tests' + depends_on: + - cocoa_fixture + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_15" + - "--no-tunnel" + - "--aws-public-ip" + - "--fail-fast" + - "features/barebone_tests.feature" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + + - label: ':bitbar: iOS 14 barebone tests' + depends_on: + - cocoa_fixture + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_14" + - "--no-tunnel" + - "--aws-public-ip" + - "--fail-fast" + - "features/barebone_tests.feature" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + + - label: ':bitbar: iOS 13 barebone tests' + depends_on: + - cocoa_fixture + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_13" + - "--no-tunnel" + - "--aws-public-ip" + - "--fail-fast" + - "features/barebone_tests.feature" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 + + - label: ':bitbar: iOS 12 barebone tests' + depends_on: + - cocoa_fixture + timeout_in_minutes: 60 + agents: + queue: opensource + plugins: + artifacts#v1.9.0: + download: "features/fixtures/ios/output/iOSTestApp.ipa" + upload: "maze_output/failed/**/*" + docker-compose#v4.7.0: + pull: cocoa-maze-runner-bitbar + run: cocoa-maze-runner-bitbar + service-ports: true + command: + - "--app=/app/build/iOSTestApp.ipa" + - "--farm=bb" + - "--device=IOS_12" + - "--no-tunnel" + - "--aws-public-ip" + - "--fail-fast" + - "features/barebone_tests.feature" + concurrency: 25 + concurrency_group: 'bitbar-app' + concurrency_method: eager + retry: + automatic: + - exit_status: -1 # Agent was lost + limit: 2 # # BrowserStack From 58a0110b24212742c28a61a65988ba4d62b55f5b Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 30 Jun 2023 09:48:49 +0100 Subject: [PATCH 31/35] Tidy-up --- features/fixtures/ios/iOSTestApp/ViewController.swift | 2 -- features/support/env.rb | 4 ---- 2 files changed, 6 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index 020acbbdf..e3b8d06a8 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -24,7 +24,6 @@ class ViewController: UIViewController { @IBOutlet var scenarioMetaDataField : UITextField! @IBOutlet var apiKeyField: UITextField! var mazeRunnerAddress: String = "" - var timerStarted: Bool = false override func viewDidLoad() { super.viewDidLoad() @@ -32,7 +31,6 @@ class ViewController: UIViewController { NotificationCenter.default.addObserver(self, selector: #selector(didEnterBackgroundNotification), name: UIApplication.didEnterBackgroundNotification, object: nil) apiKeyField.text = UserDefaults.standard.string(forKey: "apiKey") - // Poll for commands to run if #available(iOS 10.0, *) { let uiUpdater = { (scenarioName: String, eventMode: String) in diff --git a/features/support/env.rb b/features/support/env.rb index 7a3426c20..8aa7ae11d 100644 --- a/features/support/env.rb +++ b/features/support/env.rb @@ -32,10 +32,6 @@ disable_unexpectedly_quit_dialog end - - # Experiment - Maze.config.enable_retries - end # Disables the "macOSTestApp quit unexpectedly" dialog to prevent focus being stolen from the fixture. From 2e6b394924deca8c9932c66f7bb5838aa1702b41 Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 30 Jun 2023 09:51:11 +0100 Subject: [PATCH 32/35] Tidy --- .buildkite/pipeline.yml | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 227cf6b7f..993c1f4d7 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -526,30 +526,6 @@ steps: - exit_status: -1 # Agent was lost limit: 2 - - label: ':bitbar: Temp test' - depends_on: "cocoa_fixture" - timeout_in_minutes: 20 - agents: - queue: opensource - plugins: - artifacts#v1.9.0: - download: "features/fixtures/ios/output/iOSTestApp.ipa" - upload: "maze_output/failed/**/*" - docker-compose#v4.7.0: - pull: cocoa-maze-runner-bitbar - run: cocoa-maze-runner-bitbar - service-ports: true - command: - - "--app=/app/build/iOSTestApp.ipa" - - "--farm=bb" - - "--device=IOS_15" - - "--no-tunnel" - - "--aws-public-ip" - - "features/app_and_device_attributes.feature:6" - concurrency: 25 - concurrency_group: 'bitbar-app' - concurrency_method: eager - ############################################################################## # # Trigger more tests From 7c4c4287ba73e5f2dfbe636acea58a733c721cdc Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 30 Jun 2023 09:53:17 +0100 Subject: [PATCH 33/35] Tidy --- features/fixtures/ios/iOSTestApp/ViewController.swift | 5 ----- 1 file changed, 5 deletions(-) diff --git a/features/fixtures/ios/iOSTestApp/ViewController.swift b/features/fixtures/ios/iOSTestApp/ViewController.swift index e3b8d06a8..708d5306f 100644 --- a/features/fixtures/ios/iOSTestApp/ViewController.swift +++ b/features/fixtures/ios/iOSTestApp/ViewController.swift @@ -11,11 +11,6 @@ import os class FixtureConfig: Codable { var maze_address: String - - - func startBugsnag() { - sleep(1) - } } class ViewController: UIViewController { From c977b9c25401f48949ab9f30b823ed2a97f1870b Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 7 Jul 2023 10:21:58 +0100 Subject: [PATCH 34/35] Gemfile bump --- Gemfile.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index 89a5f2cfa..1c00ea98a 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -3,7 +3,7 @@ GEM specs: CFPropertyList (3.0.6) rexml - activesupport (7.0.5) + activesupport (7.0.6) concurrent-ruby (~> 1.0, >= 1.0.2) i18n (>= 1.6, < 2) minitest (>= 5.1) @@ -23,7 +23,7 @@ GEM atomos (0.1.3) bugsnag (6.25.2) concurrent-ruby (~> 1.0) - bugsnag-maze-runner (8.0.2) + bugsnag-maze-runner (8.1.4) appium_lib (~> 12.0.0) appium_lib_core (~> 5.4.0) bugsnag (~> 6.24) @@ -149,7 +149,7 @@ GEM nanaimo (0.3.0) nap (1.1.0) netrc (0.11.0) - nokogiri (1.15.2-x86_64-darwin) + nokogiri (1.15.3-x86_64-darwin) racc (~> 1.4) optimist (3.0.1) os (1.0.1) From 2b9627a728b5f66b9551d197e29f855ed793d31f Mon Sep 17 00:00:00 2001 From: Steve Kirkland-Walton Date: Fri, 7 Jul 2023 11:11:38 +0100 Subject: [PATCH 35/35] Do not set bundler local path --- .buildkite/pipeline.full.yml | 1 - .buildkite/pipeline.yml | 1 - 2 files changed, 2 deletions(-) diff --git a/.buildkite/pipeline.full.yml b/.buildkite/pipeline.full.yml index a51b28104..85792944b 100644 --- a/.buildkite/pipeline.full.yml +++ b/.buildkite/pipeline.full.yml @@ -339,7 +339,6 @@ steps: - "macOSTestApp.log" - "maze_output/failed/**/*" commands: - - bundle config set --local path 'vendor/bundle' - bundle install - bundle exec maze-runner --os=macos diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml index 993c1f4d7..1735d6220 100644 --- a/.buildkite/pipeline.yml +++ b/.buildkite/pipeline.yml @@ -251,7 +251,6 @@ steps: - "macOSTestApp.log" - "maze_output/failed/**/*" commands: - - bundle config set --local path 'vendor/bundle' - bundle install - bundle exec maze-runner features/barebone_tests.feature