App (React Native) seems to become unresponsive when running with Maestro

[matt-dalton]

I'm finding that with certain test runs, the test will get to a certain point, then the app will immediately freeze (i.e. it won't crash, but animations will stop and it will become unresponsive). Eventually maestro will timeout & fail the test.

What's weird is this exact test suite was working reliably yesterday, and a different one was failing repeatedly.

I'm guessing it's when I get to something consuming more memory/CPU:

• Yesterday it happened when opening up a deep link from app close & navigating to a screen => I guess lots things happening at once
• Today it's freezing on a loading placeholder screen while navigating => Quite a bit of data fetched in the background, but not a lot on the screen

If I follow the exact same steps manually on the simulator (without maestro running) everything works reliably & smoothly every time.

There's no visible error, but if I rebuild Xcode after the crash I get a debug report:

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Incident Identifier: 3A1127B9-D600-41D4-80BA-3BAA935E475F
CrashReporter Key:   5CDB127C-9FD0-635F-E716-77C74B52A34C
Hardware Model:      MacBookPro18,3
Process:             my_app [3211]
Path:                /Users/USER/Library/Developer/CoreSimulator/Devices/71BBA01C-2A5D-42CF-BC1C-1D14D1C3C031/data/Containers/Bundle/Application/15AC1364-CC10-43D5-B01C-D39BB5E504BB/my_app.app/my_app
Identifier:          com.my_app
Version:             0.0.0 (1)
Code Type:           X86-64 (Native)
Role:                Foreground
Parent Process:      launchd_sim [97381]
Coalition:           com.apple.CoreSimulator.SimDevice.71BBA01C-2A5D-42CF-BC1C-1D14D1C3C031 [30920]
Responsible Process: SimulatorTrampoline [37118]

Date/Time:           2023-02-16 19:05:34.3661 +0000
Launch Time:         2023-02-16 18:56:56.0366 +0000
OS Version:          macOS 12.5.1 (21G83)
Release Type:        User
Report Version:      104

Exception Type:  EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Termination Reason: FRONTBOARD 2343432205 
<RBSTerminateContext| domain:10 code:0x8BADF00D explanation:[application<com.my_app>:3211] failed to terminate gracefully after 5.0s
ProcessVisibility: Foreground
ProcessState: Running
WatchdogEvent: process-exit
WatchdogVisibility: Foreground
WatchdogCPUStatistics: (
"Elapsed total CPU time (seconds): 13.610 (user 10.590, system 3.020), 27% CPU",
"Elapsed application CPU time (seconds): 6.479, 13% CPU"
) reportType:CrashLog maxTerminationResistance:Interactive>

Triggered by Thread:  0

Thread 0 Crashed::  Dispatch queue: com.apple.accessibility.AXSideStorageQueue
0   <translation info unavailable>	       0x106a062ac ???
1   <translation info unavailable>	       0x106a06210 ???
2   libobjc.A.dylib               	       0x1104b8089 objc_retainAutoreleasedReturnValue + 45
3   AccessibilityUtilities        	       0x14fa1f78f __53-[NSObject(AXSideStorage) _accessibilityValueForKey:]_block_invoke + 47
4   libdispatch.dylib             	       0x1157b0a3a _dispatch_client_callout + 8
5   libdispatch.dylib             	       0x1157c114d _dispatch_lane_barrier_sync_invoke_and_complete + 92
6   AccessibilityUtilities        	       0x14fa1f4a7 -[NSObject(AXSideStorage) _accessibilityValueForKey:] + 615
7   UIAccessibility               	       0x152f239ac -[NSObject(UIStorage) _accessibilityGetBlockForAttribute:] + 44
8   UIAccessibility               	       0x152f207d7 -[NSObjectAccessibility accessibilityLabel] + 55
9   my_app                          	       0x102c661c1 RCTRecursiveAccessibilityLabel + 369 (RCTView.m:86)
10  my_app                          	       0x102c6601d -[RCTView accessibilityLabel] + 109 (RCTView.m:225)
11  my_app                          	       0x102c661c1 RCTRecursiveAccessibilityLabel + 369 (RCTView.m:86)
...


76  AXRuntime                     	       0x13b7056f8 _handleNonMainThreadCallback + 424
77  AXRuntime                     	       0x13b705e36 _AXXMIGCopyParameterizedAttributeValue + 662
78  AXRuntime                     	       0x13b77c0b5 _XCopyParameterizedAttributeValue + 693
79  AXRuntime                     	       0x13b723a1d mshMIGPerform + 1021
80  CoreFoundation                	       0x1171c87f4 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 41
81  CoreFoundation                	       0x1171c7d3b __CFRunLoopDoSource1 + 538
82  CoreFoundation                	       0x1171c2538 __CFRunLoopRun + 2740
83  CoreFoundation                	       0x1171c16a7 CFRunLoopRunSpecific + 560
84  GraphicsServices              	       0x11d03a28a GSEventRunModal + 139
85  UIKitCore                     	       0x127bf1ad3 -[UIApplication _run] + 994
86  UIKitCore                     	       0x127bf69ef UIApplicationMain + 123
87  my_app                          	       0x10216f748 main + 104 (main.m:15)
88  dyld_sim                      	       0x10f02c2bf start_sim + 10
89  dyld                          	       0x206c9a52e start + 462

Thread 1:: com.apple.rosetta.exceptionserver
0   ???                           	    0x7ff7ffec5944 ???
1   ???                           	    0x7ff7ffede1f0 ???

Thread 2:: com.apple.uikit.eventfetch-thread
0   ???                           	       0x1068dd940 ???
1   <translation info unavailable>	       0x106999954 ???
2   libsystem_kernel.dylib        	       0x11bb83ce8 mach_msg + 56
3   CoreFoundation                	       0x1171c788e __CFRunLoopServiceMachPort + 145
4   CoreFoundation                	       0x1171c1fdf __CFRunLoopRun + 1371
5   CoreFoundation                	       0x1171c16a7 CFRunLoopRunSpecific + 560
6   Foundation                    	       0x115dc48b4 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 213
7   Foundation                    	       0x115dc4b2d -[NSRunLoop(NSRunLoop) runUntilDate:] + 72
8   UIKitCore                     	       0x127cc7286 -[UIEventFetcher threadMain] + 535
9   Foundation                    	       0x115dee11b __NSThread__start__ + 1009
10  libsystem_pthread.dylib       	       0x11bc314e1 _pthread_start + 125
11  libsystem_pthread.dylib       	       0x11bc2cf6b thread_start + 15

Thread 3:: com.facebook.react.JavaScript
0   ???                           	       0x1068dd940 ???
1   <translation info unavailable>	       0x106999954 ???
2   libsystem_kernel.dylib        	       0x11bb83ce8 mach_msg + 56
3   CoreFoundation                	       0x1171c788e __CFRunLoopServiceMachPort + 145
4   CoreFoundation                	       0x1171c1fdf __CFRunLoopRun + 1371
5   CoreFoundation                	       0x1171c16a7 CFRunLoopRunSpecific + 560
6   oban                          	       0x102b81715 +[RCTCxxBridge runRunLoop] + 949 (RCTCxxBridge.mm:378)
7   Foundation                    	       0x115dee11b __NSThread__start__ + 1009
8   libsystem_pthread.dylib       	       0x11bc314e1 _pthread_start + 125
9   libsystem_pthread.dylib       	       0x11bc2cf6b thread_start + 15

Versions:
React Native: 0.68.5
Maestro CLI: 1.23.0
Xcode 14.2
Running on Mac M1 16GB Memory

I'm guessing this is something to do with my machine struggling with running both the app and the test runner? But not sure if there's any way of addressing. Anything else I can provide to help debug the issue?

[matt-dalton]

Not sure if it's related, but I've also now started getting this in a similar part of the test:

Exception: Failed to connect to localhost/127.0.0.1:22087
xcuitest.XCTestDriverClient$XCTestDriverUnreachable: Failed to reach out XCUITest Server
        at xcuitest.XCTestDriverClient$okHttpClient$2.invoke$lambda-2(XCTestDriverClient.kt:75)

This is happening every time I try and scroll (either scroll or scrollUntilVisible) on a certain screen. Again works perfectly when just scrolling manually on the simulator.

Not sure if this is a different symptom of the same problem, or a separate issue entirely. I can open up a different bug report if you'd prefer.

[Leland-Takamine]

Hey @matt-dalton can you please try again on the latest Maestro version? We've made a lot of improvements lately and would like to verify whether this is still an issue. Thank you for the help!

[matt-dalton]

Hey @Leland-Takamine - thanks for fixing. I'm currently on some other projects but will hopefully get back to this in a couple of weeks.

[keeslinp]

I'm on the latest release of maestro and I've seen this a few times as well. ActivityIndicators will just freeze and the app hangs. Sometimes waitForAnimationToEnd helps it get through that but not always.

[konstantinlindner]

I'm having the same issue on latest Maestro.

Our react native app will freeze on loading screens a second or so after the loading animation starts.
Is there any update or workaround?

[Dalamar]

Joining the club.
Same issue.
Will be glad to provide any helpful details to help to resolve this issue.

My observations:
Once the test gets failed the app in iOS simulator hits 100% of CPU load.
A couple of minutes later the app gets unfreeze and CPU load becomes normal. Though test by this moment has already died.

Error from maestro log file:

[ERROR] maestro.cli.runner.TestRunner - Failed to run flow
java.lang.IllegalStateException: Unable to obtain active app id
	at ios.xctest.XCTestIOSDevice.contentDescriptor(XCTestIOSDevice.kt:57)
	at ios.LocalIOSDevice.contentDescriptor(LocalIOSDevice.kt:35)
	at maestro.drivers.IOSDriver.contentDescriptor(IOSDriver.kt:184)
	at maestro.ViewHierarchy$Companion.from-c1iYVAs(ViewHierarchy.kt:29)
	at maestro.Maestro.viewHierarchy-prqvCes(Maestro.kt:416)
	at maestro.orchestra.Orchestra.findElement(Orchestra.kt:770)
	at maestro.orchestra.Orchestra.evaluateCondition(Orchestra.kt:463)
	at maestro.orchestra.Orchestra.assertConditionCommand(Orchestra.kt:253)
	at maestro.orchestra.Orchestra.executeCommand(Orchestra.kt:196)
	at maestro.orchestra.Orchestra.executeCommands(Orchestra.kt:155)
	at maestro.orchestra.Orchestra.runFlow(Orchestra.kt:98)
	at maestro.cli.runner.MaestroCommandRunner.runCommands(MaestroCommandRunner.kt:179)
	at maestro.cli.runner.TestRunner$runSingle$result$1.invoke(TestRunner.kt:45)
	at maestro.cli.runner.TestRunner$runSingle$result$1.invoke(TestRunner.kt:42)
	at maestro.cli.runner.TestRunner.runCatching(TestRunner.kt:138)
	at maestro.cli.runner.TestRunner.runSingle(TestRunner.kt:42)
	at maestro.cli.command.TestCommand$call$1.invoke(TestCommand.kt:166)
	at maestro.cli.command.TestCommand$call$1.invoke(TestCommand.kt:126)
	at maestro.cli.session.MaestroSessionManager.newSession(MaestroSessionManager.kt:99)
	at maestro.cli.session.MaestroSessionManager.newSession$default(MaestroSessionManager.kt:57)
	at maestro.cli.command.TestCommand.call(TestCommand.kt:126)
	at maestro.cli.command.TestCommand.call(TestCommand.kt:44)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
	at picocli.CommandLine.access$1200(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
	at maestro.cli.DisableAnsiMixin$Companion.executionStrategy(DisableAnsiMixin.kt:22)
	at picocli.CommandLine.execute(CommandLine.java:2058)
	at maestro.cli.AppKt.main(App.kt:136)

Dev environment:

 react-native info
info Fetching system and libraries information...
System:
    OS: macOS 13.4
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Memory: 22.93 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 14.19.0 - ~/.nvm/versions/node/v14.19.0/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 6.14.17 - ~/.nvm/versions/node/v14.19.0/bin/npm
    Watchman: 2023.05.22.00 - /usr/local/bin/watchman
  Managers:
    CocoaPods: 1.11.2 - /usr/local/bin/pod
  SDKs:
    iOS SDK:
      Platforms: DriverKit 22.4, iOS 16.4, macOS 13.3, tvOS 16.4, watchOS 9.4
    Android SDK:
      API Levels: 16, 28, 29, 30, 31, 32, 33
      Build Tools: 28.0.3, 29.0.2, 30.0.2, 30.0.3, 31.0.0, 32.0.0, 33.0.0
      System Images: android-29 | Android TV Intel x86 Atom, android-29 | Intel x86 Atom_64, android-29 | Google APIs Intel x86 Atom, android-29 | Google Play Intel x86 Atom, android-30 | Google APIs Intel x86 Atom
      Android NDK: Not Found
  IDEs:
    Android Studio: 2022.2 AI-222.4459.24.2221.9971841
    Xcode: 14.3/14E222b - /usr/bin/xcodebuild
  Languages:
    Java: 12.0.1 - /usr/bin/javac
  npmPackages:
    @react-native-community/cli: Not Found
    react: 18.2.0 => 18.2.0 
    react-native: 0.71.7 => 0.71.7 
    react-native-macos: Not Found
  npmGlobalPackages:
    *react-native*: Not Found

P.S. As a quick temporary solution you can manually launch simulator and open the app, modify your test as follows and start test:

- launchApp:
    stopApp: false

[bobinrinder]

Same issue here. 😢

P.S. As a quick temporary solution you can manually launch simulator and open the app, modify your test as follows and start test:

This just fixes the issue with launchApp sadly. @Dalamar

@Leland-Takamine Any schedule to fix this by any chance? We were just about to setup maestro cloud but if all our tests get stuck, it's not really worth the effort.

[Leland-Takamine]

@matt-dalton @keeslinp @konstantinlindner @Dalamar @bobinrinder Thank you all for bumping this issue back up! If any of you are able to send over a simulator build (Feel free to DM me on Slack) along with repro steps that would be very much appreciated!

[Leland-Takamine]

I was able to reproduce locally - we are digging in!

[artem888]

Hey @matt-dalton @keeslinp @konstantinlindner @Dalamar @bobinrinder ,

Thank you for your valuable feedback!

Upon investigating the issue, we identified multiple contributing factors. Primarily, the main thread seems to get overwhelmed by certain XCTest API calls when specific screens are presented (notably during the presentation of the 'fit52' loading screen with an animation). Although XCTest has an internal API, waitForQuiescenceIncludingAnimationsIdle, meant to prevent XCTest queries when the app isn't idle, it appears ineffective in some scenarios.

Recommendations:

• Disable Animations: Turn off animations for screens that exhibit freezing behavior.
• Use View Flattening: For ReactNative apps, consider implementing View Flattening.
• Main Thread Usage: Ensure that the main thread is exclusively reserved for UI-related logic. Tools like the Instruments Time Profiler and the combination of Thread Sanitizer + Main Thread Checker (available in Debug scheme settings) can be instrumental in this verification.

Additionally, it would immensely aid our investigation if you could share more app examples, flow cases, maestro cloud upload links, where this issue arises. Please feel free to share these details with me or @Leland-Takamine privately in our Maestro Slack channel.

Should you have further queries or concerns, don't hesitate to reach out. We're here to help!

[eeston]

Hey @artem888 @Leland-Takamine

I believe can reliably reproduce this issue on screens that are using react-native-maps with a bunch of <Marker /> components. Seems to work okay-ish with only a few markers but when I'm running e2e tests against a production dataset with just under 300 markers, the app grinds to a hault and maestro studio refuses to open.

I've created a similar reproduction here: https://github.com/eeston/maestro-issue

Hope this is useful!

[Fishbowler]

This issue has gotten a little stale, and has several slightly different problems.

@eeston You don't mention the platform you're tackling, or which of the errors you're encountering.

Maestro pulls the entire view of elements from the screen for lots of its commands. When you're running Studio, it's on a cadence. The larger the view, the larger the strain on the device to create it, and the additional traffic that needs to travel across the ADB or IDB bridge to the workstation. If the device cannot keep up with the cadence of requests, you're going to see problems.

I'd be interested to know if emulator or workstation performance is a factor here. I wonder if we increased the memory or CPU of the emulator if the same problems would exhibit?