-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Detox v18/v19 crashes sometimes on iOS in DetoxSync #3000
Comments
probably related #2641 |
Sorry for the close reopen, an incorrect mobile tap on my part. |
@mikehardy this is an excellent report, which has real potential in helping us figuring out issues with DetoxSync. |
Here's a GitHub Actions run showing it happening in case that's useful https://github.com/invertase/react-native-firebase/runs/3783157278?check_suite_focus=true#step:21:13 I have determined that it is not always immediately on startup, but it's usually pretty close to startup. Reproduction rate still seems to be about 10-20% of my local intel-mac e2e runs in the react-native-firebase suite. Not sure how to mitigate it as it doesn't always happen on startup, and I'm not sure how to catch an "app crashed" error so that I can attempt to just relaunch the app and see if it will go through afterwards / do "app crashed retries" |
Hey Mike. Just confirming that this issue is not related to Firebase Performance at all? |
I haven't tried, the whole point of my using detox is in my role maintaining react-native-firebase, and it's full e2e test harness 😁😅 |
We're having this issue despite using .e2e.ts files to mock out firebase perf and firebase analytics. So I don't think it's Firebase related. |
One thing: this only seems to happen on CircleCI for us. Could it be related to the performance of macOS containers vs. bare metal? |
Perhaps. It happens to us a lot on Github Actions which are not that fast, but it also happens to me a lot both in an OSX VM which sounds like it would be slow but is on about the fastest laptop you can purchase so is actually really fast, plus I see it on an M1 machine (bare metal, pretty quick) and MacBook Airs which are slow but bare metal. So for me doesn't seem speed-related. Surprised you don't ever see it in your local environments vs my experience then 🤔 |
I can vouch that it is not speed related. We had Detox running on physical M1 Mac Minis used as GitHub action runners for a few weeks and had this SIGNAL 11 issue repeatedly. In the end we had to revert back to Intel Mac Minis to be used only for running Detox. |
Here is my comment from the other thread: I did extensive load tests on my own machine with 100% CPU usage on bare metal and could not reproduce the issue. However, the moment Detox runs under a VM (which is 16GB mem, 8*vCPU and running at 5% load), the issue appears. What I can conclude is that this is some sort of incompatibility between Detox and a hypervisor. Detox + virtualization, not performance, seems to cause the error. |
The reason you probably see this issue on bare metal is because you've left in your Firebase dependency, where the two swizzlers (detox and firebase) seem to have some sort of underlying compatibility. Once you mock out Firebase (we removed it with an e2e.ts NOOP stub), the issue disappears on bare metal. BUT if you run Detox in a virtualized environment, firebase or not, it will crash with this error. Thus:
|
Interesting - well, I need it to work with firebase performance integrated of course :-), even if a hypervisor is only a very strong "want" vs a need. It appears that I need to run a test on bare metal without firebase performance to see if I can at least reproduce clean runs. Then I suppose to actually make this more easily debuggable for someone with appropriate skills (that being, solid Objective-C skills which I lack unfortunately) I'll need to fork here and add a test with firebase performance in it. Thanks for the info @shamilovtim |
Yeah for us it's hard blocker that Detox has to run virtualized. Circle or Github aren't going to give us bare metal containers, so it's a blocker either way. Being able to have Firebase running during Detox would also be a plus. |
In my case, this problem is reproducible on every try. x86 on macos-11 on github actions
m1 on macos-12 on mba
|
@amq I see no instance of the string DetoxSync in your stack traces in any location. I believe your crash is unrelated, and is likely only showing up with Detox at all because Detox registers itself as a general crash handler in the system so all crashes go through it. Your crash does not actually look Detox related at all. |
Detox: v18.22.1 This occurs for our team in CircleCI only:
|
I just opened a related issue on firebase-ios-sdk repo: firebase/firebase-ios-sdk#9083. Also, this seems to be an issue we may be able to solve from DetoxSync's end. I'll investigate this a bit more but the general direction is to dispose the runtime-generated class that DetoxSync created before Firebase tries to dispose their generated class (the superclass of it), which will prevent the crash of calling objc_disposeClassPair() on a class before disposing its subclasses. See Apple docs:
At the moment, the quickest solution I can offer is to disable Firebase Performance when Detox Tests are running. |
mitigation via new command line environment variable on the app launch appears to work
|
root cause hypothesis is double-swizzling w/DetoxSync and performance mitigation hypothesis is to use new GUL swizzler feature to disable some object destruction firebase/firebase-ios-sdk#9083 wix/Detox#3000
@mikehardy assuming this will this work with |
root cause hypothesis is double-swizzling w/DetoxSync and performance mitigation hypothesis is to use new GUL swizzler feature to disable some object destruction firebase/firebase-ios-sdk#9083 wix/Detox#3000
root cause hypothesis is double-swizzling w/DetoxSync and performance mitigation hypothesis is to use new GUL swizzler feature to disable some object destruction firebase/firebase-ios-sdk#9083 wix/Detox#3000
@shamilovtim unknown and I don't want to speculate, sorry - all I know is that I've got at least some mitigation with the change. I think I still see crashes under some circumstances, but I think I've got an improvement if I'm running the experiments I think I'm running and I'm interpreting results correctly. Sincere apologies for being so weasel-y / vague about it. This stuff is subtle and I'm not sure of much at the moment, takes time |
@mikehardy I see, thanks for your efforts on this. Sorry for off topic but is there a reason you execute detox as |
Probably not? Per my bio I attempt to reverse entropy as a full-time thing, but there always seems to be more of it ;-). Most likely that is unnecessary, at least I'm not aware of any reason why we reach into .bin, and I'm aware than in a yarn v2+ future that's non-functional |
This workaround solves the issue described here: firebase/firebase-ios-sdk#9083 And some of the crashes that was mentioned here: - wix#3000 - wix#3123 - wix#2641 - wix#2802
Was previous having this same stacktrace as #3000 (comment) Tried @mikehardy's solution (but beware of the typo in the env variable!) Adding Thanks @mikehardy for your investigation & work to uncover this fix! |
@asafkorem and the folks in the firebase iOS repo deserve the credit but I'm glad it works for you. I'm 100% green in CI with this mitigation |
This workaround solves the issue described here: firebase/firebase-ios-sdk#9083 And some of the crashes that was mentioned here: - #3000 - #3123 - #2641 - #2802
#3135 was merged, it solve (with a workaround) the issue of DetoxSync crashes when Firebase/Performance is integrated, it was tested on the example apps that reproduced this issue. We will release a version with this change soon. Once you have updated a version (to the latest/next version), if any of you are still having Signal 11 crashes on DetoxSync, please open a new bug report with the required details. Huge thanks @mikehardy for your support and the assistance in solving this problem. |
Description
When updating from v17.latest of Detox to v18.latest I started seeing flakiness in the react-native-firebase e2e tests.
This does not happen all the time, it only happens sometimes. It looks like a bit of a race that some object destruct code is losing sometimes?
Reproduction
Provide the steps necessary to reproduce the issue. If you are seeing a regression, try to provide the last known version where the issue did not reproduce.
git clone [email protected]:invertase/react-native-firebase.git
yarn && yarn tests:ios:pod:install && yarn tests:ios:build
yarn tests:packager:jet
andyarn tests:emulator:start
andyarn tests:ios:test
Sometimes (not every time, maybe 10% of the time? 20% fo the time?) it will crash with the stack below.
Expected behavior
It rolls through all our tests.
For the purposes of this issue, it runs at all :-) - if it is going to crash, it will do so with no tests passing, if it starts running and a single test passes, they will all roll throughIt usually crashes at startup but sometimes will make it through a handful of tests first.Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Logs
It either works through all tests, or sometime shortly after app launch it crashes like this:
If you are experiencing a timeout in your test
If you are seeing a Detox build problem (e.g. during
npm install
, notdetox build
)npm install
log below:Device and verbose Detox logs
--loglevel trace
argument and am providing the verbose log below:Reproducible Demo
In case of a bug or a crash please add an example forking from the DetoxTemplate (follow the guidelines there) which reproduce the issue and ready to clone.
Add to the DetoxTemplate (After you fork it), the minimal things which required to reproduce the issue (3rd party libraries / e2e tests).
The text was updated successfully, but these errors were encountered: