Stability issues with Realm v. 10.13.0 and 10.16.0 #4525
Comments
|
@martinpoulsen Thanks for reporting. We will look into the issue. |
|
I have investigated the stack traces. Two stack traces ( Moreover, the stack traces |
|
Thanks @kneth for investigating. I have a hard time evaluating the resemblance of the stack traces, but I see from the error report bread crumbs in our most frequent crash that - like realm/realm-core#5344 - it also happens on app launch (where we create a I get a little nervous if our stability issues share root cause with realm/realm-core#5344 as it has been open for more than a month without being updated - is it still being investigated? @fronck We still didn't find a way to reproduce the issue, but we got symbolication of the native iOS stack trace working in the Bugsnag error reports and this is an example of the stack trace of the crash which we see most (and very) frequently: Is there anything else I can do to help? E.g. would it be helpful if I pulled and attached more crash reports like the 6 reports I already attached earlier? Or more stack traces (they vary slightly) from Bugsnag like the example above? |
|
In addition to the stack trace for the realm::NoSuchTable realm::KeyNotFound |
|
We are having the same issue since upgrading. |
|
@kneth do you have any more information about this and any timeline for a possible fix? Thx! |
|
Unfortunately we still haven't been able to reproduce the crashes. But let me add some information from looking at the breadcrumbs of the To me, the stack traces look a bit different for the |
|
Hi @martinpoulsen, thanks for all the details. We are doing some investigation into this issue right now. Are you able to share more information about what your app is doing when it receives a location event? If you have some sample code for how you are handling this, that would be interesting. Our prime suspect is that the app's data could be being encrypted by iOS when it goes into the background, then when it tries to interact with the Realm in the background, the (still open) Realm is reading OS-encrypted data rather than what it expects, and so crashes in seemingly random places – because it is receiving garbage data, essentially. The Realm Swift docs talk about this encryption issue. However, I created a simple test app which is woken from the background via a silent push notification (this seemed like it would be easier to test than location) and writes and reads from the Realm when it is awoken, and I haven't seen any issues so far. If you can share a bit more about what your app is doing when awoken, then we can try to simulate a more realistic workload. I can also try using location instead of push notification, in case this has any bearing on how iOS behaves – if you have tips on how best to test location events, that would save me some research! |
|
Hi @tomduncalf, happy to hear you are investigating this issue 🙏 Essentially, what our app does is tracking trips for our users for mileage reimbursement purposes. When resumed or woken up by a location (or an iBeacon or a geofence) it decides whether it should be record as part of a trip. If that's the case (i.e. the user is currently tracking a trip), it will be stored on a Trip realm model which has a route property (i.e. a list of locations). However, if the app is not initialized when receiving the location, it will first instantiate a Realm instance for the user - we do this only in the app launch flow. One thing that seems important when talking about reproducing the crashes is that they are not evenly distributed across our users. It is "only" affecting ~ 2% of our iOS users (and no Android users) but those 2% experience enough crashes that our session stability is down from ~ 99% to ~ 90% across all iOS users. So unfortunately that might mean that it's hard to reproduce the crashes in a basic sample app. Btw, the users we have been in contact with and who have been hit by these crashes were able to get rid of them by deleting => reinstalling the app. About the iOS file encryption, I'm wondering how that suddenly could start causing this magnitude of crashes after upgrading Realm from v. 10.10.1 => 10.13.0 (and still seeing them in v. 10.16.0) while seeing them very rarely before the upgrade. I will continue to try and reproduce the crashes and also keep looking for patterns giving some clue about the cause. |
|
Thanks for the additional details @martinpoulsen, indeed that does make the iOS encryption thing seem a little less likely, though there could be some weird interaction of things. We're looking into what changed in Realm core between those versions but are yet to find anything suspicious. Are you using encrypted Realms? |
|
One other question, when you say "seeing them very rarely before the upgrade" – were you seeing these crashes in some smaller volume before the upgrade, or not at all? |
|
Hi @tomduncalf thanks again for looking into this! We are not using encrypted realms. And sorry for being vague about whether we have seen the errors before. It's only recently that we got the native side of iOS crash report stack traces symbolicated in our error reporting setup using Bugsnag (we did this when we recently saw a massive increase in iOS native crashes). We do have some SIGABRT crash reports from older app versions, but we don't know if they come from Realm. At least they are very infrequent. I have now setup symbolication for the most recent build before the Realm upgrade so for future crashes from that app version, we will know - and I will make sure to update you on that. |
|
Got it, no worries @martinpoulsen and thanks for confirming re: encryption. We're continuing to investigate as a priority, so I will keep you udpated and let you know if we have any other questions. |
|
@martinpoulsen Are you able to confirm if the It looks like the |
|
Also, if it's easy for you to supply some more stack traces (for any of the errors), that would be great so I can see if I can spot any patterns – if it's not convenient to do so via Github then you can email to [email protected] |
|
One other question @martinpoulsen, are you using Realm Sync? |
|
Sorry for the bombardment of messages @martinpoulsen, I'm asking questions as I think of them so I hope you don't mind! Are the users who hit this issue then unable to open your app at all? Or is it an intermittent issue (i.e. they get some crashes but can sometimes open the app fine)? And do you see any crash logs for starting with |
|
@tomduncalf, I'm more than happy to assist in any way I can to get these issues resolved, please keep asking anything that could be useful! The The I have attached a sample of 10 stack traces for each of the crashes in the bottom. If it's helpful with a larger sample, just let me know. We would also be happy to give you access to our Bugsnag error reporting tool - again, just let me know. We are not using Realm Sync, only a local database. One thing I thought of mentioning is that we have two realm database instances running at all times - one for some global state (e.g. to keep track of currently logged in user) and one for user data (trips, etc). I see very few errors with the The few users we have been contacted by have experienced frequent crashes but not consistently in a way where they can never open the app. The breadcrumbs of the error reports supports this - the crashes are sometimes early in a session but usually not immediately when we create the realm instance. In many cases it happens when accessing realm to read/write at (for now) seemingly random times. EXC_BAD_ACCESS_1.txt |
|
Thanks for all the details and crash reports @martinpoulsen, that's really useful. I think this is plenty to go on, but if we determine that having access to your bug reporting tool would be useful we will let you know, thanks for the offer! In the interim, I would suggest you advise users who have the issue to reinstall the app – is all your data being synced to your server by some other means so users should not lose data? Do you know if you've had any incidences of users who have had to uninstall and reinstall hitting this same problem for a second time? Especially users who reinstalled a version of your app which uses Realm v10.16.0. |
|
A related question, are you aware of any new users who started with the latest version of your app (i.e. the one using Realm v10.16.0, rather than upgrading from an older version) hitting this issue? I'm wondering if the issue could be related to an older version of Realm and is now fixed (if the file is created with v10.16.0), or if it is still occurring with v10.16.0 |
|
Thanks @tomduncalf, advicing to reinstall the app is what our support team is currently doing when we get contacted by users who face these crashes - I'm not aware of a case where a user kept seeing the crashes after a reinstall, but we have only been in contact with a relatively small amount of users so far. If I see such a case at some point, I will let you know. User data is synced to our servers via a custom sync API so luckily a reinstall will not cause loss of data. Regarding new users (who started with v.10.16.0) facing the crash, this is one of the patterns I'm currently looking into, I will get back to you on that. |
|
Btw, since adding symbolication yesterday for the most recent app version before we upgraded realm (back then we were on v. 10.10.1) a single crash has been reported and it turned out to be a realm native one: |
|
Thanks @martinpoulsen, that would be interesting to know. Thanks for your patience while we investigate this, it's obviously a tricky one until we find a way to reproduce it but we are actively investigating! |
|
@tomduncalf we're just really happy that you are working on this! - and we know very well how hard it is to investigate issues without knowing a consistent way to reproduce 😅 |
|
Hi again @tomduncalf I have looked at a sample of 200
It appears that there is a pattern of users being less likely to get the crash if they created their Realm db with 10.13.0 and much less likely if they created it with 10.16.0. Our app realm version release timeline looks like this:
Looking at the signup time of the users behind the 200 crash reports, it seems there are fewer than expected who signed up between April 1. and April 21. (where Realm was created with 10.13.0) - and much fewer than expected (none in the sample) created after April 21. (where Realm was created with 10.16.0). I'm comparing against a significant amount of users who signed up in March, February, January, etc:
I will increase the sample size and see if the pattern still holds. |
|
Thanks for your investigation @martinpoulsen, that is indeed very interesting. I'll focus my testing in on what happens when a Realm gets upgraded from 10.10.1 to 10.16.0 and let you know if I find anything |
|
Thanks @martinpoulsen! I'm wondering if the "app was in background when it happened" element is a useful clue, or irrelevant... does your app frequently write to Realm when it's in the foreground too, or would the normal use case be with it in the background? |
|
@tomduncalf it's not rare that the app writes to Realm while in foreground - e.g. when updating recorded trips (every trip is usually edited one or more times) or when the user creates trips by means of just inputting start/stop location rather than tracking GPS data. But most frequently we write GPS data to realm while the app is in the background, so I'm not sure if it's just a coincidence that the reports I have seen mostly happens in background. I will follow up and investigate this more closely. |
|
Hi @tomduncalf, I have indexed the crash reports on whether the app is in foreground/background and they happen during app launch.
The other crashes look similar in distribution though the relative amount of crashes in foreground is a little bit higher for In case it's interesting, here are a few stack traces from the SIGABRT_foreground_1.txt |
|
Thanks @martinpoulsen – seems like a pretty strong correlation there! I wonder if there's anything else that the cohort of users who are experiencing this issue have in common – for example are they all in the same country? Might sound weird but I've dealt with some really strange issues in my time and country was part of the cause of one so I never rule anything out, haha |
|
Hi @tomduncalf. I have also been wondering about what those users who experience the crashes could have in common but haven't found anything yet. The distribution of countries match our relative presence in those markets:
Btw, I am now looking at all ~850 users who experienced the
|
|
Leaving some comments here to be updated of progress as we have similar issues with random crashes in background and foreground, or even corrupted realm instances. However, we are using the beta/hermes version. |
|
@martinpoulsen Thank you for providing all the information. Did you end up working around this issue? Does upgrade work? I'm wondering if we should upgrade to v10.19.5. |
|
@0x079 We still face this issue and haven't found a way to work around it besides telling our iOS customers who contact us about the crash that they need to delete and reinstall our app. We haven't tried upgrading Realm version recently. |
|
@tomduncalf just wanted to let you know that we have observed several instances of these crashes for users who only used the app with realm v. 10.16.0. There certainly seems to be a correlation between getting these crashes and going through a realm version migration, but it doesn't seem to be a necessary condition. |
|
Thanks for the update @martinpoulsen and I'm sorry to hear you are still having issues. I'm going to discuss this with the team again in light of that information, I'm afraid we didn't make any progress on tracking down the root cause so far. |
|
Do we know how prevalent this issue is? |
|
@Pingou we've not had any other reports of this exact issue. We are aware of a few reports which seem related to memory leaks or crashes related to accessing freed memory, the ones we are tracking are:
The difficulty with all of these has been that they are relatively very rare, and we have been unable to reproduce them – we've tried various approaches to stress testing Realm JS and have not been successful. Also note that some of these are quite old and could have been fixed, but we are unable to prove that. Our upcoming Hermes release changes quite a few things related to object ownership etc. in our C++ layer, so we hope should solve many such issues. In the case of Martin's issue, we are not sure exactly what the cause is and again have not been able to reproduce it despite stress testing opening a Realm JS app in the background repeatedly, upgrading Realm between various versions etc. This is something we are still investigating, obviously any crashes are a concern for us, but what I would say is that the volume of reports of such crashes is very low compared to the number of users we have. This issue is the first one that I am aware of where there has been any hypothetical link to Realm upgrades causing issues, and we've so far not been able to prove that. |
|
@tomduncalf Hi Tom, thank you very much for the detailed reply. |
|
@Pingou 99.5% of these issues happen in the background in our case (see my update in this issue from May 23). But our app will continuously try to launch itself if terminated so that might be the reason why so many of the crashes happen while in background. |
|
@martinpoulsen Could you elaborate a little on what you mean by
please? |
|
@tomduncalf sure, our app is (under certain configurations) aiming to be active at all times to monitor the activity of the user (specifically to detect whether the user is driving). This includes launching itself from process death. So what happens for some of our users with these realm issues is that the app get's stuck in a loop where it crashes during app startup every time it tries to launch itself. |
|
Ah OK, how do you go about reopening the app when it has been terminated? |
|
We are listening for various events such as location updates, phone movements, power source change and iBeacon region state changes which will launch the app if terminated before the event gets delivered. |
|
Ahh I see, thanks! |
|
@martinpoulsen Related to iBeacon, the trick mentioned in realm/realm-core#5731 (comment) might help you. We are considering adding a configuration option to do it so you don't have to deal with Objective C or Swift code. |
|
Thanks for the pointer @kneth, we will definitely look into that. |
|
@kneth We have included the file permission trick in our latest release but unfortunately it seems to have no effect on the nr of realm crashes we see. |
@martinpoulsen Thank you for the update. |
|
We have released v11.0.0 which has a complete rewrite of the integration with JSC. I am closing the issue, and ask you to upgrade. If the issue is still observed, please create a new issue. |
Since we upgraded Realm from 10.10.1 => 10.13.0, we have experienced a big drop in stability on iOS and have been hit by what appears to be native memory management issues. Our crash reporting tool (Bugsnag) reports that the iOS app session stability dropped from ~ 99% to ~ 90%. We didn't update any other dependencies (or made any native code changes) in the release where we updated Realm and started seeing these issues. This is what the top crashes looked like with Realm 10.13.0:
Then we tried upgrading Realm from 10.13.0 => 10.16.0 (again without making changes to other app dependencies or making native code changes) but unfortunately the session stability did not improve. This is what the top crashes look like with Realm after the Realm upgrade to 10.16.0 (sorry about the non-symbolicated traces):
Both app versions with Realm 10.13.0 and 10.16.0 show realm::NoSuchTable and realm::KeyNotFound crashes. However, the most frequently reported crash comes in the form of a more general SIGABRT error report which it's my understanding could be rooted in any native module. Here's an example stack trace from the reported SIGABRT crash:
Here are a few crash reports retrieved through XCode which mention Realm in the stack trace - they are all from the app version which uses realm v. 10.16.0:
2022-04-23_11-24-19.2673_-0500-2fd0a5349d21e8624bedea7cadd05249a7ef257e.txt
2022-04-24_09-49-16.5911_+0800-5d54e08447eada3d190a81ad0583993fbed9e685.txt
2022-04-24_11-06-35.7773_+0800-1dc9a5cb656aa8f4597272dea4f4e2fd2fe4b7d3.txt
2022-04-24_11-51-44.3257_-0500-e80fd319087391b20fa73e248a7f66e80b34afa0.txt
2022-04-24_13-26-48.1968_+0200-00cc3ee3534c38956fa8190f226d1dd5d1a04af5.txt
2022-04-25_10-50-14.0976_+0200-10370900cb31f02fbe46c189a186c85709ec1532.txt
We realise that this issue is somewhat vague but unfortunately we haven't yet found a way to reproduce the crashes. Our hope is that some of the attached crash reports will be meaningful for the Realm team and either point to a Realm problem - or to something which we could be doing wrong but which only causes problems in these new realm versions.
We use Realm with react-native v. 0.65.2 (without Hermes)
The text was updated successfully, but these errors were encountered: