Started experiencing SIGKILL crashes very frequently after iOS 15 launch

[romanrudyy]

How frequently does the bug occur?

Sometimes

Description

We're having a ton of crash reports appearing after iOS 15 release that have not been present before. All of them are SIGKILL crashes and stack traces have nothing nut realm activity. Is this something that is a known issue?

Stacktrace & log output

Multiple different stack traces

Can you reproduce the bug?

Yes, sometimes

Reproduction Steps

The iOS will kill the app eventually

Version

10.7.6

What SDK flavour are you using?

Local Database only

Are you using encryption?

Yes, using encryption

Platform OS and version(s)

iOS 15

Build environment

Xcode version: ...
Dependency manager and version: ...

[romanrudyy]

Could that be due to us using the ancient 10.7.6 version?

[leemaguire]

Hi @romanrudyy you will need to give us your crash reports in order for us to help you further.

[romanrudyy]

Thanks for responding @leemaguire ! We have updated to 10.16, but the crashes seem to keep happening. I'm not fully sure the crashing is caused by Realm, but it seems like a suspect! Thanks!

Riot 07.10.2021, 23-04.crash.zip

Riot 07.10.2021, 17-23.crash.zip

Riot 07.10.2021, 16-08.crash.zip

[romanrudyy]

@leemaguire thanks! have you had a chance to look at the crash logs :)

[romanrudyy]

Some more info:

Since it's a SIGKILL crash it certainly looks like this realm operation takes too much time or resources. And we have most reports as having something to do with backgrounding the app, e.g.:

"I wasn’t getting messages on Discord so I sent a test which went through, and then I switched back to a Facebook conversation, and when I Cmd+Tabbed out of Beeper into Discord, it crashed"

"I got a message saying it crashed but I dont see any issues. Nothing user facing that I can see wrong. Actually working great so far"

This IMO indicates iOS killing the app for not being able to finish the operation in background.

I can't say I'm 100% sure it's realm but it's my best hunch.

All crashes have some sort of major realm operation present: opening, closing, resizing:

0 libsystem_kernel.dylib 0x00000001b9024888 close + 8

1 Realm 0x0000000106b52158 realm::util::File::unlock() + 104

0 libsystem_kernel.dylib 0x00000001baa47ad8 msync + 8

1 Realm 0x0000000108933cf8 realm::util::msync(int, void*, unsigned long) + 220

2 Realm 0x00000001087fd5c4 realm::GroupWriter::commit(unsigned long) + 296

0 libsystem_kernel.dylib 0x00000001b8c256e4 ftruncate + 8

1 Realm 0x0000000104ce2c94 realm::util::File::resize+ 3075220 (long long) + 56

0 libsystem_kernel.dylib 0x00000001baa45398 __munmap + 8

1 Realm 0x000000010867f830 realm::util::munmap+ 3094576 (void*, unsigned long) + 500

2 Realm 0x00000001084e70fc realm::SlabAlloc::detach() + 404

3 Realm 0x0000000108525684 realm::DB::close_internal+ 1676932

0 libsystem_kernel.dylib 0x00000001ba0c7df0 __open + 8

1 libsystem_kernel.dylib 0x00000001ba0c81d4 open + 40 (open-base.c:101)

2 Realm 0x0000000106dbf9d8 realm::util::File::lock+ 3078616 (bool, bool) + 780

[rjvir]

@romanrudyy were you able to find a fix?

[shivam-aditya]

Facing the same issue in my app. All crashes happening on iOS 15 devices only.
Realm version (RealmSwift) - 10.8.1
What SDK flavour are you using? - Local DB only
Are you using encryption? - No

Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: FRONTBOARD; [2343432205]
<RBSTerminateContext| domain:10 code:0x8BADF00D explanation:scene-create watchdog transgression: application<com.eternoinfotech.newshunt>:81007 exhausted real (wall clock) time allowance of 19.02 seconds

Not able to reproduce this issue but this is happening in high numbers

[vj1988]

Seeing the same crashes in ios 15 devices in production, but not able to reproduce it while testing. Please help as the crash numbers are very high

[vj1988]

we are using 10.8.1

[JituDeore]

I am also facing the same issue.
We are trying to insert some data in Realm DB on launch and app started crashing on production this is very specific to iOS 15.

This info might help to check the issue:-
Last time we had different realm version and for this release we have migrated to the new version which is 10.8.1.
And crashes are increased please help asap.
We do not have repro steps.

[jsflax]

Hmm... I took a look at the crash logs. Nothing sticks out to me, and I've ran our test suite locally on an iOS 15 device.

How large is the realm file? Are you using encryption? And notably, if there are locking issues (which it look like there might be), would you be able to detect any permissions issues with the file?

[JituDeore]

Size would be more than 400 MB but we are doing compaction also and limit is set to 300 MB can it be the case.
@jsflax
Attached screenshot for reference.

[jsflax]

Is your app part of an app group? Is this work happening when the app is backgrounded?

[JituDeore]

Yes we do have app groups.
But in crash log its showing most of the users are facing app crash on launch.
Suspecting in didFinishLaunching () in that we are writing some data in Realm file.

[rjvir]

@jsflax Yes, app is a part of an app group, and the Realm file is saved in the app group container.

And yes, all the crashes are happening when the app is backgrounded. It tends to happen immediately when backgrounding the app and opening another app.

[laGrave]

Also faced with same issue. I've found that in my case crashes happens on obj-c apps, but not happens on swift apps.

[rjvir]

To anyone else seeing this issue - I think moving the Realm file to the main document directory rather than the app group container fixed the issue. (Not fully confirmed, but anecdotally have been seeing fewer background crashes in the last ~3 days since attempting it).

Essentially, before initializing Realm on app launch, I perform a 1 time move operation of the Realm file to the main document directory, kind of like this:

let containerDirectory = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: Constants.sharedContainerIdentifier)

let documentsDirectory = try? FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: false)

if let oldContainerRealmURL = containerDirectory?.appendingPathComponent("app.realm") &&
    let realmUrl = directory?.appendingPathComponent("app.realm") &&
    FileManager.default.fileExists(atPath: oldContainerRealmURL.path) {
        try? FileManager.default.moveItem(atPath: oldContainerRealmURL.path, toPath: realmUrl.path)
}

// Initialize realm with the new realm URL here

[laGrave]

I have a very basic default setup, not storing Realm in app group container and still facing those crashes

RLMRealmConfiguration *config = [RLMRealmConfiguration defaultConfiguration];
    config.schemaVersion = 5;
    config.migrationBlock = ^(RLMMigration *migration, uint64_t oldSchemaVersion) {
        if (oldSchemaVersion < 5)
        {}
    };
    [RLMRealmConfiguration setDefaultConfiguration:config];
    [RLMRealm defaultRealm];

[JituDeore]

Same we also have Realm file in main Document directory still our app is crashing.
Our main reason of app crash is below.

Previously we had old version of Realm file 4.4.1.
Now we directly migrated to 10.8.1 version.
And after doing this our app started crashing.
One more thing want to highlight old users had lot of DB data and we are doing compaction also.
@jsflax

[SebC99]

Same here, as reported here: https://developer.apple.com/forums/thread/693623
Due to lock on the realm file in the shared container
Just as in the old #6671

[vj1988]

Not the same problem, @JituDeore is talking about crash on launch and not about crash on background.

[vj1988]

To simply put - When realm is upgraded from 4.* to 10.*, there is a format change in realm. Due to this the realm file is copies to the new format and this takes time on launch. Time out occurs and app crashes for a lot of users. What can we do to avoid this long delay ?

[pavel-ship-it]

Hi all,
I made some experiments on this matter and I was able to reproduce it.
I have 2 cases:
< RBSTerminateContext| domain:10 code:0x8BADF00D explanation:scene-update watchdog transgression: application<com.py.Compact>:6743 exhausted real (wall clock) time allowance of 10.00 seconds ProcessVisibility: Background
and
<RBSTerminateContext| domain:10 code:0x8BADF00D explanation:scene-create watchdog transgression: application<com.py.Compact>:6130 exhausted real (wall clock) time allowance of 19.86 seconds ProcessVisibility: Foreground
They're for entering background and for app start respectively.

How to:

1. Choose 'Edit scheme' in Xcode menu
2. Set 'Run' -> 'Build Configuration' to 'Release'
3. Install the app on the iOS device (just build and run on the device)
4. Now stop the app (You need to launch it without the debugger)
5. Manually run it on your device and do what you need

do something to reproduce the issue.

The crashlog could be found in XCode's Devices and Simulators window.

Pleae let me know if that works for you.

[pavel-ship-it]

@romanrudyy hi. any news on the issue? Can you reproduce the issue using the approach from my previous message?

[laGrave]

Hi all. Is there any information? Still facing hundreds of crashes every day. Can't see them in firebase, but they are visible in AppStoreConnect dashboard and XCode organizer

[bb-git]

We contacted Apple about the iOS 15 issues. What we were told, is that starting from iOS 15 the lifecycle methods do not necessarily need to be called in the order expected from previous iOS version. We found out, that applicationWillTerminate was called without applicationDidFinishLaunching being called. One object wasn't initalized and the app crashed. Also the crash tracker wasn't loaded, so no reports. Hope it helps!

[PetrBodnar]

May be related to https://developer.apple.com/forums/thread/693623

[pavel-ship-it]

@bb-git thank you for information. Can you confirm you were able to fix that?

@PetrBodnar thank you. It seems Apple have a fix for better crash reports on EXC_CRASH (SIGKILL). Maybe anyone can share fresh crashlogs?

@romanrudyy does anything from above helps you to fix the issue?

[IAmMichellis]

Hi @pavel-ship-it , I'm working on the same codebase / issue as @romanrudyy was working on in October.

I found this thread and I believe this to be our issue: we are storing our Realm db in a shared container and accessing it from the app and from a notification extension: https://githubhot.com/repo/realm/realm-swift/issues/7649

Can you speak to what we can do about iOS 15 killing our app because of Realm access in the shared container while in the background?

It sounds to me as though if we prevent all Realm access when the app becomes suspended, then Realm access via the notification extension shouldn't cause crashes (because there will be no file lock). Does that sound right to you?

This is quite an undertaking to fix on my side so I'd love any insight you have - I see you commented in that other thread as well :)

[IAmMichellis]

I've understood and solved the issue for our app:

TL;DR: requesting a RLMRealm via any of the realmWithConfiguration constructors (ex let realm = try RLMRealm.init(configuration: configuration) must always be wrapped in a background task, since if the app is suspended while that constructor is being executed, the app will crash with a SIGKILL on an idle main thread. Any usage of that RLMRealm must also be wrapped.

Explanation:
The issue is that (as of iOS 15) if you are holding a file lock in a shared app container when the app is suspended, the OS kills the app. This is regardless of other running processes. It's a protection mechanism to prevent deadlocks in other processes that might access the file. (This information comes from here (https://developer.apple.com/forums/thread/655225?answerId=632433022#632433022) and here (https://githubhot.com/repo/realm/realm-swift/issues/7649))

I was able to repro the crash behaviour with a simple prototype app, requesting a RLMRealm in a shared container, by inserting a sleep between the application going to background, requesting the RLMRealm, and being suspended. With the right sleep timing, which forces the RLMRealm to be created as the app is being suspended, the crash is 100% reproducible.

We solved this issue with the following RLMRealm wrapper:

@objc class BGSafeRealm: RLMRealm {

    var bgTask: BackgroundTask? // Custom implementation that handles UIApplication.shared.beginBackgroundTask overhead

    @objc static func safeRealm(with configuration: RLMRealmConfiguration) throws -> BGSafeRealm {
        let bgTask = <BGTaskAbstraction>.startBackgroundTask(withName: description != nil ? description! : "")

        do {
            let realm = try BGSafeRealm.init(configuration: configuration)
            realm.bgTask = bgTask
            return realm
        } catch {
            bgTask?.stop()
            throw(error)
        }
    }

    deinit {
        // We need to end the background task *after* realm does whatever it needs to do to cleanup. Dispatch.
        let captureTask = bgTask
        DispatchQueue.global().async {
            captureTask?.stop()
        }
    }
}

It can then be used as follows: the background task lasts until the RLMRealm instance is deallocated:

    RLMRealm *realm = [MXBGSafeRealm safeRealmWith:self.realmConfiguration description:@"Some task" error:nil];

    [realm transactionWithBlock:^{
       // Do something with the realm object
    }];

Important note: if you don't wrap up your Realm activity before the background task expiration handler is called (~30 seconds after backgrounding), you'll crash, for the same reason as before. This gives you time to wrap up what you are doing, but doesn't fix that you can't hold a file lock when the app is suspended.

[bmunkholm]

Looks like this has been resolved. If anyone see this issue again, please create another issue referencing this one. Thanks!