crashes watchOS app after foreground ("terminated due to signal 9")

[bo46]

Goals

Realm doesn't terminate watchOS app

Expected Results

above

Actual Results

v.5.4.7

• the watchOS's app 1 out of 5 times can terminate when reading\writing in background when active Workout session

v.5.5.0

• the same code as above , crash every time on start of the app

Steps for others to Reproduce

the watchos app has own database

start the watchOS app > minimize the application by touching the screen with your palm > run app (crashes every time for 5.5.0 , not so often for 5.4.7)

Code Sample

crash logs

v.5.4.7

Incident Identifier: C168003D-88A1-4EE7-8847-5E62085BD24B
CrashReporter Key: 1593602e5aa58cf94775ff6286bfdfeb3864687d
Hardware Model: Watch5,1
Process: WatchCompanion Extension [423]
Code Type: ARM (Native)
Role: unknown
Parent Process: launchd [1]

OS Version: Watch OS 7.0 (18R382)
Release Type: User
Baseband Version: n/a
Report Version: 104

Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc
Triggered by Thread: 0

Thread 0 name: Dispatch queue: com.app.realm.thread.conccurent
Thread 0 Crashed:
0 libsystem_malloc.dylib 0x1c8a1fbc tiny_size + 396
1 libsystem_malloc.dylib 0x1c890c2c szone_size + 64
2 libsystem_malloc.dylib 0x1c890c2c szone_size + 64
3 libsystem_malloc.dylib 0x1c89b7f8 free + 116
4 Realm 0x040e0820 0x3acc000 + 6375456
5 Realm 0x040e0814 0x3acc000 + 6375444
6 Realm 0x040e0820 0x3acc000 + 6375456
7 Realm 0x040e0814 0x3acc000 + 6375444
8 Realm 0x040e0820 0x3acc000 + 6375456
9 Realm 0x040e0820 0x3acc000 + 6375456
10 Realm 0x040e0820 0x3acc000 + 6375456
11 Realm 0x040e0820 0x3acc000 + 6375456
12 Realm 0x040e0820 0x3acc000 + 6375456
13 Realm 0x041198ac 0x3acc000 + 6609068
14 Realm 0x04112740 0x3acc000 + 6580032
15 Realm 0x041122c4 0x3acc000 + 6578884
16 Realm 0x041128e0 0x3acc000 + 6580448
17 Realm 0x03c06b68 0x3acc000 + 1289064
18 Realm 0x03ede4f0 0x3acc000 + 4269296
19 Realm 0x03df9768 0x3acc000 + 3331944
20 RealmSwift 0x0519fca8 0x5150000 + 326824
21 RealmSwift 0x0519f4a4 0x5150000 + 324772
22 ...ingWatchCompanion Extension 0x02ec58b0 Realm.safeWrite(:) + 219312 (RealmExtension.swift:18)
23 ...ingWatchCompanion Extension 0x02f1a300 closure #1 in static WorkoutHistoryDataManager.update(:) + 566016 (WorkoutHistoryDataManager.swift:176)
24 ...ingWatchCompanion Extension 0x02ea6f00 thunk for @callee_guaranteed () > () + 93952 (<compilergenerated>:0)
25 ...ingWatchCompanion Extension 0x02ea6f6c thunk for @escaping @callee_guaranteed () > () + 94060 (<compilergenerated>:0)
26 libdispatch.dylib 0x1c67b240 _dispatch_client_callout + 16
27 libdispatch.dylib 0x1c689118 dispatch_sync_invoke_and_complete + 60
28 ...ingWatchCompanion Extension 0x02f16a54 static WorkoutHistoryDataManager.update(:) + 551508 (WorkoutHistoryDataManager.swift:174)
29 ...ingWatchCompanion Extension 0x02f17520 closure #1 in static WorkoutHistoryDataManager.saveProgramHistory(seconds:avgSpeed:maxSpeed:) + 554272 (WorkoutHistoryDataManager.swift:93)
30 ...ingWatchCompanion Extension 0x02f15de8 thunk for @callee_guaranteed () > (@owned WorkoutHistory?, @error @owned Error) + 548328 (<compilergenerated>:0)
31 ...ingWatchCompanion Extension 0x02f1a61c thunk for @callee_guaranteed () > (@owned WorkoutHistory?, @error @owned Error)partial apply + 566812 (<compilergenerated>:0)
32 libswiftDispatch.dylib 0x4611455c partial apply for thunk for @callee_guaranteed () -> + 42332 (@out A, @error @owned Error) + 20
33 libswiftDispatch.dylib 0x46112f0c closure #1 in closure #1 in OS_dispatch_queue._syncHelper+ 36620 (fn:execute:rescue:) + 108
34 libswiftDispatch.dylib 0x4611457c partial apply for closure #1 in closure #1 in OS_dispatch_queue._syncHelper+ 42364 (fn:execute:rescue:) + 20
35 libswiftDispatch.dylib 0x461144cc partial apply for thunk for @callee_guaranteed () -> + 42188 () + 20
36 libswiftDispatch.dylib 0x46114784 thunk for @callee_guaranteed () -> + 42884 ()partial apply + 12
37 libswiftDispatch.dylib 0x46112cbc thunk for @escaping @callee_guaranteed () -> + 36028 () + 20
38 libdispatch.dylib 0x1c67b240 _dispatch_client_callout + 16
39 libdispatch.dylib 0x1c689118 _dispatch_sync_invoke_and_complete + 60
40 libswiftDispatch.dylib 0x46113750 implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync+ 38736 (execute:) + 152
41 libswiftDispatch.dylib 0x46113d0c partial apply for implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync+ 40204 (execute:) + 40
42 libswiftDispatch.dylib 0x46112dac OS_dispatch_queue._syncHelper+ 36268 (fn:execute:rescue:) + 228
43 libswiftDispatch.dylib 0x461133a4 OS_dispatch_queue.sync+ 37796 (execute:) + 140
44 ...ingWatchCompanion Extension 0x02f170e0 static WorkoutHistoryDataManager.saveProgramHistory(seconds:avgSpeed:maxSpeed:) + 553184 (WorkoutHistoryDataManager.swift:77)
45 ...ingWatchCompanion Extension 0x030aa3d4 UIProgramWalkController.finishedTrainigDay() + 2204628 (UIProgramWalkController.swift:263)
46 ...ingWatchCompanion Extension 0x030a9b2c UIProgramWalkController.sessionWasFinished() + 2202412 (UIProgramWalkController.swift:252)
47 ...ingWatchCompanion Extension 0x030a85f8 closure #1 in UIProgramWalkController.startSessionTimerWith(seconds:) + 2196984 (UIProgramWalkController.swift:199)
48 ...ingWatchCompanion Extension 0x02f2c884 thunk for @escaping @callee_guaranteed (@guaranteed NSTimer) > () + 641156 (<compilergenerated>:0)
49 Foundation 0x1d92395c __NSFireTimer + 64
50 CoreFoundation 0x1cc49378 CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION

• 28
  51 CoreFoundation 0x1cc48f7c __CFRunLoopDoTimer + 992
  52 CoreFoundation 0x1cc48478 __CFRunLoopDoTimers + 324
  53 CoreFoundation 0x1cc428b0 __CFRunLoopRun + 1896
  54 CoreFoundation 0x1cc41c28 CFRunLoopRunSpecific + 572
  55 GraphicsServices 0x20dce4c0 GSEventRunModal + 160
  56 UIKitCore 0x3df631fc -[UIApplication _run] + 1104
  57 UIKitCore 0x3df68750 UIApplicationMain + 140
  58 WatchKit 0x3272cbfc WKExtensionMain + 760
  59 libdyld.dylib 0x1c6bc94c start + 4

Thread 1 name: com.apple.uikit.eventfetch-thread
Thread 1:
0 libsystem_kernel.dylib 0x1c82e824 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x1c82dc80 mach_msg + 72
2 CoreFoundation 0x1cc480b4 __CFRunLoopServiceMachPort + 376
3 CoreFoundation 0x1cc425e8 __CFRunLoopRun + 1184
4 CoreFoundation 0x1cc41c28 CFRunLoopRunSpecific + 572
5 Foundation 0x1d7c2680 -[NSRunLoop+ 30336 (NSRunLoop) runMode:beforeDate:] + 228
6 Foundation 0x1d7c2560 -[NSRunLoop+ 30048 (NSRunLoop) runUntilDate:] + 88
7 UIKitCore 0x3dffd2e0 -[UIEventFetcher threadMain] + 152
8 Foundation 0x1d92293c NSThread__start

• 844
  9 libsystem_pthread.dylib 0x1c8cdf04 _pthread_start + 288
  10 libsystem_pthread.dylib 0x1c8d65c8 thread_start + 8

Thread 2 name: Realm notification listener
Thread 2:
0 libsystem_kernel.dylib 0x1c854304 kevent + 8
1 Realm 0x03b367ac 0x3acc000 + 436140
2 Realm 0x03b3b8c0 0x3acc000 + 456896
3 Realm 0x03b3b834 0x3acc000 + 456756
4 Realm 0x03b3b74c 0x3acc000 + 456524
5 Realm 0x03b3aaa8 0x3acc000 + 453288
6 libsystem_pthread.dylib 0x1c8cdf04 _pthread_start + 288
7 libsystem_pthread.dylib 0x1c8d65c8 thread_start + 8

Thread 3:
0 libsystem_pthread.dylib 0x1c8d65b4 start_wqthread + 0

Thread 4:
0 libsystem_pthread.dylib 0x1c8d65b4 start_wqthread + 0

Thread 0 crashed with ARM Thread State (64-bit):
x0: 0x0000000003680040 x1: 0x0000000014b94e80 x2: 0x0000000014b00000 x3: 0x0000000014b91400
x4: 0x0000000000000020 x5: 0x0000000014b911c0 x6: 0x0000000000000030 x7: 0x000000001cdadca8
x8: 0x0000000000000002 x9: 0x0000000014b93dc0 x10: 0x0000000003688500 x11: 0x0000000000000500
x12: 0x0000000010100001 x13: 0x000000000000090f x14: 0x0000000000000001 x15: 0x00000000d500dd15
x16: 0x000000001c89b784 x17: 0x0000000000028480 x18: 0x0000000000000000 x19: 0x0000000014b94e80
x20: 0x000000000367c000 x21: 0x0000000014bed2f0 x22: 0x0000000014bed2f0 x23: 0x000000004c997000
x24: 0x000000004c997000 x25: 0x0000000014050c48 x26: 0x0000000000a00000 x27: 0x0000000014050a48
x28: 0x0000000014050a18 fp: 0x000000000363ffa0 lr: 0x000000001c890c2c
sp: 0x000000000363ff90 pc: 0x000000001c8a1fbc cpsr: 0x80000000
esr: 0x56000080 Address size fault

---

v.5.5.0

Incident Identifier: 448ADEAE-1544-4F60-9F88-C34906D5D999
CrashReporter Key: 1593602e5aa58cf94775ff6286bfdfeb3864687d
Hardware Model: Watch5,1
Process: WatchCompanion Extension [501]
Code Type: ARM (Native)
Role: unknown
Parent Process: launchd [1]

OS Version: Watch OS 7.0 (18R382)
Release Type: User
Baseband Version: n/a
Report Version: 104

Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc
Triggered by Thread: 0

Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0 libsystem_kernel.dylib 0x1c82e824 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x1c82dc80 mach_msg + 72
2 CoreFoundation 0x1cc480b4 __CFRunLoopServiceMachPort + 376
3 CoreFoundation 0x1cc425e8 __CFRunLoopRun + 1184
4 CoreFoundation 0x1cc41c28 CFRunLoopRunSpecific + 572
5 GraphicsServices 0x20dce4c0 GSEventRunModal + 160
6 UIKitCore 0x3df631fc -[UIApplication _run] + 1104
7 UIKitCore 0x3df68750 UIApplicationMain + 140
8 WatchKit 0x3272cbfc WKExtensionMain + 760
9 libdyld.dylib 0x1c6bc94c start + 4

Thread 1:
0 libsystem_pthread.dylib 0x1c8d65b4 start_wqthread + 0

Thread 2 name: com.apple.uikit.eventfetch-thread
Thread 2:
0 libsystem_kernel.dylib 0x1c82e824 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x1c82dc80 mach_msg + 72
2 CoreFoundation 0x1cc480b4 __CFRunLoopServiceMachPort + 376
3 CoreFoundation 0x1cc425e8 __CFRunLoopRun + 1184
4 CoreFoundation 0x1cc41c28 CFRunLoopRunSpecific + 572
5 Foundation 0x1d7c2680 -[NSRunLoop+ 30336 (NSRunLoop) runMode:beforeDate:] + 228
6 Foundation 0x1d7c2560 -[NSRunLoop+ 30048 (NSRunLoop) runUntilDate:] + 88
7 UIKitCore 0x3dffd2e0 -[UIEventFetcher threadMain] + 152
8 Foundation 0x1d92293c NSThread__start

• 844
  9 libsystem_pthread.dylib 0x1c8cdf04 _pthread_start + 288
  10 libsystem_pthread.dylib 0x1c8d65c8 thread_start + 8

Thread 3 name: Realm notification listener
Thread 3:
0 libsystem_kernel.dylib 0x1c854304 kevent + 8
1 Realm 0x031e5b98 0x317c000 + 433048
2 Realm 0x031eacac 0x317c000 + 453804
3 Realm 0x031eac20 0x317c000 + 453664
4 Realm 0x031eab38 0x317c000 + 453432
5 Realm 0x031e9e94 0x317c000 + 450196
6 libsystem_pthread.dylib 0x1c8cdf04 _pthread_start + 288
7 libsystem_pthread.dylib 0x1c8d65c8 thread_start + 8

Thread 4:
0 libsystem_pthread.dylib 0x1c8d65b4 start_wqthread + 0

Thread 5:
0 libsystem_pthread.dylib 0x1c8d65b4 start_wqthread + 0

Thread 0 crashed with ARM Thread State (64-bit):
x0: 0x0000000010004005 x1: 0x0000000007000806 x2: 0x0000000000000000 x3: 0x0000000000000c00
x4: 0x0000000000001603 x5: 0x00000000ffffffff x6: 0x0000000000000000 x7: 0x00000000130800d0
x8: 0x00000000fffffbbf x9: 0x0000000007000906 x10: 0x000000007a2c0002 x11: 0x000000007a2c0002
x12: 0x0000000000000000 x13: 0x0000000000000000 x14: 0x0000000000000000 x15: 0x1c45e24d49d44f9a
x16: 0xffffffffffffffe1 x17: 0x0000000000000000 x18: 0x0000000000000000 x19: 0x0000000000000000
x20: 0x00000000ffffffff x21: 0x0000000000001603 x22: 0x0000000000000c00 x23: 0x0000000002f82be4
x24: 0x0000000007000806 x25: 0x0000000000000000 x26: 0x0000000007000806 x27: 0x0000000000001603
x28: 0x0000000002f82b9c fp: 0x0000000002f82aa0 lr: 0x000000001c82dc80
sp: 0x0000000002f82a50 pc: 0x000000001c82e824 cpsr: 0x60000000
esr: 0x56000080 Address size fault

Realm framework version: 5.4.7 and 5.5.0

Xcode version: 12.0.1

iOS/OSX version: iOS 14 , watchOS 7.0

[TheHmmka]

Hey, guys! The same here! =((

[ejm01]

minimize the application by touching the screen with your palm -> run app (crashes every time for 5.5.0

@bo46 @TheHmmka Would either of you be able to share a small sample reproduction in a .zip? I'm working on a simulator. So hopefully it'll clear up if the device is related.

[FelixLisczyk]

I can confirm this issue on my Apple Watch SE running watchOS 7.0.2. The affected Realm versions for my app are 5.4.8 and 5.5.0

The problem occurs when I rotate my wrist to turn off the screen. I couldn't reproduce it in the simulator yet.

The crash does not occur on my older Apple Watch Series 2 running watchOS 6.2.8 (Realm version 5.4.8).

I've started working on a sample project but couldn't recreate the crash yet. I'll attach it anyway in case someone wants to continue working on it.

WatchRealmCrash.zip

[bo46]

@ericjordanmossman
Unfortunately , I can't send the project where is I have the crash. I would check this crash on simulator if I could build it. It has not been building since Xcode 12.0 beta 5 (100 errors for Realm-watchOS). One time worked arm64 for 'Excluded Architecture', but not now.

UPD:
The crash reproduces only for watchOS 7.0 and later , for watchOS 6 works fine

[FelixLisczyk]

According to the error code 0xdead10cc, the app is terminated by watchOS because it held on to a file lock or sqlite database lock during suspension.

https://developer.apple.com/forums/thread/126438

There are a few related issues with this error code: #6749, #6671 and #5904

I can confirm that the latest Realm version 10.1.0 is still affected. The crash only occurs when I read data from Realm after the app launch and then suspend it.

Unfortunately, I was not able to reproduce the crash with a sample project yet.

[obrhoff]

Unfurtunatly I'm also affected by this issue. Did you find a workaround yet?

[gongzhang]

Totally agree @felixII. The app is terminated because it held a file lock (default.realm.lock) during suspension.

😂😂😂 I can't believe this issue is happening again on watchOS 7! I've previously been stumped by #6671 on the iOS 14 beta 1-6. (eventually Apple fixed it in iOS 14 beta 7.)

Once again, this issue can only be reproduced on a physical watch running watchOS 7 (100% reproduced). It can't be reproduced on a simulator. Just open a realm with default configuration, then go to background. The app will crash. You can learn the real cause through oslog:

Terminating with context: <RBSTerminateContext| domain:15 code:0xDEAD10CC explanation:[....watchkitapp>:356]
was suspended with locked system files:
/var/mobile/Containers/Data/PluginKitPlugin/35D506F4-77AE-4492-9978-C9CAB1328888/Documents/default.realm.lock
not in allowed directories:
/var/mobile/Containers/Data/Application/0982F723-7515-4AD9-BF36-9E5901890613/tmp
/var/mobile/Containers/Data/Application/0982F723-7515-4AD9-BF36-9E5901890613 reportType:CrashLog maxTerminationResistance:Interactive>

You can see our realm file is under /var/mobile/Containers/Data/PluginKitPlugin. That's correct because our code is running in the WatchKit extension. However, the log told us the system only allows file lock in /var/mobile/Containers/Data/Application/... This is totally wrong, because we can't even access the directory from extension!. 😤

Unfortunately, I don't know how to workaround it yet.

BTW, I don't understand why the previous filelock emulation (#3862) in realm core didn't fix this problem on watchOS. Maybe @finnschiermer can answer?

[obrhoff]

I think I successfully reproduced the issue with the attached example project. Can anyone confirm it? As noted above, it only happens on a physical watch with watchOS7 (including watchOS 7.1)

realm-lock-issue.zip

[tgoyne]

realm/realm-core#4049 should fix this, but I don't have a watch available to actually test it myself.

[obrhoff]

Unfortunately I can't confirm that this got fixed with 10.1.2. (See the updated example). Still seeing:

[application<com.obrhoff.realm-lock-issue.watchkitapp>:323] Terminating with context: <RBSTerminateContext| domain:15 code:0xDEAD10CC explanation:[application<com.obrhoff.realm-lock-issue.watchkitapp>:323] was suspended with locked system files:
/var/mobile/Containers/Data/PluginKitPlugin/0B1D0972-BEA3-4441-8DE7-8880B6151333/Documents/default.realm.lock
not in allowed directories:
/var/mobile/Containers/Data/Application/BBA1193C-7AFF-46E0-95A1-D83D3F252EE3
/var/mobile/Containers/Data/Application/BBA1193C-7AFF-46E0-95A1-D83D3F252EE3/tmp reportType:CrashLog maxTerminationResistance:Interactive>

[gongzhang]

Keep trying workaround...

On watchOS, if I can be sure that only one process in my app can operate realm (that is the watchkit extension process), can I safely release this file lock on my own?

[tgoyne]

When only one process is involved it should not be possible to hit the code path where we check the file lock while the file is actually locked. From 5.0.0 until 5.2.0 we actually did have a bug where we'd open the file twice and run into problems if the file was forcibly unlocked, but it was a pretty hard to hit edge case (a ThreadSafeReference had to be the only thing keeping the file open).

[gongzhang]

@tgoyne Thanks for the response, this is very useful information for me. 😊

(In the development environment, file lock in my app is temporarily deleted so that daily testing can continue... also added #warning(...) to remind my self to revert this later)

[gongzhang]

Confirmed REALM_FILELOCK_EMULATION flag was set correctly for Apple Watch target in core v10.1.1. However, the system still reports that the xxx.realm.lock file is locked.

@tgoyne What do you think the problem might be? Is it related to the implementation of bool File::lock(bool exclusive, bool non_blocking), or is it related to watchOS itself? I tested realm-cocoa v10.1.2 on my watch, which is running watchOS 7.1. The app doesn't crash anymore. More testing seems to be needed.

[FelixLisczyk]

I can confirm that the crash no longer occurs in my app with the latest Realm version (10.1.4) on watchOS 7.1. 👍

[obrhoff]

I can also confirm that after retesting the example. Looks good. Not sure why my last testing was still producing it.

[bo46]

Thanks! Bug doesn't reproduce (Realm v.10.1.3) 👍