Crash When Application Terminates in realm::util::set_termination_notification_callback

[LeffelMania]

Goals

Gracefully handle the system terminating our application.

Expected Results

System terminates our application peacefully.

Actual Results

We have a frequent crash reported in Fabric that Realm crashes in its termination handler.

Raw trace:

Crashed: com.apple.main-thread
0  libsystem_kernel.dylib         0x333e6df0 __pthread_kill + 8
1  libsystem_pthread.dylib        0x33465cbb pthread_kill + 62
2  libsystem_c.dylib              0x33385891 abort + 76
3  Remind101                      0x5f6edd realm::util::set_termination_notification_callback(void (*)(char const*)) + 799592
4  Remind101                      0x5f7027 realm::util::terminate_internal(std::__1::basic_stringstream<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) + 799922
5  Remind101                      0x5f7237 realm::util::terminate(char const*, char const*, long) + 800450
6  Remind101                      0x5f783d realm::util::Mutex::destroy_failed(int) + 801992
7  libsystem_malloc.dylib         0x3340f98d szone_free + 1268
8  libsystem_c.dylib              0x3338612f __cxa_finalize_ranges + 298
9  libsystem_c.dylib              0x33345abd exit + 12
10 UIKit                          0x27dbf7a7 -[UIApplication _terminateWithStatus:] + 406
11 UIKit                          0x27fa6149 -[UIApplication _handleApplicationDectivationWithScene:shouldForceExit:transitionContext:completion:] + 2416
12 UIKit                          0x27fa16e7 -[UIApplication workspaceShouldExit:] + 166
13 FrontBoardServices             0x2b2a3ec9 __31-[FBSSerialQueue performAsync:]_block_invoke_2 + 16
14 CoreFoundation                 0x24682db5 __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
15 CoreFoundation                 0x24682079 __CFRunLoopDoBlocks + 216
16 CoreFoundation                 0x24680817 __CFRunLoopRun + 790
17 CoreFoundation                 0x245ccf31 CFRunLoopRunSpecific + 476
18 CoreFoundation                 0x245ccd43 CFRunLoopRunInMode + 106
19 GraphicsServices               0x2c0a7201 GSEventRunModal + 136
20 UIKit                          0x27d9e879 UIApplicationMain + 1440
21 Remind101                      0x1a5ba3 main (main.m:16)
22 libdyld.dylib                  0x3331faaf start + 2

Steps to Reproduce

No repro steps unfortunately. All of our breadcrumbs for this crash end with us logging a call to applicationWillTerminate:, which is our last line of code in that function.

Code Sample

N/A

Version of Realm and Tooling

ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G31

/Applications/Xcode.app/Contents/Developer
Xcode 7.3.1
Build version 7D1014

/Users/alexleffelman/.rbenv/shims/pod
1.0.1
Realm (1.0.2)

/bin/bash
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)

/usr/local/bin/carthage
0.6.3
(not in use here)

/usr/bin/git
git version 2.7.4 (Apple Git-66)

[LeffelMania]

I like to believe this isn't affecting our end users because it's happening when the app terminates and is therefore theoretically backgrounded, but at the same time it's a nontrivial hit to our crash-free rate in Fabric and generates noise for us to sift through.

[bdash]

@LeffelMania, in the past we've seen similar errors when an application exits while a thread is actively writing to a Realm. The Fabric link you provided only shows the crash log for a single instance of the crash so it's difficult to tell one way or another if this is what is happening here. Does this seem like something your app could be doing? Do you have access to more than the single crash log I can see?

[LeffelMania]

@bdash I'm not sure I can get you access to the full set of crash reports, but that definitely sounds like something that can happen with our app. Other instances of the crash I'm able to view do seem to be writing on other threads when this occurs.

Other thread activity from another report:
Thread 2

RLMRealm notification listener
0  libsystem_kernel.dylib         0x21a9adac kevent + 24
1  Remind101                      0x51ee59 realm::_impl::ExternalCommitHelper::listen() (external_commit_helper.cpp:200)
2  Remind101                      0x51f13f std::__1::__async_func<realm::_impl::ExternalCommitHelper::ExternalCommitHelper(realm::_impl::RealmCoordinator&)::$_0>::operator()() (external_commit_helper.cpp:159)
3  Remind101                      0x51f057 std::__1::__async_assoc_state<void, std::__1::__async_func<realm::_impl::ExternalCommitHelper::ExternalCommitHelper(realm::_impl::RealmCoordinator&)::$_0> >::__execute() (future:1026)
4  Remind101                      0x51f2e3 std::__1::__thread_proxy<std::__1::tuple<void (std::__1::__async_assoc_state<void, std::__1::__async_func<realm::_impl::ExternalCommitHelper::ExternalCommitHelper(realm::_impl::RealmCoordinator&)::$_0> >::*)(), std::__1::__async_assoc_state<void, std::__1::__async_func<realm::_impl::ExternalCommitHelper::ExternalCommitHelper(realm::_impl::RealmCoordinator&)::$_0> >*> >(void*, void*) (__functional_base:383)
5  libsystem_pthread.dylib        0x21b50c7f _pthread_body + 138
6  libsystem_pthread.dylib        0x21b50bf3 _pthread_start + 110
7  libsystem_pthread.dylib        0x21b4ea08 thread_start + 8

Thread 8

NSOperationQueue 0x1778dbd0 :: NSOperation 0x1898b2a0 (QOS: USER_INTERACTIVE)
0  libsystem_kernel.dylib         0x21a9a6f8 flock + 8
1  Remind101                      0x60f223 realm::util::File::lock(bool, bool) + 782094
2  Remind101                      0x6e227f realm::SharedGroup::do_begin_write() + 696800
3  Remind101                      0x5e4bad auto realm::_impl::transaction::begin(realm::SharedGroup&, realm::BindingContext*, bool)::$_1::operator()<(anonymous namespace)::TransactLogValidator>((anonymous namespace)::TransactLogValidator&&) const (group_shared.hpp:964)
4  Remind101                      0x5df551 realm::_impl::transaction::begin(realm::SharedGroup&, realm::BindingContext*, bool) (vector:449)
5  Remind101                      0x5dd469 realm::Realm::begin_transaction() (shared_realm.cpp:335)
6  Remind101                      0x5c9e0b -[RLMRealm beginWriteTransaction] (RLMRealm.mm:478)
7  Remind101                      0x1505ed -[RDRealmMappingOperation performRealmMappingForJSON:mapper:request:realm:error:] (RDRealmMappingOperation.m:83)
8  Remind101                      0x15047f -[RDRealmMappingOperation main] (RDRealmMappingOperation.m:55)
9  Foundation                     0x22585f8f -[__NSOperationInternal _start:] + 774
10 Foundation                     0x22634b0d __NSOQSchedule_f + 192
11 libdispatch.dylib              0x219b9e7f _dispatch_queue_drain + 1762
12 libdispatch.dylib              0x219b2e17 _dispatch_queue_invoke + 282
13 libdispatch.dylib              0x219bb20d _dispatch_root_queue_drain + 400
14 libdispatch.dylib              0x219bb07b _dispatch_worker_thread3 + 94
15 libsystem_pthread.dylib        0x21b4ee0d _pthread_wqthread + 1024
16 libsystem_pthread.dylib        0x21b4e9fc start_wqthread + 8

Thread 11

NSOperationQueue 0x1778dbd0 :: NSOperation 0x189f7e00 (QOS: USER_INTERACTIVE)
0  libsystem_platform.dylib       0x21b4a926 _platform_memmove + 37
1  Remind101                      0x60cfd5 realm::util::EncryptedFileMapping::refresh_page(unsigned long) + 773312
2  Remind101                      0x6209af realm::util::encryption_read_barrier(void const*, unsigned long, realm::util::EncryptedFileMapping*, unsigned long (*)(char const*)) + 862
3  Remind101                      0x61e503 realm::SlabAlloc::do_translate(unsigned long) const + 844270
4  Remind101                      0x6d7299 realm::Group::update_refs(unsigned long, unsigned long) + 651770
5  Remind101                      0x6e2b15 realm::SharedGroup::commit_and_continue_as_read() + 698998
6  Remind101                      0x5df86f realm::_impl::transaction::commit(realm::SharedGroup&, realm::BindingContext*) (transact_log_handler.cpp:592)
7  Remind101                      0x5dd02d realm::Realm::commit_transaction() (memory:4013)
8  Remind101                      0x5c9f09 -[RLMRealm commitWriteTransaction:] (RLMRealm.mm:491)
9  Remind101                      0x1507ff -[RDRealmMappingOperation performRealmMappingForJSON:mapper:request:realm:error:] (RDRealmMappingOperation.m:102)
10 Remind101                      0x15047f -[RDRealmMappingOperation main] (RDRealmMappingOperation.m:55)
11 Foundation                     0x22585f8f -[__NSOperationInternal _start:] + 774
12 Foundation                     0x22634b0d __NSOQSchedule_f + 192
13 libdispatch.dylib              0x219b9e7f _dispatch_queue_drain + 1762
14 libdispatch.dylib              0x219b2e17 _dispatch_queue_invoke + 282
15 libdispatch.dylib              0x219bb20d _dispatch_root_queue_drain + 400
16 libdispatch.dylib              0x219bb07b _dispatch_worker_thread3 + 94
17 libsystem_pthread.dylib        0x21b4ee0d _pthread_wqthread + 1024
18 libsystem_pthread.dylib        0x21b4e9fc start_wqthread + 8

So I feel like that's probably the issue. Is there a clean way to tell Realm "Stop whatever you're doing on all threads right now" in our applicationWillTerminate: method?

[vlntb]

Hi @bdash @mrackwitz
We are seeing same type crashes in the same context (during application termination) as described above. I can share more logs if needed.

[bdash]

In order to avoid this crash, I think you'd need to ensure that none of your background threads have open write transactions when the app is exiting. This isn't something that Realm can do for you as we have no knowledge of what your threads are up to, or whether it's safe to interrupt them.

One thing we may be able to do is make it safe to exit while a Realm transaction is in progress. I've written up realm/realm-core#2137 to track doing that. This would result in the write transaction being canceled, so depending on your app's need you might still need to avoid having open write transactions when your app is terminating.

[jpsim]

This was fixed in #4261.