Interprocess: Support accessing realms from multiple processes

[ksmandersen]

I get a hard crash when I try to begin a transaction on a realm that is located in a shared container. The problem only occurs if both the containing app and the app extensions tries to use the realm. Both "apps" can read from the realm just fine but calling beginWriteTransaction crashes with the output:

commit_log.cpp:234: Assertion failed: is_a_known_commit(version)

Not sure If I'm doing something wrong here.

Unfortunately it is quite cumbersome to replicate a clean project with the same issue since you need to create two App IDs for the App and the extension and set up a App Group too.

[timanglade]

I know for a fact one of our users got this working in his app (on beta5 and from Objective-C though). Let us see what we can dig up… You may want to ask around on the mailing-list with more details about your setup

[roballie]

I ran into this same exact problem. The main app can read and write just fine, but once I read with the other app (an app extension), the next beginWriteTransaction() has this exact same crash.

Any findings on how to fix this?

[alazier]

There is currently a bug in the core library that prevents simultaneous access to a Realm from multiple processes, which is what is happening with shared containers. We are working on a fix and we will update this issue once we have a resolution.

[youens]

Running into this as well. 👍

[jpsim]

Small update to inform you that multiprocess support is progressing well and still at the very top of our priority list. We hope to have more to announce soon.

I've also updated the title to better reflect this issue/feature.

[youens]

VERY much appreciated. 👍

[pizthewiz]

#1075 open() not permitted - Access from Keyboard Extension to AppGroup Shared Data seems to be a dup of this too.

[youens]

I don't suppose this will be fixed in 0.88.0 already? (The WIP release notes mention some "threading" things, but wasn't sure if it perhaps was related.) I know it's not reasonable to ask for a date, but I'm just trying to make the call to wait further, or switch to Core Data. Thanks!

[timanglade]

So we have some of that done in core already, with yet more to come. The good news is that basic reading/writing should be enabled very soon. Notifications won’t work across processes for a little while longer. We’re working as fast as we can but had to divert cycles onto some important bugs recently. Our apologies for the delay!

[alazier]

@tgoyne are you working on this? What else do we need before we can make a pr to fix this?

[tgoyne]

tg-interprocess is my WIP branch. It needs a core release with realm/realm-core#592 merged, and it crashes consistently on some of the tests.

[astigsen]

Can you see what is the cause of the crashes? Are they reproducible? And
have they been reported to Finn?

On Mon, Dec 15, 2014 at 10:09 AM, Thomas Goyne [email protected]
wrote:

tg-interprocess is my WIP branch. It needs a core release with
realm/realm-core#592 realm/realm-core#592 merged,
and it crashes consistently on some of the tests.

—
Reply to this email directly or view it on GitHub
#824 (comment).

[tgoyne]

Stuff to do for full interprocess support:

• Interprocess write logs

• Interprocess change notifications

  WIP on the tg-interprocess branch, blocked on Improving wait_for_change realm-core#592)

• Audit new file initialization for multi-process race conditions

  Currently the initialization of new Realm files is guarded by a non-shared lock, so it seems likely that there needs to be some changes there. Might need to grab a write transaction earlier and roll it back if it turns out no changes are needed to avoid having two processes decide they need to initialize things. (Roll back the write transaction on schema init if nothing changed #1281)

• Decide what to do with migrations

  Changing the schema of a file that's already open will make the other process explode horribly. I don't think there's anything we can do to make it work for non-dynamic Realms, but it'd be nice to have it fail more gracefully and maybe even make it work with dynamic Realms (so that you can change the schema while a file is open in the browser). Probably not a blocker to shipping interprocess support as it doesn't apply to the use-case of sharing a file between an extension and an app.

• Browser/Simulator compatibility

  Currently opening a file currently being used by the simulator in the browser doesn't work. It crashes horribly when the simulator is 32-bits due to pthread_mutex being a different size, and simply doesn't wait for changes when the simulator is 64-bit. The browser might need to just always poll the file for changes rather than using notifications. It should be possible to only poll if there's actually another process with the file open, but it'll probably need some help from core.

[GreatApe]

To the last point: it worked fine for me, and with a quick and dirty infinite loop with a wait_for_change and a call to the notification sending method I could even get notifications. This only worked one way at a time, though, I didn't manage to set it up to use the same loop in both the sim and app bindings at the same time, that made it crash horribly.

Without notifications it worked fine, and in both cases I could make modifications in both the app and the simulator.

I used the master, the one that I think got yanked afterwards.

[tgoyne]

I spent some time investigating sharing Realm files between the simulator and the browser yesterday. The current approach of putting a pthread_mutex in shared memory has the problem of that it requires that all processes be using the exact same version of libpthread. This happens to be the case for 64-bit iOS 8.1 and OS X 10.10.0, but this isn't something that can be relied on. I think there's three potential solutions:

1. Build libpthread ourselves and statically link it.

   Would have to be OS X/simulator only and use system libpthread for devices, as it uses private APIs on iOS. Doesn't add support for opening files from 32-bit simulators. I haven't looked into if this is actually a viable thing to do yet.

2. Use some other mutex that can be shared via mmap.

   There doesn't appear to actually be any. sem_init isn't implemented on Apple platforms. sem_open can't be forced to use shared memory under our control. semaphore_t and semctl can't be shared via shared memory. libpthread's internal stuff is based on private APIs we can't use directly on iOS. lck_mtx_t doesn't exist on iOS. Comedy option: a spinlock.

3. Use named semaphores rather than sharing mutexes via mmap.

   Just implement mutexes and condition variables via sem_t and store the semaphore name in the lock file rather than the mutex. Doesn't look terribly hard to do, but it would require designing some things.

If we go with #1 (or just punt on the whole issue), we'd also need to solve the problem of that the lock file header is different sizes on 32-bit and 64-bit. Currently trying to open a Realm file in the browser which has a lock file generated by a 32-bit simulator infinite loops, since the file is non-empty but not big enough to be a valid lock file, so it assumes that some other process is in the middle of initializing it.

@finnschiermer

[timanglade]

We now have support in core for multi-process, including notifications, as long as people use compatible pthread libraries. We need to wrap up the implementation of notifications at the cocoa level, and highlight what we support (iOS 8 extensions, WatchKit, …) + caveats (incompatible pthread libraries) in our docs.

[RTimal]

@timanglade Is this support present in v0.90.4? Seeing no query results in my Watchkit extension after committing a write transaction in the parent app.

Edit: I have confirmed that multiprocess support is working in 0.90.4

[timanglade]

@RTimal Some things are working yes, but right now there are no change notifications, lots of tests are missing and there’s also an Apple bug we need to address in a more elegant way. We have some builds working internally and we’ll update this ticket as we make progress :)

[tgoyne]

This is now fully supported on master and we'll release 0.91.0 with it soon.

[nwalter08]

Could someone provide an overview of how I could get Realm to work with an extension. I am making a WatchKit extension.

[yoshyosh]

Here you go http://fancypixel.github.io/blog/2015/03/29/share-data-between-watchkit-and-your-app-with-realm/. We are working on a tutorial as well, it should be up soon

[nwalter08]

You da real MVP @yoshyosh http://stream1.gifsoup.com/view2/20140902/5101733/you-the-real-mvp-o.gif

[yoshyosh]