Delivering notifications within write transactions makes them hard to use #4818
Comments
|
Related to this: the way this mechanism is currently working can cause inconsistencies (crash) in for example a UITableView. I have created a sample app to demonstrate this. See RealmNotifications.zip The tableview shows the results of Persons that are younger than 18. ViewController1: most simple approach. Just a notification-block that calls reloadData. This crashes, because the second write triggers the notification synchronously by advancing. ViewController2: attempt to workaround this by queuing the notification on the next runloop. This seems to 'solve' the problem. ViewController3: shows the problem with the workaround of ViewController2. There is still a small gap where there could already have been cellForRows scheduled. This still crashes. ViewController4: always call refresh after each write-transaction, triggering the notification synchronously. What is the best way to workaround this problem? I don't have a really good feeling with these two 'workarounds'. There is a mechanism to not notify a specific token, but i do want to be notified actually. |
|
I've been struggling with this for a couple of days. I haven't been able to find a way to prevent table view crashes when applying updates that originate from a realm modified on a background thread. Are there any known workarounds? |
|
Any update on this? I think this should be labeled I'm not 100% sure this issue is related to #4425 which is exactly what I'm experiencing. If not, I suppose I need to open a new ticket since that one is closed. |
|
Any updates on this? |
|
I'm running into this exact same problem, with an RxSwift "event pipeline" that flatMaps notifications to new Observables (using RxRealm's Observable.from(..., properties:...)). I have tried everything I know, but cannot think of any way around creating new Observables within an Observable that is called in response to Realm notifications. Any updates on this issue or solutions / workarounds? |
|
Trying to work around this gets into other issues. Using observeOn() and passing a ThreadSafeReference fails with "Cannot obtain thread safe reference during a write transaction.", since the code that creates the ThreadSafeReference is on the main thread (same thread as the Realm transaction)... Nasty bug / issue this. |
|
@jpsim can you please tell us what's going on about this issue? |
|
When we have an update to share, we'll do so here. |
|
It is a priority task, isn't it? |
|
Running into the exact same issue as @stuartro |
|
Still don't have any solutions for this issue? |
|
We have encountered this same issue in our project as well, and our current workaround is to write to the database from separate thread pool threads, so for us this issue is clumsy but not crippling. It would be nice to have a proper solution within Realm itself however, and as has been pointed out, the fix would seemingly be a rather simple one of acquiring the write lock, delivering the collection notifications, and only then starting the write transaction. |
|
@stuartro I know this thread is quite old, but I'm running into the exact same issue with my RxSwift pipeline. Did you somehow find a workaround? |
|
I have a scenario that Person owns and creates a Dog when Person gets created, if a Person gets added to Realm DB then I observe Dog as I want to know the change of Dog. So I first observe Person collection, then if a Person get inserted, I immediately create observer to observe Dog(person.dog.observe), then the app crashes with message: I think realm.beginTransaction(person.insert()) triggers the personCollection.observe block then it calls dog.observe and that's why it has the message above. How should I observe a dog of a person when a person gets inserted/updated? |
|
@raphaklr Yes, I found a solution that works for me. The issue occurred when I tried to perform updates as a result of a notification—which was fired as a result of a preceding update, while the (original) Realm write transaction was still in progress. The solution (at least, what's worked flawlessly for me) is to detect the condition and not start a new write transaction if one is already open). See my post at the link below. If you want a more detailed code snippet lete know and I'll see what I can do. |
|
Here's what I've done that's almost completely eliminated the problem: This retries the write transaction up to 10 times with increasing delays, allowing the current write transaction to complete. When I'm trying to perform multiple transactions at once (such as updating a list of objects), I add a small random modifier to the delay (anywhere between ±0.5 seconds) to avoid continual collisions between updates. So far, I've never seen my app need more than two attempts to complete write transactions with up to ten different objects simultaneously. This way, I can call multiple write transactions from notifications without having to worry about crashing. |
|
I first ran into trouble with notifications in July last year. Our app makes heavy use of applying updates that originate from a realm modified on a background thread and the implementation has not changed since my first comment. After upgrading to Realm 3.11.2 last month, I tried switching from @tgoyne I think it is safe to say my issue is resolved but I don't know what changed to make that happen. |
|
I just tried the demo project from @TeunVR with Realm 3.13.0 and it still crashes. However, that specific issue can be resolved with by wrapping tableview update calls with |
|
I had same issue after removing condition It has complicated logic, but in general where was some object that holds reference to taken raw pics on disk. that pics ( |
|
Been struggling with a notification issue myself. My app sends messages that must be saved in draft form before sending, and then updated after upload to server. The primary key changes so I do a delete+create transaction on a background thread. Both the outgoing and final messages can also be simultaneously modified from main or background threads for other processing. When these writes are happening quickly, the size of the observed ResultSet can change while another notification is being processed and the table is being batch-updated. Because I use the value of ResultSet.count for the UITableView.numberOfSections() I end up with NSInternalInconsistencyException. (same could apply to numberOfRowsInSection()). While sitting here dreading the thought of implementing a serial queue for database updates I figured out a workaround. I added |
|
It has been more than two years, any updates on this? @bdash |
|
Any timeline for a fix on this? Its a showstopper. |
|
Please someone work on this!! |
|
This would be awesome. I'm trying to create another observer within a notification block but that logic crashes often due to "Cannot register notification blocks from within write transactions." |
|
So I took some inspiration from the above answers (starting the observer on mainThread) and made this little extension which seems to work for me... and usage Since the actual Provided as is. haha. Works nice for me but can't guarantee for others 😅 |
|
In my case without having to adopt the extension above, just wrapping .observe { } code inside dispatchQueue.main.async fixed the problem for me. Still waiting for further insight into this problem... |
|
We are still facing several of these issues, especially when we use the built in Combine publishers, where we don't actually control the queue where the notification block is created. Are there any upcoming fixes or updates on this issue? |
|
My app still facing this problem...it's been several years, will it be fixed? |
|
Same here: running into this bug for > 3 months now. Any updates on a structural solution? |
|
HI @JuliusHuizing Can you please explain the issue you are experience?, so we can keep track of this here or open a new issue. |
|
@dianaafanador3 Every now and then, my app crashes at run-time due to this error:
It might very well be that I am doing something wrong myself, but having tried all the potential fixes suggested by others above, I still run into this error on a regular basis (although it seems less frequently than before). My set-up is as follows:
The action provided to the 'whenCollectionChanges' method uses the intances of CollectionOne to compute some statistic. And that statistic is then used to update some instance of another Realm collection. E.g. in the ContentView of the SwiftApp, we perform: Where it then looks like the update method of the user often leads to the run-time error mentioned above. And the SafeWrite is implemented as follows, inspired my multiple different suggestions done by people above: Nevertheless, as said, I still run into this run-time error every now and then. Any suggestions are much appreciated! |
|
It's really strange that in 5 years and all the new versions, this most basic and fundamental feature of the entire framework is still completely broken. |
My thoughts exactly! |
|
I decided to dump Realm and switch to CoreData where possible. |
|
still facing this issue. |
|
Wow, this has been going on for so long and nothing has been done about it? |
|
Besides the workarounds above, are there usage best practices / patterns that can be avoided to sidestep this issue? I'm not encountering it but this issue frightens me from shipping my work based on Realm |
|
+1 looking for best practices as well... |
|
Seeing this issue with rapid updates to a CollectionView. Any updates on a fix for this? |
…change This fixes one of the common problems resulting from realm/realm-swift#4818, as it makes it so that adding new callbacks from within a notification callback will _usually_ work even if the notification was triggered by beginning a write transaction. It still will not work if a previously invoked callback made any changes (which would be a very strange thing to do, but possible).
…change This fixes one of the common problems resulting from realm/realm-swift#4818, as it makes it so that adding new callbacks from within a notification callback will _usually_ work even if the notification was triggered by beginning a write transaction. It still will not work if a previously invoked callback made any changes (which would be a very strange thing to do, but possible).
…change (#6560) This fixes one of the common problems resulting from realm/realm-swift#4818, as it makes it so that adding new callbacks from within a notification callback will _usually_ work even if the notification was triggered by beginning a write transaction. It still will not work if a previously invoked callback made any changes (which would be a very strange thing to do, but possible).
|
realm/realm-core#6560 makes it so that registering new notifiers inside notifications triggered by starting a write transaction will work as long as you do so before making any changes in the transaction. |
|
I've spent the last few weeks digging into this and seeing what we can do to improve things. There's a few different problems that people have run into here, with different solutions. Registering new notifiersWhen the notification system was first written, it would have been very difficult to support registering notifications inside write transactions. Back then we didn't have stable keys for objects and instead stored row numbers that could change when objects were deleted, so it'd require mapping the row number at the time you called The remaining exception is observing objects inside the same write transaction as they were created. This has the problem that object keys aren't unique until the write is committed. A naive approach to supporting this would mean that creating an object, observing it, cancelling the write, then later creating a new object in a separate write transaction would result in that new object being observed instead. This is probably solveable, but is a somewhat tricky problem and not directly related to this issue. Performing writes inside observation callbacksThis part is more complicated to support while still making the guarantees required for observation change information to be useful. To recap, the things we currently guarantee related to change information are: If you never perform writes on the same thread as you are observing on
If you perform writes on the same thread, outside of notifications
If you perform writes inside of notificationsHere we try to maintain the above guarantees, but do not entirely succeed.
The first important thing to note is that async writes now provide a reliable way to sidestep all of this complexity. If you don't need fine-grained change information inside your write block, you can do: let token = dog.observe(keyPaths: [\Dog.age]) { change in
guard case let .change(dog, _) = change else { return }
dog.realm!.writeAsync {
dog.isPuppy = dog.age < 2
}
}This will work even if the notification happens to be delivered inside a write transaction. However, because the write is async the Realm may have changed between the notification and when the write happens, so the change information passed to the notification may no longer be applicable. If you do need the fine-grained change information to be accurate inside the write transaction, it's not obvious that that's conceptually possible, even setting aside implementation difficulties. It's also not clear that anyone actually needs this, so I think maybe it's best just ignored? Updating a UITableViewThis is not really related to notifications being delivered inside write transactions, but it's come up here a few times. Here the problem is that if you only update a UITableView via notifications, in the time between a write transaction and the next notification arriving the tableview's state is out of sync with the data. If the tableview has a pending update scheduled, then this causes problems. There's a few ways to fix this. For all of these, suppose you have the following very basic view controller: class TableViewController: UITableViewController {
let realm = try! Realm()
let results = try! Realm().objects(DemoObject.self).sorted(byKeyPath: "date")
var notificationToken: NotificationToken!
override func viewDidLoad() {
super.viewDidLoad()
tableView.register(Cell.self, forCellReuseIdentifier: "cell")
notificationToken = results.observe { (changes: RealmCollectionChange) in
switch changes {
case .initial:
self.tableView.reloadData()
case .update(_, let deletions, let insertions, let modifications):
self.tableView.beginUpdates()
self.tableView.insertRows(at: insertions.map { IndexPath(row: $0, section: 0) }, with: .automatic)
self.tableView.deleteRows(at: deletions.map { IndexPath(row: $0, section: 0) }, with: .automatic)
self.tableView.reloadRows(at: modifications.map { IndexPath(row: $0, section: 0) }, with: .automatic)
self.tableView.endUpdates()
case .error(let err):
fatalError("\(err)")
}
}
}
override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
return results.count
}
override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
let object = results[indexPath.row]
let cell = tableView.dequeueReusableCell(withIdentifier: "cell", for: indexPath) as! Cell
cell.textLabel?.text = object.title
return cell
}
func delete(at index: Int) throws {
try realm.write {
realm.delete(results[index])
}
}
}Option 1: Update the UITableView directly and skip the notificationfunc delete(at index: Int) throws {
try realm.write(withoutNotifying: [token]) {
realm.delete(results[index])
}
tableView.deleteRow(at: [IndexPath(row: index, section: 0)], with: .automatic)
}This gives the most responsive UI as the tableview is updated immediately rather than after a few hops between threads that adds a small amount of lag to each update. The downside is that it requires the most manual updates to the view. Option 2: Force a refresh after the writefunc delete(at index: Int) throws {
try realm.write {
realm.delete(results[index])
}
realm.refresh()
}This results in the notifications from the write being delivered immediately rather than on a future run of the run loop, so there's no window for the table view to read out of sync values. The downside is that this makes it so that things which would normally happen on a background thread (such as rerunning the query and resorting the results) will instead happen on the main thread, and that's a bad thing. Option 3: Do the write on a background threadfunc delete(at index: Int) throws {
func delete(at index: Int) throws {
@ThreadSafe var object = results[index]
DispatchQueue.global().async {
guard let object = object else { return }
let realm = object.realm!
try! realm.write {
if !object.isInvalidated {
realm.delete(object)
}
}
}
}
}This blocks the main thread for the least amount of time, but is often more difficult to write. Since the write isn't done on the main thread, the main thread never sees the write before the notifications arrive. Option 4: Add an option to make writes not immediately visible on the writing threadOne thing that we could do now is to make it so that after |
|
Thanks for the writeup @tgoyne — would be nice to see these details in the docs. Happy with the ability to register new notifiers in a write transaction. Question about Updating a UITableView Option 3: would using the For example: func delete(at index: Int) throws {
realm.writeAsync({
realm.delete(results[index])
// (A)
// at this point `results` is updated, but no notification sent yet
// => out of sync with data
}, onComplete: { err in
// (B)
// notification has been fired
// => in sync with data
})
}I guess it is possible that between point Option 4 sounds like a nice solution to solve this, but I agree, probably too confusing and complex for most use cases. |
|
writeAsync wouldn't fix it. I suspect it might actually make you run into the problem more often. Moving the writing to disk off the main thread means that there's a longer period of time that you're in an inconsistent state with the main thread unblocked and able to schedule work. |
|
Support for adding new observers inside notifications triggered by beginning a write was released in 10.39.1 and I believe async writes are an adequate solution to performing writes inside notifications, so the problems which this issue were originally tracking have been fixed. I created realm/realm-core#6587 and realm/realm-core#6588 to track further follow-up work on the related issues that people have mentioned here. The docs team has created a ticket to incorporate the information I wrote up about the guarantees notifications provide, but that isn't publicly visible. |
and others
In v2.1.0 we started delivering collection notifications when beginning a write transaction advances the read version of a Realm. We deliver these notifications within the write transaction to prevent starvation when another thread is frequently writing to the Realm. Users have found that confusing to deal with a notification block potentially being invoked within a write transaction (#4815, #4701, #4511, and likely others). There are certain operations, such as registering for object or collection notifications, that cannot be performed within a write transaction. It's not clear how a user can ever safely use such functionality within a notification callback when it may sometimes be invoked within a write transaction (other than by never performing writes on a thread that is observing a collection or object notification?).
@tgoyne mentioned on Slack that we may be able to deal with this by separating acquisition of the write lock from beginning the write transaction. This would allow us to avoid starvation while also allowing notification callbacks to perform operations that cannot be performed within a write transaction.
The text was updated successfully, but these errors were encountered: