Defer change callbacks to a worker thread #212
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change fundamentally changes how change callbacks work on Dynamics. Prior to this change, callbacks executed on the thread that was performing the change. This could lead to situations where multiple threads were executing callback chains which leads to unpredictable locking patterns on the dynamics. The basic deadlock detection was not enough.
This change defers callbacks to a single callback thread. This thread ensures that no dynamic can have callbacks enqueued more than once. By limiting execution to one set of callbacks at any given time, this greatly reduces the surface for locks to contend with each other.
The next issue was how tuple-related functions like for_each/map_each were acquiring their locks. By calling a.read() then b.read(), this was causing a to be held in a locked state while b was being aquired. If users are careful to always acquire their locks in this order, everything is fine. But with Cushy there can be unexpected situations where these locks are being held.
This change also refactors lock acquisition for tuples to try to acquire all the locks in a non-blocking way. If any lock woould block, the initial locks are dropped while the lock that would block is waited on. After this is acquired the process starts over again to gain all the locks. This isn't perfect, but it doesn't require unsafe. With unsafe, we could in theory create a ring of callbacks that handles acquiring all of the locks into MaybeUninits. Upon successfully calling all callbacks, the values can be assumed init. But writing all of this in macro_rules isn't fun, and the current solution alleviates the main problem