Start work on avoiding table locks for upserts #2684
Conversation
Bail out early to reduce indentation
wrap the call to _simple_upsert_txn in a loop so that we retry on an integrityerror: this means we can avoid locking the table provided there is an unique index.
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the table.
|
it's surprising to me that we'd actually need a delete from |
Yes, that was option 1.
|
|
@ara4n I believe @richvdh TBH we could probably make deleted_pushers be a simple insert; given its a stream I don't think it matters at all if things get inserted multiple times. I'd also imagine that generally we'd not have many "duplicate" inserts |
|
(I'll fix the tests) |
|
Assuming that the remaining test failure is the thing erik is fixing in matrix-org/sytest#421 |
Add a loop to _simple_upsert so that it can be used without a lock in more
circumstances, and a PoC to demonstrate it in
add_pusher.WDYT?
Anyway, it's all very well but doesn't really help us for more complex
operations where we do more than one thing in the transaction (for example,
delete_pusher_by_app_id_pushkey_user_id). There we have a few options:decide that the delete from
pushersand the upsert intodeleted_pushersdon't actually need to happen in a single transaction, and use
_simple_upsertthere toocargo-cult the loop from
_simple_upsertintodelete_pusher_by_app_id_pushkey_user_idfactor out the loop to a
retry_on_integrity_erroror something and use that