-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
spanner: Poor handling of high conflict transactional workloads #11017
Comments
@richardartoul Just to be sure:
Can you confirm the above? The reason for asking the above is that:
One additional tip (this would also avoid the problem with the additional retries, as this method is specifically intended for single write operations):
|
I am following up on this (I am working with Richie):
|
FYI I tried the mutations approach directly, and that seems to help a lot. However, I am seeing the memory of my processing growing over time without stopping, so I am guessing I am doing something wrong. I am still investigating, but here is the code sample if someone sees something obvious:
|
Ok I just saw this in the
|
We are happy with your mutations workaround, feel free to close! |
Some of the examples missed a call to conn.Close(). Updates googleapis/google-cloud-go#11017
Some of the examples missed a call to conn.Close(). Updates googleapis/google-cloud-go#11017
Thanks for pointing out that :-) It has been added in googleapis/go-sql-spanner#324 |
Great thanks! |
Client
Spanner
Environment
Latest Go version, any distribution.
Code and Dependencies
Using the Golang SQL driver which wraps this library: https://github.com/googleapis/go-sql-spanner
Expected behavior
We have a workload that performs a lot of appends to a table that has a primary key with a unique constraint. 99% of the time the row doesn't exist and everything works fine, however, we rely on Spanner maintaining the primary key constraint and to propagate errors back when we violate the constraint. In our application we handle the conflicts by modifying the primary key and retrying (basically we're doing compare-and-swap).
The expected behavior is that when we perform these inserts, the spanner client would immediately propagate the error back to us so the application can handle the retry (since a primary key violation is not a fundamentally retriable error).
Actual behavior
Based on what i'm observing in our logs, tracing, and whats described in this issue: #7903 I believe what is happening is the spanner client first tries to perform the insert using an inline transaction. This fails, and the ReadWriteTransaction function sees an errInlineBeginTransactionFailed error and then immediately retries with a non-inline transaction. This transaction will fail as well, then our application receives the error. This extra retry causes unnecessary conflicts / retries.
I'd expect to get the foreign key constraint violation after the first insert attempt, not the second. I also don't see any way for me to disable this automatic retry behavior. I also don't see any way to disable the inline transaction behavior, or prevent the initial retry.
The text was updated successfully, but these errors were encountered: