Connections entering unstable state with waiting_for_connection error #2422
Assignees
Comments
|
Possibly relevant changes between 0.37.0 and 0.38.0:
Possibly relevant changes between google-cloud-spanner 6.72.0 and 6.74.1:
One interesting difference in the thread dumps:
None of the working thread dumps contain this entry. This stack dump comes from the keep-alive queries in the normal session pool. Note that this should be unrelated to the change to the core pool size for maintaining multiplexed sessions, as that is a completely separate part of the code, and only sounds/look the same due to the names. The thread dump seems to indicate that the keep-alive query is stuck while waiting for results to be returned from Spanner for a simple |
|
The thread dump where a psql query is stuck contains this stackdump for the query that is stuck: This indicates that PGAdapter is waiting for the Java client to execute and return the results of a query. The corresponding Spanner connection that is executing the query: This is an indication that also this query is just stuck waiting for results to come back from Spanner. The session pool keep-alive query does not use a custom timeout, and instead just uses the default timeouts of the Spanner library. This could be consistent with the observation that |
|
Interesting additional fact: The keep-alive queries that are executed by the session pools in the cases where the queries are stuck, are being executed by a |
olavloite
added a commit
to googleapis/java-spanner
that referenced
this issue
Oct 14, 2024
The session pool uses the gRPC transport executor for session maintenance. This executor should not be used by any other tasks than internal gRPC tasks. This change therefore creates simple executors for session maintenance and session creation. Updates GoogleCloudPlatform/pgadapter#2422
The only way so far that I've been able to reproduce this issue is by dropping all incoming network traffic to the underlying Spanner client. That means that outgoing messages do not cause any errors, but queries or other statements just wait indefinitely for a response from Spanner. |
|
The specific This is logged when the timeout occurs while the client is waiting for the stream (connection) to be established. That is: The client has started an attempt to execute a streaming RPC. This then times out while waiting for the stream to be established. grpc-core versions in the relevant PGAdapter versions are: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Current Behavior
Beginning in versions 0.38.0 and 0.39.0, we've randomly observed the following error from PGAdapter:
The deadline of ~55 seconds is expected since we run
SET statement_timeout = 55000;on every Read query to prevent Spanner from running queries longer than necessary.When a PGAdapter instance enters this unstable state, it seems some connections in the pool are tainted while others work just fine. We can run
psqlagainst a faulted instance and run simple queries (e.g.select * from table limit 1;) successfully a few times, but it will stall randomly and eventually time out with thewaiting_for_connectionerror. This stall occurs without astatement_timeoutset as well, we just won't see the error come through as it's attempting to run indefinitely.We have also seen this on write statements (e.g. UPDATE, INSERT) albeit with a lower timeout, 30 seconds, which does seem to match Spanner gRPC configuration defaults.
Reproducing the error state is quite difficult since we get all sorts of traffic to each pod and it only seems to manifest itself against one or two pods out of 120 at a time. It will generally occur 24 hours after a new Deployment, but sometimes will show up in as little as a few hours.
Reverting to version 0.37.0 fixes this behavior (so far at least).
Context (Environment)
SET statement_timeout = 55000;Other Information
We were able to gather some thread dumps (
kill -3 <PID>) in case you're able to glean insight from these:The text was updated successfully, but these errors were encountered: