PGAdapter throwing a lot of PGExceptions and SocketExceptions

[hiradp]

Sorry in advance for subpar issue creation, but I don't have much to go off.

We're seeing a ton of errors like this:

com.google.cloud.spanner.pgadapter.error.PGException: Broken pipe

at com.google.cloud.spanner.pgadapter.error.PGException$Builder.build ( com/google.cloud.spanner.pgadapter.error/PGException.java:59 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.runConnection ( com/google.cloud.spanner.pgadapter/ConnectionHandler.java:417 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.run ( com/google.cloud.spanner.pgadapter/ConnectionHandler.java:329 )
at java.lang.Thread.run
Caused by: java.net.SocketException

at sun.nio.ch.SocketDispatcher.write0
at sun.nio.ch.SocketDispatcher.write
at sun.nio.ch.NioSocketImpl.tryWrite
at sun.nio.ch.NioSocketImpl.implWrite
at sun.nio.ch.NioSocketImpl.write
at sun.nio.ch.NioSocketImpl$2.write
at java.net.Socket$SocketOutputStream.implWrite
at java.net.Socket$SocketOutputStream.write
at java.io.BufferedOutputStream.flushBuffer
at java.io.BufferedOutputStream.implFlush
at java.io.BufferedOutputStream.flush
at java.io.DataOutputStream.flush
at com.google.cloud.spanner.pgadapter.wireoutput.WireOutput.send ( WireOutput.java:75 )
at com.google.cloud.spanner.pgadapter.wireoutput.WireOutput.send ( WireOutput.java:60 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.handleError ( ConnectionHandler.java:564 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.handleMessages ( ConnectionHandler.java:495 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.runConnection ( ConnectionHandler.java:408 )

java.net.SocketException: Broken pipe

at sun.nio.ch.SocketDispatcher.write0
at sun.nio.ch.SocketDispatcher.write
at sun.nio.ch.NioSocketImpl.tryWrite
at sun.nio.ch.NioSocketImpl.implWrite
at sun.nio.ch.NioSocketImpl.write
at sun.nio.ch.NioSocketImpl$2.write
at java.net.Socket$SocketOutputStream.implWrite
at java.net.Socket$SocketOutputStream.write
at java.io.BufferedOutputStream.flushBuffer
at java.io.BufferedOutputStream.implFlush
at java.io.BufferedOutputStream.flush
at java.io.DataOutputStream.flush
at com.google.cloud.spanner.pgadapter.wireoutput.WireOutput.send ( com/google.cloud.spanner.pgadapter.wireoutput/WireOutput.java:75 )
at com.google.cloud.spanner.pgadapter.wireoutput.WireOutput.send ( com/google.cloud.spanner.pgadapter.wireoutput/WireOutput.java:60 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.handleError ( com/google.cloud.spanner.pgadapter/ConnectionHandler.java:563 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.runConnection ( com/google.cloud.spanner.pgadapter/ConnectionHandler.java:413 )
at com.google.cloud.spanner.pgadapter.ConnectionHandler.run ( com/google.cloud.spanner.pgadapter/ConnectionHandler.java:329 )
at java.lang.Thread.run

Our set up is:

• Rust application using https://github.com/launchbadge/sqlx in main container
• PGAdapter as a sidecar
• Running in a cloud run service.

It's hard for me to debug to figure out where the issue is. Any pointers would be much appreciated 🙇

[olavloite]

@hiradp Thanks for the report. We do not have any automated tests for Rust (and thereby also none for sqlx). I have a couple of quick questions up front:

1. Are you only seeing these errors in the PGAdapter logs, or are the errors also surfaced to the application?
2. Are you seeing the errors when a connection is being closed and/or when PGAdapter is shutting down, or all the time?
3. I noticed that sqlx can use LISTEN and NOTIFY when connected to PostgreSQL. That is one of the parts of the PG protocol that PGAdapter currently does not support. Do you know if you are using that?

[hiradp]

Hey @olavloite - thanks for the response.

1. They're in PGAdapter logs.
2. Pretty much all of the time.
3. We're not using LISTEN or NOTIFY

[liho00]

hi @hiradp did you get this issue resolved?

[olavloite]

@liho00 @hiradp This seems to happen when the client terminates the TCP connection without first sending a Terminate (X) message to PGAdapter (see https://www.postgresql.org/docs/current/protocol-message-formats.html). It does not cause any real harm, but we'll fix it in #2374 to reduce the log spamming and prevent any confusion regarding this.