Incomplete message error

[lklein53]

What version of gRPC and what language are you using?

grpc-dotnet client version 2.46.0 / C#

What operating system (Linux, Windows,...) and version?

Linux

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET 6 (6.0.202) SDK

What did you do?

A client sends unary grpc requests that contain some bytes and expects back a response containing two int32 values.
From time to time the client receives an error for the request with grpc status code 13 and the message Incomplete message from the server.
To me it seems to be related to the server closing the HTTP2 connection by sending a GOAWAY.
Sadly i can't provide a reproducer as the issue only occurs seldomly.
I have attached some logs and stack traces that i have gathered when the error occurs

2022-05-30 18:57:23.878 <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 56388208,0,0,SendWithVersionDetectionAndRetryAsync,Retry attempt 1 after connection failure. Connection exception: System.Net.Http.HttpRequestException: The request was aborted.
 ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'NO_ERROR' (0x0).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.ThrowRetry(String message, Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) ->

2022-05-30 18:57:24.724 <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 56388208,26131301,1,OnHeader,grpc-status: 13 ->                                                                                                                                                                                                                            
2022-05-30 18:57:24.725 <- Event Private.InternalDiagnostics.System.Net.Http - HandlerMessage : 56388208,26131301,1,OnHeader,grpc-message: Incomplete message. ->  

What did you expect to see?

A successful response for the grpc request

What did you see instead?

An error response with grpc status code 13 and the message Incomplete message

I would also be thankful for guidance on how to further investigate the issue.

Kind regards,
Lars

[Tette6969]

#2025

[spingee]

Hello,
We are facing this problem too, i have narrowed configuration which causing it.
These conditions must be met to reproduce this error (2.52 version):

• Client grpc channel is singleton (recommended by docs, and implemented in default client registration)
• RetryPolicy is set
• There is AKS ingress in network path (works properly when port of kube service/pod is forwarded to local, ingress has TLS and portforwarding not)
• Run requests in parallel

For me this is definitely bug in implementation,because it works properly without retry policy. Also there is no retrying just retry policy set.
Workarounds are make your own retry mechanism as interceptor or dont use default client registration and create your own grpcchannel with scoped lifetime.

[tarasevichvlad]

@JamesNK Hello. Do you have any update for this issue?

[kharabasz]

I believe we are running into a similar issue. @JamesNK I can try to provide more information/traces to help track this down as it is occurring very frequently in our stack - just let me know what you need.

Both client and server are running in a container based on mcr.microsoft.com/dotnet/aspnet:6.0
gRPC client version is 2.59.0

With Debug level logging turned on client and server side, the only relevant logs seem to come from the client:

 fail: External Service threw exception:
 Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="No grpc-status found on response.")
 at Grpc.Net.Client.Internal.Retry.RetryCallBase`2.GetResponseCoreAsync()
 at ... domain code ...

 fail: External Service threw exception:
 Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. InvalidDataException: Unexpected end of content while reading the message content.", DebugException="System.IO.InvalidDataException: Unexpected end of content while reading the message content.")
 ---> System.IO.InvalidDataException: Unexpected end of content while reading the message content.
 at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageContentAsync(Stream responseStream, Memory`1 messageData, Int32 length, CancellationToken cancellationToken)
 at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
 at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
 at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
 --- End of inner exception stack trace ---
 at Grpc.Net.Client.Internal.Retry.RetryCallBase`2.GetResponseCoreAsync()
 at ... domain code ...