Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999884080s. [closed=[], open=[[buffered_nanos=2001703685, waiting_for_connection]]] at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) #10365

Closed
zzq0341 opened this issue Jul 11, 2023 · 4 comments
Labels

Comments

@zzq0341
Copy link

zzq0341 commented Jul 11, 2023

I encountered some problems with grpc-java, please help, thank you very much.
My version number is 1.45.4.
My code is as follows:

originChannel is an instance variable
ManagedChannel originChannel=NettyChannelBuilder.forAddress(XXXX, XXXX)
               .idleTimeout(2, TimeUnit.MINUTES)
               .usePlaintext()
               .build();

grpc remote call code is

XXService. newBlockingStub(originChannel)
               .withInterceptors(new ClientInterceptor[]{MetadataUtils.newAttachHeadersInterceptor(meta)})
               .withDeadlineAfter(2, TimeUnit. SECONDS)
               .xx(Empty.newBuilder().build());

I get some error messages and after 20 seconds it's fine again

o.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999884080s. [closed=[], open=[[buffered_nanos=2001703685, waiting_for_connection]]]
@zzq0341 zzq0341 changed the title io.grpc.Stat usRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999884080s. [closed=[], open=[[buffered_nanos=2001703685, waiting_for_connection]]]\n\tat io.grpc.stub.ClientCa lls.toStatusRuntimeException(ClientCalls.java:262) io.grpc.Stat usRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999884080s. [closed=[], open=[[buffered_nanos=2001703685, waiting_for_connection]]] at io.grpc.stub.ClientCa lls.toStatusRuntimeException(ClientCalls.java:262) Jul 11, 2023
@zzq0341 zzq0341 changed the title io.grpc.Stat usRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999884080s. [closed=[], open=[[buffered_nanos=2001703685, waiting_for_connection]]] at io.grpc.stub.ClientCa lls.toStatusRuntimeException(ClientCalls.java:262) io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999884080s. [closed=[], open=[[buffered_nanos=2001703685, waiting_for_connection]]] at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) Jul 12, 2023
@larry-safran
Copy link
Contributor

The first time you use the channel, gRPC will connect to the server which may take some time. Since you are giving a deadline of 2 seconds, if the combination of connecting, transporting over the network both ways and server side processing exceeds 2 seconds you will get a timeout as you describe.

Since you specified an idle timeout of 2 minutes, at the 20 second point the connection would still be active so the next invocation wouldn't have the overhead of setting up the connection. Increasing the timeout on the method call to 10 seconds will probably eliminate the problem.

@zzq0341
Copy link
Author

zzq0341 commented Jul 12, 2023

The first time you use the channel, gRPC will connect to the server which may take some time. Since you are giving a deadline of 2 seconds, if the combination of connecting, transporting over the network both ways and server side processing exceeds 2 seconds you will get a timeout as you describe.

Since you specified an idle timeout of 2 minutes, at the 20 second point the connection would still be active so the next invocation wouldn't have the overhead of setting up the connection. Increasing the timeout on the method call to 10 seconds will probably eliminate the problem.

Thank you very much, but this happened during the running of the program, and there was still traffic in the last second, and then a 20-second timeout exception occurred, and then it returned to normal by itself. Is it caused by network jitter to re-establish the tcp link?

@larry-safran
Copy link
Contributor

If you use the current version of gRPC do you still see the delay?

@ejona86
Copy link
Member

ejona86 commented Sep 21, 2023

20 second timeout sounds like connect() timed out, although it would have been better to paste the exact error text. There may have been a bad address or the network was busted for a while. Either way the delay seems caused by the network.

You might end up benefiting from Happy Eyeballs, as if one address is slow it allows continuing on to the next without waiting the full 20 seconds. You could follow along on progress of grpc/proposal#356 , which looks to add Happy Eyeballs. And also #10486

@ejona86 ejona86 closed this as completed Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants