'there is no connection available' and 'the connection is draining' when using nginx

[takakawa]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

grpc-go : 1.11.3
nginx : 1.14.0

What version of Go are you using (go version)?

go version go1.10 linux/amd64

What operating system (Linux, Windows, …) and version?

CentOS Linux release 7.2.1511

What did you do?

If possible, provide a recipe for reproducing the error.

I use nginx to proxy grpc requests,And there are many

the connection is draining
there is no connection available

But nginx does not have any err logs!And the system does not have many connections,ss -s shows only 1000 connections in total.

What did you expect to see?

There should not be such errors

What did you see instead?

[takakawa]

It happens in our production envrionments from time to time

[lyuxuan]

Can you give us more info about your environment setup, your code and logs, so we can better investigate the issue? Thanks!

[takakawa]

We tried to figure out this problem,It seemed that setting the FailFast true would cause the problem. When nginx closes connection after http2_max_requests ,the client has a big possibility to show 'There is no connection available' .

When we set http2_max_requests 1000000 ,the possibility became very low.

Here is a same question:
https://trac.nginx.org/nginx/ticket/1590

[dfawley]

@takakawa I think this is working as intended from the grpc side. The fail-fast setting is intended to fail RPCs quickly if there is any temporary problem with the connection. I would recommend using grpc.FailFast(false) on any RPCs for which you are willing to tolerate connection errors like these. Wait For Ready RPCs, on the other hand, will retry once when data is written to the wire but the server does not process it, or indefinitely if the connection was chosen but receives a GOAWAY before the HTTP/2 stream can be created for the RPC. If you need retries beyond that, Go has recently implemented configurable client-side retry support as documented in gRFC A6 (in PR #2111).

Please let us know if you still think there is a problem here. Thanks!