Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server abruptly terminates connections if write buffer is small enough #2089

Closed
euroelessar opened this issue May 18, 2018 · 5 comments · Fixed by containerd/containerd#3192, docker/cli#1811, #2092, moby/moby#39014 or moby/swarmkit#2847
Closed

Server abruptly terminates connections if write buffer is small enough #2089

euroelessar opened this issue May 18, 2018 · 5 comments · Fixed by containerd/containerd#3192, docker/cli#1811, #2092, moby/moby#39014 or moby/swarmkit#2847
Assignees
@MakMukhi
Labels
P1 Type: Bug

Comments

@euroelessar
Copy link

Please answer these questions before submitting your issue.

What version of gRPC are you using?

grpc-1.12.0

What version of Go are you using (go version)?

go version go1.8.4 linux/amd64

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

Started a server with 8KB write buffer (using grpc.WriteBufferSize) in order to reduce per-connection memory usage by paying extra cpu cost.

What did you expect to see?

Server successfully processes incoming connections and requests.

What did you see instead?

Server abruptly terminates incoming connection if http2 frame size happens to be greater than write buffer size.
It happens due to combination of following factors:

  1. bufWriter gives up if incoming data is greater than its buffer size and returns non-nil error:
    https://github.com/grpc/grpc-go/blob/master/transport/http_util.go#L530
  2. http2 framer library verifies that writer doesn't violate Writer interfaces and returns io.ErrShortWrite:
    https://github.com/golang/net/blob/master/http2/frame.go#L368
  3. gRPC http server doesn't tolerate any errors and terminates tcp socket:
    https://github.com/grpc/grpc-go/blob/master/transport/http2_server.go#L278

I would assume that this is also true for outgoing client connections due to shared code but didn't try to reproduce it.
@MakMukhi MakMukhi self-assigned this May 18, 2018
@MakMukhi
Copy link
Contributor

Thanks for reporting. I'll take a look at it shortly.

@MakMukhi MakMukhi mentioned this issue May 18, 2018
Merged
@euroelessar
Copy link
Author

@MakMukhi hi, any chance this can be part of patch release?

@menghanl
Copy link
Contributor

menghanl commented Jun 6, 2018

@euroelessar Sorry, just release 1.12.1 but missed this...
Will make another release with this.

There's also another release request (#2125), but I'm a bit unsure about the mentioned PR. If you don't mind, I will want for their reply and combine the releases.
Expect a minor by tomorrow.

@euroelessar
Copy link
Author

that works for me, thanks!

@menghanl
Copy link
Contributor

menghanl commented Jun 7, 2018

@euroelessar Please see https://github.com/grpc/grpc-go/releases/tag/v1.12.2 for the release.

@lock lock bot locked as resolved and limited conversation to collaborators Dec 4, 2018
thaJeztah added a commit to thaJeztah/docker that referenced this issue Apr 6, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
6f572c8 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah added a commit to thaJeztah/cli that referenced this issue Apr 6, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
fd2d7f4 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah added a commit to thaJeztah/swarmkit that referenced this issue Apr 6, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
c3a1567 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this issue Apr 8, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
7e42ac7 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
Upstream-commit: 6f572c8154c20253474a20decc87f677feb3321e
Component: engine
adhulipa pushed a commit to adhulipa/docker that referenced this issue Apr 11, 2019
@thaJeztah @adhulipa
bump google.golang.org/grpc v1.12.2
e019c19 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah added a commit to thaJeztah/cli that referenced this issue Apr 12, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
25e6a64 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this issue Apr 18, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
7e37a58 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
Upstream-commit: 25e6a64e2a1792cb5ae5a74741f5d6f2f4531cf8
Component: cli
tiborvass pushed a commit to tiborvass/cli that referenced this issue Apr 23, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
662f6a6 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah added a commit to thaJeztah/containerd that referenced this issue May 5, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
d1c5951 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
kiku-jw pushed a commit to kiku-jw/moby that referenced this issue May 16, 2019
@thaJeztah @kiku-jw
bump google.golang.org/grpc v1.12.2
239b700 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
thaJeztah added a commit to thaJeztah/docker that referenced this issue Aug 27, 2019
@thaJeztah
bump google.golang.org/grpc v1.12.2
e6f02e4 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
(cherry picked from commit 6f572c8)
Signed-off-by: Sebastiaan van Stijn <[email protected]>
tussennet pushed a commit to tussennet/containerd that referenced this issue Sep 11, 2020
@thaJeztah @87654testtesttesttesttesttest
bump google.golang.org/grpc v1.12.2
386a547 
full diff: grpc/grpc-go@v1.12.0...v1.12.2

- grpc/grpc-go#2074 transport/server: fix race between writing status and header
  - fix grpc/grpc-go#1972 Possible race sending headers from server while receiving message over size limit
- grpc/grpc-go#2074 transport: account for user configured small io write buffer
  - fix grpc/grpc-go#2089 Server abruptly terminates connections if write buffer is small enough

Signed-off-by: Sebastiaan van Stijn <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@MakMukhi MakMukhi

Labels
P1 Type: Bug
Projects
None yet
Milestone
No milestone
3 participants
@euroelessar @menghanl @MakMukhi