kong log " Connection reset by peer " when spring cloud gateway works with it

[kongh]

Bug report

QUESTION:

We have problem while running Kong in our Docker Swarm cluster. We have one upstream, one upstream target is spring cloud gateway api

I am using spring cloud gateway after kong proxy server, but recevied an "Connection reset by peer" error 。

After a while, when we send a request to kong api endpoint we get this response from "An invalid response was received from the upstream server"

In kong log show me the error log is below:

18910 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.255.0.2, server: kong, request: "POST /one-time-password-service/v2/password HTTP/1.1", upstream: "http://x.x.x.x:y/v2/password", host: "a.b.c:8090"

In upstream server spring cloud gateway has no error logs

Enviroments:

docker version

Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Swarm: active
 NodeID:
 Is Manager: true
 ClusterID:
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
Init Binary: docker-init
Kernel Version: 4.4.0-117-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.42GiB

spring version

spring boot: 2.0.7.RELEASE
spring cloud: Finchley.SR2
spring cloud gateway:  2.0.2.RELEASE

kong version

kong: 0.14.1

How to reproduce problems

send first request to Kong proxy server , may be you can get a successful response, then wait for almost 10 minutes , send another request to Kong , may be you can get a fail response of reset connection. Try it for serval times.

What I have done to resolve

firstly, I guess this is Kong Server problem and view Kong/kong#3314 , https://discuss.konghq.com/t/104-connection-reset-by-peer-while-reading-response-header-from-upstream/249/2

then, I found reactor/reactor-netty#495 and try upgrade spring boot version to 2.0.7 (my old version is 2.0.6) , but it did not working for me.

lucky，I am new user for spring cloud gateway and have the old system working smoothly which network routes is similar with it.

End User -> Kong proxy server -> zuul gateway -> micro service
End User -> Kong proxy server -> spring cloud gateway -> micro service

finally, I replace spring cloud gateway to zuul gateway and it works smoothly for serval days.

Finally , Please make sure the problem

Why it produces the error during kong connect to upstream server. Kong upstream connection is HTTP 1.1 keep-alive default for proxy.
And spring cloud gateway console has no error logs

[ryanjbaxter]

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

[kongh]

@ryanjbaxter I am so sad , I cannot reproduce the same result using my minimal sample( just see https://github.com/kongh/lightning-gateway-issue/blob/master/READEME.MD ).

I had try it for serval days and only got issues like that

2019/01/23 10:17:28 [error] 64#0: *270199 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.99.83, server: kong, request: "GET /api/service/test.json HTTP/1.1", upstream: "http://192.168.99.165:23020/api/service/test.json", host: "kongtest.yywh.ltd"


2019/01/23 10:17:28 [debug] 64#0: *270199 [lua] init.lua:411: balancer(): setting address (try 2): 192.168.99.165:23020

I have used jmeter for testing "/api/service/test.json" endpoint, but , It works properly.

I am confused and not sure what the problem is . If I cannot provide more infomation, you can close this issue.

3KS for your reply. many 3ks.

[AaronFSJ]

I had the same problem.First request from Kong to Spring Cloud Gateway can get a successful response,then wait for almost 30 minutes , send another request to Kong ,get a fail response(An invalid response was received from the upstream server).

And Then , I changed the configuration of Kong.Drop keepalive 60 ,The Problem is fixed. I don't know why?However, keepalive is a necessary configuration and cannot be removed. It is only removed for testing this problem.

upstream kong_upstream {
server 0.0.0.1;
balancer_by_lua_block {
Kong.balancer()
}
# keepalive 60;
}