logspout stops sending logs via tcp

[akosyrev]

After 10-20mins after start container does not send any packets.

My problem was sending containers logs to logstash.

1. I did changes to modules.go as described here
2. Compile logspout on my host: GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build

go version
go version go1.7.4 linux/amd64

3. Add binary to /bin

After start container sends logs well, no drops or anything wrong. But after sometime it stops sending packets, I see it in CPU and network usage. My main container (which generates logs to stdout) works, its responses for requests, writes logs..

I verified if it is logstash-module problem. I ran luisbebop/echo-serverPorts and tried raw TCP sending and faced same beavior.

Any help is appreciated

UPD1
Restart logspout-container fixes problem for 10-20 mins.

This is my logsout output after start:

30.03.2017 13:11:56# logspout  by gliderlabs
30.03.2017 13:11:56# adapters: syslog tcp udp tls logstash raw
30.03.2017 13:11:56# options : persist:/mnt/routes
30.03.2017 13:11:56# jobs    : routes http[logs,routes]:80 pump
30.03.2017 13:11:56# routes  :
30.03.2017 13:11:56#   ADAPTER	ADDRESS		CONTAINERS	SOURCES	OPTIONS
30.03.2017 13:11:56#   raw+tcp	10.6.109.62:8800*load*			map[]

[michaelshobbs]

What version are you running? Can you also post the logspout container logs after you stop seeing packets being sent?

[michaelshobbs]

potentially related to #268

[akosyrev]

Version: v3.2-dev

There is no other logs after packets stop being sent.
Only this lines after startup:

31.03.2017 9:51:42# logspout  by gliderlabs
31.03.2017 9:51:42# adapters: syslog tcp udp tls logstash raw
31.03.2017 9:51:42# options : persist:/mnt/routes
31.03.2017 9:51:42# jobs    : http[routes,logs]:80 pump routes
31.03.2017 9:51:42# routes  :
31.03.2017 9:51:42#   ADAPTER	ADDRESS		CONTAINERS	SOURCES	OPTIONS
31.03.2017 9:51:42#   raw+tcp	10.6.109.62:8800*load*			map[]

and at 10:02 packets lost somewhere without any additional info/logs or errors.

[akosyrev]

If I use UDP, problem does not appears. Logs are transmitted for 3 days. Some lines are dropped (UDP "feature") but logspout continues sending logs and I see ~99% of them in my ElasticSearch.

Maybe this info could be helpful

[michaelshobbs]

Potential fix + tests were merged in f18e503

[akosyrev]

Sorry for my late answer.
Recompiled with logspout v3.3-dev, master-branch. The bug still exists.

The commit f18e503 contains some changes and tests about syslog adapter. And this bug is about TCP connection, I think.

[michaelshobbs]

Apologies as I assumed you were using tcp+syslog. It would seem yore using the logstash adapter. The adapters are actually the responsible code paths for connecting and reconnecting to the configured logging service.

As we are no longer seeing this issue with the syslog adapter, I'm thinking your issue probably lies with the logstash adapter. Please let us know if you have additional information or even a PR that pertaining to the tcp transport.

[renjithwarrier94]

I am facing the same issue. The version I am using is v3.2.2. I used the httpstream module and found that logspout is actually reading the logs from stdout. The following is the log for logspout and the last few lines indicate what happens after the logs are stopped being sent.

10.37.4.213 | # logspout v3.2.2 by gliderlabs
10.37.4.213 | # logspout v3.2.2 by gliderlabs
10.37.4.213 | # adapters: syslog tcp udp tls raw
10.37.4.213 | # adapters: syslog tcp udp tls raw
10.37.4.213 | # options : persist:/mnt/routes
10.37.4.213 | # options : persist:/mnt/routes
10.37.4.207 | # logspout v3.2.2 by gliderlabs
10.37.4.213 | # jobs : http[logs,routes]:80 pump routes
10.37.4.207 | # logspout v3.2.2 by gliderlabs
10.37.4.207 | # adapters: raw syslog tcp udp tls
10.37.4.207 | # adapters: udp tls raw syslog tcp
10.37.4.207 | # options : persist:/mnt/routes
10.37.4.207 | # options : persist:/mnt/routes
10.37.4.207 | # jobs : http[logs,routes]:80 pump routes
10.37.4.207 | # jobs : http[logs,routes]:80 pump routes
10.37.4.207 | # routes :
10.37.4.207 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
10.37.4.207 | # routes :
10.37.4.207 | # raw+tcp logstash:5545 map[]
10.37.4.207 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
10.37.4.207 | # raw+tcp logstash:5545 map[]
10.37.4.207 | 2017/10/26 13:54:16 raw: write tcp 10.0.9.15:57614->10.0.9.6:5545: write: no route to host
10.37.4.213 | # jobs : pump routes http[logs,routes]:80
10.37.4.213 | # routes :
10.37.4.213 | # routes :
10.37.4.213 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
10.37.4.213 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
10.37.4.213 | # raw+tcp logstash:5545 map[]
10.37.4.213 | # raw+tcp logstash:5545 map[]
10.37.4.213 | 2017/10/26 13:29:07 raw: write tcp 10.0.9.14:38782->10.0.9.6:5545: write: no route to host
10.37.4.207 | 2017/10/26 13:54:16 routes ended: %!s()
10.37.4.213 | 2017/10/26 13:29:07 routes ended: %!s()