Sidecar stops collector backends when graylog server unavailable

[bernd]

Problem description

There seems to be a problem with the collector backend process handling when the graylog server is flaky. This happened in my local development environment where I restart the server very often and where the server is unavailable for longer periods.

The following log snippet shows that the sidecar is removing and adding process runners for the rsyslog backend. But even after the last log line, the graylog server is back up running, there is no collector process running. (this also happens with a filebeat backend, so I don't think it's backend type related)

ERRO[166711] [RequestBackendList] Fetching backend list: Get http://127.0.0.1:9000/api/sidecar/collectors: dial tcp 127.0.0.1:9000: connect: connection refused 
ERRO[166711] [UpdateRegistration] Failed to report collector status to server: Put http://127.0.0.1:9000/api/sidecars/20f0f20d-a4ce-4172-954e-7c09fe6ac870: dial tcp 127.0.0.1:9000: connect: connection refused 
INFO[166724] Adding process runner for: rsyslog           
ERRO[167710] [RequestBackendList] Fetching backend list: Get http://127.0.0.1:9000/api/sidecar/collectors: dial tcp 127.0.0.1:9000: connect: connection refused 
ERRO[167710] [UpdateRegistration] Failed to report collector status to server: Put http://127.0.0.1:9000/api/sidecars/20f0f20d-a4ce-4172-954e-7c09fe6ac870: dial tcp 127.0.0.1:9000: connect: connection refused 
INFO[167710] Removing process runner: rsyslog             
INFO[167710] No configurations assigned to this instance. Skipping configuration request. 
ERRO[167715] [RequestBackendList] Fetching backend list: Get http://127.0.0.1:9000/api/sidecar/collectors: dial tcp 127.0.0.1:9000: connect: connection refused 
[...]
ERRO[168850] [UpdateRegistration] Failed to report collector status to server: Put http://127.0.0.1:9000/api/sidecars/20f0f20d-a4ce-4172-954e-7c09fe6ac870: dial tcp 127.0.0.1:9000: connect: connection refused 
INFO[168865] Adding process runner for: rsyslog

I think the sidecar should keep the collector processes running, even if the graylog server is gone. Additionally, if the sidecar process starts and finds backend configurations, it should start the collector processes, even if the graylog server is unavailable.

Steps to reproduce the problem

1. Start graylog server
2. Start sidecar
3. Configure collectors for the sidecar
4. Stop graylog server
5. Wait and check collector processes and sidecar logs

Environment

• Sidecar Version: 7164452
• Graylog Version: 3.0.0-SNAPSHOT (rev 515c8024b)