Sidecar renders configuration if Server response is not successfull

[HenryTheSir]

Problem description

If the Server responds with an error/timeout for the configuration call. Sidecar detects this as a 'new' configuration and rewrites the local configuration and restarts the specific agent. In the next iteration it detects the correct configuration again as new configuration and restarts the agent.

Steps to reproduce the problem

1. Make sure the request for the configuration times out or fails, without an HTTP Status Code or DNS Resolution times out via UDP.

Possible Solution

In this Line the Error gets detected from the sidecar, and the error gets logged to file:

collector-sidecar/api/graylog.go

Lines 109 to 113 in cd538a1

	if err != nil && resp == nil {
	msg := "Fetching configuration failed"
	system.GlobalStatus.Set(backends.StatusError, msg+": "+err.Error())
	log.Errorf("[RequestConfiguration] %s: %v", msg, err)
	}

As the resp being nil following condition does not match:

collector-sidecar/api/graylog.go

Line 115 in cd538a1

if resp != nil {

So the last return statement triggers and returns an 'nil' Error, which leads the calling procedure to 'detect' a new config:

collector-sidecar/api/graylog.go

Line 136 in cd538a1

return configurationResponse, nil

Logs

time="2021-05-20T02:39:05+02:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get GLHOST/api/sidecar/configurations/render/xxxxxxx dial tcp: lookup GLHOST  on IP:Port2 : write udp IP:Port1-> IP:Port2 : write: invalid argument" 
time="2021-05-20T02:39:05+02:00" level=info msg="[filebeat-wrapped] Configuration change detected, rewriting configuration file." 
time="2021-05-20T02:39:06+02:00" level=info msg="[filebeat-wrapped] Stopping" 
time="2021-05-20T02:39:08+02:00" level=info msg="[filebeat-wrapped] Starting (exec driver)" 
time="2021-05-20T02:39:09+02:00" level=error msg="[filebeat-wrapped] Unable to start collector after 3 tries, giving up!" 
time="2021-05-20T02:39:09+02:00" level=error msg="[filebeat-wrapped] Collector output: Exiting: No outputs are defined. Please define one under the output section.\nThu May 20 02:39:08 CEST 2021 /opt/graylog-sidecar/filebeat-sudo-wrapper.sh: started\nThu May 20 02:39:08 CE 
ST 2021 /opt/graylog-sidecar/filebeat-sudo-wrapper.sh: start collector: sudo /opt/filebeat/filebeat -c /opt/graylog-sidecar/generated/filebeat-wrapped.conf -strict.perms=false\n" 
time="2021-05-20T02:39:16+02:00" level=info msg="[filebeat-wrapped] Configuration change detected, rewriting configuration file." 
time="2021-05-20T02:39:17+02:00" level=info msg="[filebeat-wrapped] Starting (exec driver)" ```
### Environment

* Sidecar Version: 1.0.1
* Graylog Version: 4.x
* Operating System: RHEL