kafka-exporter service doesn't start after restarting kafka VMs

[to-bar]

[root@ec2-15-236-37-69 ~]# systemctl status kafka-exporter
● kafka-exporter.service - Service that runs Kafka Exporter
   Loaded: loaded (/etc/systemd/system/kafka-exporter.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2020-04-28 13:32:02 UTC; 6h ago
  Process: 3532 ExecStart=/opt/kafka_exporter/kafka_exporter --kafka.server=ec2-15-236-37-69.eu-west-3.compute.amazonaws.com:9092 --kafka.server=ec2-35-180-51-166.eu-west-3.compute.amazonaws.com:9092 --web.listen-address=:9308 --web.telemetry-path=/metrics --log.level=info --topic.filter=.* --group.filter=.* --kafka.version=2.0.0 (code=exited, status=2)
 Main PID: 3532 (code=exited, status=2)

Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Unit kafka-exporter.service entered failed state.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service failed.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service holdoff time over, scheduling restart.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Stopped Service that runs Kafka Exporter.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: start request repeated too quickly for kafka-exporter.service
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Failed to start Service that runs Kafka Exporter.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Unit kafka-exporter.service entered failed state.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service failed.

[root@ec2-15-236-37-69 ~]# journalctl -u kafka-exporter
-- Logs begin at Tue 2020-04-28 13:31:52 UTC, end at Tue 2020-04-28 20:20:02 UTC. --
Apr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Started Service that runs Kafka Exporter.
Apr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[2247]: time="2020-04-28T13:31:56Z" level=info msg="Starting kafka_exporter (version=1.2.0, branch=HEAD, revision=830660212e6c109e69Apr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[2247]: time="2020-04-28T13:31:56Z" level=info msg="Build context (go=go1.10.3, user=root@981cde178ac4, date=20180707-14:34:48)" souApr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[2247]: time="2020-04-28T13:31:56Z" level=error msg="Error Init Kafka Client" source="kafka_exporter.go:210"
Apr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Unit kafka-exporter.service entered failed state.
Apr 28 13:31:56 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service failed.
Apr 28 13:31:57 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service holdoff time over, scheduling restart.
Apr 28 13:31:57 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Stopped Service that runs Kafka Exporter.
Apr 28 13:31:57 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Started Service that runs Kafka Exporter.
Apr 28 13:31:57 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[2392]: time="2020-04-28T13:31:57Z" level=info msg="Starting kafka_exporter (version=1.2.0, branch=HEAD, revision=830660212e6c109e69Apr 28 13:31:57 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[2392]: time="2020-04-28T13:31:57Z" level=info msg="Build context (go=go1.10.3, user=root@981cde178ac4, date=20180707-14:34:48)" souApr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[2392]: time="2020-04-28T13:31:59Z" level=error msg="Error Init Kafka Client" source="kafka_exporter.go:210"
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Unit kafka-exporter.service entered failed state.
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service failed.
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service holdoff time over, scheduling restart.
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Stopped Service that runs Kafka Exporter.
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Started Service that runs Kafka Exporter.
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: time="2020-04-28T13:31:59Z" level=info msg="Starting kafka_exporter (version=1.2.0, branch=HEAD, revision=830660212e6c109e69Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: time="2020-04-28T13:31:59Z" level=info msg="Build context (go=go1.10.3, user=root@981cde178ac4, date=20180707-14:34:48)" souApr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: time="2020-04-28T13:31:59Z" level=error msg="Error Init Kafka Client" source="kafka_exporter.go:210"
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: goroutine 1 [running]:
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: main.NewExporter(0xc4201377e0, 0x2, 0x2, 0x100, 0x8f22f8, 0x0, 0x8f22f8, 0x0, 0x0, 0x8f22f8, ...)
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: /go/src/github.com/danielqsj/kafka_exporter/kafka_exporter.go:211 +0x847
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: main.main()
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com kafka_exporter[3123]: /go/src/github.com/danielqsj/kafka_exporter/kafka_exporter.go:481 +0x215b
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Unit kafka-exporter.service entered failed state.
Apr 28 13:31:59 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service failed.
Apr 28 13:32:00 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service holdoff time over, scheduling restart.
Apr 28 13:32:00 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Stopped Service that runs Kafka Exporter.
[...]
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: start request repeated too quickly for kafka-exporter.service
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Failed to start Service that runs Kafka Exporter.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: Unit kafka-exporter.service entered failed state.
Apr 28 13:32:02 ec2-15-236-37-69.eu-west-3.compute.amazonaws.com systemd[1]: kafka-exporter.service failed.

[mkyc]

@przemyslavic please check if that one still exists.

[przemyslavic]

✔️ kafka-exporter service starts with no issues after system reboot. Tested on all OSes.