This exporter can be used to export offset details of topics, get consumer offsets, get consumer counts in a group, kafka cluster brokers information, consumer lags, topic and partition details etc.
For consumer lags and offsets monitoring, it works with the new high level consumers only right now. The old high level consumer where offsets are stored in Zookeeper and where consumer-coordinations happen through Zookeeper is not tested.
You need to have golang installed to install the tool. I haven't uploaded any prebuilt binary for any platoform. I will be adding a dockerfile to build a Docker image shortly.
Run the below command:
$ go get github.com/nipuntalukdar/kafka_consumer_exporter
$ go install github.com/nipuntalukdar/kafka_consumer_exporter
An example: $ kafka_consumer_exporter -group agroup:topica,topicb -group anothergroup:topicx,topicy,topicz -topics mytopic,yourtopic
Detailed usage shown below:
$ kafka_consumer_exporter -h
Usage of ./kafka_consumer_exporter:
-group value
consumer-group and topics in the form of group1:topic1,topic2,topic3 etc
-kafka_brokers string
Comma-separated list of Kafka brokers (default "127.0.0.1:9092")
-listen_address string
http port where metrics are published (default ":10001")
-metrics_url string
URL where mettrics is accessible (default "/metrics")
-namespace string
Namespace for metrics (default "kafka")
-topics string
Comma separated list of kafka topics
-with-go-metrics
Should we import go runtime and http handler metrics
Clone this repository. Go to directory dashboardgen under the repository directory. Update the input.json with the topics and consumer groups you want to monitor. For example, if you want to monitor group "SomeGroup" which consumes from topis TopicA, TopicB, TopicC and also want to monitor another consumer group "SecretGroup" which consumes from topics SecretA, SecretB and also if you want to monitor extra topics ExtraTopicA, ExtraTopicB, then you should update the input.json as shown below:
{
"consumergroups": {
"SomeGroup": [
"TopicA",
"TopicB",
"TopicC"
],
"SecretGroup": [
"SecretA",
"SecretB"
]
},
"topics": [
"TopicA",
"TopicB",
"TopicC",
"SecretA",
"SecretB",
"ExtraTopicA",
"ExtraTopicB"
]
}
The dashboard needs an environment (so that we may cater for multiple Kafka cluster). A Prometheus target may look like as the one shown below:
- job_name: 'KafkaTopicAndConsumer'
honor_labels: true
static_configs:
- targets: ['localhost:10001']
labels:
env: 'myenv'
Now, you issue the below command (from dashboardgen directory). We are assuming environment label griven to Prometheus exporter is myenv.
$ python kafkadashboard.py myenv > kafkadashboard.json
kafkadashboard.json may be imported to Grafana now !!!
A sample dashboard screenshot is given below: