kafka exporter does not show all Consumer Group

[fedeostrit]

hello I have a problem with kafka exporter does not show all Consumer Groups, it only shows the following Consumer Group __strimzi-topic-operator-kstreams in Grafana Dashboard (examples/metrics/grafana-dashboards/strimzi-kafka-exporter.json)

this is the infrastructure installed with Azure AKS:

• Strimzi 0.24
• Installation method Strimzi operator Helm Chart
• Kubernetes Cluster 1.19.11
• Infraestructure: AKS Cluster 3 nodes
• Strimzi Kafka Cluster: 3 kafka Brokers & 3 Zookeeper

This is my cluster.yaml file to deploy the strimzi cluster with the strimzi operator:

kafkaExporter config, groupRegex: .* and topicRegex: .* :

kafkaExporter:
groupRegex: "."
topicRegex: "."
resources:
requests:
cpu: 300m
memory: 128Mi
limits:
cpu: 600m
memory: 256Mi
logging: debug
enableSaramaLogging: true
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5

---

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka-cluster
spec:
kafka:
version: 2.8.0
replicas: 3
resources:
requests:
memory: 4G
cpu: "1"
limits:
memory: 10G
cpu: "2"
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: external
port: 9094
type: loadbalancer
tls: false
configuration:
bootstrap:
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
brokers:
- broker: 0
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
- broker: 1
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
- broker: 2
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
config:
auto.create.topics.enable: "true"
delete.topic.enable: "true"
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.8"
inter.broker.protocol.version: "2.8"
default.replication.factor: 3
min.insync.replicas: 2
num.partitions: 3
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 250Gi
class: managed-standardssd-retain-sc
deleteClaim: false
- id: 2
type: persistent-claim
size: 250Gi
class: managed-standardssd-retain-sc
deleteClaim: false
- id: 3
type: persistent-claim
size: 250Gi
class: managed-standardssd-retain-sc
deleteClaim: false
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics
key: kafka-metrics-config.yml
logging:
type: inline
loggers:
kafka.root.logger.level: "INFO"
zookeeper:
replicas: 3
resources:
requests:
memory: 500M
limits:
memory: 1G
cpu: "1"
storage:
type: persistent-claim
size: 100G
class: managed-standardssd-retain-sc
deleteClaim: false
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics
key: zookeeper-metrics-config.yml
logging:
type: inline
loggers:
zookeeper.root.logger: "INFO"
kafkaExporter:
groupRegex: "."
topicRegex: "."
resources:
requests:
cpu: 300m
memory: 128Mi
limits:
cpu: 600m
memory: 256Mi
logging: debug
enableSaramaLogging: true
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
entityOperator:
userOperator: {}
topicOperator: {}

kind: ConfigMap
apiVersion: v1
metadata:
name: kafka-metrics
labels:
app: strimzi
data:
kafka-metrics-config.yml: |
# See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
lowercaseOutputName: true
rules:
# Special cases and very specific rules
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
broker: "$4:$5"
- pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
name: kafka_server_$1_connections_tls_info
type: GAUGE
labels:
listener: "$2"
networkProcessor: "$3"
protocol: "$4"
cipher: "$5"
- pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
name: kafka_server_$1_connections_software
type: GAUGE
labels:
clientSoftwareName: "$2"
clientSoftwareVersion: "$3"
listener: "$4"
networkProcessor: "$5"
- pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
name: kafka_server_$1_$4
type: GAUGE
labels:
listener: "$2"
networkProcessor: "$3"
- pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
name: kafka_server_$1_$4
type: GAUGE
labels:
listener: "$2"
networkProcessor: "$3"
# Some percent metrics use MeanRate attribute
# Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w><>MeanRate
name: kafka_$1_$2_$3_percent
type: GAUGE
# Generic gauges for percents
- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
name: kafka_$1_$2_$3_percent
type: GAUGE
- pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
name: kafka_$1_$2_$3_percent
type: GAUGE
labels:
"$4": "$5"
# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_$1_$2_$3_total
type: COUNTER
# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
name: kafka_$1_$2_$3
type: GAUGE
# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
# Note that these are missing the 'sum' metric!
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
name: kafka$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
"$6": "$7"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.), (.+)=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
quantile: "0.$8"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
labels:
"$4": "$5"
- pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
quantile: "0.$6"
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
name: kafka_$1_$2_$3_count
type: COUNTER
- pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
quantile: "0.$4"
zookeeper-metrics-config.yml: |
# See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
lowercaseOutputName: true
rules:
# replicated Zookeeper
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+)><>(\w+)"
name: "zookeeper_$2"
type: GAUGE
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+), name1=replica.(\d+)><>(\w+)"
name: "zookeeper_$3"
type: GAUGE
labels:
replicaId: "$2"
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+), name1=replica.(\d+), name2=(\w+)><>(Packets\w+)"
name: "zookeeper_$4"
type: COUNTER
labels:
replicaId: "$2"
memberType: "$3"
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+), name1=replica.(\d+), name2=(\w+)><>(\w+)"
name: "zookeeper_$4"
type: GAUGE
labels:
replicaId: "$2"
memberType: "$3"
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\d+), name1=replica.(\d+), name2=(\w+), name3=(\w+)><>(\w+)"
name: "zookeeper_$4_$5"
type: GAUGE
labels:
replicaId: "$2"
memberType: "$3"

[scholzj]

If you can properly format the YAML, it will be readable and someone could verify if you have it correct. This way nobody can read it. You should also check if the metrics are in Prometheus.

[fedeostrit]

cluster.zip

hi scholz, i just attached the cluster.yaml in a .zip file for anyone to review

the metrics would have to be in Prometheus because if not, I would be seeing Strimzi's Consumer group defualt __strimzi-topic-operator-kstreams

If not, how could I validate this?

[scholzj]

You have to go to your Prometheus client and check what you see there. It is not about the __strimzi-topic-operator-kstreams but about the other groups you are missing. You should aslo make sure the group you want to see is active.

[fedeostrit]

Good morning Scholzj, I have access to Prometheus and if I look for this metric kafka_consumergroup_current_offset that is used in the grafagna dashboards I only see this result:

In many cases I have Consumers Group that is created automatically because Databricks is used to connect and subscribe to kafka Topics, I leave a view with AKHQ of some ConsumerGroups created automatically that have already consumed messages and are in Empty state because after a while they are deleted automatically

I also leave another screen taken where another active ConsumerGroup is seen but that is not seen in the Prometheus metrics, these ConsumerGroups last a very short time active but if I connect from kafka-console-consumer and consume a topic leaving the ConsumerGroup active, I do not see the metrics

[scholzj]

As I said - I think the groups shown by the Kafka Exporter need to be active. The groups in your pictures do not seem to be active if they do not have any members. I do not know the tooling you use, but I guess Members: 0 suggests they are not active.

[fedeostrit]

hello scholzj, is how you indicate if there are no members the status is "empty" so it is not active, as I said if I connect from a server with kafka-avro-consumer and leave it consuming a topic in the console I see that it is active the ConsumerGroup but I don't see it in the kafkaExporter metrics:

This screen shows a consumerGroup called "console-consumer-88980" that is created with kafka-arvro-consumer from a server and it is seen that it is active with 1 member, this consumer I can leave it active all the time but I cannot see it in Grafagna's dashboar and I don't see it in the metric kafka_consumergroup_current_offset from Promtheus:

[scholzj]

I don't have an answer for you. The metrics are not real time. I have no idea how the other UI you are using works. All I can tell you is that for me the exporter shows all consumer groups I have.

[fedeostrit]

Could you tell me how you are validating this in Prometheus and how are you seeing it in the Grafagna Dashboard?

[scholzj]

I'm not sure I understand what you mean. In Prometheus you can just query it and see the results. The Grafana dashboard just refreshes automatically (not sure about the select boxes, but the charts and group table will).

[fedeostrit]

In prometheus I see the following if I make the query to kafka_consumergroup_current_offset, so you are saying if you throw this query in Promethues you see all the ConsumerGroups

I don't know how the kafkaExporter is receiving the ConsumerGroups or it could be that this KafkaExporter only report data for groups that are returned by the Kafka AdminClient

if in prometheus I see another metric for example kafka_consumergroup_members I see other consumerGroup

[scholzj]

I'm not sure what is the difference between those two ... for me they show the same groups more or less ... but I do not know how they are designed.

[Zarathustra2]

I am having the same issue, consumer groups from a Flink program & a python program show up but consumers from a rust & elixir program do not show up. I am wondering if there are some missing meta information which are not being sent/added in the latter programs in the underlying implementations(?) and thus causing the consumer groups to not show up in the metrics with their consumer group.

Do you know of any needed meta information that need to be included to show up in the metrics as consumer group @scholzj ? I will also open up tickets in the related connectors and see if someone else has the issue, if I find a solution I will post it here as well

[scholzj]

AFAIK, Kafka Exporter decodes the commited consumer offsets from the __consumer_offsets topic. Keep in mind that not every Kafka client has to use the consumer group protocol and not every client has to commit the consumed messages into Kafka.

[Zarathustra2]

I have opened up an issue in the elixir kafka library I use: kafkaex/kafka_ex#471

Unfortunately I am not too knowledgable about kafka under the hood but when a consumer group commits offset, can it also commit the offset without the __consumer_offsets topic? Because every consumer I have commits its offset, wondering if it uses a different mechanism to commit it

[scholzj]

I know nothing about Elixir and only very little about Flink. So I have no idea if this is relevant for them or not. In general, you can store the commit wherever you want ... in a file, in a SQL database, etc. Most commonly, they are stored in Kafka in the __consumer_offsets topic, because that is the easiest and is already available to any Kafka consumer (since, well, they consume form Kafka). But it is not something you have to do to consume from Kafka. As I said, I have no knowledge about the tools you use how they work and what they do, so I'm just covering the general options.

[Zarathustra2]

Thanks appreciate your help, I will try to find out where they are being stored and hopefully that will help troubleshoot why they do not appear

[scholzj]

PS: The Kafka Exporter is not developed by Strimzi - it is a separate project: https://github.com/danielqsj/kafka_exporter ... so it might also make sense to check the issues there.