Collect metrics of Kafka consumer groups without active members

[jsoriano]

consumergroup metricset of the Metricbeat kafka module collects consumer group metrics per member in the consumer group. When a consumer group doesn't have any active member, it doesn't collect any data, but this can be important to detect that all members are down, and to keep visualizing the last processed offsets.

Metricbeat should be able to collect metrics both, per consumergroup and per member.

I see two options for that:

• Split the consumergroup metricset in two, one for consumergroups, and another one for members. The one for members should take care of collecting the consumer lag for each member.
• In current consumergroup, generate an event for consumergroups that don't have any member, without the member-specific fields.

Example consumer group without members:

# ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --command-config /kafka/bin/sasl-consumer.properties --describe --group foo-1
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
Consumer group 'foo-1' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
foo             0          1               2               1               -               -               -
foo             1          1               3               2               -               -               -

[MeijerM1]

Is there an update on this?

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[MeijerM1]

👍

[evilezh]

I just hit the same problem. As soon we stop consumer .. there are no metrics anymore coming in. I'm just wondering how this is not hitting thousdans of people.
If there are no metrics - you can't really detect anything.

[wim82]

Same here. I would like to monitor consumer group lag - but if a consumer goes down, i do not see any lag anymore.