Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide per-partition first and last offset #28

Open
CAFxX opened this issue Oct 2, 2016 · 1 comment
Open

Provide per-partition first and last offset #28

CAFxX opened this issue Oct 2, 2016 · 1 comment

Comments

@CAFxX
Copy link

CAFxX commented Oct 2, 2016

The offset shell allows to dump the oldest and newest offsets currently available in a partition:
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-GetOffsetShell
The same information is also available from the SimpleConsumer: https://apache.googlesource.com/kafka/+/trunk/core/src/main/scala/kafka/consumer/SimpleConsumer.scala#184

This information seems to be currently missing in the data returned by kazoo-go (you can ask for the current offset of a consumergroup, but not for the per-partition oldest/newset offsets).

@kamermans
Copy link

kamermans commented Apr 18, 2018

I would also like to be able to get the partition offset details, as there seems to be no other way to compute consumer group lag with this library. I am happy to contribute but I also don't know where this information is stored in Zookeeper.

[Edit] it seems this information (first/last offset) is not stored in Zookeeper at all, and needs to be pulled from the Kafka broker. In Sarama this is done with sarama.Client.GetOffset(topic, partitionID, sarama.OffsetOldest) and sarama.Client.GetOffset(topic, partitionID, sarama.OffsetNewest).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants