Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support delegation of offset management before consumer group topic consumption begins #120

Merged
merged 5 commits into from
Jan 6, 2022

Conversation

michaelwilner
Copy link
Contributor

One of the features preventing full adoption of franz-go in my application was the ability for consumer groups to swizzle offsets upon assignment of topics & partitions before consumption on those topics begins. In confluent-kafka-go this is illustrated via this example.

This PR offers a means via callback function (set with a GroupOpt) to intercept those assignments and change them before cursors are created.

@twmb
Copy link
Owner

twmb commented Jan 3, 2022

This looks good, I think I may have a comment around some doc string aspect, but I plan to look more either tomorrow or Tuesday. Thanks for the PR, interesting idea! I also need to look into if relative offsets are supported in this code path (I think they may be, but just need to dive in).

@michaelwilner
Copy link
Contributor Author

@twmb thanks for taking a look! As far as I can tell, once we hit assignPartitions right after this callback, relative offsets are indeed calculated:

offset.at = offset.at + offset.relative

@michaelwilner
Copy link
Contributor Author

@twmb anything noteworthy I should be aware of?

pkg/kgo/config.go Outdated Show resolved Hide resolved
pkg/kgo/config.go Outdated Show resolved Hide resolved
pkg/kgo/config.go Outdated Show resolved Hide resolved
@twmb
Copy link
Owner

twmb commented Jan 6, 2022

Looks good, thanks! I'll issue a tag shortly, I have some changes locally I'm working on as well that I aim to finish tonight.

@twmb twmb merged commit 3cb68bf into twmb:master Jan 6, 2022
@twmb
Copy link
Owner

twmb commented Jan 6, 2022

Tagged in v1.3.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants