Consumer registration and rebalancing

[dim]

Are there plans for sarama to directly support automated consumer registration and rebalancing, as described on http://kafka.apache.org/documentation.html#distributionimpl.

Use case: assuming I have N topics with M partitions, and I would to run a consumer group and distribute these across O consumers as evenly as possible (with each message being consumed only once).

The process steps are nicely documented, but is purely based on ZK communication. Should this be implemented directly into sarama or would it be better to create a wrapper library?

I volunteer to help with the development, but I would like to know if there as any existing work or plans I can (should) rely on?

[dim]

I went ahead and created a library on top of sarama: http://godoc.org/github.com/bsm/sarama/cluster

It integrates Sarama with Zookeeper and implements Consumer Group functionality. Should it be merged into the main sarama tree or do you want me to maintain it separately?

[wvanbergen]

Sorry for the very late reply. I think we should include this in Sarama: without it, you don't get many of the advantages that Kafka offers. Let's get it merged!

[dim]

So I'm still working out the API. While the code works just fine, my main concern is currently error/event handling. There is no concise way to feed back errors and state changes to the client. I could sneak in a few ConsumerEvents with an Err, but I would almost prefer to allow users to register event listener callbacks. Thoughts?

[wvanbergen]

Maybe we can do something like this:

cg.Process(chan sarama.ConsumerEvent, chan error)

[rsrsps]

very nice but the name collision makes things a little annoying to pull a fork.

On Mar 19, 2014, at 11:17 AM, Dimitrij Denissenko [email protected] wrote:

I went ahead and created a library on top of sarama: http://godoc.org/github.com/bsm/sarama/cluster

It integrates Sarama with Zookeeper and implements Consumer Group functionality.

—
Reply to this email directly or view it on GitHub.

[wvanbergen]

I will start the proces sof moving this code over in a branch.

[wvanbergen]

See #84

[dim]

Thanks. I also added a more generic event notification in bsm/sarama@3ad706907810c16.

[dim]

I think it's better to keep this separate from the main sarama code and not pollute it with zk dependencies. I have created a new project for it: https://github.com/bsm/sarama-cluster

[arianitu]

@dim now that zookeeper isn't a dependency to implement consumer groups, what are the chances you can move this into the official sarama repo?

[dim]

@arianitu maybe, but there is quite a bit of effort involved and the integration is not 100% straight-forwards as the test cases/setups are more complicated. more importantly: I probably won't be able to find the time for such an effort any time soon, but help is more than welcome