-
Notifications
You must be signed in to change notification settings - Fork 437
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kafka singleton consumer #947
Conversation
I was wondering if it's the responsibility of enqueue to keep consumers cached. Theoretically one should never want to create a second consumer to the same broker, but I could see a potential use-case when one creates two consumers with different I think this caching should be done in adapter instead. WDYT @dirk39 ? |
@Steveb-p caching in adapter could be a good solution, but taking in examples the sroze implementation, he uses the same class (QueueInteropTransport) to execute the |
@dirk39 I was thinking about this and eventually came to the conclusion that there really is no good approach other than the one you provided. We can't really cache consumers in Therefore, I'll review your code and see if there is anything that catches my attention. I'll request a review from @makasim but will merge this if there is no response from him in a couple of days if it's ok with you? |
good to me. One note: clean cached consumers on context close. |
d656b83
to
130e934
Compare
I believe using multiple consumers attached to the same topic with different Theoretically in newer versions of librdkafka |
Hi @Steveb-p , do you have an idea of when do you expect to merge the PR? |
@makasim ? |
There are two properties |
@makasim Otherwise we should introduce a getter method to wrappers (which is fine and probably should be prefered?) |
It could be done as a separate PR |
Hi, introduce a
rdKafkaConsumers
collection to avoid the creation of multiple consumers for the same queue. This PR solve the issue #894 and #893 (I've tried the solution on this project and it works)