-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pause and Resume Functionality in Processor #404
Comments
@johnmehan Yes it is possible to add the pause and resume functionality. |
@roubanakhle So, in order to gain a performance improvement, we have our callback queuing messages into a queue. We then have a separate process consuming from this queue and committing messages when processed. We have achieved a substantial speed improvement but we do need to pause/resume our consumer in order to manage memory usage. |
@johnmehan sorry for the long delay, I gave it a quick shot in this PR: #411. |
This seems to be exactly what we need. Our usecase is that we have a service A that is dependent on another service B for its lookups. Service A consumes diagnostics but for each consumed message it needs to check if the source of the diagnostic message is still in active service (we want to ignore diagnostics from inactive sources). Whether the source is active or inactive is determined by service B. To service A, these two scenarios will look identical:
To alleviate this, we can provide a heartbeat from service B, and if the heartbeats stop, pause consuming diagnostics in service A until heartbeats resume. |
Would it be possible to add functionality to pause and resume the Processor?
The sarama.Consumer already has these methods: Pause, PauseAll, Resume, ResumeAll.
Thanks
The text was updated successfully, but these errors were encountered: