stream: made it easier to run a batch consumer and added methods to convert between normal and batch consumer callbacks #1180
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This exposes additional options for AWS clients to the config files. These include: - Disabling request compression and min compression request size for cloudwatch - Disabling response checksum validation and accepting gzip compression for dynamodb - The usePathStyle setting was documented for S3 (but it was already exposed before) These settings are needed when talking to services implementing the AWS API like localstack or ScyllaDB as these services sometimes can't map all features exactly.
…onvert between normal and batch consumer callbacks; This allows you to use `application.RunBatchConsumer` similary to how you would use `application.RunConsumer` to run a batch consumer. It also adds `stream.ConsumerToBatchConsumer` and `stream.ConsumerToParallelBatchConsumer` to adapt single consumers to batch consumers. You should set stream.consumer.<name>.batch_size to take advantage of batching as it defaults to 1.
ajscholl
force-pushed
the
make-batch-consumer-available
branch
from
201943f to
d113b65
Compare
…unner; Right now, `stream.consumer.<consumer>.runner_count` must be equal to 1 when using the batch consumer as the batch consumer is not thread safe. This commit fixes this and thus allows processing and acknowledging more than one batch at once.
This commit allows the batch consumer to acknowledge aggregate messages. Until now, the consumer would struggle to find the correct message ids to delete the messages from something like SQS, making it impossible to actually acknowledge an aggregate message in a batch. We now correctly preserve the original message as well as added some delayed context cancels in case of needing to write a message to a retry queue (otherwise, the normal consumer would just lose the message if the queue doesn't have any native retry functionality).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows you to use
application.RunBatchConsumersimilary to how you would useapplication.RunConsumerto run a batch consumer. It alsostream.ConsumerToBatchConsumerandstream.ConsumerToParallelBatchConsumeradapt single consumers to batch consumers.You should set stream.consumer..batch_size to take advantage of batching as it defaults to 1.
TODO: