-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: [SRE-517] create a consumer module #63
Conversation
@@ -0,0 +1,47 @@ | |||
defmodule Kafee.Consumer.BrodBackendIntegrationTest do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For those interested in seeing how it's actually used: here are two integration tests. Once for brod and one for broadway.
Yet to do:
Things outside this PR:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've looked through the code and everything looks good. Given we've discussed this quite a bit in Slack we should be good to go. Excited to try this out in one of the WMS services.
An automated release has been created for you. --- ## [3.0.0](v2.6.2...v3.0.0) (2023-10-12) ### ⚠ BREAKING CHANGES * `Kafee.Testing` is now reworked to `Kafee.Test` ### Features * [SRE-515] use testing pid for better kafka produce testing ([#58](#58)) ([cae7bce](cae7bce)) * [SRE-517] create a consumer module ([#63](#63)) ([d012734](d012734)) * [SRE-518] setup encoder decoder modules ([#62](#62)) ([b494049](b494049)) * Integrate data-streams into kafee producer ([#52](#52)) ([ffdd5da](ffdd5da)) ### Miscellaneous * **deps:** Update outdated dependencies ([#59](#59)) ([5439acb](5439acb)) * Update codeowners ([#61](#61)) ([10f0290](10f0290)) * Update README code examples ([#64](#64)) ([50bed75](50bed75)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). [SRE-515]: https://stord.atlassian.net/browse/SRE-515?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SRE-517]: https://stord.atlassian.net/browse/SRE-517?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SRE-518]: https://stord.atlassian.net/browse/SRE-518?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
An automated release has been created for you. --- ## [3.0.0](v2.6.2...v3.0.0) (2023-11-05) ### ⚠ BREAKING CHANGES * `Kafee.Producer` configuration is done differently to match how `Kafee.Consumer` works. * the use of "backend" has been renamed to "adapter" * `Kafee.Testing` is now reworked to `Kafee.Test` ### Features * [SRE-515] use testing pid for better kafka produce testing ([#58](#58)) ([cae7bce](cae7bce)) * [SRE-517] create a consumer module ([#63](#63)) ([d012734](d012734)) * [SRE-518] setup encoder decoder modules ([#62](#62)) ([b494049](b494049)) * Integrate data-streams into kafee producer ([#52](#52)) ([ffdd5da](ffdd5da)) * Update producer to match consumer style ([#70](#70)) ([39fc85a](39fc85a)) ### Bug Fixes * Fix consumer directory name typo ([#71](#71)) ([44de1d6](44de1d6)) ### Miscellaneous * **deps:** Update outdated dependencies ([#59](#59)) ([5439acb](5439acb)) * **main:** Release 3.0.0 ([#60](#60)) ([3c1b238](3c1b238)) * Sync files with stordco/common-config-elixir ([#72](#72)) ([2d458ba](2d458ba)) * Update backend copy to adapter to align with Elixir more ([#69](#69)) ([1c2da6b](1c2da6b)) * Update codeowners ([#61](#61)) ([10f0290](10f0290)) * Update codeowners ([#65](#65)) ([e352bc6](e352bc6)) * Update README code examples ([#64](#64)) ([50bed75](50bed75)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). [SRE-515]: https://stord.atlassian.net/browse/SRE-515?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SRE-517]: https://stord.atlassian.net/browse/SRE-517?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SRE-518]: https://stord.atlassian.net/browse/SRE-518?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
This creates a
Kafee.Consumer
module that can take multiple different backends (similar to the producer.) This determines the whole backend process tree. Aside from that, thehandle_message/1
andhandle_failure/2
functions are generic and don't require any code changes based on the backend picked.Kafee.Consumer.BroadwayBackend
This backend uses the Broadway library to process Kafka messages. It's made to be a freight train of maximum throughput that won't stop for anything. This is great for processing a ton of messages that require no order guarantee and have their own dead letter failure logic.
Kafee.Consumer.BrodBackend
This backend will use
:brod
directly. It will spin up a process for each partition to process messages, but will only process them one at a time. This means if a message fails to process, it will be retried and not committed. This sacrifices throughput for safer message handling and order guarantees.nil
Because it's super annoying to try starting a service locally only for it to start throwing "I can't connect to Kafka" errors. By default the
Kafee.Consumer
module will have anil
backend and not start any tree. This makes it easier for development and testing.Different approaches
This module goes a bit different that the producer in that I'm using
NimbleOptions
to parse and validate options given, and avoiding creating aConfig
module to hold everything. This should end up being cleaner code in the long term. If successful, I'll probably take a deeper look into the producer code and try to clean it up before 3.0 release.I also took some inspiration from how Broadway handles
start_link
and__using__
to try and avoid huge macros and cleaner code. I think it's working out better.