You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Mar 29, 2023. It is now read-only.
Describe the bug
When running the report-parser and slack-connector components with more than one instance, sending a ToolReport to Kiln will result in duplicate messages being sent to Slack. When tested with 2 report-parser instances and 2 slack-connector instances, messages were replicated an additional 3 times.
To Reproduce
Follow instructions in Integration Testing guide to prepare a local Kiln stack, but instead of running the documented command to start the report parser and slack connector, run docker-compose up --scale report-parser=2 report-parser and docker-compose up --scale slack-connector=2 slack-connector.
Additional context
Possibly because the kafka crate we're using doesn't seem to handle multiple consumers in a group properly, because calling consumer.commit_consumed() "Persists the so-far "marked as consumed" messages (on behalf of this consumer's group for the underlying topic - if any.)"
Which I think means the whole topic, not just the partitions this consumer is subscribed to
There's an alternative Kafka crate which wraps the librdkafka C library, which as a bonus also supports the new async await syntax (in master fede1024/rust-rdkafka#187, which the maintainers said should be released in a few days)
Based on their docs, that crate seems to handle committing consumed messages within a topic partition correctly
The text was updated successfully, but these errors were encountered:
Describe the bug
When running the report-parser and slack-connector components with more than one instance, sending a ToolReport to Kiln will result in duplicate messages being sent to Slack. When tested with 2 report-parser instances and 2 slack-connector instances, messages were replicated an additional 3 times.
To Reproduce
docker-compose up --scale report-parser=2 report-parser
anddocker-compose up --scale slack-connector=2 slack-connector
.docker run -it --net=host -v "${PWD}:/code" -e SCAN_ENV="Local" -e APP_NAME="RailsGoat" -e DATA_COLLECTOR_URL="http://localhost:8081" kiln/bundler-audit:master-latest
Expected behavior
Each vulnerability in Bundler-audit output is posted to Slack once
Screenshots
If applicable, add screenshots to help explain your problem.
Environment details
Additional context
Possibly because the kafka crate we're using doesn't seem to handle multiple consumers in a group properly, because calling
consumer.commit_consumed()
"Persists the so-far "marked as consumed" messages (on behalf of this consumer's group for the underlying topic - if any.)"Which I think means the whole topic, not just the partitions this consumer is subscribed to
There's an alternative Kafka crate which wraps the librdkafka C library, which as a bonus also supports the new async await syntax (in master fede1024/rust-rdkafka#187, which the maintainers said should be released in a few days)
Based on their docs, that crate seems to handle committing consumed messages within a topic partition correctly
The text was updated successfully, but these errors were encountered: