Skip to content
This repository has been archived by the owner on Mar 29, 2023. It is now read-only.

[BUG] Running Kiln in a HA configuration causes message duplication #139

Closed
DanHatesNumbers opened this issue Jan 8, 2020 · 0 comments · Fixed by #178
Closed

[BUG] Running Kiln in a HA configuration causes message duplication #139

DanHatesNumbers opened this issue Jan 8, 2020 · 0 comments · Fixed by #178
Assignees
Labels
bug Something isn't working report-parser slack-connector WIP Work In Progress
Milestone

Comments

@DanHatesNumbers
Copy link
Contributor

DanHatesNumbers commented Jan 8, 2020

Describe the bug
When running the report-parser and slack-connector components with more than one instance, sending a ToolReport to Kiln will result in duplicate messages being sent to Slack. When tested with 2 report-parser instances and 2 slack-connector instances, messages were replicated an additional 3 times.

To Reproduce

  1. Follow instructions in Integration Testing guide to prepare a local Kiln stack, but instead of running the documented command to start the report parser and slack connector, run docker-compose up --scale report-parser=2 report-parser and docker-compose up --scale slack-connector=2 slack-connector.
  2. docker run -it --net=host -v "${PWD}:/code" -e SCAN_ENV="Local" -e APP_NAME="RailsGoat" -e DATA_COLLECTOR_URL="http://localhost:8081" kiln/bundler-audit:master-latest
  3. Observe 4 messages posted for each vulnerability found

Expected behavior
Each vulnerability in Bundler-audit output is posted to Slack once

Screenshots
If applicable, add screenshots to help explain your problem.

Environment details

  • Component: Report parser and Slack connector
  • Commit: 6c2acb4

Additional context
Possibly because the kafka crate we're using doesn't seem to handle multiple consumers in a group properly, because calling consumer.commit_consumed() "Persists the so-far "marked as consumed" messages (on behalf of this consumer's group for the underlying topic - if any.)"
Which I think means the whole topic, not just the partitions this consumer is subscribed to

There's an alternative Kafka crate which wraps the librdkafka C library, which as a bonus also supports the new async await syntax (in master fede1024/rust-rdkafka#187, which the maintainers said should be released in a few days)
Based on their docs, that crate seems to handle committing consumed messages within a topic partition correctly

@DanHatesNumbers DanHatesNumbers added this to the v0.3 milestone Jan 10, 2020
@DanHatesNumbers DanHatesNumbers added the WIP Work In Progress label Jan 31, 2020
@DanHatesNumbers DanHatesNumbers self-assigned this Jan 31, 2020
@DanHatesNumbers DanHatesNumbers removed the WIP Work In Progress label Jan 31, 2020
@DanHatesNumbers DanHatesNumbers removed their assignment Jan 31, 2020
@DanHatesNumbers DanHatesNumbers self-assigned this Feb 20, 2020
@DanHatesNumbers DanHatesNumbers added the WIP Work In Progress label Feb 20, 2020
This was referenced Feb 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working report-parser slack-connector WIP Work In Progress
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant