Bugfix/disconnect due to slow consumer event drop #289
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
luckyj5
reviewed
Sep 20, 2021
luckyj5
reviewed
Sep 20, 2021
|
How far did we test it? Did we test it with reproducible scenario with slow-consumers errors on tc? |
Yes, tested with a low values of the queue size and HEC workers to reproduce the errors while reading and verified that the new build drops events in the same scenario instead of any errors from web-socket. |
luckyj5
added a commit
that referenced
this pull request
Oct 14, 2021
* update docs (#276) * Update Go version and migrate to Go modules (#287) * Adding go.mod and go.sum to migrate to Go modules from Glide * Removed old Vendor folder after migrating to Go modules * Adding new Vendor folder after migrating to Go mudules * Removed Glide files, references in readme and updated CI/CD * Migrated from bolt to bbolt for cache module to resolve race issue * Added tags from the envelope to the events * Updated tile.yml with new config * Updated README * Updated migration script to prevent unnecessary migration (#288) * Updated cache.GetApp logic to retrieve app info from boltdb database as well * Bugfix/disconnect due to slow consumer event drop (#289) * dropping events when the queue is full * Corrected the newly added test case with data race condition * minor change in logging * added event drop warning threshold in config * minor update in the logging * Updated comments and log info * Trimmed white space for ApiEndPoint and SplunkHost * Updated docs with required field & optional value * Added default value in README * Updated events doc * Removed SPLUNK_VERSION config as not needed anymore * Removed SplunkVersion from eventsink.splunk_test * Updated README with app level index routing (#292) * Updated README with app level index routing * Removed unnecessary changes * added required config to enable index routing * updated index routing doc * added warning in index routing docs * upgraded module version * upgraded gogo/protobuf version * executed go mod tidy * upgraded indirect dependency * updated version in a build example in readme * Corrected index routing docs * Added more details (#299) * Updated docs around - Add Tags: Performance impact of adding tags in events - How to detect slow downstream related issues and event drops * added 'drop_warn_threshold' config in tile (#300) Co-authored-by: Matthew Heidemann <[email protected]> Co-authored-by: Shubham Jain <[email protected]> Co-authored-by: harshit-splunk <[email protected]> Co-authored-by: harshit-splunk <[email protected]>
kashyap-splunk
deleted the
bugfix/disconnect-due-to-slow-consumer-event-drop
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding mechanism to drop events if the events queue is full to avoid the blocking of the main nozzle goroutine and hence avoid passing the back pressure back to Doppler when the downstream is slow.
There is still room to further improve it by moving the event parsing from main thread to writer threads. This will be addressed in later pull-requests targeted in subsequent releases.