Added more details #299
Merged
Added more details #299
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added details for: - Add Tags: Performance impact of adding tags in events - How to detect slow downstream related issues and event drops
luckyj5
requested changes
Oct 14, 2021
README.md
Outdated
| @@ -81,6 +81,7 @@ This is recommended for dev environments only. | |||
| * `FIREHOSE_KEEP_ALIVE`: Keep alive duration for the Firehose consumer. (Default: 25s) | |||
| * `ADD_APP_INFO`: Enrich raw data with app info. A comma separated list of app metadata (AppName,OrgName,OrgGuid,SpaceName,SpaceGuid). (Default: "") | |||
| * `ADD_TAGS`: Add additional tags from envelope to splunk event. (Default: false) | |||
| (Note: Adding tags in the events will increase the event-size and can impact the performance of the Nozzle and Splunk. (During tests with simulator, the event size was increased by ~50% and the resource usage by the nozzle increased by 5-10%. This would vary with environment or CF configurations)) | |||
There was a problem hiding this comment.
It shouldn't impact the overall thruput (mbps), but will impact the expected EPS because of the increased event-size. If thats the correct understanding, please update it acc.
There was a problem hiding this comment.
In the perf testing I did, the EPS were the same with/without tags. Only the resource usage changed, and the indexer throughput as event size increased.
If the nozzle is at 100% capacity and then if tags are added then, theoretically, it would be possible that EPS of nozzle egress get decreased, but we have not been able to reproduce this either. I will try to update the note not to rule out EPS change either.
README.md
Outdated
| ### 4. Check for data loss inside the Splunk Firehose Nozzle: | ||
| ### 4. Check for dropped events due to slow downstream(Network/Splunk): | ||
|
|
||
| If the nozzle emits the ‘dropped events’ warning saying that downstream is slow, then the Splunk HEC might needs to be scaled. |
There was a problem hiding this comment.
Downstream can be slow for multiple reasons:
Splunk HEC receiver node, Splunk Indexer, LB etc
There was a problem hiding this comment.
ok, will update accordingly
luckyj5
added a commit
that referenced
this pull request
Oct 14, 2021
* update docs (#276) * Update Go version and migrate to Go modules (#287) * Adding go.mod and go.sum to migrate to Go modules from Glide * Removed old Vendor folder after migrating to Go modules * Adding new Vendor folder after migrating to Go mudules * Removed Glide files, references in readme and updated CI/CD * Migrated from bolt to bbolt for cache module to resolve race issue * Added tags from the envelope to the events * Updated tile.yml with new config * Updated README * Updated migration script to prevent unnecessary migration (#288) * Updated cache.GetApp logic to retrieve app info from boltdb database as well * Bugfix/disconnect due to slow consumer event drop (#289) * dropping events when the queue is full * Corrected the newly added test case with data race condition * minor change in logging * added event drop warning threshold in config * minor update in the logging * Updated comments and log info * Trimmed white space for ApiEndPoint and SplunkHost * Updated docs with required field & optional value * Added default value in README * Updated events doc * Removed SPLUNK_VERSION config as not needed anymore * Removed SplunkVersion from eventsink.splunk_test * Updated README with app level index routing (#292) * Updated README with app level index routing * Removed unnecessary changes * added required config to enable index routing * updated index routing doc * added warning in index routing docs * upgraded module version * upgraded gogo/protobuf version * executed go mod tidy * upgraded indirect dependency * updated version in a build example in readme * Corrected index routing docs * Added more details (#299) * Updated docs around - Add Tags: Performance impact of adding tags in events - How to detect slow downstream related issues and event drops * added 'drop_warn_threshold' config in tile (#300) Co-authored-by: Matthew Heidemann <[email protected]> Co-authored-by: Shubham Jain <[email protected]> Co-authored-by: harshit-splunk <[email protected]> Co-authored-by: harshit-splunk <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added details for: