[output.wavefront] Introduced "immediate_flush" flag #8165
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required for all PRs:
Addresses an issue mentioned in #7409 concerning the "double buffering" done by the Wavefront SDK. The SDK is designed to send metrics asynchronously in order to minimize the delay experienced by a caller. This behavior is not ideal when used with Telegraf, since Telegraf will assume the connection to Wavefront can accept data very quickly. When the internal buffer in the Wavefront SDK overflows, it will drop metrics. Meanwhile, Telegraf will still think the metrics are flowing at a very high rate and will keep flooding the SDK.
To avoid extensive redesign of the SDK, this PR adds a flag,
immediate_flush
that flushes the internal SDK buffers after each batch. This will ensure that the buffer never fills with more records than the batch size and also has the side effect of providing "back pressure" to the Telegraf core. Thanks to that back pressure, Telegraf will no longer be "tricked" into thinking the connection has virtually unlimited capacity and can take whatever the action it needs to manage the throughput gracefully.