Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(outputs.influxdb_v2): Add rate limit implementation #15742

Merged
merged 1 commit into from
Dec 6, 2024

Conversation

srebhan
Copy link
Member

@srebhan srebhan commented Aug 14, 2024

Summary

This PR adds the ability to limit the output rate to InfluxDBv2 instances. Currently only a fixed window rate-limit for the amount of uncompressed data is supported.

Checklist

  • No AI generated code was used in this PR

Related issues

resolves #14802

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Aug 14, 2024
@srebhan srebhan added the plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins label Aug 14, 2024
@srebhan srebhan self-assigned this Aug 16, 2024
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srebhan - this is a great piece of work. Clearly this needs to be broken up into a variety of PRs to make reviewing easier, however, it accomplishes the goal nicely. I love that you were able to make the size + period config option work.

From the performance side, this nicely appears to have no affect on other existing use cases or if it no set.

I would have you consider working with support to have someone there test this out as well.

@srebhan
Copy link
Member Author

srebhan commented Aug 20, 2024

Thanks @powersj for your review. I'll adapt the spec then to reflect the possibility to set byte-based rate limits. I'll also break down the PR as it includes at least

  • rework InfluxDBv2 plugin
  • cleanup InfluxDBv2 plugin tests
  • spec for partial write errors (being able for an output plugin to selectively drop metrics from a batch)
  • implementation of partial write errors
  • implement a rate limiter
  • adapt InfluxDBv2 to use the ratelimiter

Thanks again for the review!

@srebhan srebhan changed the title feat(output): Add rate limit implementation feat(output.influxdbv2): Add rate limit implementation Aug 21, 2024
@srebhan srebhan changed the title feat(output.influxdbv2): Add rate limit implementation feat(output.influxdb_v2): Add rate limit implementation Aug 21, 2024
@srebhan srebhan changed the title feat(output.influxdb_v2): Add rate limit implementation feat(output.influxdb): Add rate limit implementation Aug 21, 2024
@srebhan srebhan changed the title feat(output.influxdb): Add rate limit implementation feat(outputs.influxdb): Add rate limit implementation Aug 21, 2024
@srebhan srebhan changed the title feat(outputs.influxdb): Add rate limit implementation feat(outputs.influxdb_v2): Add rate limit implementation Sep 11, 2024
@srebhan srebhan force-pushed the outputs_ratelimit branch 3 times, most recently from 1896438 to aa5c7ee Compare December 6, 2024 10:49
@srebhan srebhan marked this pull request as ready for review December 6, 2024 10:50
@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Dec 6, 2024
@srebhan srebhan assigned DStrand1 and unassigned srebhan Dec 6, 2024
@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Dec 6, 2024

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 4.69 % for linux amd64 (new size: 268.7 MB, nightly size 256.7 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

Copy link
Member

@DStrand1 DStrand1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work! Glad to see this landed for next release 😄

@DStrand1 DStrand1 merged commit 1170985 into influxdata:master Dec 6, 2024
27 checks passed
@github-actions github-actions bot added this to the v1.33.0 milestone Dec 6, 2024
justinwwhuang pushed a commit to justinwwhuang/telegraf_fork that referenced this pull request Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/influxdb feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rate limit multiple batch sends
3 participants