Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added MaxOpenRequests to kafka output config #42515

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

flexitrev
Copy link

@flexitrev flexitrev commented Jan 30, 2025

Enhancement

Proposed commit message

Added MaxOpenRequests with a default of 5 to configuration options. This will increase throughput, but could have an impact on event ordering.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 30, 2025
@botelastic
Copy link

botelastic bot commented Jan 30, 2025

This pull request doesn't have a Team:<team> label.

Copy link
Contributor

mergify bot commented Jan 30, 2025

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @flexitrev? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Jan 30, 2025

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Jan 30, 2025
@flexitrev flexitrev marked this pull request as ready for review January 31, 2025 19:41
@flexitrev flexitrev requested a review from a team as a code owner January 31, 2025 19:41
@flexitrev flexitrev requested review from AndersonQ and faec January 31, 2025 19:41
CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
===== `max_open_requests`

The maximum number of unacknowledged requests the client will send on a single connection before blocking.
Throughput can improve but message ordering is not guaranteed if Idempotent is disabled, see:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont expose idempotent as a setting and the default value for this also impacts message ordering so I think we can get rid of the comment about idempotent and message ordering?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed reference to idempotent, but I think it's worth keeping the note about the impact on message ordering. If we aren't explicit about the lack of guarantee, it will be easy for someone to get the wrong idea

better documentation

Co-authored-by: William Easton <[email protected]>
Comment on lines 27 to +28
*Filebeat*

- Added max_open_requests to kafka output config {pull}42515[42515]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't it apply to all beats? Here it's listed as only affecting filebeat

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants