[libbeat] kafka message support headers feature #29940

herbxu · 2022-01-21T05:17:21Z

What does this PR do?

kafka message support headers feature

Why is it important?

Kafka Message Header lets consumers process message metadata without having to deserialize the message value/payload. This could be useful in the routing of message use case.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[email protected]

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service · 2022-01-21T05:17:25Z

💚 CLA has been signed

elasticmachine · 2022-01-21T05:22:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-11T22:30:06.048+0000
Duration: 60 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28215
Skipped	3032
Total	31247

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2022-01-21T05:26:38Z

This pull request does not have a backport label. Could you fix it @herbxu? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-01-24T12:45:53Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mergify · 2022-02-01T15:54:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka-output-add-headers upstream/kafka-output-add-headers
git merge upstream/master
git push upstream kafka-output-add-headers

libbeat/outputs/kafka/client.go

rdner

Thank you for your contribution, @herbxu!

In order to merge this PR, we would need a test confirming the new functionality works (see https://github.com/elastic/beats/blob/main/libbeat/outputs/kafka/kafka_integration_test.go). Once it's there, I'm happy to merge the PR.

herbxu · 2022-02-10T06:55:47Z

Thank you for your contribution, @herbxu!

In order to merge this PR, we would need a test confirming the new functionality works (see https://github.com/elastic/beats/blob/main/libbeat/outputs/kafka/kafka_integration_test.go). Once it's there, I'm happy to merge the PR.

I have added the integration test for this feature, thanks !

rdner

Thank you for addressing the comments and adding the test case. I noticed another small improvement that you might want to apply.

It looks like you have not signed the CLA https://www.elastic.co/contributor-agreement

Since you use the privacy feature on GitHub and your commits have a different email address, the automation is not able to detect the CLA signature. However, I checked manually in our contributor database and I could find neither your nickname herbxu nor your email address [email protected]

After the CLA is signed I'll run the CI pipeline and we can merge this PR.

libbeat/outputs/kafka/client.go

Accepted this pre-allocate memory for kafka headers buffer Co-authored-by: Denis Rechkunov <[email protected]>

herbxu · 2022-02-11T12:52:43Z

Thank you for addressing the comments and adding the test case. I noticed another small improvement that you might want to apply.

It looks like you have not signed the CLA https://www.elastic.co/contributor-agreement

Since you use the privacy feature on GitHub and your commits have a different email address, the automation is not able to detect the CLA signature. However, I checked manually in our contributor database and I could find neither your nickname herbxu nor your email address [email protected]

After the CLA is signed I'll run the CI pipeline and we can merge this PR.

I have signed with the email [email protected], thanks for your suggestion.

rdner · 2022-02-11T13:16:01Z

jenkins run tests

rdner · 2022-02-11T22:29:30Z

/test

rdner · 2022-02-14T07:17:14Z

@herbxu just FYI, I had to modify the configuration structure a little bit, see #30369

…into feature/use-with-kind-k8s-env * 'feature/use-with-kind-k8s-env' of github.com:v1v/beats: (52 commits) ci: home is declared within withBeatsEnv ci: use withKindEnv step ci: use getBranchesFromAliases and support next-patch-8 (elastic#30400) Update fields.yml (elastic#29609) Heartbeat: fix browser metrics and trace mappings (elastic#30258) Apply light edits to 8.0 changelog (elastic#30351) packetbeat/beater: make sure Npcap installation runs before interfaces are needed (elastic#30396) Add a ring-buffer reporter to libbeat (elastic#28750) Osquerybeat: Add install verification for osquerybeat (elastic#30388) update windows matrix support (elastic#30373) Refactor of metricbeat process-gathering metrics and system/process (elastic#30076) adjust next changelog wording (elastic#30371) [Metricbeat] azure: move event report into loop validDim loop (elastic#29945) fix: report GitHub Check before the cache (elastic#30372) Add support for non-unique keys in Kafka output headers (elastic#30369) ci: 6 major branch reached EOL (elastic#30357) reduce Elastic Agent shut down time by stopping processes concurrently (elastic#29650) [Filebeat] Add message to register encode/decode debug logs (elastic#30271) [libbeat] kafka message header support (elastic#29940) Heartbeat: set duration to zero for syntax errors (elastic#30227) ...

kafka message support headers feature

c8b6cc7

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 21, 2022

herbxu changed the title ~~kafka message support headers feature~~ [libbeat] kafka message support headers feature Jan 21, 2022

add kafka message headers to CHANGELOG

25733b2

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Jan 21, 2022

mtojek added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jan 24, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 24, 2022

mergify bot assigned herbxu Feb 8, 2022

Merge branch 'main' into kafka-output-add-headers

e88d64d

rdner self-requested a review February 9, 2022 15:20

rdner reviewed Feb 9, 2022

View reviewed changes

libbeat/outputs/kafka/client.go Outdated Show resolved Hide resolved

rdner reviewed Feb 9, 2022

View reviewed changes

add integration test for kafka header feature

92545b1

remove sarama cfg with version

eb1aea1

herbxu requested a review from rdner February 11, 2022 01:59

rdner reviewed Feb 11, 2022

View reviewed changes

libbeat/outputs/kafka/client.go Outdated Show resolved Hide resolved

Update libbeat/outputs/kafka/client.go

e90c2c1

Accepted this pre-allocate memory for kafka headers buffer Co-authored-by: Denis Rechkunov <[email protected]>

rdner added the v8.2.0 label Feb 11, 2022

rdner added the libbeat label Feb 11, 2022

rdner approved these changes Feb 12, 2022

View reviewed changes

rdner merged commit 1bba206 into elastic:main Feb 12, 2022

rdner mentioned this pull request Feb 12, 2022

Add support for non-unique keys in Kafka output headers #30369

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat] kafka message support headers feature #29940

[libbeat] kafka message support headers feature #29940

herbxu commented Jan 21, 2022 •

edited

Loading

cla-checker-service bot commented Jan 21, 2022 •

edited

Loading

elasticmachine commented Jan 21, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Jan 21, 2022

elasticmachine commented Jan 24, 2022

mergify bot commented Feb 1, 2022

rdner left a comment

herbxu commented Feb 10, 2022

rdner left a comment

herbxu commented Feb 11, 2022

rdner commented Feb 11, 2022 •

edited

Loading

rdner commented Feb 11, 2022

rdner commented Feb 14, 2022

[libbeat] kafka message support headers feature #29940

[libbeat] kafka message support headers feature #29940

Conversation

herbxu commented Jan 21, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Jan 21, 2022 • edited Loading

elasticmachine commented Jan 21, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Jan 21, 2022

elasticmachine commented Jan 24, 2022

mergify bot commented Feb 1, 2022

rdner left a comment

Choose a reason for hiding this comment

herbxu commented Feb 10, 2022

rdner left a comment

Choose a reason for hiding this comment

herbxu commented Feb 11, 2022

rdner commented Feb 11, 2022 • edited Loading

rdner commented Feb 11, 2022

rdner commented Feb 14, 2022

herbxu commented Jan 21, 2022 •

edited

Loading

cla-checker-service bot commented Jan 21, 2022 •

edited

Loading

elasticmachine commented Jan 21, 2022 •

edited by jenkins-beats-ci bot

Loading

rdner commented Feb 11, 2022 •

edited

Loading