[Enhancement][libbeat] Add append processor #33364

Vinit023 · 2022-10-16T11:53:38Z

Type of change

Enhancement

What does this PR do?

This PR here is to add the append processor in the libbeat. using this processor users will be able to append values from either a field or static values to either an existing or a new target key in the response.

The code will get 3 parameters target_field, fields and values. And It will append the values of each field listed under the fields section and then it will append all the values listed under the values key.
If the target_key is an existing key with values, It will be appended in an array first.

Why is it important?

As a user I will be able to append values to a field using beats processors.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Filebeat] [Processors] Add Append Processor #29934

Use cases

Screenshots

Logs

cla-checker-service · 2022-10-16T11:53:42Z

💚 CLA has been signed

botelastic · 2022-10-16T11:53:44Z

This pull request doesn't have a Team:<team> label.

mergify · 2022-10-16T11:54:14Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @WinIT23? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2022-10-16T12:00:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-11-16T20:07:42.266+0000
Duration: 77 min 14 sec

Test stats 🧪

Test	Results
Failed	0
Passed	24107
Skipped	1951
Total	26058

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

leehinman · 2022-10-18T18:04:11Z

libbeat/processors/actions/append.go

+	if err != nil {
+		f.logger.Debugf("could not fetch value for key: %s. all the values will be appended in a new key %s.", target, target)
+	} else {
+		arr = append(arr, val)


Do we need to check the type of val here? I'm thinking we need to do the same as line 117 below.

Either way, it would be nice to add a test where target already exists as both a scalar and a list.

I don't think a type check is required. as the event.GetValue returns interface{} and we will have to append if it's a scaler. We just have to iterate in case of a list. Hance the type check on L117.

Let me know if you think otherwise. 🤔

You were right. I have added a type check in this line to avoid appending and array inside an array.

leehinman · 2022-10-18T18:06:51Z

libbeat/processors/actions/docs/append.asciidoc

+<titleabbrev>append_processor</titleabbrev>
++++
+
+The `append_processor` processor appends static values and values from the listed fields to target field.


Can you expand description so it is like the append ingest node processor? I think we want to behavior of these two append implementations to be the same.

https://www.elastic.co/guide/en/elasticsearch/reference/current/append-processor.html

Actually, I have that particular processor in mind while developing this one. I'll update the description in this one though.

PS. could you please let me know if I'm missing any other doc file? CHANGELOG or any other? I'm unaware of the doc files that needed to be changed for libbeat. 😅

…ed test case.

leehinman

Looks good.

Just needs an entry in CHANGELOG.next.asciidoc in the Added section for all beats.

Vinit023 · 2022-11-14T03:33:10Z

Hey @leehinman - Added a changelog entry let me know if anything else is required. 😄

mr1716 · 2022-11-17T15:53:59Z

Will this also be added into Filebeat so Filebeat users can implement it?

Vinit023 · 2022-11-17T17:48:02Z

Hey @mr1716 - It will be accessible in all the Beats. So yes, you'll be able to use it in Filebeat.

mr1716 · 2023-01-12T14:07:47Z

@vinit-chauhan thanks. I dont see the documentation for the append processor here: https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html Does that need to be updated to reflect this new processor?

Vinit023 · 2023-01-12T14:27:11Z

Hey @mr1716 - it is available in the main branch docs here https://www.elastic.co/guide/en/beats/filebeat/master/append.html

@leehinman can you or anyone from your team confirm if the change would be part of 8.7.0? moreover, can we backport it to 8.6.0?

mr1716 · 2023-01-12T14:42:53Z

@vinit-chauhan Thanks/ Much appreciated. It says that it is a part of all beats for 8.6. So if it isnt, there might need to be a change in the release notes as it isnt there for filebeat.

Vinit023 · 2023-01-12T16:39:15Z

Hey @mr1716 you are right, for some reason code is not part of 8.6.0 and I'm not sure why?! I believe someone from elastic team might be able to answer it better.

leehinman · 2023-01-12T18:19:33Z

@leehinman can you or anyone from your team confirm if the change would be part of 8.7.0? moreover, can we backport it to 8.6.0?

The merge didn't happen before feature freeze for 8.6.0, so that is why it isn't there. It will be part of 8.7.0.

mr1716 · 2023-01-12T18:42:27Z

@leehinman can you or anyone from your team confirm if the change would be part of 8.7.0? moreover, can we backport it to 8.6.0?

The merge didn't happen before feature freeze for 8.6.0, so that is why it isn't there. It will be part of 8.7.0.

@leehinman If that's the case, the document should be updated accordingly

Add append processor that matches the behavior of the elasticsearch ingest node append processor

vinit-chauhan added 3 commits October 16, 2022 15:50

initial draft for append processor.

15de94d

Added Unit tests and handled some corner cases.

98e3831

Added unit test for nested fields.

379d304

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2022

mergify bot assigned Vinit023 Oct 16, 2022

vinit-chauhan added 2 commits October 16, 2022 17:59

Added asciidoc for the processor.

7b49c7c

Fixed linting issues.

020214a

Vinit023 force-pushed the add-append-processor branch from 79197aa to 020214a Compare October 16, 2022 22:50

fixed linting issue. (removed package errors)

5029ac4

Vinit023 changed the title ~~Add append processor~~ [Enhancement][libbeat] Add append processor Oct 16, 2022

Vinit023 marked this pull request as ready for review October 16, 2022 23:08

Vinit023 requested a review from a team as a code owner October 16, 2022 23:08

Vinit023 requested review from faec and leehinman and removed request for a team October 16, 2022 23:08

Merge branch 'main' into add-append-processor

7ea6402

leehinman reviewed Oct 18, 2022

View reviewed changes

vinit-chauhan added 2 commits October 19, 2022 01:00

update the description. Added type check in target-field array. Updat…

89c108e

…ed test case.

Added 'allow_duplicate' parameter and code cleanup.

41fbb71

Vinit023 requested review from leehinman and removed request for faec October 19, 2022 19:03

vinit-chauhan added 2 commits October 30, 2022 22:08

Merge branch 'main' into add-append-processor

b39f1af

Merge branch 'main' into add-append-processor

0e1eb3a

leehinman requested changes Nov 10, 2022

View reviewed changes

vinit-chauhan added 2 commits November 14, 2022 09:01

Added entry in CHANGELOG.next

2f6c296

Merge branch 'main' into add-append-processor

ce9c3e9

Merge branch 'main' into add-append-processor

e2b11b3

Vinit023 requested a review from leehinman November 15, 2022 16:15

leehinman approved these changes Nov 16, 2022

View reviewed changes

Merge branch 'main' into add-append-processor

24492ea

leehinman merged commit edc4e07 into elastic:main Nov 17, 2022

Vinit023 mentioned this pull request Jan 23, 2023

[docs] Fix changelog entry for 8.6.0 #34340

Closed

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

[Enhancement][libbeat] Add append processor (#33364)

94bf189

Add append processor that matches the behavior of the elasticsearch ingest node append processor

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement][libbeat] Add append processor #33364

[Enhancement][libbeat] Add append processor #33364

Vinit023 commented Oct 16, 2022 •

edited

Loading

cla-checker-service bot commented Oct 16, 2022 •

edited

Loading

botelastic bot commented Oct 16, 2022

mergify bot commented Oct 16, 2022

elasticmachine commented Oct 16, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

leehinman Oct 18, 2022

Vinit023 Oct 18, 2022

Vinit023 Oct 19, 2022

leehinman Oct 18, 2022

Vinit023 Oct 18, 2022

leehinman left a comment

Vinit023 commented Nov 14, 2022

mr1716 commented Nov 17, 2022

Vinit023 commented Nov 17, 2022

mr1716 commented Jan 12, 2023 •

edited

Loading

Vinit023 commented Jan 12, 2023

mr1716 commented Jan 12, 2023

Vinit023 commented Jan 12, 2023

leehinman commented Jan 12, 2023

mr1716 commented Jan 12, 2023

[Enhancement][libbeat] Add append processor #33364

[Enhancement][libbeat] Add append processor #33364

Conversation

Vinit023 commented Oct 16, 2022 • edited Loading

Type of change

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Oct 16, 2022 • edited Loading

botelastic bot commented Oct 16, 2022

mergify bot commented Oct 16, 2022

elasticmachine commented Oct 16, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

leehinman Oct 18, 2022

Choose a reason for hiding this comment

Vinit023 Oct 18, 2022

Choose a reason for hiding this comment

Vinit023 Oct 19, 2022

Choose a reason for hiding this comment

leehinman Oct 18, 2022

Choose a reason for hiding this comment

Vinit023 Oct 18, 2022

Choose a reason for hiding this comment

leehinman left a comment

Choose a reason for hiding this comment

Vinit023 commented Nov 14, 2022

mr1716 commented Nov 17, 2022

Vinit023 commented Nov 17, 2022

mr1716 commented Jan 12, 2023 • edited Loading

Vinit023 commented Jan 12, 2023

mr1716 commented Jan 12, 2023

Vinit023 commented Jan 12, 2023

leehinman commented Jan 12, 2023

mr1716 commented Jan 12, 2023

Vinit023 commented Oct 16, 2022 •

edited

Loading

cla-checker-service bot commented Oct 16, 2022 •

edited

Loading

elasticmachine commented Oct 16, 2022 •

edited by jenkins-beats-ci bot

Loading

mr1716 commented Jan 12, 2023 •

edited

Loading