Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Merge with AppendValues option when merging default config with hints generated config #36857

Merged
merged 4 commits into from
Oct 24, 2023
Merged

Use Merge with AppendValues option when merging default config with hints generated config #36857

merged 4 commits into from
Oct 24, 2023

Conversation

MichaelKatsoulis
Copy link
Contributor

This PR is the resolution of #36838 issue where the problem is described in details.

In filebeat, when kubernetes autodisocver is used with hints enabled, processors and other array fields (paths, parsers) are not merged correctly.
The values coming from default config and the ones generated from hints should be merged by appending the hints generated values to the default config's ones in case of arrays like processors.

The Merge method used does not append array values by default.
Instead MergeWithOpts and AppendValues option should be used.

@MichaelKatsoulis MichaelKatsoulis requested a review from a team as a code owner October 16, 2023 07:59
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2023
@mergify
Copy link
Contributor

mergify bot commented Oct 16, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @MichaelKatsoulis? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@MichaelKatsoulis MichaelKatsoulis requested review from ChrsMark and removed request for faec October 16, 2023 08:00
@MichaelKatsoulis MichaelKatsoulis added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label Oct 16, 2023
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2023
@MichaelKatsoulis MichaelKatsoulis added needs_team Indicates that the issue/PR needs a Team:* label backport-skip Skip notification from the automated backport with mergify labels Oct 16, 2023
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2023
gizas
gizas reviewed Oct 16, 2023
View reviewed changes
filebeat/autodiscover/builder/hints/logs.go
@@ -139,7 +139,8 @@ func (l *logHints) CreateConfig(event bus.Event, options ...ucfg.Option) []*conf
kubernetes.ShouldPut(tempCfg, json, jsonOpts, l.log)
}
// Merge config template with the configs from the annotations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this now. Is not needed probably

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is still a valid comment as it describes what is merged. And then we describe why AppendValues option is used.

@pierrehilbert pierrehilbert added the Team:Elastic-Agent Label for the Agent team label Oct 16, 2023
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 16, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-16T11:45:27.808+0000

  • Duration: 74 min 46 sec

Test stats 🧪

Test Results
Failed 0
Passed 8297
Skipped 753
Total 9050

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

ChrsMark
ChrsMark approved these changes Oct 16, 2023
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I think a changelog entry would be needed here since it fixes a user facing issue.

@mergify
Copy link
Contributor

mergify bot commented Oct 16, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hints-autodiscover-merge-config-with-appendvalues upstream/hints-autodiscover-merge-config-with-appendvalues
git merge upstream/main
git push upstream hints-autodiscover-merge-config-with-appendvalues

@MichaelKatsoulis
Copy link
Contributor Author

@elastic/elastic-agent-data-plane could you please review this PR as you are the code owners ?

@MichaelKatsoulis MichaelKatsoulis merged commit 670b682 into elastic:main Oct 24, 2023
8 checks passed
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
@MichaelKatsoulis @gizas @Scholar-Li
Use Merge with AppendValues option when merging default config with h…
db3d0ac 
…ints generated config (elastic#36857)

* Use Merge with AppendValues option when merging default config with hints generated config

* Update filebeat/autodiscover/builder/hints/logs.go

Co-authored-by: Andrew Gizas <[email protected]>

* Update changelog

---------

Co-authored-by: Andrew Gizas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

@cmacknz cmacknz cmacknz approved these changes

@gizas gizas gizas approved these changes

@leehinman leehinman Awaiting requested review from leehinman leehinman is a code owner automatically assigned from elastic/elastic-agent-data-plane

@faec faec Awaiting requested review from faec

@belimawr belimawr Awaiting requested review from belimawr

@rdner rdner Awaiting requested review from rdner

Assignees

@MichaelKatsoulis MichaelKatsoulis

Labels
backport-skip Skip notification from the automated backport with mergify Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team Team:Elastic-Agent Label for the Agent team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@MichaelKatsoulis @elasticmachine @gizas @cmacknz @ChrsMark @pierrehilbert