Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow setting index field limit during stack monitoring e2e test. #8284

Merged
merged 2 commits into from
Dec 3, 2024

Conversation

naemono
Copy link
Contributor

@naemono naemono commented Dec 2, 2024

Related: elastic/beats#41640
Resolves: #8250
Reverts: #8251

This change allows the TestMetricbeatStackMonitoringRecipe test to set a custom index field limit as Metricbeat versions >= 8.16.x seem to have an index template that exceeds the default limit of 10,000 fields per index. This sets this limit on all versions of this test, and doesn't limit to >= 8.16.x, but that can be enabled if we feel it's needed. It seems harmless to set this across all versions.

@naemono naemono added >bug Something isn't working >test Related to unit/integration/e2e tests labels Dec 2, 2024
@naemono
Copy link
Contributor Author

naemono commented Dec 2, 2024

buildkite test this -f p=gke,t=Test.*StackMonitoring,s=8.16.0

@pebrc
Copy link
Collaborator

pebrc commented Dec 3, 2024

Why are we not using the workaround described in the known issue in the beats repo?

    setup.template.settings:
        index.mapping.total_fields.limit: 12500

As a conditional element in https://github.com/elastic/cloud-on-k8s/blob/main/pkg/controller/kibana/stackmon/metricbeat.tpl.yml

@naemono
Copy link
Contributor Author

naemono commented Dec 3, 2024

Why are we not using the workaround described in the known issue in the beats repo?

    setup.template.settings:
        index.mapping.total_fields.limit: 12500

As a conditional element in https://github.com/elastic/cloud-on-k8s/blob/main/pkg/controller/kibana/stackmon/metricbeat.tpl.yml

I didn't consider this approach. Definitely simpler. I'll get this implemented, and ping you when done. Thanks for the pointer.

@naemono
Copy link
Contributor Author

naemono commented Dec 3, 2024

buildkite test this -f p=gke,t=Test.*StackMonitoring,s=8.16.0

@naemono
Copy link
Contributor Author

naemono commented Dec 3, 2024

@pebrc This has been updated, and the e2e tests are passing for stack monitoring.

Copy link
Collaborator

@pebrc pebrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@naemono naemono merged commit 7f816b6 into elastic:main Dec 3, 2024
5 checks passed
@naemono naemono deleted the adjust-total-fields-limit-beat-e2e branch December 3, 2024 21:25
naemono added a commit to naemono/cloud-on-k8s that referenced this pull request Dec 3, 2024
…astic#8284)

* Allow setting index field limit during stack monitoring e2e test.
* Use beats setup to handle initial index settings.

---------

Signed-off-by: Michael Montgomery <[email protected]>
(cherry picked from commit 7f816b6)
@naemono
Copy link
Contributor Author

naemono commented Dec 3, 2024

💚 All backports created successfully

Status Branch Result
2.16

Questions ?

Please refer to the Backport tool documentation

naemono added a commit that referenced this pull request Dec 4, 2024
) (#8288)

* Allow setting index field limit during stack monitoring e2e test.
* Use beats setup to handle initial index settings.

---------

Signed-off-by: Michael Montgomery <[email protected]>
(cherry picked from commit 7f816b6)
@naemono naemono added the v2.16.0 label Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug Something isn't working >test Related to unit/integration/e2e tests v2.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Beats ingest_pipeline e2e test for stack monitoring seems broken in 8.16.
2 participants