Only watch metadata for ReplicaSets in metricbeat k8s module #41289

swiatekm · 2024-10-17T14:51:48Z

Proposed commit message

Use metadata watchers for ReplicaSets in the metricbeats Kubernetes module. This is a similar change to #41100, with two major differences:

State metrics for ReplicaSets may actually need all the metadata, so our transform function keeps it.
Due to the way the update handler function switches over the resource type, our resource needs to have TypeMeta present. The informer doesn't set this, so we need to do it ourselves.

A decent chunk of the changes is just passing around a metadata-only k8s client.

For more details see elastic/elastic-agent#5623.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

You need a full K8s cluster, unfortunately. I've tested it using the default elastic-agent standalone manifest, by building a custom container image and loading it into a local kind cluster.

Related issues

Relates Agent and beats store too much ReplicaSet data in K8s elastic-agent#5623

mergify · 2024-10-17T14:52:26Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @swiatekm? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-10-17T14:52:27Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

elasticmachine · 2024-10-17T17:03:39Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

elasticmachine · 2024-10-17T17:03:40Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

swiatekm · 2024-10-17T17:06:30Z

metricbeat/module/kubernetes/util/kubernetes.go

+			return map[string]mapstr.M{id: generalMetaGen.Generate(PersistentVolumeClaimResource, r)}
+		case *kubernetes.StorageClass:
+			return map[string]mapstr.M{id: generalMetaGen.Generate(StorageClassResource, r)}
+		default:


Note: We can't have a branch for PartialObjectMetadata here, because this may represent different resource types. So this falls into the default branch, and that does the right thing as long as TypeMeta is set. This is why our transform function adds the TypeMeta.

Does not this fall under the case of kubernetes.replicaset rather than default?

No, the actual struct we get here has type PartialObjectMetadata.

gizas · 2024-10-18T07:36:29Z

metricbeat/module/kubernetes/util/kubernetes_test.go

+
+	updateFunc := getEventMetadataFunc(log, generalMetaGen, nil, nil)
+
+	deleteFunc := func(r kubernetes.Resource) []string {


Why we need the deleteFunc here in the test?

We are testing the delete trigger, so we need some function there. I copied the one we actually use in NewResourceMetadataEnricher.

(cherry picked from commit 9e6a942)

…#41289)

…#41323) (cherry picked from commit 9e6a942) Co-authored-by: Mikołaj Świątek <[email protected]>

…beat k8s module (#41321) * Only watch metadata for ReplicaSets in metricbeat k8s module (#41289) (cherry picked from commit 9e6a942) * Fix Changelog --------- Co-authored-by: Mikołaj Świątek <[email protected]>

…#41322) (cherry picked from commit 9e6a942) Co-authored-by: Mikołaj Świątek <[email protected]>

…#41289)

swiatekm added enhancement Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Oct 17, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 17, 2024

mergify bot assigned swiatekm Oct 17, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 17, 2024

swiatekm force-pushed the feat/k8s-metadata-replicaset-partial-metricbeats branch from 394678c to 993ef88 Compare October 17, 2024 14:54

Only watch metadata for ReplicaSets in metricbeat k8s module

7cd4d05

swiatekm force-pushed the feat/k8s-metadata-replicaset-partial-metricbeats branch from 993ef88 to 7cd4d05 Compare October 17, 2024 16:07

swiatekm added backport-8.15 Automated backport to the 8.15 branch with mergify backport-8.16 Automated backport with mergify labels Oct 17, 2024

swiatekm marked this pull request as ready for review October 17, 2024 17:03

swiatekm requested a review from a team as a code owner October 17, 2024 17:03

swiatekm requested review from gizas and constanca-m October 17, 2024 17:03

swiatekm requested review from pkoutsovasilis and MichaelKatsoulis October 17, 2024 17:03

swiatekm commented Oct 17, 2024

View reviewed changes

gizas reviewed Oct 18, 2024

View reviewed changes

gizas approved these changes Oct 18, 2024

View reviewed changes

pkoutsovasilis approved these changes Oct 18, 2024

View reviewed changes

swiatekm merged commit 9e6a942 into main Oct 18, 2024
38 checks passed

swiatekm deleted the feat/k8s-metadata-replicaset-partial-metricbeats branch October 18, 2024 15:20

mergify bot pushed a commit that referenced this pull request Oct 18, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (#41289)

abfdae4

(cherry picked from commit 9e6a942)

mergify bot pushed a commit that referenced this pull request Oct 18, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (#41289)

1aa0aa9

(cherry picked from commit 9e6a942)

mergify bot pushed a commit that referenced this pull request Oct 18, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (#41289)

c25f49a

(cherry picked from commit 9e6a942)

swiatekm mentioned this pull request Oct 18, 2024

Agent and beats store too much ReplicaSet data in K8s elastic/elastic-agent#5623

Closed

6 tasks

swiatekm added a commit that referenced this pull request Oct 18, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (#41289)

c3f345f

(cherry picked from commit 9e6a942)

belimawr pushed a commit to belimawr/beats that referenced this pull request Oct 18, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (elastic…

1630413

…#41289)

swiatekm added a commit that referenced this pull request Oct 21, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (#41289) (…

b3abb30

…#41323) (cherry picked from commit 9e6a942) Co-authored-by: Mikołaj Świątek <[email protected]>

swiatekm added a commit that referenced this pull request Oct 21, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (#41289) (…

4c90e88

…#41322) (cherry picked from commit 9e6a942) Co-authored-by: Mikołaj Świątek <[email protected]>

oakrizan pushed a commit to oakrizan/beats that referenced this pull request Oct 21, 2024

Only watch metadata for ReplicaSets in metricbeat k8s module (elastic…

12eb021

…#41289)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only watch metadata for ReplicaSets in metricbeat k8s module #41289

Only watch metadata for ReplicaSets in metricbeat k8s module #41289

swiatekm commented Oct 17, 2024 •

edited

Loading

mergify bot commented Oct 17, 2024

mergify bot commented Oct 17, 2024

elasticmachine commented Oct 17, 2024

elasticmachine commented Oct 17, 2024

swiatekm Oct 17, 2024

gizas Oct 18, 2024

swiatekm Oct 18, 2024

gizas Oct 18, 2024

swiatekm Oct 18, 2024


		updateFunc := getEventMetadataFunc(log, generalMetaGen, nil, nil)

		deleteFunc := func(r kubernetes.Resource) []string {

Only watch metadata for ReplicaSets in metricbeat k8s module #41289

Only watch metadata for ReplicaSets in metricbeat k8s module #41289

Conversation

swiatekm commented Oct 17, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

mergify bot commented Oct 17, 2024

mergify bot commented Oct 17, 2024

elasticmachine commented Oct 17, 2024

elasticmachine commented Oct 17, 2024

swiatekm Oct 17, 2024

Choose a reason for hiding this comment

gizas Oct 18, 2024

Choose a reason for hiding this comment

swiatekm Oct 18, 2024

Choose a reason for hiding this comment

gizas Oct 18, 2024

Choose a reason for hiding this comment

swiatekm Oct 18, 2024

Choose a reason for hiding this comment

swiatekm commented Oct 17, 2024 •

edited

Loading