Add support for kubernetes provider to recognize namespace level defaults #16321

vjsamuel · 2020-02-14T07:15:34Z

This PR allows namespace level defaults to be configured on the namespace object for either service or pod hints autodiscover. This will need:

add_resource_metadata.namespace.enabled: true

to be set on the kubernetes autodiscover provider.

This PR has a known gap where a pod is deployed before the namespace annotations are added. In this case, autodiscover wouldnt be able to add in the defaults. To fix that behavior we would need to ensure that we handle resync events as updates. Currently we ignore resync events as the resource version remains the same.

elasticmachine · 2020-02-14T07:15:36Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-02-14T07:15:36Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

blakerouse · 2020-02-18T18:15:04Z

We have a template that is generally used when creating a PR, could you update this PR to fill out all those sections.

Also did (or could) you file a bug for the behavior your trying to solve and can you provide steps on how to test this PR?

exekias · 2020-02-24T09:56:39Z

libbeat/autodiscover/providers/kubernetes/pod.go

+
+	// Fall back to defaults on the namespace if there were no hints on the pods
+	if len(hints) == 0 {
+		if rawAnn, ok := kubeMeta["defaults"]; ok {


It sounds to me that we should do a mix of namespace hints + pod specific hints? Example:

I may have a setting to do json decode for all pods, but for one specific pod in the namespace I may want to tune how multiline goes, without affecting the JSON parsing?

exekias

Thank you for contributing! I commented in the PR. I think we need to define how namespace hints and Pod hints should interact with each other (order of precedence, full override..), let's keep the conversation going!

Once settled, this will need tests & docs

vjsamuel · 2020-02-24T22:25:42Z

I have added changes such that annotations that are missing on the pod/service are defaulted with the help of namespace level annotations. i have added test cases for the same. @exekias PTAL

exekias · 2020-02-25T10:43:43Z

libbeat/autodiscover/providers/kubernetes/kubernetes.go

 			event["config"] = config
+		} else {
+


empty else?

exekias

Thank you for working on this! I left a few comments

exekias · 2020-02-25T11:38:40Z

libbeat/autodiscover/providers/kubernetes/pod.go

+					for k, v := range namespace.GetAnnotations() {
+						safemapstr.Put(defaults, k, v)
+					}
+					kubemeta["defaults"] = defaults


I don't think we need to create a new key for this. metadata package is already able to put namespace annotations under kubernetes.namespace.annotations right? We can add that to the kubemeta object, in the same way we always add pod annotations.

onl in 8.0 we will have namespace as a MapStr. until then it would be namespace_annotations

good catch, that's true! I think we should do the same here

exekias · 2020-02-25T11:39:07Z

libbeat/autodiscover/providers/kubernetes/pod.go

+		}
+
+		// Look at all the namespace level default annotations and do a merge with priority going to the pod annotations.
+		if rawNsAnn, ok := kubeMeta["defaults"]; ok {


we would be getting namespace.annotations here

exekias · 2020-02-27T12:53:37Z

Thank you for the changes, I think this is looking good! let's have the docs in and it's ready to go 🎉

vjsamuel · 2020-03-03T06:26:54Z

@exekias added docs.

exekias · 2020-03-03T17:13:33Z

ok to test

…ult hints

…ults (elastic#16321) * Add support for kubernetes provider to recognize namespace level default hints (cherry picked from commit 004d36c)

…ognize namespace level defaults (#17054) * Add support for kubernetes provider to recognize namespace level defaults (#16321) * Add support for kubernetes provider to recognize namespace level default hints (cherry picked from commit 004d36c) * Update CHANGELOG.next.asciidoc Co-authored-by: Vijay Samuel <[email protected]>

exekias · 2020-03-26T14:52:47Z

How to test this:

Deploy Filebeat (or Metricbeat, but Filebeat makes more sense here) in kubernetes, enable hints based autodiscover with add_resource_metadata.namespace.enabled: true. Now adding namespace hints should apply to all pods in the namespace.

For instance, adding this annotation to the namespace:

co.elastic.logs/enabled: false

Should result on Filebeat stop retrieving logs from the pods in that namespace, while the others should still be retrieved. Also we should test that this can be overridden per pod, for instance, disable collection for a whole namespace, but enable it for one of the pods only

vjsamuel changed the title ~~Add support for kubernetes provider to recognize namespace level defa…~~ Add support for kubernetes provider to recognize namespace level defaults Feb 14, 2020

vjsamuel force-pushed the add_namespace_defaults branch from dcbe36e to 6ac6d69 Compare February 14, 2020 07:17

exekias added autodiscovery enhancement review Team:Platforms Label for the Integrations - Platforms team needs tests needs_docs labels Feb 14, 2020

andresrc added [zube]: In Review [zube]: Inbox and removed [zube]: In Review [zube]: Inbox labels Feb 14, 2020

blakerouse added the containers Related to containers use case label Feb 18, 2020

exekias reviewed Feb 24, 2020

View reviewed changes

vjsamuel force-pushed the add_namespace_defaults branch from 24d5180 to 21af677 Compare February 24, 2020 22:24

exekias reviewed Feb 25, 2020

View reviewed changes

libbeat/autodiscover/providers/kubernetes/kubernetes.go Outdated

event["config"] = config

} else {

Copy link

Contributor

exekias Feb 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty else?

exekias suggested changes Feb 25, 2020

View reviewed changes

vjsamuel force-pushed the add_namespace_defaults branch from 21af677 to 7440678 Compare February 27, 2020 06:30

exekias mentioned this pull request Feb 27, 2020

Define mapping for new k8s resources #16558

Closed

vjsamuel force-pushed the add_namespace_defaults branch from 7440678 to 9ae203a Compare March 3, 2020 06:26

exekias approved these changes Mar 3, 2020

View reviewed changes

vjsamuel added 3 commits March 9, 2020 22:59

Add support for kubernetes provider to recognize namespace level defa…

0bba2ae

…ult hints

Incorporate review comments

c377c01

Add documentation

54d0b5c

vjsamuel force-pushed the add_namespace_defaults branch from 9ae203a to 54d0b5c Compare March 10, 2020 06:00

exekias merged commit 004d36c into elastic:master Mar 10, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Mar 10, 2020

andresrc removed the [zube]: Done label Mar 10, 2020

vjsamuel deleted the add_namespace_defaults branch March 10, 2020 17:20

exekias added test-plan Add this PR to be manual test plan and removed needs tests needs_docs labels Mar 17, 2020

exekias mentioned this pull request Mar 17, 2020

Cherry-pick #16321 to 7.x: Add support for kubernetes provider to recognize namespace level defaults #17054

Merged

exekias added the v7.7.0 label Mar 17, 2020

andresrc added the test-plan-added This PR has been added to the test plan label Mar 21, 2020

jsoriano mentioned this pull request Sep 15, 2020

Allow hints based autodiscover based on namespace metadata #13913

Closed

tetianakravchenko mentioned this pull request May 5, 2022

Hints autodiscovery:Misleading documentation regarding namespace defaults #31530

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for kubernetes provider to recognize namespace level defaults #16321

Add support for kubernetes provider to recognize namespace level defaults #16321

vjsamuel commented Feb 14, 2020 •

edited

Loading

elasticmachine commented Feb 14, 2020

elasticmachine commented Feb 14, 2020

blakerouse commented Feb 18, 2020

exekias Feb 24, 2020

exekias left a comment

vjsamuel commented Feb 24, 2020

exekias Feb 25, 2020

exekias left a comment

exekias Feb 25, 2020

vjsamuel Feb 26, 2020

vjsamuel Feb 26, 2020

exekias Feb 26, 2020

exekias Feb 25, 2020

exekias commented Feb 27, 2020

vjsamuel commented Mar 3, 2020

exekias commented Mar 3, 2020

exekias commented Mar 26, 2020

Add support for kubernetes provider to recognize namespace level defaults #16321

Add support for kubernetes provider to recognize namespace level defaults #16321

Conversation

vjsamuel commented Feb 14, 2020 • edited Loading

elasticmachine commented Feb 14, 2020

elasticmachine commented Feb 14, 2020

blakerouse commented Feb 18, 2020

exekias Feb 24, 2020

Choose a reason for hiding this comment

exekias left a comment

Choose a reason for hiding this comment

vjsamuel commented Feb 24, 2020

exekias Feb 25, 2020

Choose a reason for hiding this comment

exekias left a comment

Choose a reason for hiding this comment

exekias Feb 25, 2020

Choose a reason for hiding this comment

vjsamuel Feb 26, 2020

Choose a reason for hiding this comment

vjsamuel Feb 26, 2020

Choose a reason for hiding this comment

exekias Feb 26, 2020

Choose a reason for hiding this comment

exekias Feb 25, 2020

Choose a reason for hiding this comment

exekias commented Feb 27, 2020

vjsamuel commented Mar 3, 2020

exekias commented Mar 3, 2020

exekias commented Mar 26, 2020

vjsamuel commented Feb 14, 2020 •

edited

Loading