Add support for processors in hints based k8s autodiscover #3107

gizas · 2023-07-20T15:33:04Z

Enhancement

What does this PR do?

This PR adds the support of processors autodiscovery based on hints annotations that are provided from kubernetes pods

Why is it important?

Part of [Meta] Improving Hints Autodiscovery with Elastic Agent #3063

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Author's Checklist

Check processors provided as list data structure
Check processors configuration uses map data structure
Create Unit test for processor
Update online documentation to include examples of hints/processors ---> To be done as part of the [Elastic Agent ] Support of processors functionality with hints autodiscovery #2959
Perform more tests e2e to see that processors work

How to test this PR locally

Install a local kubernetes cluster
Build Elastic stack: elastic-package stack up -d -vvv --version=8.9.0-SNAPSHOT
Connect network of kind and ELK: docker network connect elastic-package-stack_default kind-control-plane
Download Pr code
Build agent image with DEV=true EXTERNAL=true SNAPSHOT=true PLATFORMS=linux/arm64 PACKAGES=docker mage -v package
Create docker image to load to your kind cluster

cd build/package/elastic-agent/elastic-agent-linux-arm64.docker/docker-build
docker build --platform linux/amd64 -t custom-agent-image .

kind load docker-image custom-agent-image:latest --name kind
Load the needed container.yml template in inputs.d folder of agent (Mount the /etc/elastic-agent/inputs.d/container_logs.yml template of https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/container_logs.yml)

docker cp container_logs.yml  kind-control-plane:/var/log/test

Install elastic-agent-standalone
Sample provided:
elastic-agent.txt
Install ngin-x pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        co.elastic.hints/package: "container_logs"
        co.elastic.hints/processors.1.add_fields.target: "project"
        co.elastic.hints/processors.1.add_fields.fields.name: "myproject"
        co.elastic.hints/processors.rename.fields.0.from: "a.g"
        co.elastic.hints/processors.rename.fields.1.to: "e.d"
        co.elastic.hints/processors.rename.fail_on_error: 'false'  
    spec:
      containers:
      - image: nginx
        name: nginx

Then if agent is up and running connect to Elastic-Agent and run inspect command. You should be able to see the new processors added

❯ kubectl exec -ti -n kube-system elastic-agent-7fnbl -- bash
root@kind-control-plane:/usr/share/elastic-agent# elastic-agent inspect -v --variables --variables-wait 2s

[output truncated ...]
- add_fields:
      fields:
        name: myproject
      target: project
  - rename:
      fail_on_error: "false"
      fields:
        "0":
          from: a.g
        "1":
          to: e.d

Related issues

Relates to [Elastic Agent ] Support of processors functionality with hints autodiscovery #2959

Screenshots

Map populated correctly based on hints/processors provided

…rhints_enhancement

mergify · 2023-07-20T15:33:40Z

This pull request does not have a backport label. Could you fix it @gizas? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-07-20T15:58:18Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-01T09:33:11.322+0000
Duration: 26 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6237
Skipped	55
Total	6292

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

ChrsMark

Looks quite good!

I have left a minor comment about the code organisation.
Please add some unit tests and remember to update the respective dev+user docs.

internal/pkg/composable/providers/kubernetes/pod.go

Co-authored-by: Chris Mark <[email protected]>

elasticmachine · 2023-07-31T09:15:28Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.765% (`80/81`)	👍
Files	67.368% (`192/285`)	👍
Classes	66.105% (`353/534`)	👍
Methods	53.04% (`1108/2089`)	👍 0.022
Lines	38.635% (`12653/32750`)	👍 0.016
Conditionals	100.0% (`0/0`)	💚

Co-authored-by: Chris Mark <[email protected]>

ChrsMark

Hey @gizas , thanks for addressing the comments.
I left some minor comments but it's getting there.

changelog/fragments/1691060806-Processor-support-for-hints-autodiscover.yaml

internal/pkg/composable/providers/kubernetes/pod.go

internal/pkg/composable/providers/kubernetes/hints_test.go

…odiscover.yaml Co-authored-by: Chris Mark <[email protected]>

ChrsMark

🔥 lgtm, great to see this landed! 🚀 🚀 🚀

ChrsMark · 2023-08-17T11:05:09Z

internal/pkg/composable/providers/kubernetes/hints_test.go

+
+	hintData := GetHintsMapping(mapping, logger, "co.elastic", "asdfghjkl")
+
+	assert.Equal(t, expected_hints, hintData.composableMapping)


internal/pkg/composable/providers/kubernetes/pod.go

Co-authored-by: Chris Mark <[email protected]>

gizas · 2023-08-17T11:43:52Z

@elastic/elastic-agent-control-plane I need one review from yourside as well in order to merge. Can you please have a look?

ChrsMark · 2023-08-21T15:39:58Z

I guess we should also update the code ownership here as well @cmacknz , right?

…ic-agent into processorhints_enhancement

…rhints_enhancement

michalpristas · 2023-08-22T09:24:55Z

flakyness in a way how maps are used/compared. keys are not sorted.

gizas · 2023-08-22T09:48:29Z

flakyness in a way how maps are used/compared. keys are not sorted.

Fixing this with 21d095b

gizas · 2023-08-23T08:49:35Z

/test

gizas · 2023-08-24T06:59:57Z

/test

gizas · 2023-09-01T09:32:53Z

/test

pierrehilbert · 2023-09-01T12:02:02Z

Unrelated failures. I force merge.

) * adding processors block * Updating code with last eg that retrievs processors * Updating code with processors with reference form utils * fixing initialisation of variable * Update internal/pkg/composable/providers/kubernetes/pod.go Co-authored-by: Chris Mark <[email protected]> * Update internal/pkg/composable/providers/kubernetes/pod.go Co-authored-by: Chris Mark <[email protected]> * adding unit test and fixing struct * fixing initialisation of struct * adding function updateprocessors also to default_containers * adding fragment file * Update changelog/fragments/1691060806-Processor-support-for-hints-autodiscover.yaml Co-authored-by: Chris Mark <[email protected]> * Updating variable names and fixing test to call public GetHintsMapping * Update internal/pkg/composable/providers/kubernetes/pod.go Co-authored-by: Chris Mark <[email protected]> * Updating tests * Updating tests * Updating tests to avoid sorting of mappings --------- Co-authored-by: Chris Mark <[email protected]>

bartoszcisek · 2023-10-02T12:40:10Z

Would this be released in 8.10.3? Is there an ETA for 8.10.3?

gizas · 2023-10-03T07:48:42Z

Would this be released in 8.10.3? Is there an ETA for 8.10.3?

If there is no delay, next week will be the release of both 8.10.3 and 8.11.0. Either way I will notify you in which version this is in

gizas added 5 commits July 7, 2023 15:50

adding processors block

5c20888

Updating code with last eg that retrievs processors

e4d1c95

Merge branch 'main' of github.com:elastic/elastic-agent into processo…

643ac4c

…rhints_enhancement

Updating code with processors with reference form utils

93c8eed

fixing initialisation of variable

0bbf419

gizas requested a review from a team as a code owner July 20, 2023 15:33

gizas requested review from michalpristas, blakerouse, a team, ChrsMark and constanca-m and removed request for a team July 20, 2023 15:33

mergify bot assigned gizas Jul 20, 2023

mergify bot added the backport-skip label Jul 20, 2023

gizas requested review from MichaelKatsoulis, tetianakravchenko and tommyers-elastic July 20, 2023 15:33

ChrsMark requested changes Jul 24, 2023

View reviewed changes

internal/pkg/composable/providers/kubernetes/pod.go Outdated Show resolved Hide resolved

internal/pkg/composable/providers/kubernetes/pod.go Outdated Show resolved Hide resolved

ChrsMark reviewed Jul 24, 2023

View reviewed changes

internal/pkg/composable/providers/kubernetes/pod.go Outdated Show resolved Hide resolved

ChrsMark reviewed Jul 24, 2023

View reviewed changes

internal/pkg/composable/providers/kubernetes/pod.go Outdated Show resolved Hide resolved

ChrsMark reviewed Jul 24, 2023

View reviewed changes

internal/pkg/composable/providers/kubernetes/pod.go Outdated Show resolved Hide resolved

ChrsMark mentioned this pull request Jul 26, 2023

[Elastic Agent ] Support of processors functionality with hints autodiscovery #2959

Closed

Update internal/pkg/composable/providers/kubernetes/pod.go

944962a

Co-authored-by: Chris Mark <[email protected]>

Update internal/pkg/composable/providers/kubernetes/pod.go

84649bb

Co-authored-by: Chris Mark <[email protected]>

gizas added the skip-changelog label Jul 31, 2023

gsantoro mentioned this pull request Aug 1, 2023

Support hints Autodiscovery for input configs per container #3161

Closed

adding unit test and fixing struct

288aa6c

ChrsMark requested changes Aug 16, 2023

View reviewed changes

ChrsMark changed the title ~~Processorhints enhancement~~ Add support for processors in hints based k8s autodiscover Aug 16, 2023

gizas and others added 2 commits August 16, 2023 12:15

Update changelog/fragments/1691060806-Processor-support-for-hints-aut…

2c1db90

…odiscover.yaml Co-authored-by: Chris Mark <[email protected]>

Updating variable names and fixing test to call public GetHintsMapping

89bc94d

ChrsMark approved these changes Aug 17, 2023

View reviewed changes

Update internal/pkg/composable/providers/kubernetes/pod.go

505c3e0

Co-authored-by: Chris Mark <[email protected]>

ChrsMark mentioned this pull request Aug 21, 2023

Add cnative team as codeowner for k8s provider #3271

Merged

michalpristas approved these changes Aug 22, 2023

View reviewed changes

gizas added 3 commits August 22, 2023 10:48

Updating tests

19982fb

Merge branch 'processorhints_enhancement' of github.com:elastic/elast…

b4f325a

…ic-agent into processorhints_enhancement

Merge branch 'main' of github.com:elastic/elastic-agent into processo…

2f218d7

…rhints_enhancement

gizas added 2 commits August 22, 2023 12:28

Updating tests

9904bab

Updating tests to avoid sorting of mappings

21d095b

gizas added the 8.10-candidate label Aug 23, 2023

gizas mentioned this pull request Aug 24, 2023

Processor hints doc elastic/ingest-docs#379

Merged

pierrehilbert merged commit 9bf74e4 into main Sep 1, 2023

pierrehilbert deleted the processorhints_enhancement branch September 1, 2023 12:02

gizas mentioned this pull request Sep 14, 2023

Support Hints per container input #3416

Merged

7 tasks

gizas mentioned this pull request Mar 5, 2024

[Elastic Agent]-PR- Introduce log message for not supported annotations for Hints based autodiscover #4360

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for processors in hints based k8s autodiscover #3107

Add support for processors in hints based k8s autodiscover #3107

gizas commented Jul 20, 2023 •

edited

Loading

mergify bot commented Jul 20, 2023

elasticmachine commented Jul 20, 2023 •

edited

Loading

Build stats

Test stats 🧪

ChrsMark left a comment

elasticmachine commented Jul 31, 2023 •

edited

Loading

ChrsMark left a comment

ChrsMark left a comment

ChrsMark Aug 17, 2023

gizas commented Aug 17, 2023

ChrsMark commented Aug 21, 2023

michalpristas commented Aug 22, 2023 •

edited

Loading

gizas commented Aug 22, 2023

gizas commented Aug 23, 2023

gizas commented Aug 24, 2023

gizas commented Sep 1, 2023

pierrehilbert commented Sep 1, 2023

bartoszcisek commented Oct 2, 2023

gizas commented Oct 3, 2023


		hintData := GetHintsMapping(mapping, logger, "co.elastic", "asdfghjkl")

		assert.Equal(t, expected_hints, hintData.composableMapping)

Add support for processors in hints based k8s autodiscover #3107

Add support for processors in hints based k8s autodiscover #3107

Conversation

gizas commented Jul 20, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

mergify bot commented Jul 20, 2023

elasticmachine commented Jul 20, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

ChrsMark left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 31, 2023 • edited Loading

🌐 Coverage report

ChrsMark left a comment

Choose a reason for hiding this comment

ChrsMark left a comment

Choose a reason for hiding this comment

ChrsMark Aug 17, 2023

Choose a reason for hiding this comment

gizas commented Aug 17, 2023

ChrsMark commented Aug 21, 2023

michalpristas commented Aug 22, 2023 • edited Loading

gizas commented Aug 22, 2023

gizas commented Aug 23, 2023

gizas commented Aug 24, 2023

gizas commented Sep 1, 2023

pierrehilbert commented Sep 1, 2023

bartoszcisek commented Oct 2, 2023

gizas commented Oct 3, 2023

gizas commented Jul 20, 2023 •

edited

Loading

elasticmachine commented Jul 20, 2023 •

edited

Loading

elasticmachine commented Jul 31, 2023 •

edited

Loading

michalpristas commented Aug 22, 2023 •

edited

Loading