[Kubernetes] Add kustomize template for hints auto discover #5643

MichaelKatsoulis · 2024-10-01T08:04:01Z

What does this PR do?

This PR

adds a Kustomize template to enable hints based auto discovery by default in standalone elastic agent.
It adds a init container block in elastic agent daemonset.
For all the rest resources needed for the deployment of elastic agent it relies on the resources from the default kustomize templates.
Updates the volume mount of the init container and adds a shared volume between init and main container for storing the input templates. This allows for removal of root privileges in init container which can be a security vulnerability.

Why is it important?

To enable users of the Observability Kubernetes quickstart to make use of hints auto discovery without the need of configuring and redeploying their manifests.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Follow instructions of https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-kustomize/default#remote-usage-of-kustomize-templates

Use https://github.com/MichaelKatsoulis/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone\?ref\=kustomize_templates_hints_autodiscover for the kustomization file link.

Related issues

Closes https://github.com/elastic/ingest-dev/issues/3760

mergify · 2024-10-01T08:04:51Z

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-10-01T08:04:51Z

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone/kustomization.yaml

pkoutsovasilis · 2024-10-02T07:39:22Z

deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone/kustomization.yaml

+                runAsUser: 0
+              volumeMounts:
+                - name: elastic-agent-state
+                  mountPath: /usr/share/elastic-agent/state


Could you extend the integration k8s tests by adding a new one with hints enabled? it would be nice to check that the agent and all components are starting and reporting healthy with this feature enabled

Which tests are you referring to Panos?

I had in mind these tests https://github.com/elastic/elastic-agent/blob/main/testing/integration/kubernetes_agent_standalone_test.go#L63 🙂

Thank you !! I wanted to double check as we dont actually own them.

But I created an issue here: #5660. We can discuss internally for prioritisation

AFAICT the k8s ownership is shared, and the agent repo now has k8s integration tests. So ideally anything that has to do with different k8s deployments should be covered by a test but sure discuss #5660 and try to couple this integration test with this PR

swiatekm · 2024-10-02T10:45:03Z

I'm not sure I fully understand the intent of this PR, but shouldn't we also be modifying https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml?

gizas · 2024-10-02T12:19:22Z

@swiatekm no the main standalone manifest should not be affected

The users should run a command like the following and make an installation of agent+k8s integration+system integration+kube-state-metrics up and running:

kubectl kustomize https://github.com/MichaelKatsoulis/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone\?ref\=kustomize_templates_hints_autodiscover | sed -e "s/JUFQSV9LRVkl/UTWamNtcGFkdz09/g" -e "s/%ES_HOST%/https:\/\/e56ee37c53644df6940a8ae217ab5960.us-central1.gcp.cloud.es.io:443/g" -e "s/%CA_TRUSTED%/ca_trusted_fingerprint/g" -e "s/%ONBOARDING_ID%/12345/g" | kubectl apply -f

Please read some more info here

gizas · 2024-10-02T14:01:16Z

Tested successfully the PR with a current Elastic Cloud setup.

Prereq: Go to Stack Management > API Keys and create a new key

Important use the beats format key

Create the encoded value by running:

echo -n oB1lTZIBdpMiLQUgtFM4:Z1bs7_ufTuOj5O8pg0Sxmg | base64
b0IxbFRaSUJkcE1pTFFVZ3RGTTQ6WjFiczdfdWZUdU9qNU84cGcwU3htZw==

Then run the following:

kubectl kustomize https://github.com/MichaelKatsoulis/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone\?ref\=kustomize_templates_hints_autodiscover | sed -e "s/JUFQSV9LRVkl/b0IxbFRaSUJkcE1pTFFVZ3RGTTQ6WjFiczdfdWZUdU9qNU84cGcwU3htZw==/g" -e "s/%ES_HOST%/https:\/\/<CloudURL>:443/g" -e "/{CA_TRUSTED}/c\ "  > k2.yaml

Replace the image referneces:
9.0 > 8.15
9.0.0 > 8.15.0

Then run kubectl apply -f k2.yaml

Metrics and logs collection is successfull. I will run some more tests with autodiscovery templates and annotations in pods.
But main fucntionality is ok

elasticmachine · 2024-10-02T23:33:08Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

gizas · 2024-10-03T12:54:54Z

Continuation of testing, the autodiscovery works as expected:

Once the pod gets annotated with:

annotations:
        co.elastic.hints/enabled: 'true'
        co.elastic.hints/package: "nginx"

The datastream changes and no duplication happens:

I can see two issues with current template:

We need to comment out / remove the dataset: kubernetes.container_logs that is produced
- Discuss if we should keep audit.logs
We need to uncomment following lines

providers.kubernetes:
      hints.enabled: true
      hints.default_container_logs: true

MichaelKatsoulis · 2024-10-09T12:42:12Z

@gizas and @pkoutsovasilis could you please review this PR again after latest changes ?

gizas · 2024-10-10T14:10:07Z

changelog/fragments/1727790202-kustomize-templates-with-hints.yaml

+kind: feature
+
+# Change summary; a 80ish characters long description of the change.
+summary: Provide a Kustomize template which enables hints based autodiscovery by default when deploying standalone elastic-agent into a Kubernetes cluster.


nit: Updates the mount path of autodiscovery init container to /etc/elastic-agent/inputs.d in order to eliminate root access needs

To add sth like that in the reasoning as well

gizas · 2024-10-10T14:10:39Z

LGTM Mike! Great work

pkoutsovasilis

LGTM! For the future I strongly recommend to extent the agent k8s integration tests to include testing these templates (tracked by this issue #5660)

…ichaelKatsoulis/elastic-agent into kustomize_templates_hints_autodiscover

elastic-sonarqube · 2024-10-11T12:47:31Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

* Add template for hints * Create fragment * fixing comments for provider * removing container_logs * Remove root capabilites from initcontainer * Update kustomize patches for hints * Update changelog fragment * Update configmap for standalone agent --------- Co-authored-by: Andrew Gizas <[email protected]> (cherry picked from commit eedce01) # Conflicts: # deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-daemonset.yaml # deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-daemonset.yaml # deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/extra/elastic-agent-standalone-statefulset.yaml # deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

…uto discover (#5772) * [Kubernetes] Add kustomize template for hints auto discover (#5643) * Add template for hints * Create fragment * fixing comments for provider * removing container_logs * Remove root capabilites from initcontainer * Update kustomize patches for hints * Update changelog fragment * Update configmap for standalone agent --------- Co-authored-by: Andrew Gizas <[email protected]> (cherry picked from commit eedce01) # Conflicts: # deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-daemonset.yaml # deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-daemonset.yaml # deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/extra/elastic-agent-standalone-statefulset.yaml # deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml * Fix conflicts --------- Co-authored-by: Michael Katsoulis <[email protected]>

Add template for hints

07b80c1

MichaelKatsoulis requested a review from a team as a code owner October 1, 2024 08:04

MichaelKatsoulis requested review from gizas and tetianakravchenko October 1, 2024 08:04

mergify bot assigned MichaelKatsoulis Oct 1, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 1, 2024

MichaelKatsoulis marked this pull request as draft October 1, 2024 08:05

Create fragment

c3e55e2

MichaelKatsoulis marked this pull request as ready for review October 1, 2024 13:47

MichaelKatsoulis requested a review from a team as a code owner October 1, 2024 13:47

MichaelKatsoulis requested review from pkoutsovasilis and andrzej-stencel October 1, 2024 13:47

MichaelKatsoulis and others added 2 commits October 1, 2024 16:47

Merge branch 'main' into kustomize_templates_hints_autodiscover

7574236

Merge branch 'main' into kustomize_templates_hints_autodiscover

435f1d9

ycombinator requested review from swiatekm and removed request for andrzej-stencel October 1, 2024 23:46

Merge branch 'main' into kustomize_templates_hints_autodiscover

d1051cc

pkoutsovasilis reviewed Oct 2, 2024

View reviewed changes

deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone/kustomization.yaml Outdated Show resolved Hide resolved

pkoutsovasilis reviewed Oct 2, 2024

View reviewed changes

deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone/kustomization.yaml Outdated Show resolved Hide resolved

pkoutsovasilis reviewed Oct 2, 2024

View reviewed changes

This was referenced Oct 2, 2024

Include hints autodiscovery configuration inside integration tests for k8s #5660

Closed

Investigate the copy of autodiscovery templates inside Elastic Agent image #5661

Closed

Merge branch 'main' into kustomize_templates_hints_autodiscover

ca148dc

ycombinator added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Oct 2, 2024

gizas and others added 6 commits October 3, 2024 17:18

fixing comments for provider

a502473

Merge branch 'main' into kustomize_templates_hints_autodiscover

a2f800b

removing container_logs

5b7eb20

Remove root capabilites from initcontainer

491de5c

Update kustomize patches for hints

e67a3c5

Merge branch 'main' into kustomize_templates_hints_autodiscover

127e1c2

MichaelKatsoulis requested a review from pkoutsovasilis October 9, 2024 12:40

MichaelKatsoulis changed the title ~~Add kustomise template for hints auto discover~~ [Kubernetes]Add kustomize template for hints auto discover Oct 10, 2024

MichaelKatsoulis changed the title ~~[Kubernetes]Add kustomize template for hints auto discover~~ [Kubernetes] Add kustomize template for hints auto discover Oct 10, 2024

gizas reviewed Oct 10, 2024

View reviewed changes

gizas approved these changes Oct 10, 2024

View reviewed changes

pkoutsovasilis approved these changes Oct 10, 2024

View reviewed changes

MichaelKatsoulis added 5 commits October 11, 2024 10:27

Update changelog fragment

cc5ec22

Merge branch 'kustomize_templates_hints_autodiscover' of github.com:M…

8051c09

…ichaelKatsoulis/elastic-agent into kustomize_templates_hints_autodiscover

Merge branch 'main' into kustomize_templates_hints_autodiscover

f758edc

Merge branch 'main' into kustomize_templates_hints_autodiscover

f3f0770

Update configmap for standalone agent

3be9606

swiatekm approved these changes Oct 11, 2024

View reviewed changes

MichaelKatsoulis merged commit eedce01 into elastic:main Oct 14, 2024
9 checks passed

mergify bot mentioned this pull request Oct 14, 2024

[8.x](backport #5643) [Kubernetes] Add kustomize template for hints auto discover #5772

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Kubernetes] Add kustomize template for hints auto discover #5643

[Kubernetes] Add kustomize template for hints auto discover #5643

MichaelKatsoulis commented Oct 1, 2024 •

edited

Loading

mergify bot commented Oct 1, 2024

mergify bot commented Oct 1, 2024

pkoutsovasilis Oct 2, 2024

gizas Oct 2, 2024

pkoutsovasilis Oct 2, 2024

gizas Oct 2, 2024

pkoutsovasilis Oct 2, 2024

swiatekm commented Oct 2, 2024

gizas commented Oct 2, 2024 •

edited

Loading

gizas commented Oct 2, 2024

elasticmachine commented Oct 2, 2024

gizas commented Oct 3, 2024 •

edited

Loading

MichaelKatsoulis commented Oct 9, 2024

gizas Oct 10, 2024

gizas commented Oct 10, 2024

pkoutsovasilis left a comment

elastic-sonarqube bot commented Oct 11, 2024

[Kubernetes] Add kustomize template for hints auto discover #5643

[Kubernetes] Add kustomize template for hints auto discover #5643

Conversation

MichaelKatsoulis commented Oct 1, 2024 • edited Loading

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

mergify bot commented Oct 1, 2024

mergify bot commented Oct 1, 2024

pkoutsovasilis Oct 2, 2024

Choose a reason for hiding this comment

gizas Oct 2, 2024

Choose a reason for hiding this comment

pkoutsovasilis Oct 2, 2024

Choose a reason for hiding this comment

gizas Oct 2, 2024

Choose a reason for hiding this comment

pkoutsovasilis Oct 2, 2024

Choose a reason for hiding this comment

swiatekm commented Oct 2, 2024

gizas commented Oct 2, 2024 • edited Loading

gizas commented Oct 2, 2024

elasticmachine commented Oct 2, 2024

gizas commented Oct 3, 2024 • edited Loading

MichaelKatsoulis commented Oct 9, 2024

gizas Oct 10, 2024

Choose a reason for hiding this comment

gizas commented Oct 10, 2024

pkoutsovasilis left a comment

Choose a reason for hiding this comment

elastic-sonarqube bot commented Oct 11, 2024

Quality Gate passed

MichaelKatsoulis commented Oct 1, 2024 •

edited

Loading

gizas commented Oct 2, 2024 •

edited

Loading

gizas commented Oct 3, 2024 •

edited

Loading