Use Kustomize to build elastic-agent manifests for both managed and standalone more #2104

gsantoro · 2023-01-16T12:06:19Z

What does this PR do?

Use Kustomize to simplify building manifest for elastic-agent both for managed and standalone mode.

Why is it important?

This PR reduces the amount of boiler plate code needed to build manifests for elastic-agent. Some enhancements:

we can also install kube-state-metrics just by adding an extra resource github.com/kubernetes/kube-state-metrics?ref=main to common/kustomization.yml
most of the resources (aka roles, role-binding, service accounts, cluster-role, cluster-role binding) are exactly the same for both standalone and managed mode. This reduces massively the amount of duplication.
some resources (like config maps and extra roles and role bindings) are only necessary in standalone mode. There is no need to duplicates them in both mode
it makes possible to customise environment variables without changing the manifests . Default values for those environment variables are provided in .env files under dev-tools/kubernetes/base//.env (added to git) while custom values are under dev-tools/kubernetes/overlays//.env (those should not be under git. that's why they are added to .gitignore).
namespace references are only present in kustomization.yml files. It is easier to deploy all resources to a separate namespace
overlays can be used to customize the docker image tag of the elastic-agent without changing the manifest parts
Common resources have generic names so that you can use prefixes specific to the mode you want to deploy (eg. elastic-agent-standalone- or elastic-agent-managed). This enables to deploy standalone and managed agents side by side.
comments in manifest parts are removed in the final manifest
yaml code is pretty formatted when the final manifest is generated

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool

Author's Checklist

You can deploy elastic-agent in both standalone and managed modes

How to test this PR locally

Related issues

Closes Use Kustomize.io to build elastic-agent manifests for k8s #1668

Use cases

Screenshots

Logs

mergify · 2023-01-16T12:06:53Z

This pull request does not have a backport label. Could you fix it @gsantoro? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-01-16T12:22:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-17T16:28:02.524+0000
Duration: 18 min 18 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4825
Skipped	13
Total	4838

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

gizas · 2023-01-16T13:10:46Z

dev-tools/kubernetes/base/elastic-agent-managed/daemonset.yaml

+            requests:
+              cpu: 100m
+              memory: 200Mi
+          volumeMounts:


I think following part can also be splitted to multilple overalys.

We want to have needed mounts for observability and extra ones that imposed from security

do you mind elaborate a bit more on the topic. I vaguely recollect some suggestions from the security team but I wouldn't know how to change this part of the manifest

gizas · 2023-01-16T13:13:50Z

dev-tools/kubernetes/base/elastic-agent-standalone/.env

@@ -0,0 +1,3 @@
+ES_USERNAME=elastic
+ES_PASSWORD=changeme
+ES_HOST=https://elasticsearch:9200


I would suggest to put namespace as variable here

I don't think you can use a variable from an .env file in the kustomization.yml file (which is where I set the namespace for all the resources)

gizas · 2023-01-16T13:15:04Z

dev-tools/kubernetes/base/elastic-agent-standalone/.env

@@ -0,0 +1,3 @@
+ES_USERNAME=elastic
+ES_PASSWORD=changeme
+ES_HOST=https://elasticsearch:9200


Can we have as boolean on/off options the ability to install kube state metrics here?
By default to be on

I don't think so. This env file is only used to create a configmap and I don't think we can use an env variable from a configmap to decide if we want to add a manifest based on that. We could use an overlay with an extra resource to install or not the kube-state-metrics

But then we will need to edit the kustomize.yaml.

Maybe we can think that we can provide diffrent kustomzation files ?
eg kustomize-ksmenabled, kustomize-ksmdisabled?

yes, I believe we have to go that way. Different overlays for different behaviours

1st Option:
A tool worth looking:
https://carvel.dev/ytt/
So we would like to provide some basic YAML templating and then kustomize installation.

2nd Option:
Maybe our new MAKEFILE that we can run and produce specific kustomize folders and run as oneliner command?

I heard about ytt but it seemed way too complicated compared to kustomize. Should we create another issue with the list of requirements before diving into ytt?

gizas · 2023-01-16T13:21:30Z

Summarising some small comments from relevant slack commnication:

For first iteration of this effort:

Only default elastic-agent managed and standalone scenarios to be implemented. Any extra scenarios to be defined in extra stories
Include installation of KSM as part of the resources file. We would like ideally the UX to be that with one command to install all needed resources
Namespace and toggle option of KSM installation to be provided
Correlation between our manifest files with specific KSM by default. I guess this also be an env variable which can be overrided if user wants on runtime

Extra stories:

Documentation story - TBD
Story to add this functionality to kibana TBD
Alighnement with main https://github.com/elastic/observability-dev/issues/2410

elasticmachine · 2023-01-16T13:54:46Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.305% (`58/59`)	👍
Files	69.268% (`142/205`)	👍
Classes	69.231% (`270/390`)	👍
Methods	54.047% (`828/1532`)	👍
Lines	39.229% (`9001/22945`)	👎 -0.031
Conditionals	100.0% (`0/0`)	💚

…c-agent labels, namespace and other changes from the various kustomizations.yaml

…ther kustomization and overlays

ChrsMark · 2023-01-23T00:12:53Z

dev-tools/kubernetes/base/elastic-agent-managed/daemonset.yaml

+          - configMapRef:
+              # Fleet Server URL to enroll the Elastic Agent into
+              # FLEET_URL can be found in Kibana, go to Management > Fleet > Settings
+              # - name: FLEET_URL


Just out of curiosity, why all these settings are commented out?

they are mostly left there for documentation. We should address them (one way or another) in the next update about on this topic.

But when those vars should be uncommented?

I have created this PR to fix this

gsantoro added 11 commits January 16, 2023 09:58

elastic-agent-managed

ee5c133

simplified resource names

707265d

split into multiple files

ff98adc

standalone mode

d4f8b6f

removed namespace from common resources

ef8b285

removed namespace from common resources

4a19fc7

more configs in .env files

806a0db

removed labels from selector and metadata

e627762

updated version of the agent

7d021d3

differentiate standalone from managed

2584adf

add back .env files with default values

b880760

gsantoro added enhancement New feature or request draft labels Jan 16, 2023

gsantoro requested a review from a team January 16, 2023 12:06

gsantoro self-assigned this Jan 16, 2023

gsantoro requested a review from a team as a code owner January 16, 2023 12:06

gsantoro requested review from aleksmaus and michalpristas and removed request for a team January 16, 2023 12:06

mergify bot added the backport-skip label Jan 16, 2023

gizas reviewed Jan 16, 2023

View reviewed changes

added kube-state-metrics from main branch to common kustomization.yml

26e5649

gsantoro added 2 commits January 16, 2023 14:59

moved kube-state-metrics resource out of common to avoid extra elasti…

2f73bfb

…c-agent labels, namespace and other changes from the various kustomizations.yaml

some extra configs for standalone mode

efc4e46

disable configmap suffix hash to avoid leaving clutter around

1fffd1d

gizas approved these changes Jan 17, 2023

View reviewed changes

gsantoro added 2 commits January 17, 2023 12:41

updated config map for standalone from elastic#2113

895dc68

removed namespace from base/common since it's already provided in o…

c1d5ea0

…ther kustomization and overlays

gsantoro merged commit b1b5a43 into elastic:main Jan 17, 2023

gsantoro deleted the feature/1668_kustomize branch January 17, 2023 18:16

constanca-m mentioned this pull request Jan 19, 2023

[K8s testing] Support use of Kustomize elastic/elastic-package#1102

Closed

ChrsMark reviewed Jan 23, 2023

View reviewed changes

gsantoro mentioned this pull request Jan 24, 2023

pull out env vars from daemonset and document them #2160

Merged

6 tasks

gizas mentioned this pull request Jan 25, 2023

[Cloud Native] Use Kustomize to deploy resources #2175

Open

29 tasks

gsantoro mentioned this pull request Feb 1, 2023

remove quotes from FLEET_ENROLL that prevented kustomize to create valid manifest #2216

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Kustomize to build elastic-agent manifests for both managed and standalone more #2104

Use Kustomize to build elastic-agent manifests for both managed and standalone more #2104

gsantoro commented Jan 16, 2023 •

edited

Loading

mergify bot commented Jan 16, 2023

elasticmachine commented Jan 16, 2023 •

edited

Loading

Build stats

Test stats 🧪

gizas Jan 16, 2023

gsantoro Jan 16, 2023

gizas Jan 16, 2023

gsantoro Jan 16, 2023

gizas Jan 16, 2023

gsantoro Jan 16, 2023

gizas Jan 16, 2023

gsantoro Jan 16, 2023

gizas Jan 16, 2023 •

edited

Loading

gsantoro Jan 17, 2023

gizas commented Jan 16, 2023 •

edited

Loading

elasticmachine commented Jan 16, 2023 •

edited

Loading

ChrsMark Jan 23, 2023

gsantoro Jan 23, 2023

ChrsMark Jan 24, 2023

gsantoro Jan 24, 2023

Use Kustomize to build elastic-agent manifests for both managed and standalone more #2104

Use Kustomize to build elastic-agent manifests for both managed and standalone more #2104

Conversation

gsantoro commented Jan 16, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Jan 16, 2023

elasticmachine commented Jan 16, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gizas Jan 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gizas commented Jan 16, 2023 • edited Loading

elasticmachine commented Jan 16, 2023 • edited Loading

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsantoro commented Jan 16, 2023 •

edited

Loading

elasticmachine commented Jan 16, 2023 •

edited

Loading

gizas Jan 16, 2023 •

edited

Loading

gizas commented Jan 16, 2023 •

edited

Loading

elasticmachine commented Jan 16, 2023 •

edited

Loading