Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch K8s tests to Wolfi by default and make it configurable #41222

Merged
merged 2 commits into from
Oct 15, 2024

Conversation

rdner
Copy link
Member

@rdner rdner commented Oct 14, 2024

Proposed commit message

By default we test Wolfi-based images but there is now an environment variable IMAGE_MODIFIER that can be used for specifying different image prefixes when calling the make target.

Also, now the test verifies the actual container readiness and deletes resources after itself.

How to test this PR locally

Run the packaging command in the following folders:

  • x-pack/filebeat
  • x-pack/metricbeat
  • x-pack/auditbeat
  • x-pack/heartbeat

(put your architecture instead)

PLATFORMS=linux/arm64 PACKAGES=docker mage package

Then you need a running K8s cluster with the pre-configured kubectl. I used Docker Desktop.

then go to ./deploy/kubernetes and run

make test

by default we test against the Wolfi-based images but it can be switched by:

IMAGE_MODIFIER="-ubi" mage test

or test the main image instead:

IMAGE_MODIFIER="" mage test

The output should be something like this:

Generating filebeat-kubernetes.yaml for version 9.0.0 and image modifier '-wolfi'
file: filebeat/filebeat-service-account.yaml
file: filebeat/filebeat-role.yaml
file: filebeat/filebeat-role-binding.yaml
file: filebeat/filebeat-configmap.yaml
file: filebeat/filebeat-daemonset.yaml
Generating metricbeat-kubernetes.yaml for version 9.0.0 and image modifier '-wolfi'
file: metricbeat/metricbeat-service-account.yaml
file: metricbeat/metricbeat-role.yaml
file: metricbeat/metricbeat-role-binding.yaml
file: metricbeat/metricbeat-configmap.yaml
file: metricbeat/metricbeat-daemonset.yaml
Generating auditbeat-kubernetes.yaml for version 9.0.0 and image modifier '-wolfi'
file: auditbeat/auditbeat-service-account.yaml
file: auditbeat/auditbeat-role.yaml
file: auditbeat/auditbeat-role-binding.yaml
file: auditbeat/auditbeat-configmap.yaml
file: auditbeat/auditbeat-daemonset.yaml
Generating heartbeat-kubernetes.yaml for version 9.0.0 and image modifier '-wolfi'
file: heartbeat/heartbeat-service-account.yaml
file: heartbeat/heartbeat-role.yaml
file: heartbeat/heartbeat-role-binding.yaml
file: heartbeat/heartbeat-configmap.yaml
file: heartbeat/heartbeat-deployment.yaml

filebeat
serviceaccount/filebeat created
clusterrole.rbac.authorization.k8s.io/filebeat created
role.rbac.authorization.k8s.io/filebeat created
role.rbac.authorization.k8s.io/filebeat-kubeadm-config created
clusterrolebinding.rbac.authorization.k8s.io/filebeat created
rolebinding.rbac.authorization.k8s.io/filebeat created
rolebinding.rbac.authorization.k8s.io/filebeat-kubeadm-config created
configmap/filebeat-config created
daemonset.apps/filebeat created
Testing filebeat container for readiness...
pod/filebeat-wsn22 condition met
Deleting filebeat...
serviceaccount "filebeat" deleted
clusterrole.rbac.authorization.k8s.io "filebeat" deleted
role.rbac.authorization.k8s.io "filebeat" deleted
role.rbac.authorization.k8s.io "filebeat-kubeadm-config" deleted
clusterrolebinding.rbac.authorization.k8s.io "filebeat" deleted
rolebinding.rbac.authorization.k8s.io "filebeat" deleted
rolebinding.rbac.authorization.k8s.io "filebeat-kubeadm-config" deleted
configmap "filebeat-config" deleted
daemonset.apps "filebeat" deleted

metricbeat
serviceaccount/metricbeat created
clusterrole.rbac.authorization.k8s.io/metricbeat created
role.rbac.authorization.k8s.io/metricbeat created
role.rbac.authorization.k8s.io/metricbeat-kubeadm-config created
clusterrolebinding.rbac.authorization.k8s.io/metricbeat created
rolebinding.rbac.authorization.k8s.io/metricbeat created
rolebinding.rbac.authorization.k8s.io/metricbeat-kubeadm-config created
configmap/metricbeat-daemonset-config created
configmap/metricbeat-daemonset-modules created
daemonset.apps/metricbeat created
Testing metricbeat container for readiness...
pod/metricbeat-lfrqq condition met
Deleting metricbeat...
serviceaccount "metricbeat" deleted
clusterrole.rbac.authorization.k8s.io "metricbeat" deleted
role.rbac.authorization.k8s.io "metricbeat" deleted
role.rbac.authorization.k8s.io "metricbeat-kubeadm-config" deleted
clusterrolebinding.rbac.authorization.k8s.io "metricbeat" deleted
rolebinding.rbac.authorization.k8s.io "metricbeat" deleted
rolebinding.rbac.authorization.k8s.io "metricbeat-kubeadm-config" deleted
configmap "metricbeat-daemonset-config" deleted
configmap "metricbeat-daemonset-modules" deleted
daemonset.apps "metricbeat" deleted

auditbeat
serviceaccount/auditbeat created
clusterrole.rbac.authorization.k8s.io/auditbeat created
role.rbac.authorization.k8s.io/auditbeat created
role.rbac.authorization.k8s.io/auditbeat-kubeadm-config created
clusterrolebinding.rbac.authorization.k8s.io/auditbeat created
rolebinding.rbac.authorization.k8s.io/auditbeat created
rolebinding.rbac.authorization.k8s.io/auditbeat-kubeadm-config created
configmap/auditbeat-config created
configmap/auditbeat-daemonset-modules created
daemonset.apps/auditbeat created
Testing auditbeat container for readiness...
pod/auditbeat-jv9l5 condition met
Deleting auditbeat...
serviceaccount "auditbeat" deleted
clusterrole.rbac.authorization.k8s.io "auditbeat" deleted
role.rbac.authorization.k8s.io "auditbeat" deleted
role.rbac.authorization.k8s.io "auditbeat-kubeadm-config" deleted
clusterrolebinding.rbac.authorization.k8s.io "auditbeat" deleted
rolebinding.rbac.authorization.k8s.io "auditbeat" deleted
rolebinding.rbac.authorization.k8s.io "auditbeat-kubeadm-config" deleted
configmap "auditbeat-config" deleted
configmap "auditbeat-daemonset-modules" deleted
daemonset.apps "auditbeat" deleted

heartbeat
serviceaccount/heartbeat created
clusterrole.rbac.authorization.k8s.io/heartbeat created
role.rbac.authorization.k8s.io/heartbeat created
role.rbac.authorization.k8s.io/heartbeat-kubeadm-config created
clusterrolebinding.rbac.authorization.k8s.io/heartbeat created
rolebinding.rbac.authorization.k8s.io/heartbeat created
rolebinding.rbac.authorization.k8s.io/heartbeat-kubeadm-config created
configmap/heartbeat-deployment-config created
deployment.apps/heartbeat created
Testing heartbeat container for readiness...
pod/heartbeat-77f95d9b7-6rb9n condition met
Deleting heartbeat...
serviceaccount "heartbeat" deleted
clusterrole.rbac.authorization.k8s.io "heartbeat" deleted
role.rbac.authorization.k8s.io "heartbeat" deleted
role.rbac.authorization.k8s.io "heartbeat-kubeadm-config" deleted
clusterrolebinding.rbac.authorization.k8s.io "heartbeat" deleted
rolebinding.rbac.authorization.k8s.io "heartbeat" deleted
rolebinding.rbac.authorization.k8s.io "heartbeat-kubeadm-config" deleted
configmap "heartbeat-deployment-config" deleted
deployment.apps "heartbeat" deleted

This means K8s manifests were successfully generated, applied and the containers were running successfully.

Check the generated manifests for the images to have -wolfi suffix:

-rw-r--r--  1 rdner  staff   7.9K Oct 14 16:00 auditbeat-kubernetes.yaml
-rw-r--r--  1 rdner  staff   5.8K Oct 14 16:00 filebeat-kubernetes.yaml
-rw-r--r--  1 rdner  staff   5.3K Oct 14 16:00 heartbeat-kubernetes.yaml
-rw-r--r--  1 rdner  staff   9.3K Oct 14 16:00 metricbeat-kubernetes.yaml

Related issues

By default we test Wolfi-based images but there is now an environment
variable `IMAGE_MODIFIER` that can be used for specifying different
image prefixes when calling the make target.

Also, now the test verifies the actual container readiness and deletes
resources after itself.
@rdner rdner added enhancement Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Oct 14, 2024
@rdner rdner self-assigned this Oct 14, 2024
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 14, 2024
Copy link
Contributor

mergify bot commented Oct 14, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @rdner? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Oct 14, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 14, 2024
@rdner rdner marked this pull request as ready for review October 14, 2024 14:35
@rdner rdner requested review from a team as code owners October 14, 2024 14:35
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@rdner rdner requested review from cmacknz and pazone and removed request for faec October 14, 2024 14:35
@pierrehilbert pierrehilbert added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label Oct 15, 2024
Copy link
Contributor

@pazone pazone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gizas
Copy link
Contributor

gizas commented Oct 15, 2024

nit: Maybe add to the descprition that something like export IMAGE_MODIFIER=-pas; make test can work to modify the variable.

Looks good

@rdner rdner merged commit 3492089 into elastic:main Oct 15, 2024
16 checks passed
@rdner rdner deleted the wolfi-images-in-tests branch October 15, 2024 13:40
mergify bot pushed a commit that referenced this pull request Oct 15, 2024
By default we test Wolfi-based images but there is now an environment
variable `IMAGE_MODIFIER` that can be used for specifying different
image prefixes when calling the make target.

Also, now the test verifies the actual container readiness and deletes
resources after itself.

(cherry picked from commit 3492089)

# Conflicts:
#	deploy/kubernetes/auditbeat-kubernetes.yaml
#	deploy/kubernetes/filebeat-kubernetes.yaml
#	deploy/kubernetes/heartbeat-kubernetes.yaml
#	deploy/kubernetes/metricbeat-kubernetes.yaml
rdner added a commit that referenced this pull request Oct 15, 2024
#41237)

By default we test Wolfi-based images but there is now an environment
variable `IMAGE_MODIFIER` that can be used for specifying different
image prefixes when calling the make target.

Also, now the test verifies the actual container readiness and deletes
resources after itself.

(cherry picked from commit 3492089)

# Conflicts:
#	deploy/kubernetes/auditbeat-kubernetes.yaml
#	deploy/kubernetes/filebeat-kubernetes.yaml
#	deploy/kubernetes/heartbeat-kubernetes.yaml
#	deploy/kubernetes/metricbeat-kubernetes.yaml

Co-authored-by: Denis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify enhancement Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants