Add release scripts for upgrading sidecar dependencies and collating image digests #1792

AndrewSirenko · 2023-10-19T18:08:39Z

Is this a bug fix or adding new feature?
Release automation

What is this PR about? / Why do we need it?
This PR adds:

image-digests-template.yaml: A helper file for collating images, tags, and manifest digests used in the rest of the repository. Scripts in hack/release-scripts will generate an image-digests.yaml file from this template and propagating those updates to the rest of the repository.
get-latest-sidecar-images: Generates a image-digests.yaml file with the latest tags and associated manifest digests for each sidecar image by using crane.
generate-sidecar-tags in hack/release-scripts: Generates the sidecar image tags in deploy/kubernetes/overlays/stable/gcr/kustomization.yaml and charts/aws-ebs-csi-driver/values.yaml based off of the values in hack/release-scripts/image-digests.yaml
update-sidecar-dependencies Makefile target: Fetches latest sidecar tags and syncs helm/kustomize files via the following Makefile targets
- update-truth-sidecars
- generate-sidecar-tags
- generate-kustomize

See previous discussion here: [Automate-Release-1] Add image-source-of-truth.yaml and update-truth-sidecars scripts #1791

What testing is done?
Running make update-sidecar-dependencies will and produce the following diff:

diff --git a/charts/aws-ebs-csi-driver/values.yaml b/charts/aws-ebs-csi-driver/values.yaml
index 27280e7e..cbc240ee 100644
--- a/charts/aws-ebs-csi-driver/values.yaml
+++ b/charts/aws-ebs-csi-driver/values.yaml
@@ -7,11 +7,9 @@ image:
   # Overrides the image tag whose default is v{{ .Chart.AppVersion }}
   tag: ""
   pullPolicy: IfNotPresent
-
 # -- Custom labels to add into metadata
-customLabels:
-  {}
-  # k8s-app: aws-ebs-csi-driver
+customLabels: {}
+# k8s-app: aws-ebs-csi-driver
 
 sidecars:
   provisioner:
@@ -19,7 +17,7 @@ sidecars:
     image:
       pullPolicy: IfNotPresent
       repository: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner
-      tag: "v3.5.0-eks-1-28-6"
+      tag: "v3.6.0-eks-1-28-7"
     logLevel: 2
     # Additional parameters provided by external-provisioner.
     additionalArgs: []
@@ -44,7 +42,7 @@ sidecars:
     image:
       pullPolicy: IfNotPresent
       repository: public.ecr.aws/eks-distro/kubernetes-csi/external-attacher
-      tag: "v4.4.0-eks-1-28-6"
+      tag: "v4.4.0-eks-1-28-7"
     # Tune leader lease election for csi-attacher.
     # Leader election is on by default.
     leaderElection:
@@ -71,7 +69,7 @@ sidecars:
     image:
       pullPolicy: IfNotPresent
       repository: public.ecr.aws/eks-distro/kubernetes-csi/external-snapshotter/csi-snapshotter
-      tag: "v6.3.0-eks-1-28-6"
+      tag: "v6.3.0-eks-1-28-7"
     logLevel: 2
     # Additional parameters provided by csi-snapshotter.
     additionalArgs: []
@@ -85,7 +83,7 @@ sidecars:
     image:
       pullPolicy: IfNotPresent
       repository: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe
-      tag: "v2.10.0-eks-1-28-6"
+      tag: "v2.11.0-eks-1-28-7"
     # Additional parameters provided by livenessprobe.
     additionalArgs: []
     resources: {}
@@ -97,7 +95,7 @@ sidecars:
     image:
       pullPolicy: IfNotPresent
       repository: public.ecr.aws/eks-distro/kubernetes-csi/external-resizer
-      tag: "v1.9.0-eks-1-28-6"
+      tag: "v1.9.0-eks-1-28-7"
     # Tune leader lease election for csi-resizer.
     # Leader election is on by default.
     leaderElection:
@@ -122,7 +120,7 @@ sidecars:
     image:
       pullPolicy: IfNotPresent
       repository: public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar
-      tag: "v2.9.0-eks-1-28-6"
+      tag: "v2.9.0-eks-1-28-7"
     logLevel: 2
     # Additional parameters provided by node-driver-registrar.
     additionalArgs: []
@@ -151,20 +149,19 @@ sidecars:
     securityContext:
       readOnlyRootFilesystem: true
       allowPrivilegeEscalation: false
-
+  snapshotController:
+    image:
+      tag: v6.3.0-eks-1-28-7
 proxy:
   http_proxy:
   no_proxy:
-
 imagePullSecrets: []
 nameOverride:
 fullnameOverride:
-
 awsAccessSecret:
   name: aws-secret
   keyId: key_id
   accessKey: access_key
-
 controller:
   volumeModificationFeature:
     enabled: false
@@ -175,24 +172,24 @@ controller:
   affinity:
     nodeAffinity:
       preferredDuringSchedulingIgnoredDuringExecution:
-      - weight: 1
-        preference:
-          matchExpressions:
-          - key: eks.amazonaws.com/compute-type
-            operator: NotIn
-            values:
-            - fargate
+        - weight: 1
+          preference:
+            matchExpressions:
+              - key: eks.amazonaws.com/compute-type
+                operator: NotIn
+                values:
+                  - fargate
     podAntiAffinity:
       preferredDuringSchedulingIgnoredDuringExecution:
-      - podAffinityTerm:
-          labelSelector:
-            matchExpressions:
-            - key: app
-              operator: In
-              values:
-              - ebs-csi-controller
-          topologyKey: kubernetes.io/hostname
-        weight: 100
+        - podAffinityTerm:
+            labelSelector:
+              matchExpressions:
+                - key: app
+                  operator: In
+                  values:
+                    - ebs-csi-controller
+            topologyKey: kubernetes.io/hostname
+          weight: 100
   # The default filesystem type of the volume to provision when fstype is unspecified in the StorageClass.
   # If the default is not set and fstype is unset in the StorageClass, then no fstype will be set
   defaultFsType: ext4
@@ -251,7 +248,7 @@ controller:
     limits:
       memory: 256Mi
   serviceAccount:
-  # A service account will be created for you if set to true. Set to false if you want to use your own.
+    # A service account will be created for you if set to true. Set to false if you want to use your own.
     create: true
     name: ebs-csi-controller-sa
     annotations: {}
@@ -308,7 +305,6 @@ controller:
   otelTracing: {}
   #  otelServiceName: ebs-csi-controller
   #  otelExporterEndpoint: "http://localhost:4317"
-
 node:
   env: []
   envFrom: []
@@ -320,19 +316,19 @@ node:
     nodeAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         nodeSelectorTerms:
-        - matchExpressions:
-          - key: eks.amazonaws.com/compute-type
-            operator: NotIn
-            values:
-            - fargate
+          - matchExpressions:
+              - key: eks.amazonaws.com/compute-type
+                operator: NotIn
+                values:
+                  - fargate
   nodeSelector: {}
   podAnnotations: {}
   podLabels: {}
   tolerateAllTaints: true
   tolerations:
-  - operator: Exists
-    effect: NoExecute
-    tolerationSeconds: 300
+    - operator: Exists
+      effect: NoExecute
+      tolerationSeconds: 300
   resources:
     requests:
       cpu: 10m
@@ -381,16 +377,14 @@ node:
   otelTracing: {}
   #  otelServiceName: ebs-csi-node
   #  otelExporterEndpoint: "http://localhost:4317"
-
 additionalDaemonSets:
-  # Additional node DaemonSets, using the node config structure
-  # See docs/additional-daemonsets.md for more information
-  #
-  # example:
-  #   nodeSelector:
-  #     node.kubernetes.io/instance-type: c5.large
-  #   volumeAttachLimit: 15
-
+# Additional node DaemonSets, using the node config structure
+# See docs/additional-daemonsets.md for more information
+#
+# example:
+#   nodeSelector:
+#     node.kubernetes.io/instance-type: c5.large
+#   volumeAttachLimit: 15
 storageClasses: []
 # Add StorageClass resources like:
 # - name: ebs-sc
diff --git a/deploy/kubernetes/base/controller.yaml b/deploy/kubernetes/base/controller.yaml
index 8ebe5469..a749ea9e 100644
--- a/deploy/kubernetes/base/controller.yaml
+++ b/deploy/kubernetes/base/controller.yaml
@@ -127,7 +127,7 @@ spec:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
         - name: csi-provisioner
-          image: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v3.5.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v3.6.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=$(ADDRESS)
@@ -152,7 +152,7 @@ spec:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
         - name: csi-attacher
-          image: public.ecr.aws/eks-distro/kubernetes-csi/external-attacher:v4.4.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/external-attacher:v4.4.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=$(ADDRESS)
@@ -174,7 +174,7 @@ spec:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
         - name: csi-snapshotter
-          image: public.ecr.aws/eks-distro/kubernetes-csi/external-snapshotter/csi-snapshotter:v6.3.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/external-snapshotter/csi-snapshotter:v6.3.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=$(ADDRESS)
@@ -196,7 +196,7 @@ spec:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
         - name: csi-resizer
-          image: public.ecr.aws/eks-distro/kubernetes-csi/external-resizer:v1.9.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/external-resizer:v1.9.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=$(ADDRESS)
@@ -219,7 +219,7 @@ spec:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
         - name: liveness-probe
-          image: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.10.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.11.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=/csi/csi.sock
diff --git a/deploy/kubernetes/base/node.yaml b/deploy/kubernetes/base/node.yaml
index 7b1b12a0..195499b7 100644
--- a/deploy/kubernetes/base/node.yaml
+++ b/deploy/kubernetes/base/node.yaml
@@ -92,7 +92,7 @@ spec:
               exec:
                 command: ["/bin/aws-ebs-csi-driver", "pre-stop-hook"]
         - name: node-driver-registrar
-          image: public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.9.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.9.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=$(ADDRESS)
@@ -129,7 +129,7 @@ spec:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
         - name: liveness-probe
-          image: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.10.0-eks-1-28-6
+          image: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.11.0-eks-1-28-7
           imagePullPolicy: IfNotPresent
           args:
             - --csi-address=/csi/csi.sock
diff --git a/deploy/kubernetes/overlays/stable/gcr/kustomization.yaml b/deploy/kubernetes/overlays/stable/gcr/kustomization.yaml
index 38c674c4..2de3e3de 100644
--- a/deploy/kubernetes/overlays/stable/gcr/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable/gcr/kustomization.yaml
@@ -7,13 +7,13 @@ images:
     newName: registry.k8s.io/provider-aws/aws-ebs-csi-driver
   - name: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner
     newName: registry.k8s.io/sig-storage/csi-provisioner
-    newTag: v3.5.0
+    newTag: v3.6.0
   - name: public.ecr.aws/eks-distro/kubernetes-csi/external-attacher
     newName: registry.k8s.io/sig-storage/csi-attacher
     newTag: v4.4.0
   - name: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe
     newName: registry.k8s.io/sig-storage/livenessprobe
-    newTag: v2.10.0
+    newTag: v2.11.0
   - name: public.ecr.aws/eks-distro/kubernetes-csi/external-snapshotter/csi-snapshotter
     newName: registry.k8s.io/sig-storage/csi-snapshotter
     newTag: v6.3.0
diff --git a/hack/release-scripts/image-source-of-truth.yaml b/hack/release-scripts/image-source-of-truth.yaml
index 9ba68e18..a6814b0b 100644
--- a/hack/release-scripts/image-source-of-truth.yaml
+++ b/hack/release-scripts/image-source-of-truth.yaml
@@ -10,33 +10,33 @@ driver:
 sidecars: # sidecar names match upstream helm chart values.yaml
   snapshotter:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/external-snapshotter/csi-snapshotter"
-    tag: ""
-    manifestDigest: ""
+    tag: "v6.3.0-eks-1-28-7"
+    manifestDigest: "sha256:61ebea9d396ad9ef0767bd697b50ed2615c9fd49c2625e64c102d916030f7369"
   attacher:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/external-attacher"
-    tag: ""
-    manifestDigest: ""
+    tag: "v4.4.0-eks-1-28-7"
+    manifestDigest: "sha256:955fd72b5b77cffdf785c7c110de0a927906a8765c19160fdb5d7cd74cdc20a6"
   provisioner:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner"
-    tag: ""
-    manifestDigest: ""
+    tag: "v3.6.0-eks-1-28-7"
+    manifestDigest: "sha256:91158c7bf17832d03d7472f4ceb111564d61674ac1aea10c3699f0c55545782d"
   resizer:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/external-resizer"
-    tag: ""
-    manifestDigest: ""
+    tag: "v1.9.0-eks-1-28-7"
+    manifestDigest: "sha256:991ffd5221c340168fbcd77148fd4098df469c2bc78aa84bed4eb323673f87ca"
   livenessProbe:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe"
-    tag: ""
-    manifestDigest: ""
+    tag: "v2.11.0-eks-1-28-7"
+    manifestDigest: "sha256:1ee7f20beaf76a57c5446dc41b0718172e8beb69da8ca2343804309e3a58a367"
   nodeDriverRegistrar:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar"
-    tag: ""
-    manifestDigest: ""
+    tag: "v2.9.0-eks-1-28-7"
+    manifestDigest: "sha256:a51e121d046e459a4315894be43b13ba7348038a67d3e9fb4e3d44cda3dbed1a"
   volumemodifier:
     image: "public.ecr.aws/ebs-csi-driver/volume-modifier-for-k8s"
-    tag: ""
-    manifestDigest: ""
+    tag: "v0.1.3"
+    manifestDigest: "sha256:5452144bfc75cb986ccf266cc967773801c91d47ff3b51326904fa516765c914"
   snapshotController:
     image: "public.ecr.aws/eks-distro/kubernetes-csi/external-snapshotter/snapshot-controller"
-    tag: ""
-    manifestDigest: ""
+    tag: "v6.3.0-eks-1-28-7"
+    manifestDigest: "sha256:a6f0bf7d1f16991bbd85764b2e2791f7812d13b04ed8596904cd3a6a9fbb87b1"

Running make verify successfully passes afterwards

hack/release-scripts/image-digests-template.yaml

hack/release-scripts/get-latest-sidecar-images

hack/release-scripts/image-digests-template.yaml

ConnorJC3 · 2023-11-20T16:20:06Z

hack/release-scripts/generate-sidecar-tags

+
+# Force macOS users to use gsed due to -i incompatibility
+  export SED="sed"
+  if [[ $(uname) = "Darwin" ]]; then


Instead of checking the OS directly, is there a way that we could instead check if the local sed has the support we need? What if a linux user has BSD sed installed, for example?

Looking into that the best solution I found involves grep-ing the sed man page Shell — How to detect some command using GNU or BSD ? ( man sed | grep GNU ). Therefore I think this workaround is fine enough to solve this common MacOs gotcha unless you have a cleaner solution ready.

hack/release-scripts/generate-sidecar-tags

hack/release-scripts/get-latest-sidecar-images

Makefile

ConnorJC3 · 2023-11-20T16:27:08Z

Makefile

+	./hack/release-scripts/generate-sidecar-tags
+
+.PHONY: update-sidecar-dependencies
+update-sidecar-dependencies: update-truth-sidecars generate-sidecar-tags generate-kustomize


Why does this run generate-kustomize?

Because generate-kustomize will propagate the updates in deploy/kubernetes/overlays/stable/gcr/kustomization.yaml to deploy/kubernetes/base/controller.yaml and deploy/kubernetes/base/node.yaml

e.g. csi-provisioner in base/controller.yaml

github-actions · 2023-11-25T00:59:05Z

Code Coverage Diff

This PR does not change the code coverage

ConnorJC3 · 2023-12-01T19:24:18Z

/lgtm

torredil

/lgtm
/approve

k8s-ci-robot · 2023-12-01T19:29:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

AndrewSirenko · 2023-12-01T20:42:53Z

/retest

k8s-ci-robot · 2023-12-01T21:24:58Z

@AndrewSirenko: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-aws-ebs-csi-driver-e2e-single-az	`081c936`	link	unknown	`/test pull-aws-ebs-csi-driver-e2e-single-az`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

AndrewSirenko · 2023-12-01T21:25:30Z

/retest

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 19, 2023

k8s-ci-robot requested review from ConnorJC3 and torredil October 19, 2023 18:08

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 19, 2023

AndrewSirenko marked this pull request as draft October 19, 2023 18:09

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 19, 2023

AndrewSirenko mentioned this pull request Oct 20, 2023

[Automate-Release-3] Add generate-release-pr to /hack/release-scripts #1793

Closed

ConnorJC3 force-pushed the master branch 2 times, most recently from 24a8e7b to bddbe0b Compare November 1, 2023 18:08

AndrewSirenko force-pushed the autoRelease2 branch from 06a3360 to ae54aaf Compare November 2, 2023 14:36

AndrewSirenko marked this pull request as ready for review November 2, 2023 14:52

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 2, 2023

AndrewSirenko changed the title ~~[Automate-Release-2] Add make target and scripts for upgrading sidecar dependencies~~ Add release scripts for upgrading sidecar dependencies and collating image digests Nov 2, 2023

AndrewSirenko commented Nov 15, 2023

View reviewed changes

hack/release-scripts/image-digests-template.yaml Outdated Show resolved Hide resolved

torredil reviewed Nov 15, 2023

View reviewed changes

hack/release-scripts/image-digests-template.yaml Outdated Show resolved Hide resolved

ConnorJC3 reviewed Nov 20, 2023

View reviewed changes

AndrewSirenko force-pushed the autoRelease2 branch 2 times, most recently from 2017d9a to 56b26a0 Compare November 25, 2023 00:58

AndrewSirenko requested review from ConnorJC3 and torredil November 25, 2023 00:59

Add release scripts for upgrading sidecar dependencies

081c936

AndrewSirenko force-pushed the autoRelease2 branch from 56b26a0 to 081c936 Compare November 25, 2023 01:02

k8s-ci-robot assigned ConnorJC3 Dec 1, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 1, 2023

torredil approved these changes Dec 1, 2023

View reviewed changes

k8s-ci-robot assigned torredil Dec 1, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 1, 2023

k8s-ci-robot merged commit 7ec4800 into kubernetes-sigs:master Dec 1, 2023
18 of 19 checks passed

AndrewSirenko deleted the autoRelease2 branch December 5, 2023 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add release scripts for upgrading sidecar dependencies and collating image digests #1792

Add release scripts for upgrading sidecar dependencies and collating image digests #1792

AndrewSirenko commented Oct 19, 2023 •

edited

Loading

ConnorJC3 Nov 20, 2023

AndrewSirenko Nov 24, 2023

ConnorJC3 Nov 20, 2023

AndrewSirenko Nov 24, 2023

github-actions bot commented Nov 25, 2023

ConnorJC3 commented Dec 1, 2023

torredil left a comment

k8s-ci-robot commented Dec 1, 2023

AndrewSirenko commented Dec 1, 2023

k8s-ci-robot commented Dec 1, 2023

AndrewSirenko commented Dec 1, 2023

Add release scripts for upgrading sidecar dependencies and collating image digests #1792

Add release scripts for upgrading sidecar dependencies and collating image digests #1792

Conversation

AndrewSirenko commented Oct 19, 2023 • edited Loading

ConnorJC3 Nov 20, 2023

Choose a reason for hiding this comment

AndrewSirenko Nov 24, 2023

Choose a reason for hiding this comment

ConnorJC3 Nov 20, 2023

Choose a reason for hiding this comment

AndrewSirenko Nov 24, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 25, 2023

Code Coverage Diff

ConnorJC3 commented Dec 1, 2023

torredil left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Dec 1, 2023

AndrewSirenko commented Dec 1, 2023

k8s-ci-robot commented Dec 1, 2023

AndrewSirenko commented Dec 1, 2023

AndrewSirenko commented Oct 19, 2023 •

edited

Loading