Override the klipper-helm image used in default HelmCharts

[mddamato]

Is your feature request related to a problem? Please describe.
I need to be able to override the klipper-helm image to use a custom registry without adding a docker.io mirror

Describe the solution you'd like
I can pass in --klipper-helm-image=X which adds the spec.jobImage property to the default HelmChart files in /var/lib/rancher/rke2/server/manifests

Describe alternatives you've considered

add to config (RKE2 in the example but applies to k3s):

disable:
- rke2-canal
- rke2-coredns
- rke2-ingress-nginx
- rke2-metrics-server

cp /var/lib/rancher/rke2/data/v1.21.3-rke2r1-789f04a3cde2/charts/rke2-canal.yaml /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml (repeat for other charts).

add the spec.jobImage parameter to the 4 charts.

start rke2-server.

Additional context
This would allow for a rebuilt image to be used without setting up a mirror for docker.io such as using IronBank images

Backporting
no

• Needs backporting to older releases

[FruityWelsh]

I believe the "disable" causes the new charts to be deleted.

[brandond]

@FruityWelsh is correct, if you use the same names for your custom manifest as the rke2 bundled chart manifests that you've disabled, your custom manifests will be disabled and deleted as well. If you're providing custom manifests, you should be sure that they don't use the rke2- prefix, as that is essentially reserved for packaged components.

[FruityWelsh]

Are there some good methods to trouble shoot the add-ons resource not adding the helm installation jobs?
I changed the charts to no longer have the rke2 prefix, and it does show up successfully when running kubectl get addon -A, but does not install the service or the helm install pod at all in kube-system.

Quick description (kubectl -n kube-system describe addon canal):

Name:         canal
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>
API Version:  k3s.cattle.io/v1
Kind:         Addon
Metadata:
  Creation Timestamp:  2021-09-02T17:57:03Z
  Generation:          1
  Managed Fields:
    API Version:  k3s.cattle.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:source:
      f:status:
    Manager:         [email protected]
    Operation:       Update
    Time:            2021-09-02T17:57:03Z
  Resource Version:  749
  UID:               06c2de18-e8b8-4147-8e5e-646c77c78636
Spec:
  Source:  /var/lib/rancher/rke2/server/manifests/canal.yaml
Status:
Events:  <none>

[brandond]

You can look at the k3s service log on the server to see if perhaps the yaml is malformed. With 1.22 releases, you should see events as well when describing, to save having to go grep through log files.

[FruityWelsh]

is that available in any of the rke2 releases as well?

I am not seeing anything besides Traifek erroring from helm.

What my canal.yaml looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  jobImage: {{ RKE2_KILPPER_HELM_IMAGE }}:{{ RKE2_KLIPPER_HELM_IMAGE_TAG }}
  valuesContent: |-
    flannel:
      image:
        repository: {{ RKE2_FLANNEL_IMAGE }}
        tag: {{ RKE2_FLANNEL_IMAGE_TAG }}
    calico:
      cniImage:
        repository: {{ RKE2_CALICO_IMAGE }}
        tag: {{ RKE2_CALICO_IMAGE_TAG }}
      nodeImage:
        repository: {{ RKE2_CALICO_IMAGE }}
        tag: {{ RKE2_CALICO_IMAGE_TAG }}
      flexvolImage:
        repository: {{ RKE2_CALICO_IMAGE }}
        tag: {{ RKE2_CALICO_IMAGE_TAG }}

Edit:
added jobImage that is included now and removes Config

[brandond]

Does it literally look like that, including the templating placeholders?

[FruityWelsh]

No the those are replaced with images from a private docker registry, and tag strings

[brandond]

Your HelmChart is does not seem to contain an actual chart to install? There's no name, version, repo, or inline chart data. Take a look at the manifest created by RKE2 when you don't disable the packaged Canal CNI; it will have the chart data base64-encoded inline. You can at the very least copy that file as a starting point for your own manifest and modify that before disabling the packaged CNI.

If you don't need to use a custom JobImage, you can just use a HelmChartConfig manifest to apply customizations to the stock chart: https://docs.rke2.io/helm/#customizing-packaged-components-with-helmchartconfig

[FruityWelsh]

Further success, though now the helm charts are pending, they are failing to schedule because the node is not ready.

[brandond]

CNI charts need to be flagged as bootstrap, otherwise they won't be installed until after the nodes are ready. Another thing that's handled in the stock chart.

[FruityWelsh]

Success I believe (my custom image is having issues, but I have no reason to believe related to the workaround):
Example chart template:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  jobImage: {{ HELM_IMAGE }}
  bootstrap: true
  chart: {{ RKE2_COREDNS_CHART }}
  valuesContent: |-
    image:
      repository: {{ RKE2_COREDNS_IMAGE }}
      imageTag: {{ RKE2_COREDNS_IMAGE_TAG }}

[ShylajaDevadiga]

Validated on master branch using commit id 2eddfe6

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
Single node
Steps

• Install k3s passing --helm-job-image to override klipper-helm image other than the default

Here in the example we have overridden using the older version of klipper-helm

Default image: rancher/klipper-helm:v0.8.0-build20230510

$ kubectl describe pod -n kube-system  helm-install-traefik-pq45j |grep klipper
    Image:         rancher/klipper-helm:v0.8.0-build20230510
    Image ID:      docker.io/rancher/klipper-helm@sha256:4d2ec9ac78f6e3ca3d4dd0a1c3b754aec2b4f19e3a922c6ebcb0d74bb5ac674a

Overridden with: rancher/klipper-helm:v0.7.7-build20230403

curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=2eddfe6cf4e67b7d6aab0fe8311d72372030459e sh -s - server --cluster-init --helm-job-image=rancher/klipper-helm:v0.7.7-build20230403

$ kubectl describe pod -n kube-system helm-install-traefik-9n762 |grep klipper
    Image:         rancher/klipper-helm:v0.7.7-build20230403
    Image ID:      docker.io/rancher/klipper-helm@sha256:3cd71ccc3cce5010865d9ab3548788a3a8cf5e9cfc5b48cc54b5b815675dd121
    
$ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-77ccd57875-vhqfj                 1/1     Running     0          25m
kube-system   helm-install-traefik-9n762               0/1     Completed   1          25m
kube-system   helm-install-traefik-crd-g58ld           0/1     Completed   0          25m
kube-system   local-path-provisioner-957fdf8bc-csrf8   1/1     Running     0          25m
kube-system   metrics-server-648b5df564-4t5hj          1/1     Running     0          25m
kube-system   svclb-traefik-cad2b75b-4nlx9             2/2     Running     0          25m
kube-system   traefik-64f55bb67d-kxwgg                 1/1     Running     0          25m