[Release-1.27] - Registry TLS configuration from registries.yaml is only honored for mirror endpoints

[brandond]

Backport fix for Registry TLS configuration from registries.yaml is only honored for mirror endpoints

• Registry TLS configuration from registries.yaml is only honored for mirror endpoints #9839

[aganesh-suse]

Validated on release-1.27 branch with version v1.27.13-rc1+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

registries.yaml:

 $ sudo cat /etc/rancher/k3s/registries.yaml
mirrors:
  pvt-registry.com:
    endpoint:
      - pvt-registry.com
  docker.io:
    endpoint:
      - pvt-registry.com      
  k8s.gcr.io:
    endpoint:
      - pvt-registry.com      
configs:
  pvt-registry.com:
    auth:
      username: xxxx
      password: xxxx
    tls:
      ca_file: /home/user/ca.pem

test-image.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: pvt-reg-test
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pvt-reg-test
  namespace: pvt-reg-test
spec:
  selector:
    matchLabels:
      k8s-app: nginx-app-clusterip
  replicas: 2
  template:
    metadata:
      labels:
        k8s-app: nginx-app-clusterip
    spec:
      containers:
      - name: nginx
        image: pvt-registry.com/nginx:latest
        ports:
        - containerPort: 8080

Testing Steps

1. Copy config.yaml and registries.yaml

$ sudo mkdir -p /etc/rancher/k3s 
$ sudo cp config.yaml /etc/rancher/k3s
$ sudo cp registries.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.27.13-rc1+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Push an image onto the private registry and try to deploy a pod with said image.
   The image should get pulled and pod should come up without any tls certificate errors.

$ kubectl apply -f test-image.yaml
$ kubectl get pods -n pvt-reg-test
$ kubectl describe pod/pvt-reg-test-abcd -n pvt-reg-test

5. Check the hosts.toml files for host section

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.27.12+k3s1 (78ad5756)
go version go1.21.8

$ kubectl get pods -A
kube-system      coredns-77ccd57875-fh97z                  1/1     Running            0          4m42s
kube-system      helm-install-traefik-crd-vw7xb            0/1     Completed          0          4m42s
kube-system      helm-install-traefik-ggglg                0/1     Completed          1          4m42s
kube-system      local-path-provisioner-79ffd768b5-prbzn   1/1     Running            0          4m42s
kube-system      metrics-server-c44988498-8tkrv            1/1     Running            0          4m42s
kube-system      svclb-traefik-94c80479-6x95r              2/2     Running            0          3m27s
kube-system      svclb-traefik-94c80479-kcv7d              2/2     Running            0          2m39s
kube-system      svclb-traefik-94c80479-lfkhl              2/2     Running            0          4m27s
kube-system      svclb-traefik-94c80479-mgtjd              2/2     Running            0          4m27s
kube-system      traefik-768bdcdcdd-bgj2r                  1/1     Running            0          4m27s
pvt-reg-test     pvt-reg-test-74c6894c78-hvtlr             0/1     ErrImagePull       0          28s
pvt-reg-test     pvt-reg-test-74c6894c78-nrvgg             0/1     ErrImagePull       0          28s

Pod Events:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m52s                  default-scheduler  Successfully assigned pvt-reg-test/pvt-reg-test-74c6894c78-nrvgg to ip-172-31-21-68
  Normal   Pulling    3m27s (x4 over 4m52s)  kubelet            Pulling image "pvt-registry-.com/nginx:latest"
  Warning  Failed     3m27s (x4 over 4m52s)  kubelet            Failed to pull image "pvt-registry-.com/nginx:latest": rpc error: code = Unknown desc = failed to pull and unpack image "pvt-registry-.com/nginx:latest": failed to resolve reference "pvt-registry-.com/nginx:latest": failed to do request: Head "https://pvt-registry-.com/v2/nginx/manifests/latest": tls: failed to verify certificate: x509: certificate signed by unknown authority
  Warning  Failed     3m27s (x4 over 4m52s)  kubelet            Error: ErrImagePull
  Warning  Failed     3m15s (x6 over 4m51s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    3m3s (x7 over 4m51s)   kubelet            Back-off pulling image "pvt-registry-.com/nginx:latest"

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.27.13-rc1+k3s1 (33ee1687)
go version go1.21.9

$ kubectl get pods -A 
kube-system      coredns-77ccd57875-5d8pj                  1/1     Running     0          4m45s
kube-system      helm-install-traefik-crd-n96rv            0/1     Completed   0          4m45s
kube-system      helm-install-traefik-pbgcr                0/1     Completed   1          4m45s
kube-system      local-path-provisioner-79ffd768b5-qpw9k   1/1     Running     0          4m45s
kube-system      metrics-server-c44988498-dx9bz            1/1     Running     0          4m45s
kube-system      svclb-traefik-646f1763-7fd8k              2/2     Running     0          4m30s
kube-system      svclb-traefik-646f1763-85494              2/2     Running     0          2m40s
kube-system      svclb-traefik-646f1763-pmgz6              2/2     Running     0          3m29s
kube-system      svclb-traefik-646f1763-rz458              2/2     Running     0          4m27s
kube-system      traefik-7d5c94d587-4c246                  1/1     Running     0          4m30s
pvt-reg-test     pvt-reg-test-6675b7ccbb-fvrks             1/1     Running     0          29s
pvt-reg-test     pvt-reg-test-6675b7ccbb-ssgsn             1/1     Running     0          29s

Check hosts.toml file contents:

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/pvt-registry-.com/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://pvt-registry-.com/v2"
capabilities = ["pull", "resolve", "push"]

ca = ["/home/ubuntu/ca.pem"]


[host]

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://pvt-registry-.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://k8s.gcr.io/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://pvt-registry-.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]