Using alternate location data-dir crictl.yaml still resides in the default data-dir path

[ShylajaDevadiga]

Version:
rke2 version v1.18.9+rke2r1

Describe the Issue:

While using /mnt dir as the param for data-dir all content of data dir is relocated to new /mnt dir, except for crictl file in the agent dir

# ls -R  /var/lib/rancher/rke2/
/var/lib/rancher/rke2/:
agent

/var/lib/rancher/rke2/agent:
etc

/var/lib/rancher/rke2/agent/etc:
crictl.yaml

# ls /mnt
agent  bin  data  server

Cluster is up

/mnt/bin/kubectl get nodes
NAME              STATUS   ROLES         AGE     VERSION
ip-172-31-23-79   Ready    <none>        2m14s   v1.18.9+rke2r1
ip-172-31-30-69   Ready    etcd,master   41m     v1.18.9+rke2r1

[brandond]

This appears to be coming from here:

k3s-io/k3s@58aae57#diff-e86b77690979bd28019f01c993abec39R44

[ShylajaDevadiga]

While validating on k3s version v1.19.3+k3s-58b5b21f, found the behavior is different while passing data-dir and passing data-dir through config file
Node OS: Ubuntu 20.04, sudo user
Using data-dir flag in cli, alternate location is used and /var/lib/rancher does not exist

ubuntu@ip-172-31-34-47:~$ ls /var/lib/rancher
ls: cannot access '/var/lib/rancher': No such file or directory
ubuntu@ip-172-31-34-47:~$ ls /mnt
agent                      client-kube-proxy.crt  containerd                kubelet.kubeconfig    server-ca.crt
client-ca.crt              client-kube-proxy.key  data                      kubeproxy.kubeconfig  serving-kubelet.crt
client-k3s-controller.crt  client-kubelet.crt     etc                       pod-manifests         serving-kubelet.key
client-k3s-controller.key  client-kubelet.key     k3scontroller.kubeconfig  server

Using config file, /var/lib/rancher contains k3s/data dir

cat /etc/rancher/k3s/config.yaml 
data-dir: /mnt
 ls /var/lib/rancher/k3s/data/
76b839842c85db78d477b73c3dcb656159d31d91081517102614dad69c3a340c  current
ls /mnt
agent                      client-k3s-controller.key  client-kubelet.crt  etc                       kubeproxy.kubeconfig  server-ca.crt
client-ca.crt              client-kube-proxy.crt      client-kubelet.key  k3scontroller.kubeconfig  pod-manifests         serving-kubelet.crt
client-k3s-controller.crt  client-kube-proxy.key      containerd          kubelet.kubeconfig        server                serving-kubelet.key

[brandond]

@ShylajaDevadiga I've opened a new issue for the extracted binaries being in the wrong location; lets keep this one for just crictl.yaml

[ShylajaDevadiga]

rke2 version v1.18.12-rc1+rke2r1
crictl.yaml is present using alternate data-dir as /mnt.

ls /var/lib/rancher/rke2/agent/etc/crictl.yaml 
/var/lib/rancher/rke2/agent/etc/crictl.yaml

[brandond]

I don't believe that the required patches have been pulled through from k3s yet.

[ShylajaDevadiga]

Validated on rke2 version v1.19.5-rc1+rke2r1

On Centos and well as Ubuntu

sudo /mnt/bin/crictl -r /var/run/k3s/containerd/containerd.sock images
IMAGE                                                       TAG                     IMAGE ID            SIZE
docker.io/rancher/hardened-calico                           v3.13.3                 21c345f395893       158MB
docker.io/rancher/hardened-coredns                          v1.6.9                  b8eb1106d5ef1       48.4MB
docker.io/rancher/hardened-etcd                             v3.4.13-k3s1            f24e72a2007f8       49.8MB
docker.io/rancher/hardened-flannel                          v0.13.0-rancher1        f96df637bbc7e       147MB
docker.io/rancher/hardened-k8s-metrics-server               v0.3.6                  1afed3a414103       45.8MB
docker.io/rancher/hardened-kube-proxy                       v1.19.5                 198762105ed42       94.4MB
docker.io/rancher/hardened-kubernetes                       v1.19.5-rc1-rke2r1      250a08f4d3db1       172MB
docker.io/rancher/klipper-helm                              v0.3.2                  4be09ab862d40       50.7MB
docker.io/rancher/nginx-ingress-controller-defaultbackend   1.5-rancher1            b5af743e59849       2MB
docker.io/rancher/nginx-ingress-controller                  nginx-0.30.0-rancher1   2166188b0dba1       114MB
docker.io/rancher/pause                                     3.2                     e004ddc1b078f       300kB

[brandond]

Just confirming:

• SELinux policies are hardcoded to apply labels to default paths
• Any user who wants to change the data-dir from /var/lib/rancher/rke2 on a system with SELinux will need to clean up the prelabeled directories created by the rke2-selinux package, and either:
  • Disable selinux
  • Apply the appropriate labels to their custom data-dir

Failure to do either of the above will cause RKE2 to fail to start when using a custom data-dir.