k3s certificate rotate doesn't fully respect data_dir

[cortices]

Environmental Info:
K3s Version:
k3s version v1.24.4+k3s-7d66e419-dirty (7d66e41)
go version go1.18.5

Node(s) CPU architecture, OS, and Version:
Linux moria 5.15.62+truenas #1 SMP Mon Sep 12 18:25:31 UTC 2022 x86_64 GNU/Linux

Cluster Configuration:
1 server, administered via TrueNAS
k3s Data dir in /mnt/Knapsack/ix-applications/k3s (default location for TrueNAS on chosen App pool).

Describe the bug:
k3s certificate rotate fails to respect the Data_Dir value in all commands, regardless how it's provided.

Steps To Reproduce:

• Installed K3s: TrueNAS base k3s
• Debugging separate issue with certs when trying to fetch logs (to be filed separately when more info gathered).
• Try running any of:
  • k3s cert rotate -d /mnt/Knapsack/ix-applications/k3s (definitely correct location)
  • export K3S_CONFIG_FILE=/etc/rancher/k3s/config.yaml; k3s cert rotate (config.yaml contains correct data_dir)

Expected behavior:
Correct rotation of certificates.

Actual behavior:
Fails with:

FATA[0000] stat /var/lib/rancher/k3s/server/tls: no such file or directory

Additional context / logs:
To get around the issue, I created a symbolic link: sudo mkdir /var/lib/rancher; sudo ln -s /mnt/Knapsack/ix-applications/k3s /var/lib/rancher/k3s
This solved the issue and certs were correctly rotated, showing that it's not an issue with my config or data.

More verbose shell transcript

sparky@moria[/mnt/Knapsack/ix-applications/k3s/data]# sudo k3s -d /mnt/Knapsack/ix-applications/k3s certificate rotate 
FATA[0000] stat /var/lib/rancher/k3s/server/tls: no such file or directory

root@moria[/mnt/Knapsack/ix-applications/k3s]# export K3S_CONFIG_FILE=/etc/rancher/k3s/config.yaml
root@moria[/mnt/Knapsack/ix-applications/k3s]# echo $K3S_CONFIG_FILE 
/etc/rancher/k3s/config.yaml
root@moria[/mnt/Knapsack/ix-applications/k3s]# sudo k3s certificate rotate
FATA[0000] stat /var/lib/rancher/k3s/server/tls: no such file or directory

root@moria[/mnt/Knapsack/ix-applications/k3s]# ls /var/lib
apache2/   consul/  dkms/   glusterd/   kdump/        libxml-sax-perl/  mibs/   ntp/        polkit-1/   samba/      smartmontools/  syslog-ng/  vim/
apt/       ctdb/    dpkg/   grub/       kubelet/      logrotate/        misc/   nut/        portables/  scst/       snmp/           systemd/    waagent/
cni/       dbus/    exim4/  haproxy/    letsencrypt/  machines/         nfs/    os-prober/  private/    setserial/  sntp/           ucf/        xfonts/
collectd/  dhcp/    git/    ieee-data/  libvirt/      man-db/           nginx/  pam/        python/     sgml-base/  sudo/           usbutils/   xml-core/
root@moria[/mnt/Knapsack/ix-applications/k3s]# sudo mkdir /var/lib/rancher
root@moria[/mnt/Knapsack/ix-applications/k3s]# sudo ln -s /mnt/Knapsack/ix-applications/k3s /var/lib/rancher/k3s
root@moria[/mnt/Knapsack/ix-applications/k3s]# l /var/lib/rancher 
total 512
lrwxrwxrwx 1 root 33 Oct  1 13:46 k3s -> /mnt/Knapsack/ix-applications/k3s
root@moria[/mnt/Knapsack/ix-applications/k3s]# sudo k3s certificate rotate                                      
INFO[0000] Server detected, rotating server certificates 
INFO[0000] Rotating certificates for admin service      
INFO[0000] Rotating certificates for etcd service       
INFO[0000] Rotating certificates for api-server service 
INFO[0000] Rotating certificates for controller-manager service 
INFO[0000] Rotating certificates for cloud-controller service 
INFO[0000] Rotating certificates for scheduler service  
INFO[0000] Rotating certificates for k3s-server service 
INFO[0000] Rotating dynamic listener certificate        
INFO[0000] Rotating certificates for k3s-controller service 
INFO[0000] Rotating certificates for auth-proxy service 
INFO[0000] Rotating certificates for kubelet service    
INFO[0000] Rotating certificates for kube-proxy service 
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/k3s/server/tls-1664595979, please restart k3s server or agent to rotate certificates

[brandond]

k3s version v1.24.4+k3s-7d66e419-dirty

This would appear to be a custom build of K3s. Did you build it yourself, or get it from somewhere else?

Try running any of:
k3s cert rotate -d /mnt/Knapsack/ix-applications/k3s (definitely correct location)
export K3S_CONFIG_FILE=/etc/rancher/k3s/config.yaml; k3s cert rotate (config.yaml contains correct data_dir)

I can verify that config loading doesn't appear to be working properly for some reason, but I can pass it on the CLI just fine:

systemd-node-1:/ # cat /etc/rancher/k3s/config.yaml
data-dir: /var/lib/rancher/custom-data-dir

systemd-node-1:/ # k3s certificate rotate
FATA[0000] stat /var/lib/rancher/k3s/server/tls: no such file or directory

systemd-node-1:/ # k3s certificate rotate --data-dir /var/lib/rancher/custom-data-dir
INFO[0000] Server detected, rotating server certificates
INFO[0000] Rotating certificates for admin service
INFO[0000] Rotating certificates for etcd service
INFO[0000] Rotating certificates for api-server service
INFO[0000] Rotating certificates for controller-manager service
INFO[0000] Rotating certificates for cloud-controller service
INFO[0000] Rotating certificates for scheduler service
INFO[0000] Rotating certificates for k3s-server service
INFO[0000] Rotating dynamic listener certificate
INFO[0000] Rotating certificates for k3s-controller service
INFO[0000] Rotating certificates for auth-proxy service
INFO[0000] Rotating certificates for kubelet service
INFO[0000] Rotating certificates for kube-proxy service
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/custom-data-dir/server/tls-1664603843, please restart k3s server or agent to rotate certificates

[cortices]

The build is from TrueNAS 22.12 Beta 1. I'm not exactly sure where their CI and codebase are located.

[brandond]

OK. Well I can reproduce part of the bug, but in general I would recommend running our supported builds as opposed to other ones built from dirty repos with unknown changes.

[brandond]

@dereknola you might take a look at this as part of your work to clean up flag parsing on #6420

[dereknola]

This a problem around subcommands and cli context. You need to ensure in your script that any flags are pass after the subcommand. So you can use k3s certificate rotate -d /tmp/k3s1 not k3s -d /tmp/k3s1 certificate rotate. This ensures that the subcommand actually has access to the -d flag value, otherwise is locked up in the parent commands context.

[brandond]

Somehow we've made this function properly for other things like --debug, but I think it took a surprising amount of work to use it consistently. urfave/cli doesn't do flag inheritance in a very intuitive way.

[caroline-suse-rancher]

@dereknola will create an issue about adding this expectation for flags to be past at the end of the command; closing this issue