[release-1.24] "FATA [0000] permission denied" when /var is mounted noexec

[brandond]

• Backport "FATA [0000] permission denied" when /var is mounted noexec #7372

[est-suse]

Validated on branch with commit / version

f575bd1cc6ac19995e014bb5fde8bd98164dad05

k3s version v1.24.13+k3s-f575bd1c (f575bd1c)
go version go1.19.8

Environment Details
Infrastructure

Cloud
Hosted
Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-43-6 5.15.0-1033-aws #37~20.04.1-Ubuntu SMP Fri Mar 17 11:39:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: 644
token: test
prefer-bundled-bin: true
k3s_use_unsupported_config: true
cluster-init: true

Testing Steps:

Copy config.yaml
$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
Add a tmpfs mount to /var in fstab with noexec: by adding the following line: vi /etc/fstab
tmpfs /var tmpfs defaults,nodev,nosuid,noexec 0 0
reboot the VM
4. Install k3s: curl https://get.k3s.io/ | INSTALL_K3S_COMMIT=2b24c9917cdfec92439ac68dd6706fffe20195a5 sh -s - server
5. Check the logs for the new error msg

Validation Results:

May 16 22:40:28 ip-172-31-8-184 k3s[1185]: time="2023-05-16T22:40:28Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:40:33 ip-172-31-8-184 k3s[1194]: time="2023-05-16T22:40:33Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:40:38 ip-172-31-8-184 k3s[1202]: time="2023-05-16T22:40:38Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:40:44 ip-172-31-8-184 k3s[1210]: time="2023-05-16T22:40:44Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:40:49 ip-172-31-8-184 k3s[1218]: time="2023-05-16T22:40:49Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:40:54 ip-172-31-8-184 k3s[1226]: time="2023-05-16T22:40:54Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:40:59 ip-172-31-8-184 k3s[1234]: time="2023-05-16T22:40:59Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:41:05 ip-172-31-8-184 k3s[1242]: time="2023-05-16T22:41:05Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:41:10 ip-172-31-8-184 k3s[1250]: time="2023-05-16T22:41:10Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:41:15 ip-172-31-8-184 k3s[1258]: time="2023-05-16T22:41:15Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"
May 16 22:41:20 ip-172-31-8-184 k3s[1266]: time="2023-05-16T22:41:20Z" level=fatal msg="exec /var/lib/rancher/k3s/data/78d63c205d5cd5e710d1250c5751d24e946dddd2e8b84cd7a382d4dfe20dbebd/bin/k3s-server failed: permission denied"