k3s server not passing arguments to kube controller, scheduler, and proxy.

[AshtonKem]

Environmental Info:
k3s version v1.23.10+k3s1 (826b949)
go version go1.17.13

Node(s) CPU architecture, OS, and Version:
Linux r420 5.15.47 #1-NixOS SMP Tue Jun 14 16:36:28 UTC 2022 x86_64 GNU/Linux

Cluster Configuration:
Single server.

Describe the bug:
I am trying to pass through arguments to the kube scheduler, controller manager, and proxy in order to fix the known prometheus-stack issue here. But the values I provide to k3s server do not seem to be working.

Since I'm using NixOS, most of my configuration is passed via command line flags. To avoid duplicating my Nix config here, here's my ExecStartEx from SystemD.

ExecStartEx={ path=/nix/store/c11xma06i9n9lxaipf4pdpbb8dka1q0w-k3s-1.23.10+k3s1/bin/k3s ; argv[]=/nix/store/c11xma06i9n9lxaipf4pdpbb8dka1q0w-k3s-1.23.10+k3s1/bin/k3s server --node-label app=zigbee-controller --etcd-expose-metrics true --kube-proxy-arg metrics-bind-address=0.0.0.0 --kube-controller-manager-arg address=0.0.0.0 --kube-controller-manager-arg bind-address=0.0.0.0 --kube-scheduler-arg bind-address=0.0.0.0 --kube-scheduler-arg address=0.0.0.0>

But when I check the logs for these arguments, I see that they are not being passed directly to the underlying services. For example, here are the logs for kube-controller-manager.

Oct 02 11:23:38 r420 k3s[2181326]: time="2022-10-02T11:23:38-06:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"

Steps To Reproduce:

• Installed K3s
• Set kube-proxy-arg, kube-controller-manager-arg, or kube-scheduler-arg via command line
• See that arguments are not passed.

Expected behavior:

I expected the bind-address to be set to 0.0.0.0 for all services.

Actual behavior:

No arguments appear to have been passed.

Additional context / logs:

Note: Some command line flags are working, so it's not just a nixos issue. The --node-label app=zigbee-controller is working as expected.

[brandond]

Just to pull that out, since it's kind of hard to read as posted...

k3s server \
--node-label app=zigbee-controller \
--etcd-expose-metrics true \
--kube-proxy-arg metrics-bind-address=0.0.0.0 \
--kube-controller-manager-arg address=0.0.0.0 \
--kube-controller-manager-arg bind-address=0.0.0.0 \
--kube-scheduler-arg bind-address=0.0.0.0 \
--kube-scheduler-arg address=0.0.0.0

[brandond]

This is a problem with how urfave/cli parses args and flags. Boolean flags (such as --etcd-expose-metrics) don't consume the following argument from argv, so everything from true onwards is treated as args, not as flags. Any leftover args are ignored.

tl;dr you can do --etcd-expose-metrics or --etcd-expose-metrics=true but not --etcd-expose-metrics true

Once you fix that, k3s will exit with another error, due to your use of an invalid controller-manager flag:

FATA[0004] controller-manager exited: unknown flag: --address

You should probably verify that all the args you're passing are valid according to the upstream docs: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

[AshtonKem]

Aha, I suspected it might be some quirk in the way that args are being parsed. Thank you!