Kubeadm should run kube-proxy under its' own identity

[andrewrynhard]

kubeadm runs kube-proxy under the kubelet's identity. It should run under its' own identity.

[luxas]

@liggitt @deads2k Does it sound like a good idea to create a new flag to kube-proxy (--use-inclusterconfig-creds or something) where it reads the ca.crt and token from /var/run/secrets?

It would also take the --master parameter for knowing where the apiserver is.

Optionally, we could set --kubernetes-service-node-port=30000 and set the --master flag to localhost:30000 so kubelet will loadbalance the requests to the endpoints for the kubernetes service.

Does this sound good to you?

[deads2k]

@liggitt @deads2k Does it sound like a good idea to create a new flag to kube-proxy (--use-inclusterconfig-creds or something) where it reads the ca.crt and token from /var/run/secrets?

I haven't looked at the kube-proxy arg handling in a very long time, but I'd expect it to take a --kubeconfig and the resolution of that kubeconfig works like kubectl, where if its missing, it attempts to use an in-cluster-config if it is available.

[mikedanese]

InClusterConfig uses the kubernetes service vip which isn't setup til the kube-proxy runs. We could however use a service account token.

[luxas]

Yes, @mikedanese @deads2k @liggitt are you in favor of a PR that adds a flag so kube-proxy reads the ca.crt and token from /var/run/secrets so we can use a kube-proxy-specific serviceaccount?

kube-proxy can't use the full inclusterconfig because of the chicken-and-egg situation, and we can't either add this functionality by default because it could break existing users.

I could send a such PR next week if that sounds good, it should be pretty straightforward I guess

[liggitt]

I'd rather keep kube-proxy simple and make it the responsibility of the thing starting kube-proxy to build a kubeconfig for it.

if you want to run it in a daemonset, that would mean injecting a kubeconfig with the ca/token/master address you want. you can reference the service account token and ca.crt from that kubeconfig if you like:

kind: Config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server: https://master:masterport
  name: default
contexts:
- context:
    cluster: default
    namespace: default
    user: default
  name: default
current-context: default
users:
- name: default
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

[luxas]

Ah, of course. I totally forgot the possibility to point to files in the kubeconfig, that's the best approach indeed. Thanks!

What do you think about --kubernetes-service-node-port=30000 and --master=localhost:30000 vs hard-coding server: https://master:masterport in the kubeconfig file? The former could possibly be used with HA, at least it's more dynamic (especially when we're selfhosting things etc.)

[mikedanese]

Maybe I misunderstand what you are suggesting. The kube-proxy isn't running at this point so there is no node port.

[luxas]

I thought it was the kubelet that was opening the nodePort, I didn't ever check it was the proxy which it indeed is.
Makes sense, then that's obviously not an option :)

I guess we bet everything on HA kubeconfig then ;)

[luxas]

Work in progress here: kubernetes/kubernetes#40556

[luxas]

This is now fixed in kubernetes/kubernetes#40556, kube-proxy injects a ConfigMap with the kubeconfig file and runs in a separate ServiceAccount that has granted the system:node-proxier ClusterRole to it.

So now it runs under its own identity, thanks for the pointers 👍