Support labels on Provisioner set outside of labels

[olemarkus]

Tell us about your request
What do you want us to build?

Karpenter currently only supports label selectors that either karpenter manage (by assuming it can configure kubelet) or that are well-known (those kubelet set itself, such as kubernetes.io/os and kubernetes.io/arch).

But there is a host of other components that put labels on nodes: KCM, CCM, CNIs, device managers and others.
It should be possible to tell karpenter that instances launched by a given Provisioner is expected to get these labels.

This is the label analog to #628

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

One example would be a Pod that is scheduled to run only on worker nodes and that would have node-role.kubernetes.io/node as selector.

Note that many of these selectors are forbidden to be set by kubelet (e.g the above-mentioned label), so this will most likely need a new field in the API.

Are you currently working around this issue?
Not possible.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ellistarn]

I'd love to see another example other than kop's node-role.kubernetes.io/node. One might be the node labels from https://github.com/NVIDIA/k8s-device-plugin. Seeing more examples will help us design the right solution.

I'll be honest that this feels like a hack to me. One option (at least in this case) would be to use a NotIn NodeAffinity rule to avoid the kops masters, rather than attracting to kops nodes.

[olemarkus]

The node role label is a wellknown label that all installers use. Kops has a role for spots though that is very popular for using with cluster autoscaler to ensure a pod are either scheduled on or off spot instances.

master (and apiserver) nodes are tainted and doesn't automatically attract any pods.

CAS has templates that support this functionality.

[olemarkus]

It's worth mentioning that karpenter cannot schedule nodeAffinity that uses NotIn. There is no way for karpenter to know what labels a node may not have.

[ellistarn]

Would this "just work" if kops didn't use the kubernetes.io domain?

[olemarkus]

What are you referring to? If you mean the node-role.kubernetes.io/role label then that is a label that all nodes have in a cluster. It is e.g used by kubectl get nodes. All installers have mechanisms for setting this label. That also goes for the older kubernetes.io/role label.

[billrayburn]

@olemarkus We're not sure this is still a requirement. Please reopen or comment if still needed.