document Karpenter's reaction to taints/tolerations

[geoffcline]

as of #643, Karpenter will "Generate matching taints on nodes for pods that contain tolerations"

This behavior should be added to the docs.

[stevehipwell]

Are there always ignored taints and customisable ignored taints?

I'll read #628 before asking any further questions there.

[ellistarn]

We desperately need to publish docs for this. They are currently WIP. This is the best explanation I can give (if you can read go). https://github.com/awslabs/karpenter/blob/main/pkg/controllers/allocation/scheduling/taints.go#L26.

Feel free to contact me on Kubernetes slack for quick questions.

[stevehipwell]

Thanks @ellistarn. I was specifically thinking about the CriticalAddonsOnly=true:NoSchedule taint which gets added to EKS nodes and shows up in a number of AWS Helm charts by default.

[ellistarn]

Hey @stevehipwell, I've temporarily removed taint generation from karpenter (for our next release). I'm currently leaning towards a solution of only tainting nodes for tolerations with the key karpenter.sh/isolation. Similarly, we would generate labels for the same label key. This avoids the undesirable taint generation you mentioned above. It also helps avoid confusing behavior from typos in label generation like node.kubernetes.i/instance-type.

Does this make sense to you?

[stevehipwell]

Hey @stevehipwell, I've temporarily removed taint generation from karpenter (for our next release). I'm currently leaning towards a solution of only tainting nodes for tolerations with the key karpenter.sh/isolation. Similarly, we would generate labels for the same label key. This avoids the undesirable taint generation you mentioned above. It also helps avoid confusing behavior from typos in label generation like node.kubernetes.i/instance-type.

Does this make sense to you?

@ellistarn this makes a lot of sense to me. I like the fact that we're talking "isolation" and that the taints and labels are just the implementation. I assume there is still the ability to define static taints and labels on a provisioner?

[ellistarn]

Correct!