Relax PreferNoSchedule Taint

[dewjam]

1. Issue, if available:
#1433

2. Description of changes:
When taints with the effect of PreferNoSchedule are applied to nodes, the behavior of kube-scheduler is to only schedule pods to said nodes when no other scheduling options exist. Currently, Karpenter treats taints with the effect of PreferNoSchedule as NoSchedule and will not provision a node with the PreferNoSchedule taint unless the pod tolerates it explicitly.

This change adds a blanket PreferNoSchedule toleration to pods when no other Provisioners are selectable and after attempting to relax Affinity terms. In short, Provisioners with PreferNoSchedule taints which would otherwise meet the pods requirements will be used, but only as a last resort.

3. How was this change tested?
A test was implemented in the test suite. Additionally, manual tests were executed with a combination of provisioners configured.

Example using a matching provisioner (PreferNoSchedule toleration is not added to pod):

2022-03-08T17:23:29.999Z	INFO	controller.provisioning	Batched 5 pods in 1.074480675s	{"commit": "774d5ed", "provisioner": "default"}

Example where Affinity terms exist and are relaxed which then match a provisioner (PreferNoSchedule toleration is not added to pod):

2022-03-08T18:06:52.002Z	DEBUG	controller.selection	Could not schedule pod, matched 0/2 provisioners, tried provisioner/restricted: did not tolerate fake=fake:NoSchedule; tried provisioner/zonal: incompatible requirements, [us-west-2d] not in [us-west-2a], key topology.kubernetes.io/zone	{"commit": "774d5ed", "pod": "default/inflate-7c98f5b68d-kd78b"}
2022-03-08T18:06:52.007Z	DEBUG	controller.selection	Relaxing soft constraints for pod since it previously failed to schedule, removing: spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[0]={"weight":1,"preference":{"matchExpressions":[{"key":"topology.kubernetes.io/zone","operator":"In","values":["us-west-2a"]}]}}	{"commit": "774d5ed", "pod": "default/inflate-7c98f5b68d-kd78b"}
2022-03-08T18:06:53.008Z	INFO	controller.provisioning	Batched 1 pods in 1.000311298s	{"commit": "774d5ed", "provisioner": "zonal"}

Example where only matching provisioner has PreferNoSchedule taint (PreferNoSchedule toleration is added to pod):

2022-03-08T17:31:32.796Z	DEBUG	controller.selection	Could not schedule pod, matched 0/2 provisioners, tried provisioner/prefer-no-schedule: did not tolerate fake=fake:PreferNoSchedule; tried provisioner/restricted: did not tolerate fake=fake:NoSchedule	{"commit": "774d5ed", "pod": "default/inflate-6b88c9fb68-t7cvm"}
2022-03-08T17:31:32.801Z	DEBUG	controller.selection	Relaxing soft constraints for pod since it previously failed to schedule, adding: toleration for PreferNoSchedule taints	{"commit": "774d5ed", "pod": "default/inflate-6b88c9fb68-t7cvm"}
2022-03-08T17:31:33.802Z	INFO	controller.provisioning	Batched 1 pods in 1.000636795s	{"commit": "774d5ed", "provisioner": "prefer-no-schedule"}

Example where no provisioners match (PreferNoSchedule toleration is added to pod, but still no provisioners match):

2022-03-08T17:36:28.079Z	DEBUG	controller.selection	Could not schedule pod, matched 0/1 provisioners, tried provisioner/restricted: did not tolerate fake=fake:NoSchedule	{"commit": "774d5ed", "pod": "default/inflate-6b88c9fb68-5jj8f"}
2022-03-08T17:36:28.085Z	DEBUG	controller.selection	Relaxing soft constraints for pod since it previously failed to schedule, adding: toleration for PreferNoSchedule taints	{"commit": "774d5ed", "pod": "default/inflate-6b88c9fb68-5jj8f"}
2022-03-08T17:36:28.085Z	DEBUG	controller.selection	Could not schedule pod, matched 0/1 provisioners, tried provisioner/restricted: did not tolerate fake=fake:NoSchedule	{"commit": "774d5ed", "pod": "default/inflate-6b88c9fb68-5jj8f"}

4. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: link to issue
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: f6da842

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/622a31d46559e40008e364c4

[ellistarn]

Nice fix! A couple comments.

Also -- does this require docs updates?