Tunnelfront and coredns fails to schedule when all nodes are tainted

[folkol]

What happened:

Tunnelfront and coredns pods failed to be scheduled when all nodes were tainted.

What you expected to happen:

To have the ability of letting the kube system deployments tolerate a tainted node.

How to reproduce it (as minimally and precisely as possible):

Set up a cluster e.g.:

$ az aks create \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $CLUSTER_NAME \
    --node-count 1 \
    --nodepool-name nodepool1 \
    --vm-set-type VirtualMachineScaleSets

• Taint the only node: kubectl taint nodes -l agentpool=nodepool1 zone=nodepool1:NoSchedule
• Delete the tunnelfront pod: kubectl delete pod -n kube-system tunnelfront-xyz-123
• New pod will be stuck in pending because it fails to be schedule with message: 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.

Anything else we need to know?:

There is this similar issue: #363

Environment:

• Server version: v1.14.8
• Client version: v1.16.2
• Location: francecentral

[yamansama]

Hi, any news on this ?

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[palma21]

This is by design to allow you to move your system compoents that are not daemon sets to a specific pool.

For the record, they already have affinity to pools of mode System.
https://docs.microsoft.com/en-us/azure/aks/use-system-pools

If you want to taint a pool to only have system components use the taint CriticalAddonsOnly which is already tolerated:

 tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      - key: CriticalAddonsOnly
        operator: Exists