Assigning node taints to system nodepool not working anymore, failing with SystemPoolHasRestrictedTaint

[nbusseneau]

Hello 😄

What happened: Creating system nodepools with node taints is not working since 2021-09-28, failing with the following error:

(SystemPoolHasRestrictedTaint) Operation failed due to insufficient nodes for system pod scheduling. Placing custom taints on system pool is not supported(except 'CriticalAddonsOnly' taint or taint effect is 'PreferNoSchedule'). Please refer to https://aka.ms/aks/system-taints for detail

What you expected to happen: System nodepool is created with the node taint applied.

How to reproduce it (as minimally and precisely as possible): Using the CLI:

az group create --name foo --location westeurope
az aks create --resource-group foo --name bar
az aks nodepool add --name nodepool2 --resource-group foo --cluster-name bar \
  --mode system --node-taints foo=bar:NoSchedule

Anything else we need to know?:

We recommend users to apply taint node.cilium.io/agent-not-ready=true:NoSchedule to nodepools when using cilium/cilium (CNI plugin) to prevent application pods from being managed by the default AKS CNI plugin.

• Rationale: if users want to use Cilium, there's a concurrency issue where application pods might be managed by Cilium or by the default CNI plugin. Thanks to the node taint, the application pods will not run until Cilium is deployed and removes the taint. A more detailed explanation can be found in Preventing unmanaged Cilium endpoints on newly-created nodes cilium/cilium#16602.
• Caveat: it is not possible to apply node taints to the initial nodepool of AKS clusters at creation time (see CLI request: set --node-taints for primary node pool via CLI #1402).
• Solution: as of writing, Cilium documentation recommended the users to create a cluster, then add a secondary system nodepool with the taint applied, then delete the initial nodepool.

We test this behaviour in our AKS CI, via an automated GitHub Actions workflow using the Azure CLI (see source command lines here).

• This was working until 2021-09-28 around 20:00 CEST: https://github.com/cilium/cilium/actions/runs/1283753297
• This stopped working with the error mentioned above soon after (here around 21:45 CEST): https://github.com/cilium/cilium/actions/runs/1284063464

Thanks!

[ghost]

Hi nbusseneau, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[palma21]

AKS System pools only allow 1 taint CriticalAddonsOnly=true:NoSchedule or soft taints, so that behavior is by design per the documentation.
https://docs.microsoft.com/en-us/azure/aks/use-system-pools#system-and-user-node-pools

AKS User pools support any taint.

You can still any any taint manually via kubectl in a per node / temporary basis, but not to the node pool model/permanently.

caveat: it is not possible to apply node taints to the initial nodepool of AKS clusters at creation time

This is possible now, that linked issue is not just waiting for update taints to be supported.

[nbusseneau]

@palma21 Thanks for your answer.

AKS System pools only allow 1 taint CriticalAddonsOnly=true:NoSchedule or soft taints

Does that mean that when it was working (before 2021-09-28), this was actually not intended and should never have been working in the first place?

If that is indeed the case, do you have any recommendation on pod management by third-party CNI plugins?

so that behavior is by design per the documentation.

Nit: I might have missed where this is specified, but I don't see where in the documentation it is specified that system nodepools can only use CriticalAddonsOnly=true:NoSchedule or soft taints => I would suggest clarifying this in the list of system nodepools restrictions.

caveat: it is not possible to apply node taints to the initial nodepool of AKS clusters at creation time

This is possible now, that linked issue is not just waiting for update taints to be supported.

Oh, is it? I don't see --node-taints in the az aks create documentation, and it does not seem to work with CLI version 2.28.0. Am I missing something?

[ghost]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.

[nbusseneau]

I still had questions though... :/