CLI request: set --node-taints for primary node pool via CLI

[folkol]

What happened:

$ az aks create \
    --name foo \
    --resource-group bar \
    --node-taints "foo=bar:NoSchedule"
az: error: unrecognized arguments: --node-taints foo=bar:NoSchedule
usage: az [-h] [--verbose] [--debug]
          [--output {json,jsonc,table,tsv,yaml,none}] [--query JMESPATH]
          {aks} ...

What you expected to happen:

I expected the primary node pool to be tainted in the same way as additional node pools can be tainted like this:

$ az aks nodepool add \
    --name foo \
    --resource-group bar \
    --cluster-name baz \
    --node-taints "foo=bar:NoSchedule"

How to reproduce it (as minimally and precisely as possible):

See "What you expected".

Anything else we need to know?:

This is related to #1401.

[jluk]

Edit: First pool taints can only be set during cluster creation, since taints are create-time only right now.

Thanks for the request, today you can add your own taints to the first node pool when the cluster is being created. You are correct that functionality is not in the CLI yet, but can be set via ARM template after the cluster/pool is created.

Could you share what taints you plan to apply to primary pool and the scenario of tagging goals? Is it to try to isolate system-pods (or the like) to the primary pool?

[folkol]

I was not aware of the primary node pool taint through ARM templates — that's convenient. (We have instead tainted the individual nodes.)

Our goal with the taints is compartmentalization of the different components of our workload, and not directly related to the system pods. We wouldn't mind the systems pods to be scheduled on the same nodes as our public components — but right now it seems like we need to have one "untainted" node pool for some of the system pods (see the related ticket about this: #1401).

[jluk]

Manually adding node taints via kubectl won't guarantee persistence/reapplication of those to new nodes created during scale or upgrade events, hence if you rely on a taint logic for scheduling you should set it through the AKS API definition for an agent pool.

Will leave this ticket open for tracking this CLI gap and jump to the related issue.

[ondrejhlavacek]

@jluk How do you add a node taint to an existing node pool using ARM templates? I'm getting

Deployment failed. Correlation ID: ***. {
  "code": "PropertyChangeNotAllowed",
  "message": "Changing property 'properties.nodeTaints' is not allowed.",
  "target": "properties.nodeTaints"
}

[jluk]

@ondrejhlavacek apologies, my previous comment was incorrect that you can do a post-create update of pool taints. I've updated it.

This property is only set during pool creation - so you will need to define the taints on the first agent pool when the cluster is created in the ARM template.

[ondrejhlavacek]

@jluk Thanks for the clarification! Why is this property read-only during update?

[jluk]

Today it's not a mutable property because there is no reconciliation mechanism for cleanup/reapplication handling. We have that capability on the backlog, but we need to get some more important work done before supporting that.

[ernani]

Weird thing here, I've added taints to another nodepool I've just added and it shows up as:

TLDR: Any reason why in my case it shows foo=bar:Noschedule and in the doc it shows up as "foo": "bar:NoSchedule"

az aks nodepool add \
    --resource-group foo \
    --cluster-name bar \
    --name footaint \
    --node-count 1 \
    --node-taints "foo=bar:NoSchedule"

But after creating it, the list shows as:

...
"nodeTaints": [
    "foo=bar:NoSchedule"
],
...

Where as in the doc linked here:
https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools#specify-a-taint-label-or-tag-for-a-node-pool

It shows as:

...
"nodeTaints":  {
  "foo": "bar:NoSchedule"
},
...

[poochwashere]

@jluk Sorry for the silly question but I have been trying to deploy an arm template pointed to the api "apiVersion": "2020-03-01" to "type": "Microsoft.ContainerService/managedClusters/agentPools" with this taint setting..."nodeTaints": "CriticalAddonsOnly=true:NoSchedule" but I get this error, no matter what I try.
Any clues would be GREATLY appreciated!

{
    "code": "UnmarshalError",
    "message": "UnmarshalEntity encountered error: json: cannot unmarshal string into Go struct field AgentPoolProperties.properties.nodeTaints of type []string."
}

[felipecruz91]

@poochwashere the nodeTaints property is defined as type array. See below an example of how to use it in the ARM template:

"nodeTaints": [
    "CriticalAddonsOnly=true:NoSchedule"
]

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[palma21]

Moving to feature request to update taints.

[jeanfrancoislarente]

Any update on this @palma21?

We're trying to create a system-only node pool with taints at cluster creation but taints are not supported during cluster creation and not supported on nodepool update.

--node-taints CriticalAddonsOnly=true:NoSchedule

Thanks in advance!!

[palma21]

@jeanfrancoislarente I have a separate issue for that one which we are working on right now. #1833

Will provide updates there soon

[bplasmeijer]

@jeanfrancoislarente I have a separate issue for that one which we are working on right now. #1833

Will provide updates there soon

Thanks @palma21

[ramazankilimci]

Is there any update for this?

[dmeytin]

+1 on this feature

[guiliguili]

+1

[samuel-form3]

+1

[haitch]

We are working on a change allow --node-taints to be updated without triggering the VMSS re-image, expect to have it ready, CLI will be updated after that.

[michelefa1988]

@haitch when will this fix be out? It is quite critical that we can prevent pods from being scheduled on the system node, especially when this one node is getting full

[mezzofix]

any updates on this ?

[palma21]

All items in this should be concluded

[ghost]

Thank you for the feature request. I'm closing this issue as this feature has shipped and it hasn't had activity for 7 days.