Modifying node pool for deprecated k8s version is not possible

[futureviperowner]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.26

• provider.azuread v0.11.0
• provider.azurerm v2.18.0
• provider.helm v1.2.4
• provider.kubernetes v1.12.0
• provider.random v2.3.0

Affected Resource(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

No special configuration is necessary but you must have an AKS using a version of k8s that is no longer supported.

Debug Output

I have debug output but haven't removed any sensitive data from it. If it's necessary to share, I will do so.

Expected Behavior

Terraform modification of a node pool using an unsupported k8s version should be successful.

Actual Behavior

An error is displayed:

The Kubernetes/Orchestrator Version "1.16.8" is not available for Node Pool "default".

Please confirm that this version is supported by the Kubernetes Cluster "aks-staging"
(Resource Group "rg-staging") - which may need to be upgraded first.

The Kubernetes Cluster is running version "1.16.8".

The supported Orchestrator Versions for this Node Pool/supported by this Kubernetes Cluster are:

• 1.15.12
• 1.15.11

Steps to Reproduce

1. Hopefully you already have an AKS deployed for an unsupported version. I was using 1.16.8.
2. Modify your terraform script to make a change to the node pool (change the node count).
3. Apply

Important Factoids

This error appears to have been introduced with the June AKS updates.

One thing in particular to call out is updating Node Pools - where a Kubernetes Cluster/Control Plane must be updated before the Node Pools can be updated

I hope this isn't meant to imply that you must be on a supported k8s version in order to make modifications through terraform. I can still make modifications to my unsupported cluster version through the Portal UI. I would think this is a pretty common scenario where a tweak may need to be done and upgrading isn't possible due to the impact a k8s upgrade has on the cluster and deployed services.

[fgarcia-cnb]

we are seeing the same issue with v1.18.4, which stopped being supported about a week ago. we had to completely destroy and rebuild the cluster (v1.18.6) which is not ideal

[ekhaydarov]

We hit this issue too and simply updating tags in the node pool forced a replacement of the whole node. The node pool was on an outdated version of k8s.

We "fixed" this by passing through the k8s version both to master and node pool, which meant we did not need to recreate the cluster but to update it in place. Patch versions are usually supported for a while and do not have any breaking changes in the api so it should be safe to upgrade k8s version

[Miouge1]

I ran into this when changing the node_count of a nodepool running an unsupported version. My workaround has been to apply the change manually in the portal and run terraform apply to update the state.

This is sort of painful since there is a number of things we can update on an existing nodepool without having to check the supported version (tags, node_count, etc...).

An example scenario is to scale a nodepool running an unsupported version to 0 nodes, verify everything is good, then delete the nodepool.

[switchboardOp]

Ran into this today enabling autoscaling for my node pool. Control plane and node pool on 1.20.2 (deprecated)
got a similar error when running terraform, but was able to enable autoscaling without upgrading through the portal without issue.

Using v2.64.0 of the azure provider

The Kubernetes/Orchestrator Version "1.20.2" is not available for Node Pool "***".
Please confirm that this version is supported by the Kubernetes Cluster "***"
(Resource Group "***") - which may need to be upgraded first.
The Kubernetes Cluster is running version "1.20.2".
The supported Orchestrator Versions for this Node Pool/supported by this Kubernetes Cluster are:
 * 1.18.19
 * 1.18.17
 * 1.19.11
 * 1.19.9
Node Pools cannot use a version of Kubernetes that is not supported on the Control Plane. More
details can be found at https://aka.ms/version-skew-policy.
  on ../../modules/azure/cluster/main.tf line 13, in resource "azurerm_kubernetes_cluster" "cluster":
  13: resource azurerm_kubernetes_cluster cluster {

[favoretti]

Thank you for opening this issue. Unfortunately there's little we can do from the provider side, since we're just talking to the APIs. In-place update of the clusters seems to be the only option here, so I'm going to close this issue.

[futureviperowner]

@favoretti - can you explain your reasoning a bit more? This issue did not exist before an update to the provider was done (as linked in my original report). So it seems strange that nothing can be done on the provider side to resolve the issue.

[favoretti]

Checked out for today, sorry, but I'll come back to it either Sunday or early next week. Thank you for your patience!

[favoretti]

From my perspective, if the APIs of Azure deprecate a version of k8s to the point that it doesn't exist - what's there we can do? Guess a version you'd need and start an upgrade? Sounds kinda dangerous?

[psibi]

Guess a version you'd need and start an upgrade?

Since that specific version was working previously, I don't think we are guessing a version. We know that the version works and already have at-least one node running on that version.

[futureviperowner]

Correct me if I'm wrong, but the error message that appears isn't saying that the k8s version is unsupported by Azure. It's saying it may not supported by the cluster, which isn't true at all.

This original report pertains to a valid change that can be done through the Azure portal, through ARM, or through Azure CLI. So why wouldn't I be able to do it through the Azure terraform provider?

[tombuildsstuff]

Reopening since this is a Terraform bug which needs to be fixed - we should look to add the current version of Kube used on the control plane/node pools into the validation list which I believe should fix this issue

[favoretti]

There's an API call for that, although it's region-specific, but we could populate it somewhere before initializing the client or so? That said, even if we do I still don't follow how this will help. Say, your TF config says you want a cluster version 1.20.1. That version disappears off the list and 1.20.3 comes instead. One still would need to update the config. What am I missing here?

[favoretti]

Coming to think of it... Would setting automatic_channel_upgrade not help in this case? Looking at the code it will auto-update the cluster to the next available version? (I haven't tried this myself yet, just brainstorming)

[github-actions]

This functionality has been released in v3.4.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.