Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

k8s cluster upgrade fails with template error #4078

Closed
chreichert opened this issue Oct 19, 2018 · 2 comments · Fixed by #4084
Closed

k8s cluster upgrade fails with template error #4078

chreichert opened this issue Oct 19, 2018 · 2 comments · Fixed by #4084
Assignees
@tariq1890

Comments

@chreichert
Copy link

chreichert commented Oct 19, 2018 •
edited
Loading

Is this a request for help?:
yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?:
0.24.0

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.10.7

What happened:
When trying to upgrade Cluster to 1.10.9 the following error is thrown:

INFO[0000] validating...
INFO[0004] Name suffix: 12658618
INFO[0004] Gathering agent pool names...
INFO[0005] VM k8s-dynamic-12658618-vmss_0 in VMSS k8s-dynamic-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-dynamic-12658618-vmss_1 in VMSS k8s-dynamic-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-dynamic-12658618-vmss_2 in VMSS k8s-dynamic-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-dynamic-12658618-vmss_3 in VMSS k8s-dynamic-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-graph-12658618-vmss_0 in VMSS k8s-graph-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_0 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_1 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_2 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_3 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_4 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_5 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_6 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_7 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_8 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_9 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_10 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_11 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_12 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_13 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_14 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_15 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_16 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_17 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_18 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_19 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] VM k8s-static-12658618-vmss_20 in VMSS k8s-static-12658618-vmss has a current tag of Kubernetes:1.10.7 and a desired tag of Kubernetes:1.10.9. Upgrading this node.

INFO[0005] Master VM name: k8s-master-12658618-0, orchestrator: Kubernetes:1.10.7 (MasterVMs)

INFO[0005] Upgrading to Kubernetes version 1.10.9

INFO[0005] Master nodes StorageProfile: ManagedDisks
INFO[0005] Prepping master nodes for upgrade...
INFO[0005] Resource count before running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0005] Evaluating if agent pool: master, resource: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')))] needs to be removed
INFO[0005] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')),'/cse', '-master-', copyIndex(variables('masterOffset')))] needs to be removed
INFO[0005] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')), '/computeAksLinuxBilling')] needs to be removed
INFO[0005] Resource count after running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0005] Total expected master count: 1
INFO[0005] Master nodes that need to be upgraded: 1
INFO[0005] Master nodes that have been upgraded: 0
INFO[0005] Starting upgrade of master nodes...
INFO[0005] masterNodesInCluster: 1
INFO[0005] Upgrading Master VM: k8s-master-12658618-0
INFO[0005] fetching VM: dev_004_QKNOWS_K8s/k8s-master-12658618-0
INFO[0006] found nic name for VM (dev_004_QKNOWS_K8s/k8s-master-12658618-0): k8s-master-12658618-nic-0
INFO[0006] deleting VM: dev_004_QKNOWS_K8s/k8s-master-12658618-0
INFO[0006] waiting for vm deletion: dev_004_QKNOWS_K8s/k8s-master-12658618-0
INFO[0102] deleting nic: dev_004_QKNOWS_K8s/k8s-master-12658618-nic-0
INFO[0102] waiting for nic deletion: dev_004_QKNOWS_K8s/k8s-master-12658618-nic-0
INFO[0102] deleting managed disk: dev_004_QKNOWS_K8s/k8s-master-12658618-0_OsDisk_1_08bcf62084bd4234858234979722d2ba
INFO[0163] Master offset: 0

INFO[0163] Master pool set count to: 1 temporarily during upgrade...

INFO[0163] Starting ARM Deployment (master-18-10-18T20.58.38-194173334). This will take some time...
INFO[0164] Error creating upgraded master VM: k8s-master-12658618-0
FATA[0164] Error upgrading cluster: resources.DeploymentsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidTemplate" Message="Deployment template validation failed: 'The value for the template parameter 'kubernetesDashboardCPULimit' at line '1' and column '16921' is not provided. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'."

This unfortunately happens after the Master node was already deleted.

What you expected to happen:
Cluster upgrade to complete successfully

How to reproduce it (as minimally and precisely as possible):
Initial setup of Cluster with acs-engine 0.21.2:

  • Kubernetes 1.10.7
  • Private cluster
  • RBAC enabled
  • Calico Networkpolicy
  • Disable addons:
    • blobfuse-flexvolume
    • smb-flexvolume
  • 1 Master
  • 3 Agentpools as vmss

Command used to upgrade:
acs-engine upgrade --subscription-id --resource-group --location northeurope --deployment-dir deployment-20181004_104850/arm-deploy --upgrade-version 1.10.9 --auth-method client_secret --client-id --client-secret

Anything else we need to know:
Cluster could be brought back up with using acs-engine 0.21.2 and upgrading to original k8s version 1.10.7.
@tariq1890
Copy link
Contributor

tariq1890 commented Oct 19, 2018 •
edited
Loading

@chreichert Thanks for reporting. This is indeed a regression that we need to fix. The fix for it will be available in the next release

@tariq1890 tariq1890 mentioned this issue Oct 19, 2018
Merged
@ghost ghost assigned tariq1890 Oct 19, 2018
@ghost ghost added the in progress label Oct 19, 2018
@ghost ghost removed the in progress label Oct 22, 2018
@tariq1890
Copy link
Contributor

@chreichert We have a patch release that fixes this issue, https://github.com/Azure/acs-engine/releases/tag/v0.24.1

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tariq1890 tariq1890

Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@tariq1890 @chreichert