Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AKS cluster-autoscaler stucks in "Initializing". #2236

Closed
monaka opened this issue Aug 4, 2019 · 1 comment · Fixed by #2237
Closed

AKS cluster-autoscaler stucks in "Initializing". #2236

monaka opened this issue Aug 4, 2019 · 1 comment · Fixed by #2237

Comments

@monaka
Copy link
Contributor

monaka commented Aug 4, 2019

On AKS (not with VMSS) 1.14.3, cluster-autoscaler stucks under "Initializing" state.

Nodes information on my node is:

NAME                       STATUS   ROLES   AGE     VERSION                             INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-39680657-1   Ready    agent   8d      v1.14.3                             10.240.0.35   <none>        Ubuntu 16.04.6 LTS   4.15.0-1052-azure   docker://3.0.6
aks-agentpool-39680657-2   Ready    agent   8d      v1.14.3                             10.240.0.4    <none>        Ubuntu 16.04.6 LTS   4.15.0-1052-azure   docker://3.0.6
virtual-node-aci-linux     Ready    agent   2d21h   v1.13.1-vk-v0.9.0-1-g7b92d1ee-dev   10.240.0.48   <none>        <unknown>            <unknown>           <unknown>

After inspecting code, I convinced it is caused only in case Virtual Kubelet was deployed.
I'll send the PR for fixing this soon.

@monaka
Copy link
Contributor Author

monaka commented Aug 4, 2019

Here is the root cause of this issue.

$ kubectl get nodes aks-agentpool-39680657-1 -o=jsonpath='{.spec.providerID}'
azure:///subscriptions/{{subscription-id}}/resourceGroups/mc_camino_camino-wus-2_westus2/providers/Microsoft.Compute/virtualMachines/aks-agentpool-39680657-1
$ kubectl get nodes virtual-node-aci-linux -o=jsonpath='{.spec.providerID}'
$  #no output

The controller doesn't set spec.providerID to Virtual Kubelet node.
At the same, there is no care whether spec.providerID is empty.

klog.V(6).Infof("Searching for node group for the node: %s\n", node.Spec.ProviderID)
ref := &azureRef{
Name: node.Spec.ProviderID,
}
return azure.azureManager.GetAsgForInstance(ref)

So every RunOnce() calls are exited with error.

k8s-ci-robot added a commit that referenced this issue Aug 22, 2019
[CA][Azure] Ignore nodes that have no ProviderID. (#2236)
feiskyer pushed a commit to feiskyer/autoscaler that referenced this issue Sep 12, 2019
feiskyer pushed a commit to feiskyer/autoscaler that referenced this issue Sep 12, 2019
feiskyer pushed a commit to feiskyer/autoscaler that referenced this issue Sep 12, 2019
feiskyer pushed a commit to feiskyer/autoscaler that referenced this issue Sep 12, 2019
mikkeloscar pushed a commit to zalando-incubator/autoscaler that referenced this issue Sep 4, 2020
mikkeloscar pushed a commit to zalando-incubator/autoscaler that referenced this issue Aug 24, 2022
tim-smart pushed a commit to arisechurch/autoscaler that referenced this issue Nov 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant