By default CLUSTER_HEALTH_RETRY=1 so it will fail soon (you can open README and see it). you need to increase this value by export CLUSTER_HEALTH_RETRY=10. That means it will try to check 10 times and it have enough time to verify cluster health check instead 1 by default.

It helps, but it doesn't necessarily solve the issue, as sometimes some other ASG in cluster is resized in the meantime and it will never reach the wanted value.

I've modified the code to it simply gives up after some tries and does the changes anyway. It fits our case and the whole process is monitored by a human, anyway.

@thorro could you please share with me which code you already modified, maybe I also applied for my case.

Just a simple hack, hardcoded to 5 tries.

diff --git a/eksrollup/lib/k8s.py b/eksrollup/lib/k8s.py
index ad286cf..9cf73f7 100644
--- a/eksrollup/lib/k8s.py
+++ b/eksrollup/lib/k8s.py
@@ -226,6 +226,9 @@ def k8s_nodes_count(desired_node_count, max_retry=app_config['GLOBAL_MAX_RETRY']
     """
     logger.info('Checking k8s expected nodes are online after asg scaled up...')
     retry_count = 1
+    retry_count2 = 0
+    retry_count2_max = 5
+
     nodes_online = False
     while retry_count < max_retry:
         nodes_online = True
@@ -233,10 +236,16 @@ def k8s_nodes_count(desired_node_count, max_retry=app_config['GLOBAL_MAX_RETRY']
         nodes = get_k8s_nodes()
         logger.info('Current k8s node count is {}'.format(len(nodes)))
         if len(nodes) != desired_node_count:
+            retry_count2 += 1
+            if retry_count2 >= retry_count2_max:
+                logger.info('Not waiting for k8s nodes to reach count {} anymore, continuing anyway '.format(desired_node_count))
+                break
+

@thorro - are you using any other flags or features? Currently, I'm using ASG_NAMES and seeing this issue. I'm also using: EXCLUDE_NODE_LABEL_KEYS and label couple of nodes before running eks-rolling-update. Would be interested to know if you use EXCLUDE_NODE_LABEL_KEYS too?

EDIT: Looks like there is an issue with EXCLUDE_NODE_LABEL_KEYS that is fixed in #117

Not using it anymore as we switched to EKS managed node groups.

Noticed the problem with this is caused by: if len(nodes) != desired_node_count: - it should probably be:if len(nodes) < desired_node_count: