Nodes are Deleted after Unsuccessful Drain

[dlouks]

Environment:

• Cloud provider or hardware configuration: vsphere

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): CentoOS 7

• Version of Ansible (ansible --version): 2.9.16

• Version of Python (python --version): 2.7.5

Kubespray version (commit) (git rev-parse --short HEAD): 5517e62

Network plugin used: Calico

Command: ansible-playbook remove-node.yml -i hosts.ini -e node=NODE_NAME -e reset_nodes=false

Task with Bug

kubespray/roles/remove-node/pre-remove/tasks/main.yml

Line 13 in 5517e62

- name: remove-node | Drain node except daemonsets resource # noqa 301

When Draining nodes from a Kubespray cluster, it is possible to hit a scenario where the drain does not complete and the drain_grace_period and drain_timeout are hit and the task fails. Since ignore_errors is set to true, things move on as if no error occurred and ultimately the role remove-node/post-remove which runs deletes the node prior to being drained.

In my case the pod being drained was waiting for a another pod to come up to satisfy a disruption budget, and ideally the drain would have failed and the node would not have been deleted. To further complicate the issue, the pod that had the node deleted from under it had a persistent volume, so the pod fails to start up on another node because the volume can't be mounted twice.

I'd like to propose the following improvements to handle node draining with more care

• Add retries to the drain task to give more than one chance to connect and drain.
• Remove ignore_errors from drain tasks

[dlouks]

/assign

[dlouks]

I don't see the need for the extra cordon step since draining does that already. Removing it.

kubespray/roles/remove-node/pre-remove/tasks/main.yml

Line 2 in 5517e62

- name: cordon-node | Mark all nodes as unschedulable before drain # noqa 301