Drain Timeout is not respected during client side throttling

[himanshu-kun]

How to categorize this issue?

/area robustness
/kind bug
/priority 2

What happened:
A case was seen where drain timeout was set to 2hrs but the drain ended up going till 11hrs.
If drainTimeout is 2hrs, then using podEvictionInterval(20sec) , we calculate maxEvictRetries as 360.
In case of client side thottling , the evict call takes huge time and thus the interval b/w 2 pod eviction requests becomes more than 100sec also.
Since currently we don't have context based cancellation for evictPodWithoutPVInternal , we rely just on maxEvictRetries to exhaust to come out of the loop. This can happen in the case where pod eviction runs into a timeout.
This leads to 11hrs or more of drain, and no force delete of the machine is done.

What you expected to happen:
Drain timeout to be respected in every situtation.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
Context based cancellation should be used effectively.

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

[gardener-robot]

@himanshu-kun Label area/todo does not exist.

[gardener-robot]

@elankath Label area/todo does not exist.