Fix taint removal retry for non-swallowed errors

[ConnorJC3]

Is this a bug fix or adding new feature?

Bug fix

What is this PR about? / Why do we need it?

https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait#ExponentialBackoff

When the condition func returns an error, it isn't retried:

Stops and returns as soon as: 1. the condition check returns true or an error

This change configures the conditional func used by removeTaintInBackground to always swallow the error

What testing is done?

Added basic unit test confirming the function retries

Manually tested, output from real world retry failures:

I0119 15:57:29.455540       1 metadata_ec2.go:25] "Retrieving EC2 instance identity metadata" regionFromSession=""
E0119 15:57:29.472054       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:57:29.975040       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:57:30.979037       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:57:32.980967       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:57:36.984689       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:57:44.987308       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:58:00.995860       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:58:33.014379       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 15:59:37.017078       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 16:01:45.026438       1 node.go:861] "Unexpected failure when attempting to remove node taint(s)" err="nodes \"i-0e0b9229e1b239ef0\" is forbidden: User \"system:serviceaccount:kube-system:ebs-csi-node-sa\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"
E0119 16:01:45.026490       1 node.go:868] "Retries exhausted, giving up attempting to remove node taint(s)" err="timed out waiting for the condition"

[github-actions]

Code Coverage Diff

File	Old Coverage	New Coverage	Delta
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/node.go	78.7%	80.3%	1.6

[ConnorJC3]

/retest

[torredil]

node.go lgtm

[torredil]

Thanks @ConnorJC3
/lgtm

[AndrewSirenko]

/retry

[AndrewSirenko]

/retest

[AndrewSirenko]

Thanks for this

[AndrewSirenko]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AndrewSirenko

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [AndrewSirenko]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment