Update sidecar timeout values

[torredil]

What is this PR about? / Why do we need it?

• This PR changes the default timeout value (15s) for the attacher sidecar, responsible for exercising ControllerPublishVolume / ControllerUnpublishVolume. The default value of 15s used today is not a sensible default, as a result the following error is observed frequently:

E1101 14:21:54.345145       1 driver.go:124] "GRPC error" err="rpc error: code = Internal desc = Could not detach volume \"vol-07ec886465a316b9b\" from node \"i-0e137107f5012fb1f\": context deadline exceeded"

In the vast majority of cases the volume is successfully detached just mere seconds after the attacher times out.

closes #1671

For context:

--timeout <duration>: Timeout of all calls to CSI driver. It should be set to value that accommodates majority of ControllerPublish and ControllerUnpublish calls. See [CSI error and timeout handling](https://github.com/kubernetes-csi/external-attacher#csi-error-and-timeout-handling) for details. 15 seconds is used by default.

What testing is done?

• Manual testing
• CI

[AndrewSirenko]

/lgtm
Left non-blocking comment.

[AndrewSirenko]

/retest
/lgtm

[wmesard]

When an operation does take a long time, how will the cx change as a result of this PR? Fewer annoying error messages in the log, sure. But what else? Will it reduce the number of AWS API calls? Increase it? Will it reduce the length of time that it takes for K8s to notice that the volume attachment state has changed? Increase it?

[ConnorJC3]

How will this behave if the user is already passing --timeout via additionalArgs? Do we need a guard against that?

[AndrewSirenko]

/lgtm
as two way door

[ConnorJC3]

Missing snapshotter?

[torredil]

Missing snapshotter?

The snapshotter sidecar already uses a timeout value of 60s by default.

[ConnorJC3]

/lgtm

[torredil]

The primary goal of adjusting the timeout values in this PR is to prevent premature context cancellations for CSI operations. With longer timeout values, Kubernetes (more specifically the sidecars) will wait a longer duration (60s) before retrying cancelled operations - this does not inherently change the speed at which (as an example) the volume attachment state changes - it simply provides the respective EC2 API call more time to complete before the sidecar retries.

In terms of operational performance, the increased timeout values would result in delays for operations that are genuinely stuck (which would likely be an indicator of a bug or inefficiency in the driver code. In this regard, increasing the timeout values improves the resiliency of our code because bugs can hide underneath the current set of values). However, in all other cases the operational performance is improved as the state is updated sooner because there is no need to wait for a retry.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment