Consolidate request handling in RecordRequestsMiddleware

[torredil]

Is this a bug fix or adding new feature?

Bug fix

What is this PR about? / Why do we need it?

This PR ensures that throttled requests are correctly captured in the cloudprovider_aws_api_throttled_requests_total metric instead of cloudprovider_aws_api_request_errors metric.

RecordThrottledRequestsMiddleware has been removed as it is no longer needed.

What testing is done?

// RecordRequestsHandler is added to the Complete chain; called after any request
func RecordRequestsMiddleware() func(*middleware.Stack) error {
	return func(stack *middleware.Stack) error {
		return stack.Finalize.Add(middleware.FinalizeMiddlewareFunc("RecordRequestsMiddleware", func(ctx context.Context, input middleware.FinalizeInput, next middleware.FinalizeHandler) (output middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
			start := time.Now()
			output, metadata, err = next.HandleFinalize(ctx, input)
			labels := createLabels(ctx)
			if err != nil {
	->			klog.InfoS("Request error", "request", labels["request"], "error", err)
				metrics.Recorder().IncreaseCount("cloudprovider_aws_api_request_errors", labels)
			} else {
				duration := time.Since(start).Seconds()
				metrics.Recorder().ObserveHistogram("cloudprovider_aws_api_request_duration_seconds", duration, labels, nil)
			}
			return output, metadata, err
		}), middleware.Before)
	}
}

I0418 18:55:46.707008       1 handlers.go:41] "Request error" request="AttachVolume" error="exceeded maximum number of attempts, 3, https response error StatusCode: 503, RequestID: 6b017ada-3ba9-473c-855d-cf37c4d413fd, api error RequestLimitExceeded: Request limit exceeded."

[github-actions]

Code Coverage Diff

This PR does not change the code coverage

[ConnorJC3]

/lgtm

[AndrewSirenko]

Good news, actual api request errors are no longer also including throttled request errors (nice!)

Bad news, this now significantly undercounts actual throttled requests at the sdk level. Off by factor of 100. (I believe it is only counting the FINAL RateLimitExceeded error for EC2 API Call.)

Scalability 1000 pod test results:

cloudprovider_aws_api_request_errors{request="DetachVolume"} 82
# HELP cloudprovider_aws_api_throttled_requests_total [ALPHA] ebs_csi_aws_com metric
# TYPE cloudprovider_aws_api_throttled_requests_total counter
cloudprovider_aws_api_throttled_requests_total{operation_name="AttachVolume"} 19
cloudprovider_aws_api_throttled_requests_total{operation_name="CreateVolume"} 44
cloudprovider_aws_api_throttled_requests_total{operation_name="DeleteVolume"} 25
cloudprovider_aws_api_throttled_requests_total{operation_name="DetachVolume"} 65

"CloudWatch" shows the following actual numbers

[AndrewSirenko]

Connor's Diff works correctly, well done team!

# HELP cloudprovider_aws_api_request_errors [ALPHA] ebs_csi_aws_com metric
# TYPE cloudprovider_aws_api_request_errors counter
cloudprovider_aws_api_request_errors{request="DetachVolume"} 111
# HELP cloudprovider_aws_api_throttled_requests_total [ALPHA] ebs_csi_aws_com metric
# TYPE cloudprovider_aws_api_throttled_requests_total counter
cloudprovider_aws_api_throttled_requests_total{operation_name="AttachVolume"} 281
cloudprovider_aws_api_throttled_requests_total{operation_name="CreateVolume"} 135
cloudprovider_aws_api_throttled_requests_total{operation_name="DeleteVolume"} 1347
cloudprovider_aws_api_throttled_requests_total{operation_name="DetachVolume"} 1764

---

Though we should convey to customers that cloudprovider_aws_api_request_errors no longer includes terminal throttled requests as part of that count as part of PR description?

[ConnorJC3]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ConnorJC3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ConnorJC3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment