Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x-pack/filebeat/input/cel: suppress and log max http request retry errors #37160

Merged
merged 3 commits into from
Nov 22, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
x-pack/filebeat/input/cel: suppress and log max http request retry er…
…rors

The retryablehttp.Client will return an error based on the ErrorHandler
field when the number of requests exceeds the maximum configuration.
In the case that the field is nil, this is a non-nil error that causes
issues with some APIs and does not allow the CEL code to evaluate the
HTTP response status. So add a retryablehttp.ErrorHandler that will log
the failure, but return the response unaltered and a nil error.
  • Loading branch information
efd6 committed Nov 21, 2023
commit a43bb96070fc7df69905afc87d51b375e034ac96
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
@@ -213,6 +213,7 @@ Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will d
- Made Azure Blob Storage input GA and updated docs accordingly. {pull}37128[37128]
- Add request trace logging to http_endpoint input. {issue}36951[36951] {pull}36957[36957]
- Made GCS input GA and updated docs accordingly. {pull}37127[37127]
- Suppress and log max HTTP request retry errors in CEL input. {pull}37160[37160]

*Auditbeat*

14 changes: 13 additions & 1 deletion x-pack/filebeat/input/cel/input.go
Original file line number Diff line number Diff line change
@@ -730,14 +730,16 @@ func newClient(ctx context.Context, cfg config, log *logp.Logger) (*http.Client,
c.CheckRedirect = checkRedirect(cfg.Resource, log)

if cfg.Resource.Retry.getMaxAttempts() > 1 {
maxAttempts := cfg.Resource.Retry.getMaxAttempts()
c = (&retryablehttp.Client{
HTTPClient: c,
Logger: newRetryLog(log),
RetryWaitMin: cfg.Resource.Retry.getWaitMin(),
RetryWaitMax: cfg.Resource.Retry.getWaitMax(),
RetryMax: cfg.Resource.Retry.getMaxAttempts(),
RetryMax: maxAttempts,
CheckRetry: retryablehttp.DefaultRetryPolicy,
Backoff: retryablehttp.DefaultBackoff,
ErrorHandler: retryErrorHandler(maxAttempts, log),
bhapas marked this conversation as resolved.
Show resolved Hide resolved
}).StandardClient()
}

@@ -831,6 +833,16 @@ func checkRedirect(cfg *ResourceConfig, log *logp.Logger) func(*http.Request, []
}
}

// retryErrorHandler returns a retryablehttp.ErrorHandler that will log retry resignation
// but return the last retry attempt's response and a nil error so that the CEL code
bhapas marked this conversation as resolved.
Show resolved Hide resolved
// can evaluate the response status itself.
func retryErrorHandler(max int, log *logp.Logger) retryablehttp.ErrorHandler {
return func(resp *http.Response, err error, numTries int) (*http.Response, error) {
bhapas marked this conversation as resolved.
Show resolved Hide resolved
log.Warnw("giving up retries", "method", resp.Request.Method, "url", resp.Request.URL, "retries", max+1)
return resp, nil
}
}

func newRateLimiterFromConfig(cfg *ResourceConfig) *rate.Limiter {
r := rate.Inf
b := 1