x-pack/filebeat/input: Fix truncation of bodies in request tracing

[chrisberkhout]

Proposed commit message

x-pack/filebeat/input: Fix truncation of bodies in request tracing

When logging request traces, truncate the request/response body to 10%
of the maximum log file size.

Previously, bodies were truncated to the maximum file size, less 10kB.
10kB is a reasonable number for the other trace details, but space is
also required for encoding the body data as a JSON string value.

One example JSON body was 15% larger after encoding, but the 10kB
margin is 1% or less of the total limit. A body approaching the size
limit would typically generate a log entry that exceeded the limit.

Truncating large log entries to fit the file size limit means there may
only be one such entry per file. By truncating body data to 10% of the
file limit, we can expect to see entries for several request/response
pairs in each file.

The default maximum file size of 1MB gives a default maximum body size
of 100kB.

The behavior of request tracing for the HTTP Endpoint input is
unchanged: it always truncates request bodies to a size of 10kiB.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates [Filebeat] http trace logger - truncate response bodies larger than request.tracer.maxsize #37826

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @chrisberkhout? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[andrewkroh]

That's an excellent commit message. Thank you.

The default maximum file size of 1MB gives a default maximum body size
of 100kB.

I find this behavior a bit surprising. I would have expected that if I needed to capture a full 5 MiB response body that I would need to increase the max size to something a little larger than 5 MiB, but not 10x the size. What do others think? At a minimum, I think we need to mention this behavior in the documentation associated with the tracer settings.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-req-tracing-truncation upstream/fix-req-tracing-truncation
git merge upstream/main
git push upstream fix-req-tracing-truncation

[chrisberkhout]

That's an excellent commit message. Thank you.

The default maximum file size of 1MB gives a default maximum body size
of 100kB.

I find this behavior a bit surprising. I would have expected that if I needed to capture a full 5 MiB response body that I would need to increase the max size to something a little larger than 5 MiB, but not 10x the size. What do others think? At a minimum, I think we need to mention this behavior in the documentation associated with the tracer settings.

@andrewkroh I get that perspective. There are two things that make me lean towards the current version:

Often getting several responses is more important than getting full response bodies. One per file isn't a great experience. If you're getting truncated, you can double the limit (or better, shorten the page length) until you get what you need.

More importantly, since we're doing the truncation on the raw data, we need a fair bit of spare space to avoid problems. The real-world JSON I checked expanded by 15%, but it could need significantly more. A body of backslashes would double in size and bytes encoded as \u00XX grow by 6x.

It would be nice to truncate to a specific length as the data is written into the log but that's a much more complicated change.

I'm interested to know what others think.

If we go with this version I can improve the documentation.

[kcreddy]

The default maximum file size of 1MB gives a default maximum body size
of 100kB.

@chrisberkhout , here are my thoughts.

Couple of things that may impact us.

1. With new default max body size of 100kB, we will be getting more initial diagnostics where responses are truncated and may need to go-back and request customer to increase their resource.tracer.maxsize if we are in need of full response (some APIs have the pagination links/numbers at the end of the response which are going to fall in this category.). We may need a new resource.tracer.maxsize default, probably 5MB, to reduce this impact.
2. We don't have resource.tracer.maxsize exposed in most of our integrations. Until now, we have been selectively adding them as per the integration needs, which needs to be made available just like enable_request_tracer.

[chrisberkhout]

As discussed in the team meeting, we'll go with the current approach unless @efd6 has a better idea.
We can add the option and set higher defaults in integrations as necessary.

I've updated the documentation to be clear about the body limit being 10% of the file size limit. I added a tracer.maxsize entry in the Entity Analytics documentation.

The HTTP Endpoint truncation always limits request bodies to 10kiB. That behavior is unchanged by this PR but I noted it in the documentation.

[efd6]

Thanks