Add log.flags and object metadata to aws-s3 input events

[andrewkroh]

What does this PR do?

This adds the log.flags field created by the line readers to aws-s3 events. log.flags contains metadata like multiline and truncated to indicate how the data was processed.

This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the Last-Modified timestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback.

Why is it important?

It makes the S3 log reader behave more similar to the log reader. And having access to the S3 headers gives users a bit more flexibility in their use-cases.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #26267 updated

• Start Time: 2021-06-14T15:19:01.307+0000

• Duration: 103 min 25 sec

• Commit: ae25a20

Test stats 🧪

Test	Results
Failed	0
Passed	14052
Skipped	2292
Total	16344

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14052
Skipped	2292
Total	16344

[leehinman]

What do you think about adding the meta data to the s3Info struct?

The meta data feels like a good fit with the other data in s3Info, and we already passing an s3Info into all the functions so it would shorten the argument list.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/fb/s3-log-flags-last-modified upstream/feature/fb/s3-log-flags-last-modified
git merge upstream/master
git push upstream feature/fb/s3-log-flags-last-modified

[andrewkroh]

What do you think about adding the meta data to the s3Info struct?

@leehinman Good idea. Looks better after that change.