libbeat: add support for defining analyzers in-line in fields.yml files

[efd6]

What does this PR do?

This adds support for defining custom text analyzers in fields.yml files. For example:

- key: powershell
  title: PowerShell module
  description: >
    These are the event fields specific to the module for the Microsoft-Windows-PowerShell/Operational and Windows PowerShell logs.
  release: beta
  analyzer:
    powershell_script_analyzer:
      type: pattern
      pattern: "[\\W&&[^-]]+"
  fields:
  ...

Why is it important?

Not being able to define custom analyzers is a blocker for processing some documents containing syntactically meaningful non-standard token structures (for example captured script text as shown above).

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

No specific recommendations.

How to test this PR locally

Running go test in the relevant packages tests this change.

Related issues

• Closes [libbeat] Support custom analyzers in fields.yml #28540
• Relates x-pack/winlogbeat/module/powershell: don't split tokens on hyphen #28483
• Relates packages/windows/data_stream/powershell_operations: don't split tokens on hyphen integrations#1931

Use cases

Use case is shown above.

Screenshots

N/A

Logs

N/A

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-11-15T21:39:25.895+0000

• Duration: 103 min 35 sec

• Commit: 69a09cb

Test stats 🧪

Test	Results
Failed	0
Passed	54249
Skipped	5345
Total	59594

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[efd6]

It's not obvious to me where documentation for this should go. So suggestions welcomed.

[andrewkroh]

It's not obvious to me where documentation for this should go. So suggestions welcomed.

Personally if I were looking for info on how to use it I would be reading the Field struct or the Analyzer struct so I suggest some godocs in one of those places.

Now I also see we have docs in https://www.elastic.co/guide/en/beats/devguide/current/event-fields-yml.html. That's probably what most contributors read rather than reading Go source.

[andrewkroh]

LGTM

[efd6]

Now I also see we have docs in https://www.elastic.co/guide/en/beats/devguide/current/event-fields-yml.html.

That's perfect. Thanks.

That's probably what most contributors read rather than reading Go source.

:sadpanda:

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b libbeat/inlineanalyzer upstream/libbeat/inlineanalyzer
git merge upstream/master
git push upstream libbeat/inlineanalyzer

[efd6]

@andrewkroh PTAL for docs.

[andrewkroh]

LGTM