ES package log pipelines

[crespocarlos]

Closes #4046

Summary

Updates the log pipelines so we correctly ingest all elasticsearch log types.

Also adds a docker-compose with an elasticsearch service that can be started with elastic-package service up. The service is configured to generate logs out of the box, with an additional container continuously executing a script (generate-logs.sh) that sends requests to ES to generate log types not created ootb

Testing

• build the elasticsearch package: cd packages/elasticsearch && elastic-package build
• start a stack: elastic-package stack up -v -d --version 8.5.0-SNAPSHOT
• start the elasticsearch service: cd packages/elasticsearch && elastic-package service up -v
• install the elasticsearch package
  • a script is available to automate the installation with pre-configured values ./packages/elasticsearch/_dev/deploy/docker/scripts/install-package.sh,
  • otherwise install the package manually. The logs basepath is /tmp/service_logs/.
• connect to Kibana at https://localhost:5601 and verify all 5 log types are correctly ingested. One can lookup the data_stream.dataset for logs-* pattern in discover. Note that 4 log dataset should be elasticsearch.* while deprecation is the other way around deprecation.elasticsearch. Asked the ES team whether we should take the opportunity to reverse it in [elasticsearch] Verify logs mappings and pipelines #4046 (comment)

These logs are also surfaced in the Stack Monitoring UI. You can connect a local kibana (from main) to the elastic-package stack (howto)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-01T16:57:11.531+0000

• Duration: 13 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	31
Skipped	0
Total	31

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 2.809
Classes	100.0% (0/0)	💚 2.809
Methods	49.057% (26/53)	👎 -40.329
Lines	100.0% (0/0)	💚 9.072
Conditionals	100.0% (0/0)	💚

[klacabane]

@elastic/ecosystem I've added helper scripts to work with the package under the _dev/scripts directory but this is not supported as per the linter. Where would you recommend to put these scripts ?

[2022-08-24T16:20:47.598Z] Error: checking package failed: linting package failed: found 1 validation error:

[2022-08-24T16:20:47.598Z]    1. item [scripts] is not allowed in folder [/var/lib/jenkins/workspace/est-manager_integrations_PR-4033/src/github.com/elastic/integrations/packages/elasticsearch/_dev]

[matschaffer]

Some initial thoughts from visual inspection. Will give it a test run next.

[matschaffer]

Have we tried minimizing this file? Basically I like to keep this sort of thing as short as possible so we can pick up on new defaults as new versions of ES come out rather than having to figure out what the "latest" file might be.

[klacabane]

Not really, I'll look into it

[klacabane]

I've created a followup #4116

[matschaffer]

Test worked nicely!

I opened crespocarlos#1 with the script fixes

[matschaffer]

I think this is mergable. Definitely some follow ups I could see but what's here works and it's nice to have a rough cut of the automation.