[elasticsearch] Verify logs mappings and pipelines

[klacabane]

Summary

Let's verify that every elasticsearch log types is properly ingested when running the elasticsearch package. The package should only support log formats from versions >= 8.0.

Note

• if a log type is failing to ingest, we should look at the corresponding filebeat implementation and verify both pipelines and mappings are aligned.
• if the logs are generated with an ecs format, we should look at simplifying the pipeline like the platform-observability package

[klacabane]

Questions for @elastic/es-core-infra

• the deprecation logs have a data_stream.dataset property set to deprecation.elasticsearch when one would expect the reverse like the other type of logs (eg elasticsearch.server). Is there any reason for this naming pattern and should we take the opportunity to fix it ?

• our documentation recommends to use json logs when running version >= 7.0, since we're releasing this package exclusively for versions >= 8.5, should we still support plain text logs ? If we only support json then we won't have to port over the plain text pipelines

  If you’re running against Elasticsearch >= 7.0.0, configure the var.paths setting to point to JSON logs. Otherwise, configure it to point to plain text logs.

[dakrone]

@klacabane using @elastic/elasticsearch-team pings 115 people, so it's probably not the best as there may be a lot of bystander effect for it.

I think these questions might make the most sense for the @elastic/es-core-infra team to answer as they own the logging infrastructure as well as the deprecation logger.

[pgomulka]

@klacabane we are looking into fixing this. I raised an issue a while back elastic/elasticsearch#83251
however this is a breaking change and is still being discussed

[klacabane]

Thanks @pgomulka - any thoughts on the second bullet point ? If we don't remove plaintext logs support we should at least remove the default configuration paths to only point to json

[pgomulka]

@klacabane by plaintext logs support you mean emitting them in ES or support to parse them?
in 8.x plaintext is only remaining for server logs and is intended for the "visual investigation" by admin/developer. It won't contain all the information (additional fields like x-opaque-id, traceparent etc)
If _server.json is already shipped I don't think we need to ship .log file. In fact it would be confusing to see both ingested later into logs cluster.

[klacabane]

I meant support to parse them

If _server.json is already shipped I don't think we need to ship .log file. In fact it would be confusing to see both ingested later into logs cluster.

Yes, we don't have control over that on the ingestion side however. If user configures both plaintext and json paths and the ingestion supports these two formats we'll end up with duplication. Now given we recommend using json format logs for versions >= 7.0 and that the plaintext logs are incomplete, I'm wondering if we should drop support for ingesting plaintext and remove the corresponding pipelines (only in the Integration, not filebeat). This will prevent duplication and clean the integration

[pgomulka]

if we should drop support for ingesting plaintext and remove the corresponding pipelines (only in the Integration, not filebeat).

sounds good to me