[Stack Monitoring] Align integration packages ingest pipelines with filebeat

[klacabane]

Summary

We should verify that the logs mappings and ingest pipelines defined in filebeat for the elasticsearch, kibana and logstash modules are aligned with the ones defined in the corresponding integration packages.

• [elasticsearch] Verify logs mappings and pipelines integrations#4046
• [kibana] Verify logs mappings and pipelines integrations#4047
• [logstash] Verify logs mappings and pipelines integrations#4048

AC

• filebeat and packages ingest pipelines are aligned
• filebeat and packages mappings are aligned

Related UI work #120416

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

Interesting! I was under the assumption that the current packages used the filebeat modules under the hood, but if this is the case I guess not. Looking forward to trying it out.

[miltonhultgren]

They do but not the mappings. The metricbeat mappings are joined and put into a huge mega mapping, while the integrations keep their own single index mappings to keep it slim (also in line with the index naming schema).
I think it's done this way also to allow future versions of these integrations to not use Beats under the hood.

So same document but different mappings, and maybe different ingest pipelines too?

[klacabane]

I'm not completely sure but I'm basing this ticket on the fact that the logs datastreams include a definition of pipelines and mappings. It also make sense because the logs are shipped to their individual datastreams at logs-elasticsearch.{logtype} just like the metrics do, and have pipelines defined similary under the logs-elasticsearch.{logtype}-{version} pattern

[crespocarlos]

I've been looking into this since yesterday, because it will make this one #120416 work.

Interesting! I was under the assumption that the current packages used the filebeat modules under the hood, but if this is the case I guess not. Looking forward to trying it out.

Assuming you are wondering why we need duplicate the ingestion pipelines on the integration packages, the best way I can answer this, also based on this doc, is that with integration packages we don't enable modules in filebeat, we use inputs which is a more manual configuration style, therefore, we don't use filebeat predefined pipelines.

An example of how the agent's filebeat config looks like, when elasticsearch package log ingestion is enabled:

>sudo elastic-agent inspect output --output default --program filebeat -v 

>filebeat:
  inputs:
  - exclude_files:
    - .gz$
    - _slowlog.log$
    - _access.log$
    - _deprecation.log$
    id: logfile-elasticsearch.server-66cf0158-333f-439c-a5fe-920a7e2c5d17
    index: logs-elasticsearch.server-default
    meta:
      package:
        name: elasticsearch
        version: 0.4.2
    multiline:
      match: after
      negate: true
      pattern: ^(\[[0-9]{4}-[0-9]{2}-[0-9]{2}|{)
    name: elasticsearch-1
    paths:
    - /tmp/es/8.5.0/logs/ elasticsearch*.log
    - /tmp/es/8.5.0/logs/*_server.json
    processors:
    - add_locale.when.not.regexp.message: ^{
    - add_fields:
        fields:
          service.type: elasticsearch
        target: message
    - add_fields:
        fields:
          dataset: elasticsearch.server
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: elasticsearch.server
        target: event
    - add_fields:
        fields:
          id: dfc83057-24a2-4b1c-b87d-ef1c79b1c25d
          snapshot: false
          version: 8.3.3
        target: elastic_agent
    - add_fields:
        fields:
          id: dfc83057-24a2-4b1c-b87d-ef1c79b1c25d
        target: agent
    revision: 20
    type: log
output:
  elasticsearch:
    api_key: dXVRq4IBb3H3XlXTClBB:xsSRhP6ISeikWs3ACEC6HA
    hosts:
    - http://localhost:9200

[crespocarlos]

There is a few more "catches" because agent's filebeat isn't using modules config:

service.type field

I've been mostly looking at the Cluster Overview UI, more specifically at the Logs tile. It filters logs by service.type. This field is not assigned when using agent, because filebeat relies on the module config too. Considering that the index pattern we will use is 'logs-*' and will continue to use the same filters, we need to manually assign a value to service.type with a module name on the packages pipelines

[klacabane]

So we should be able to copy the raw assets (mappings/pipelines) from metricbeat modules over to the package ? I suppose the pipelines should work right away but the mappings probably need more work because the log stream are specialized and separated per type of log

[crespocarlos]

Yes. We will need to adapt a few things for pipelines, because some modules support v7 and v8 and packages will only support v8. For Elasticsearch we can even remove the support for .log files for most filesets.

Mappings will need more work.

[crespocarlos]

Draft of ES logs pipeline elastic/integrations#4033

[crespocarlos]

We've checked all three tasks. Closing this ticket since there is nothing left to do.