Filebeat's Kibana module doesn't support new 8.0 logging format

[adriansr]

For confirmed bugs, please report:

• Version: Kibana 8.0+
• Operating System: n/a
• Discuss Forum URL: n/a
• Steps to Reproduce:

Starting with v8.0, Kibana introduces a new logging system that changes the JSON fields that are generated.

The kibana module doesn't support the new fields: expects a pid field, while the new format outputs ECS-compatible process.pid. Other fields need review.

This results in an ingestion:

"error.message": "field [kibana.log.meta.pid] doesn't exist"

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[insukcho]

Do we have any workarounds before fixing the bug?

[klacabane]

@insukcho Ignoring failures of the kibana.log.meta.pid processor would allow the documents to be processed further down the pipeline. I'm still ramping up on the kibana logs to filebeat ingestion so I can't tell yet if any other processor will fail with the new format

[klacabane]

Raised a potential bug in Kibana log entries #31576

[MSSP-BLKing]

After implementing the workaround we were presented with field [kibana.log.meta.tags] doesn't exist. The same workaround worked, but I wanted to make sure you knew this field was in error too. Cheers!

[klacabane]

Fix was merged in main and backported to 8.2

[MSSP-BLKing]

I'm still seeing this in 8.2, but this time it appears to be only occurring on Machine Learning entries.

@klacabane should this be a new issue or a recurrence of this same issue?

[muratkucuktepe]

I got also the same error. Kibana uses 8.4.0 ecs and filebeat 1.12.0
error.message field [kibana.log.meta.pid] doesn't exist

[klacabane]

Hi @muratkucuktepe, is this happening in cloud or onprem environment ? Which filebeat/kibana version are you using ? The kibana.log pipeline was reworked in 8.8 (#35268), are you able to test this version to see if it fixes the issue ?

[muratkucuktepe]

Hi @klacabane. We have production & monitoring cluster.
Production Cluster -> Filebeat 7.16.3, Kibana 8.5.3, ELS 8.5.3
Monitoring Cluster -> ELS 7.16.3
Filebeat uses ECS 1.12 Kibana has 8.4 (at least in logs I have seen).
Kibana has at the moment registry issue. As far as I understood, it works alone and not attached to cluster. ( I am currently on it) Filebeat reads the Kibana logs in json format.

kibana.yml

  root:
    level: warn
    appenders: [rolling-file]
    appenders: [ rolling-file-default,rolling-file-json ]
  appenders:
    rolling-file:
    rolling-file-default:
      type: rolling-file
      fileName: /../kibana/logs/kibana.log
      policy:
        type: size-limit
        size: 100mb
      strategy:
        type: numeric
        pattern: '-%i'
        max: 10
      layout:
        type: pattern
    rolling-file-json:
      type: rolling-file
      fileName: /../kibana/logs/kibana.json
      policy:
        type: size-limit
        size: 100mb
      strategy:
        type: numeric
        pattern: '-%i'
        max: 10
      layout:
        type: json```

filebeat.yml
```- module: kibana
    # Server logs
    log:
      enabled: true
      var.paths:
        - /../kibana/logs/kibana.json```
      
I do not know if this is relevant. Kibana error: 
`[2023-01-13T16:13:59.900+01:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Error connecting to package registry: request to https://epr.elastic.co/search?package=synthetics&experimental=true&kibana.version=8.5.3 failed, reason: connect ENETUNREACH XX.XXX.XXX.XXX:XXX - Local (0.0.0.0:0)`

[klacabane]

@muratkucuktepe The filebeat version running in production does not fully support 8.0+ log format and should be upgraded. I've reproduced the issue and using a filebeat > v8.2 fixes it. You can use 8.5.3 to align with the production stack