[filebeat] add 8.x kibana logs ingest pipeline #31286
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| description: Pipeline for parsing Kibana logs | ||
| processors: | ||
| - set: | ||
| field: event.ingested | ||
| value: '{{_ingest.timestamp}}' | ||
| - set: | ||
| copy_from: '@timestamp' | ||
| field: event.created | ||
| - rename: | ||
| field: json | ||
| target_field: kibana.log.meta | ||
| - date: | ||
| field: kibana.log.meta.@timestamp | ||
| formats: | ||
| - ISO8601 | ||
| target_field: '@timestamp' | ||
| - remove: | ||
| field: kibana.log.meta.@timestamp | ||
| - rename: | ||
| field: kibana.log.meta.message | ||
| target_field: message | ||
| - rename: | ||
| field: kibana.log.meta.state | ||
| target_field: kibana.log.state | ||
| ignore_missing: true | ||
| - rename: | ||
| field: kibana.log.meta.pid | ||
| target_field: process.pid | ||
| - rename: | ||
| field: kibana.log.meta.tags | ||
| target_field: kibana.log.tags | ||
| - rename: | ||
| field: kibana.log.meta.res.statusCode | ||
| target_field: http.response.status_code | ||
| ignore_missing: true | ||
| - script: | ||
| lang: painless | ||
| source: ctx.event.duration = Math.round(ctx.kibana.log.meta.res.responseTime * 1000000L) | ||
| if: ctx?.kibana?.log?.meta?.res?.responseTime != null | ||
| - remove: | ||
| field: kibana.log.meta.res.responseTime | ||
| ignore_missing: true | ||
| - rename: | ||
| field: kibana.log.meta.res.contentLength | ||
| target_field: http.response.body.bytes | ||
| ignore_missing: true | ||
| - rename: | ||
| field: kibana.log.meta.req.method | ||
| target_field: http.request.method | ||
| ignore_missing: true | ||
| - rename: | ||
| field: kibana.log.meta.req.headers.referer | ||
| target_field: http.request.referrer | ||
| ignore_missing: true | ||
| - rename: | ||
| field: kibana.log.meta.req.headers.user-agent | ||
| target_field: user_agent.original | ||
| ignore_missing: true | ||
| - rename: | ||
| field: kibana.log.meta.req.remoteAddress | ||
| target_field: source.address | ||
| ignore_missing: true | ||
| - set: | ||
| field: source.ip | ||
| value: '{{source.address}}' | ||
| ignore_empty_value: true | ||
| - rename: | ||
| field: kibana.log.meta.req.url | ||
| target_field: url.original | ||
| ignore_missing: true | ||
| - remove: | ||
| field: kibana.log.meta.req.referer | ||
| ignore_missing: true | ||
| - remove: | ||
| field: kibana.log.meta.statusCode | ||
| ignore_missing: true | ||
| - remove: | ||
| field: kibana.log.meta.method | ||
| ignore_missing: true | ||
| - append: | ||
| field: service.name | ||
| value: kibana | ||
| - set: | ||
| field: event.kind | ||
| value: event | ||
| - script: | ||
| lang: painless | ||
| source: >- | ||
| if (ctx?.kibana?.log?.state != null) { | ||
| if (ctx.kibana.log.state == "red") { | ||
| ctx.event.type = "error"; | ||
| } else { | ||
| ctx.event.type = "info"; | ||
| } | ||
| } | ||
|
|
||
| - set: | ||
| field: event.outcome | ||
| value: success | ||
| if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code < 400" | ||
| - set: | ||
| field: event.outcome | ||
| value: failure | ||
| if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code >= 400" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| description: Pipeline for parsing Kibana ecs logs | ||
| processors: | ||
| - set: | ||
| field: event.ingested | ||
| value: '{{_ingest.timestamp}}' | ||
| - set: | ||
| copy_from: '@timestamp' | ||
| field: event.created | ||
| - script: | ||
| lang: painless | ||
| inline: 'ctx.json.keySet().each (key -> ctx[key] = ctx.json.get(key))' | ||
| - remove: | ||
| field: json | ||
| - set: | ||
| field: event.outcome | ||
| value: success | ||
| if: 'ctx?.http?.response?.status_code != null && ctx.http.response.status_code < 400' | ||
| - set: | ||
| field: event.outcome | ||
| value: failure | ||
| if: 'ctx?.http?.response?.status_code != null && ctx.http.response.status_code >= 400' | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,105 +4,9 @@ on_failure: | |
| field: error.message | ||
| value: '{{ _ingest.on_failure_message }}' | ||
| processors: | ||
| - pipeline: | ||
| if: '!(ctx.containsKey("json") && ctx.json.containsKey("ecs") && ctx.json.ecs.containsKey("version"))' | ||
| name: '{< IngestPipeline "pipeline-7" >}' | ||
| - pipeline: | ||
| if: 'ctx.containsKey("json") && ctx.json.containsKey("ecs") && ctx.json.ecs.containsKey("version")' | ||
| name: '{< IngestPipeline "pipeline-ecs" >}' | ||
|
There was a problem hiding this comment. Is it possible for these 3 conditions to be in mixed state, causing these two if statements to both miss and we fall through to no pipeline? There was a problem hiding this comment. mixed state of these condition would mean an issue with the logs ingested and a fallthrough is likely to be the best option. Now I'm more surprised that we don't have a plaintext ingest pipeline considering kibana can be configured to log with There was a problem hiding this comment. This is basically There was a problem hiding this comment. So looks like we could use this here. That's probably more idiomatic anyway. The pattern @klacabane copied from me was probably not my greatest moment 😛 |
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was surprised to see this rather than https://www.elastic.co/guide/en/elasticsearch/reference/master/json-processor.html - wondering what's special about the kibana module that causes them to land like this rather than as a
messageJSON string like they land in the elasticsearch module.Guess I'll give it a go and see if I can work out what's happening.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see. It's because of
beats/filebeat/module/kibana/log/config/log.yml
Line 8 in cca59ba
I tried doing something like this but the fields end up as just strings rather than actual values.
Maybe painless is the only option to merge objects in an ingest pipeline. Would be nice to have an ingest pipeline SME weigh-in. Maybe @jbaiera can recommend one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for late response! Yeah, I don't think there is an easier way to merge two object fields together in ingest at the moment. You could use the append processor, but I think you'd need to know all the field names ahead of time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries @jbaiera ! thanks for the cross check.
Maybe interesting food for thought, but the lack of such a thing makes me wonder if we should stick to json parsing at the ES side (since the https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html can merge/expand objects).