support kv processor

[Wing924]

Describe the enhancement:

Both logstash and elasticsearch have kv filter while beats don't have.

example input

level=info ts=2019-10-23T03:00:07.774549823Z caller=shipper.go:349 msg=\"upload new block\" id=01DQV8X2Y6H7JGZEE8CDQXT3YS

example output

{
  "level": "info",
  "ts": "2019-10-23T03:00:07.774549823Z",
  "caller": "shipper.go:349",
  "msg": "upload new block",
  "id": "01DQV8X2Y6H7JGZEE8CDQXT3YS"
}

Describe a specific use case for the enhancement or feature:

many programs such as prometheus use key1=value1 key2=value2 as log format.

[vjsamuel]

this can be done using a dissect processor right?
https://www.elastic.co/guide/en/elasticsearch/reference/master/dissect-processor.html

[philippkahr]

Hi @vjsamuel

I am not quite sure, I run into the same problem with ingest pipelines in elasticsearch whilst writing a filebeat module for Fortinet. #13245 Basically, the Elasticsearch KV processor does not honor quotes the same as the logstash one. elastic/elasticsearch#31786 (comment) I think that the dissect processor is nice but it has one problem Dissect matches a single text field against a defined pattern. let's take something like firewall logs that can contain sometimes more and sometimes fewer fields. I would have to build a lot of patterns that might interfere. I guess @Wing924 has the same idea, that he wants to use the KV processor to automatically split the fields and do a rename on the fields afterward to make them ECS compliant.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.