Default mapping for "log" field used in examples

[zytek]

Hello again,

I've setup Filebeat 6.0.1 on Kubernetes, based on https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml
which seems to follow best practices for kubernetes deployments.

I do not use any ingest pipelines nor any other processing of log entries. I just wish to send pure log lines from docker logs to elasticsearch. Example config part:

    - type: log
      paths:
        - /var/lib/docker/containers/*/*.log
      json.message_key: log
      json.keys_under_root: true

This configuration parses docker logs and sends log message to ES index under "log" field. This field has type: keyword as seen in mapping

              "log" : {
                "properties" : {
                  "level" : {
                    "type" : "keyword",
                    "ignore_above" : 1024
                  },

I believe this should either:
a) be of type "text" for more meaningful user experience and easier "grepping" of log entries
b) or these log lines should be send under "message" key which has type: text already set

Could you advice how to handle that? Either default examples should make filebeat send log lines under "message" field or mapping of "log" field should be changed. As I am not experienced here I cannot tell which approach is better.

As a workaround, can you provide info how to achieve a) ? I can't tell from the docs. The json.message_key setting applies to source file (docker json logs from json-file log driver) and there doesn't seem to be any setting to tell it to send this under message field

Filebeat version 6.0.1

[zytek]

FYI, I've manually loaded template that I altered to mention "log" field and set it type to "text", the same as "message" field. Kibana now behaves properly when searching for log events. This should be the default configuration, I can assist in fixing this if you'll guide me a lil bit ;-)

[exekias]

Good catch! and thank you for reporting, we should actually switch defaults to use the new docker prospector, it will both decode JSON automatically, extract the timestamp, and put log lines under message key, which is mapped.

I have created #5934, you are more than welcome to contribute with this change, I'll take it at some point soon if you are not in the mood :)

[exekias]

closed by #5963