cef module mapping to ecs mapper_parsing_exception

[undelete84]

When sending events from an ArcSight SmartConnector with cefsyslog to a Filebeat agent, via Logstash, Logstash gets mapper_parsing_exception when trying to index the events.
Index Template from Filebeats has been imported in to Elasticsearch successfully.

Feb 24 15:54:55 server.utv.m logstash[31272]: [2020-02-24T15:54:55,501][WARN ][logstash.outputs.elasticsearch][beats] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.6.0-cef-2020.02.24", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x667d2782], :response=>{"index"=>{"_index"=>"filebeat-7.6.0-cef-2020.02.24", "_type"=>"_doc", "_id"=>"BuCxd3ABlGu3p_9DogkU", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [source.user.group] tried to parse field [group] as object, but found a concrete value"}}}}

Mar 19 05:58:47 server.utv.m logstash[29409]: [2020-03-19T05:58:47,767][WARN ][logstash.outputs.elasticsearch][beats] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.6.0-cef-2020.03.17", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x167d0435], :response=>{"index"=>{"_index"=>"filebeat-7.6.0-cef-2020.03.17", "_type"=>"_doc", "_id"=>"6uAo8XABlGu3p_9DfYia", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [destination.user.group] tried to parse field [group] as object, but found a concrete value"}}}}

Index template for Filebeat 7.60 looks like this:

  "source": {
        "properties": {
 ---removed to save space---
          "user": {
            "properties": {
 ---removed to save space---
              "group": {
                "properties": {
                  "domain": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "name": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "id": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              }
            }
          }
        }
      },

"destination" : {
  "properties" : {
 ---removed to save space---
            "user" : {
			  "properties" : {
 ---removed to save space---
                "group" : {
                  "properties" : {
                    "domain" : {
                      "ignore_above" : 1024,
                      "type" : "keyword"
                    },
                    "name" : {
                      "ignore_above" : 1024,
                      "type" : "keyword"
                    },
                    "id" : {
                      "ignore_above" : 1024,
                      "type" : "keyword"
                    }
                  }
                }
              }
            }
          }
        },

By changing the default index template to not have [group] for these values the mapping_parsing_exceptions disappear.

        "source" : {
          "properties" : {
 ---removed to save space---
            "user" : {
              "properties" : {
 ---removed to save space---
                "group" : {
					"ignore_above" : 1024,
					"type" : "keyword"
				}
              }
            }
          }
        },

Same index mapping in ECS is made for:
host, server, client and user. But I have not encountered errors for these yet.

For confirmed bugs, please report:

• Version: 7.6.0 Filebeats, ArcSight SmartConnector 7.13.0.8194.1
• Operating System: RHEL
• Discuss Forum URL:
• Steps to Reproduce:

1. Install Beats 7.6.0
2. Activate CEF module
3. Change config file ./module/cef/log/config/input.yml to use following instead of type syslog
   type: tcp
   host: "{{.syslog_host}}:{{.syslog_port}}"
4. Send events from ArcSight SmartConnector to the configued port via Type: cefsyslog.

[kvch]

Github Issues are reserved for verified bugs. Please post your question on Discuss: https://discuss.elastic.co/c/beats

[undelete84]

Github Issues are reserved for verified bugs. Please post your question on Discuss: https://discuss.elastic.co/c/beats

Hi, is it step 3 in steps to reproduce that makes you count this as not a bug? That is the only setting that I made, otherwise it's default behaviour.

[undelete84]

Hi, this was fixed here [https://github.com//issues/17216]. Don't know why #17102 issue was not classified as a "verified bug".