Crashing 5.0.0-alpha1

[LoungeFlyZ]

I am hitting a problem where filebeat is crashing 5.0.0-alpha1 like this:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0xa0 pc=0x4bb6a2]

goroutine 25 [running]:
panic(0x9b5320, 0xc820018080)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/elastic/beats/libbeat/common.ConvertToGenericEvent(0xc8204012c0, 0x7)
/go/src/github.com/elastic/beats/libbeat/common/event.go:47 +0x252
github.com/elastic/beats/libbeat/publisher.(*client).filterEvent(0xc820148bc0, 0xc8204012c0, 0xc820026e20)
/go/src/github.com/elastic/beats/libbeat/publisher/client.go:184 +0x30
github.com/elastic/beats/libbeat/publisher.(*client).PublishEvents(0xc820148bc0, 0xc8203c0c00, 0x7d, 0x7d, 0xc820419490, 0x2, 0x2, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/client.go:134 +0xc6
github.com/elastic/beats/filebeat/beater.(*syncLogPublisher).Start.func1(0xc820149140)
/go/src/github.com/elastic/beats/filebeat/beater/publish.go:103 +0x3bb
created by github.com/elastic/beats/filebeat/beater.(*syncLogPublisher).Start
/go/src/github.com/elastic/beats/filebeat/beater/publish.go:113 +0x58

I am playing with 5.0.0-alpha1 for the JSON support. I am using the same filebeat.yml file as with 1.2 except with the added JSON section:

filebeat:

  prospectors:
    -
      paths:
        - "logs/*.*.log.*"

      document_type: antrea

      json:
        message_key: message
        keys_under_root: true
        add_error_key: true

output:
  elasticsearch:

    hosts: ["http://REMOVED:9200"]

    username: "REMOVED"
    password: "REMOVED"

    index: "filebeat"

logging:
  level: debug

Repro'd on OSX and linux (debian:jessie)

Here are the log files i am testing this with that repro this issue:
https://dl.dropboxusercontent.com/u/27238389/logs.zip

Any ideas?

[ruflin]

For reference: https://discuss.elastic.co/t/crashing-5-0-0-alpha1/48212

[ruflin]

@LoungeFlyZ Thanks for the report. Could you post the full panic you get? Sometimes there is some more information below about the line that it exactly happens.

@monicasarbu Based on the panic I would think the error is filter related, but it seems like no filters are configured?

[monicasarbu]

The panic happens before the event is passed to the generic filtering when it's trying to convert the fields of the event to basic types. It might be due to a strange type that is found in the json object.

[LoungeFlyZ]

@ruflin that's the full panic. There isn't anything logged after this. Prior it logs out the log entries like this:

2016/04/22 23:28:26.584227 client.go:193: DBG Publish: { "@timestamp": "2016-04-22T23:28:21.575Z", "beat": { "hostname": "antrea-filebeat-1", "name": "antrea-filebeat-1" }, "input_type": "log", "level": "info", "message": "Got auth result", "offset": 22325314, "source": "/antrea/logs/api.antrea-api-1.log.2016-04-22", "timestamp": "2016-04-22T22:58:30.931Z", "type": "antrea" } 2016-04-22T23:28:26.597244800Z panic: runtime error: invalid memory address or nil pointer dereference

Then the rest of the panic is listed.

If i understood correctly, you are right ... i'm not using any filters.

[tsg]

@LoungeFlyZ thanks for the good bug report. This is an example line causing the panic:

{"method":"GET","path":"/auth/authorize","headers":{"content-type":"application/json","content-length":4,"location":"http://devhost:3000/auth/return?backend=http%3A%2F%2Fdevhost%3A4000?code=2f4818fd-a396-4150-a96b-3ee430ebef17","access-control-allow-origin":"*","access-control-allow-headers":"Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, Api-Version, Response-Time","access-control-allow-methods":"GET","access-control-expose-headers":"Api-Version, Request-Id, Response-Time","connection":"Keep-Alive","content-md5":"N6YlnMDB2uKZp4Zkid/wvQ==","date":"Fri, 04 Mar 2016 00:34:16 GMT","server":"hypeapi","api-version":"1.0.0","request-id":"0f4b264f-c259-4fc3-bd23-facf4021c0be","response-time":682},"res":null,"level":"info","message":"Sent response: ","timestamp":"2016-03-04T00:34:16.846Z"}

More precisely, it's the "res": null key that we need to treat better.

[ruflin]

@tsg I assume this can be closed as #1489 was merged?

[tsg]

That's right, thanks.

[LoungeFlyZ]

Thanks everyone for you help with this! I have the nightly running fine now. Much appreciated!