Winlogbeat: Experimental API wrong message extraction with invalid events

[theodiefenthal]

I have an event with holding two data attributes, one is a string, the other is binary. The binary type currently isn't supported ( See also: #19337 )

Having the default api enabled, the event I receive looks like the one referenced in Issue 19337:

{
  "@timestamp": "2020-06-07T23:12:13.205Z",
  "@metadata": {
    "beat": "winlogbeat",
    "type": "_doc",
    "version": "7.2.0",
    "topic": "winevents"
  },
  "event": {
    "original": "\u003cEvent xmlns\u003d\u0027http://schemas.microsoft.com/win/2004/08/events/event\u0027\u003e\u003cSystem\u003e\u003cProvider Name\u003d\u0027dokan1\u0027/\u003e\u003cEventID Qualifiers\u003d\u002716390\u0027\u003e1\u003c/EventID\u003e\u003cLevel\u003e4\u003c/Level\u003e\u003cTask\u003e0\u003c/Task\u003e\u003cKeywords\u003e0x80000000000000\u003c/Keywords\u003e\u003cTimeCreated SystemTime\u003d\u00272020-06-07T23:12:13.205194300Z\u0027/\u003e\u003cEventRecordID\u003e637256\u003c/EventRecordID\u003e\u003cChannel\u003eSystem\u003c/Channel\u003e\u003cComputer\u003eTheo\u003c/Computer\u003e\u003cSecurity/\u003e\u003c/System\u003e\u003cEventData\u003e\u003cData\u003e\u003c/Data\u003e\u003cData\u003eMounting disk device.\u003c/Data\u003e\u003cBinary\0D000030000028003000040001000140000000050000000700000000000000000003000000000000\u003c/Binary\u003e\u003c/EventData\u003e\u003cRenderingInfo Culture\u003d\u0027de-DE\u0027\u003e\u003cMessage\u003eMounting disk device.\u003c/Message\u003e\u003cLevel\u003eInformationen\u003c/Level\u003e\u003cTask\u003e\u003c/Task\u003e\u003cOpcode\u003e\u003c/Opcode\u003e\u003cChannel\u003e\u003c/Channel\u003e\u003cProvider\u003e\u003c/Provider\u003e\u003cKeywords\u003e\u003cKeyword\u003eKlassisch\u003c/Keyword\u003e\u003c/Keywords\u003e\u003c/RenderingInfo\u003e\u003c/Event\u003e",
    "kind": "event",
    "code": 1,
    "created": "2020-06-23T16:22:18.669Z"
  },
  "log": {
    "level": "informationen"
  },
  "message": "Mounting disk device.",
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "Theo",
    "hostname": "Theo",
    "architecture": "x86_64",
    "os": {
      "kernel": "10.0.18362.900 (WinBuild.160101.0800)",
      "build": "18363.900",
      "platform": "windows",
      "version": "10.0",
      "family": "windows",
      "name": "Windows 10 Pro"
    },
    "id": "doesItMatter"
  },
  "agent": {
    "version": "7.2.0",
    "type": "winlogbeat",
    "ephemeral_id": "doesItMatter",
    "hostname": "Theo",
    "id": "doesItMatter"
  },
  "winlog": {
    "keywords": [
      "Klassisch"
    ],
    "channel": "System",
    "task": "",
    "api": "wineventlog",
    "event_data": {
      "param2": "Mounting disk device.",
      "Binary": "0D000030000028003000040001000140000000050000000700000000000000000003000000000000"
    },
    "provider_name": "dokan1",
    "computer_name": "Theo",
    "record_id": 637256,
    "event_id": 1
  }
}

Using the experimental API, one can see that the message field is still wrong, even though that was part of a parseable data field from the event data (I have many events of the same type on my system, so I pick a possibly different one here compared to the above one with a different API, but makes the point still clear):

{
  "@timestamp": "2020-06-07T23:25:23.242Z",
  "@metadata": {
    "beat": "winlogbeat",
    "type": "_doc",
    "version": "7.8.0"
  },
  "host": {
    "name": "Theo",
    "hostname": "Theo",
    "architecture": "x86_64",
    "os": {
      "version": "10.0",
      "family": "windows",
      "name": "Windows 10 Pro",
      "kernel": "10.0.18362.900 (WinBuild.160101.0800)",
      "build": "18363.900",
      "platform": "windows"
    },
    "id": "doesItMatter",
    "ip": [
      "ip6",
      "ip4",
      "ip6",
      "192.168.221.177",
      "ip6",
      "ip6",
      "ip6",
      "ip4",
      "ip6",
      "ip4",
      "ip6",
      "ip4",
      "ip6",
      "ip4",
      "ip6",
      "ip4",
      "ip6",
      "ip4"
    ],
    "mac": [
      "mac1",
      "mac2",
      "mac3",
      "mac4",
      "mac5",
      "mac6",
      "mac7",
      "mac8"
    ]
  },
  "log": {
    "level": "information"
  },
  "message": "{{eventParam $ 2}}",
  "ecs": {
    "version": "1.5.0"
  },
  "agent": {
    "version": "7.8.0",
    "hostname": "Theo",
    "ephemeral_id": "doesItMatter",
    "id": "doesItMatter",
    "name": "Theo",
    "type": "winlogbeat"
  },
  "winlog": {
    "computer_name": "Theo",
    "channel": "System",
    "api": "wineventlog-experimental",
    "event_id": 1,
    "task": "None",
    "keywords": [
      "Classic"
    ],
    "event_data": {
      "param2": "Mounting disk device.",
      "Binary": "\u003cnil\u003e"
    },
    "provider_name": "dokan1",
    "record_id": 639574
  },
  "event": {
    "kind": "event",
    "code": 1,
    "provider": "dokan1",
    "action": "None",
    "created": "2020-06-23T16:43:58.775Z"
  }
}

Looks like, message lacks some kind of template extraction logic here...

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)