[Filebeat] Journald event format changed

[andrewkroh]

While testing the journald input from master I found that the event format changed from what it produced in 7.x. I don't think this was intentional. It no longer translates the field names as described in the docs.

{
  "@timestamp": "2022-01-26T15:45:33.876Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.1.0"
  },
  "_HOSTNAME": "ubuntu",
  "agent": {
    "ephemeral_id": "487ca4cf-400e-4b10-93ac-dd181aa48a04",
    "id": "30326e16-e7ec-4db4-ace4-6afabefc84cb",
    "name": "ubuntu-impish",
    "type": "filebeat",
    "version": "8.1.0"
  },
  "host": {
    "name": "ubuntu-impish"
  },
  "PRIORITY": "6",
  "_MACHINE_ID": "d0bf0d000d034a4e93307255268a3a69",
  "_TRANSPORT": "kernel",
  "ecs": {
    "version": "8.0.0"
  },
  "_BOOT_ID": "b8d3fca6f9f44ad1acdefef51ce2b8b7",
  "_SOURCE_MONOTONIC_TIMESTAMP": "389271",
  "tags": [
    "NEW"
  ],
  "message": "Kprobes globally optimized",
  "SYSLOG_FACILITY": "0",
  "SYSLOG_IDENTIFIER": "kernel",
  "input": {
    "type": "journald"
  }
}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[andrewkroh]

@belimawr Did you test that your change fixes this issue?

[belimawr]

@belimawr Did you test that your change fixes this issue?

Yes I did ;)

[andrewkroh]

I tested from 522c862 and still see the original journald field names rather than the translated names.

{
  "@timestamp": "2022-01-13T00:17:01.964Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.1.0"
  },
  "_MACHINE_ID": "28c5bf41aa984b5889fbc9de96f504c0",
  "_SOURCE_MONOTONIC_TIMESTAMP": "1840802579931",
  "PRIORITY": "7",
  "SYSLOG_FACILITY": "0",
  "SYSLOG_IDENTIFIER": "kernel",
  "agent": {
    "version": "8.1.0",
    "ephemeral_id": "8884a04c-5406-45e5-b7e2-fb924f990910",
    "id": "095e9e1e-b7dc-4f69-b782-4fefa529b064",
    "name": "ubuntu-impish",
    "type": "filebeat"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "input": {
    "type": "journald"
  },
  "host": {
    "name": "ubuntu-impish"
  },
  "_HOSTNAME": "compute04.hc.va.local.example.com",
  "_BOOT_ID": "c2f79f985830406a9e08241d015eff05",
  "_TRANSPORT": "kernel",
  "message": "iptables DROP_INPUT: IN=eth1 OUT= MAC=33:33:00:00:00:01:b4:fb:e4:10:0b:51:86:dd SRC=fe80:0000:0000:0000:b6fb:e4ff:fe10:0b51 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=234 TC=0 HOPLIMIT=1 FLOWLBL=232529 PROTO=UDP SPT=35624 DPT=10001 LEN=194 ",
  "tags": [
    "OLD"
  ]
}

[andrewkroh]

The linter is giving hints.

$ pwd
/home/vagrant/go/src/github.com/elastic/beats/filebeat/input/journald

$ golangci-lint run -v --build-tags=withjournald ./...
conv.go:31:6: func `eventFromFields` is unused (unused)

[belimawr]

I see it now. I fixed the cursor issue you mentioned here, but missed the translation of the fields 🤦‍♂️ Sorry for the confusion!

I'll look at this translation issue now.

[belimawr]

I found the issue with the translation of the fields, when introducing the multiline parser I changed the way the entry is converted into a beat.Event, the new one does not do this name translation. I'll fix that tomorrow. It seem the current tests on beats focus on the message field, so they didn't break. I'll make sure to add some tests to cover this bug.

[belimawr]

@andrewkroh here is the fix: #30167