issues with reading systemd v237 on Ubuntu Bionic

[arno01]

Trying this for the first time:

1

$ ~/go/bin/SystemdJournal2Gelf 127.0.0.1:12201 --merge
panic: could not parse journal output: json: cannot unmarshal array into Go struct field entryAlias.SYSLOG_FACILITY of type string

goroutine 1 [running]:
main.main()
	/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:207 +0x5b3

2

$ ~/go/bin/SystemdJournal2Gelf 127.0.0.1:12201 _TRANSPORT=kernel
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
main.(*SystemdJournalEntry).isJsonMessage(...)
	/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:128
main.(*SystemdJournalEntry).toGelf(0xc4200e86e0, 0x7f2110535d90)
	/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:57 +0x78c
main.(*SystemdJournalEntry).send(0xc4200e86e0)
	/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:115 +0x2f
main.(*pendingEntry).Push(0xc42009e160, 0xc4200a4000, 0x7b, 0x571c27d944a13, 0xc4200a60d0, 0xb, 0xc4200c2000, 0x20, 0xc4200a6008, 0x6, ...)
	/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:140 +0xc9
main.main()
	/home/arno/go/src/github.com/parse-nl/SystemdJournal2Gelf/SystemdJournal2Gelf.go:210 +0x396

It looks like that only the --follow works well:

$ ~/go/bin/SystemdJournal2Gelf 127.0.0.1:12201 --follow
$ echo 'test message to journald' | systemd-cat

Probably people are configuring journald write to files and feed them to filebeats, since filebeats isn't reading the systemd journald directly yet elastic/beats#7955

[SjonHortensius]

Thanks for your report. So what's happening is that there might be some unexpected input in your journal - which get skipped when you pass --follow as that only pushes any new entries (plus the last 30 or so).

Unfortunately I recently simplified processing of the journal making it harder to trace the source in the running application - but you can definitely help me fix this. For the first error, please run this and tell me what the output is:

journalctl --merge --output=json|grep -oP '"SYSLOG_FACILITY":.*?,'|sort|uniq

I'd expect only numeric values, but apparently there can be more. If you could find the complete line that would be great (after finding a non-numeric value, grep for that and past the json line here)

The second one I just fixed

[justin-gerhardt]

I'm running Systemd v241 (v241-8.git9ef65cb.fc30) on Fedora 30.
The SYSLOG_FACILITY value can also be an array.
I modified your command to

journalctl --merge --output=json | jq ".SYSLOG_FACILITY" -c | sort | uniq -c | sort -r -n

Running it yields

3213763 null
 572954 "3"
 231197 "4"
 187718 "0"
  28438 "DEVICE"
  15888 "9"
   7677 "10"
   7206 "1"
   6793 ["DEVICE","WIFI"]
   1617 "DHCP4"
   1461 "CORE"
   1227 "SETTINGS"
    603 "PLATFORM"
    485 "RFKILL"
    368 "BT"
    225 "SUPPLICANT"
    172 "MB"
    135 ["PLATFORM","WIFI"]
     98 "5"
     97 "DNS"
     97 ["DHCP4","DHCP6"]
     32 ["WIFI","PLATFORM"]
      1 "AGENTS"

In the format "<number of messages> <facility>"
Note: null indicates the key was not present on the message, not a string containing "null".

If you want a sample message with a non-numeric facility:

{
  "NM_DEVICE": "br-e4a1b7d8b8d2",
  "_MACHINE_ID": "1f50048cb385435998043f3a81395b91",
  "NM_LOG_DOMAINS": "DEVICE",
  "_SOURCE_REALTIME_TIMESTAMP": "1555796237704379",
  "CODE_LINE": "14055",
  "_HOSTNAME": "Justin-fedora",
  "CODE_FILE": "src/devices/nm-device.c",
  "_COMM": "NetworkManager",
  "_EXE": "/usr/sbin/NetworkManager",
  "SYSLOG_PID": "1478",
  "TIMESTAMP_BOOTTIME": "18.869759",
  "__MONOTONIC_TIMESTAMP": "18895214",
  "TIMESTAMP_MONOTONIC": "9.869759",
  "NM_LOG_LEVEL": "INFO",
  "_SELINUX_CONTEXT": "system_u:system_r:NetworkManager_t:s0",
  "_CMDLINE": "/usr/sbin/NetworkManager --no-daemon",
  "_GID": "0",
  "_BOOT_ID": "6be951d979fc42ccba47d98c8034b39a",
  "__CURSOR": "s=e4417a70c7e64ed4a52f47d87699e514;i=5b28;b=6be951d979fc42ccba47d98c8034b39a 20516e;t=586fd0a067045;x=2ebed3bf0301365d",
  "_SYSTEMD_CGROUP": "/system.slice/NetworkManager.service",
  "MESSAGE": "<info>  [1555796237.7043] device (br-e4a1b7d8b8d2): state change: ip-check -> ondaries (reason 'none', sys-iface-state: 'external')",
  "_SYSTEMD_UNIT": "NetworkManager.service",
  "_UID": "0",
  "_SYSTEMD_SLICE": "system.slice",
  "SYSLOG_IDENTIFIER": "NetworkManager",
  "_PID": "1478",
  "_CAP_EFFECTIVE": "200534e2",
  "__REALTIME_TIMESTAMP": "1555796236595269",
  "_SYSTEMD_INVOCATION_ID": "0f8be39f3b5742488320abb4152daa0e",
  "PRIORITY": "6",
  "_TRANSPORT": "journal",
  "SYSLOG_FACILITY": "DEVICE"
}

[SjonHortensius]

Thanks for the information @justin-gerhardt . Since I'm using SYSLOG_IDENTIFIER for the Facility field I've decided to drop other / unused fields for now. This should complete fix above errors