Journalbeat

[kvch]

Journalbeat is going to be introduced, so Elastic stack users can forward journal entries to outputs.

Features

• Seeking to head, tail or to cursor
• Exact match for fields of journal entries using include_matches option
• ...

Configuration

journalbeat.inputs:
  # Paths that should be crawled and fetched.
  # When empty starts to read from local journal.
- paths: []

  # The number of seconds to wait before trying to read again from journals.
  #backoff: 1s
  # The maximum number of seconds to wait before attempting to read again from journals.
  #max_backoff: 20s

  # Position to start reading from journal. Possible values: head, tail, cursor
  seek: cursor

  # Exact matching for field values of events.
  # Matching for nginx entries: "systemd.unit=nginx"
  #include_matches: []

#========================= Journalbeat global options ============================
#journalbeat:
  # Name of the registry file. If a relative path is used, it is considered relative to the
  # data path.
  #registry_file: registry

TODO

6.5

• README
• add fields.yml && index template Initialize Journalbeat #8277
• ability to add filters to inputs Journalbeat matches support && minor additions #8324
• simple E2E tests Initialize Journalbeat #8277
• more comlex E2E tests with registry Initialize Journalbeat #8277
• investigate errors during reading (operation not permitted)
• adding journald specific metrics (e.g. journal size) Journalbeat matches support && minor additions #8324
• adding dashboard
• add Jenkins job
• add Dockerfile to build Journalbeat matches support && minor additions #8324
• support processors Journalbeat matches support && minor additions #8324
• add Fields support Journalbeat matches support && minor additions #8324
• convert numbers in events Minor improvements to Journalbeat #8618
• tie version to vendored lib Minor improvements to Journalbeat #8618

6.6 (needs backward-compatibility)

• prefix fields with journald.
• add Dockerimage Add Journalbeat docker image beats-docker#58
• add cursor_seek_fallback Add cursor_seek_fallback option #9234

later

• support filtering of entries like community Journalbeat: unit, kernel, identifier: Initialize filtering functionality #10985
• support for multiline and other readers of Filebeat
• support autodiscovery
• support input file globbing: Support for path globbing #21128
• support more boolean expressions in case of include_matches
• support wildcards in include_matches

[kvch]

Example event

{
  "@timestamp": "2018-09-05T16:51:46.798Z",
  "@metadata": {
    "beat": "journalbeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "read_timestamp": "2018-10-01T06:57:16.424Z",
  "beat": {
    "name": "sleipnir",
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1"
  },
  "process": {
    "uid": "0",
    "name": "dhclient",
    "executable": "/sbin/dhclient",
    "cmd": "dhclient -i wlp4s0",
    "capabilites": "3fffffffff",
    "audit": {
      "login_uid": "1000",
      "session": "1"
    },
    "pid": "10145"
  },
  "syslog": {
    "identifier": "dhclient",
    "pid": "10145",
    "priority": "6",
    "facility": "3"
  },
  "systemd": {
    "slice": "user-1000.slice",
    "owner_uid": "1000",
    "user_slice": "-.slice",
    "cgroup": "/user.slice/user-1000.slice/session-1.scope",
    "unit": "session-1.scope",
    "invocation_id": "95e83dc146c446c5880c56270101993c",
    "session": "1",
    "transport": "syslog"
  },
  "host": {
    "boot_id": "a5b9a1fe16874501b7552bbcc5efc911",
    "name": "sleipnir",
    "id": "f53e8c67094c443c88c2f5e1f130ceb6"
  },
  "message": "bound to 192.168.0.80 -- renewal in 1620 seconds.",
}

[kvch]

Example configurations

Monitoring multiple journals under the same directory

Journals under /path/to/journal/directory are merged into a single journal and read. By setting cursor to seek, Journalbeat reads from the beginning of the journal. Then after reload/restart it continues from where it left off.

journalbeat.inputs:
- paths: ["/path/to/journal/directory"]
  seek: cursor

Get redis events from a Docker container tagged redis

Using the translated fields by Journald.

journalbeat.inputs:
- paths: []
  include_matches:
    - "container.image.tag=redis"
    - "process.name=redis"

Using the field names of systemd journal.

journalbeat.inputs:
- paths: []
  include_matches:
    - "CONTAINER_TAG=redis"
    - "_COMM=redis"

[tardyp]

Hi, how is this going on? looks like the docker images are not available anymore in the docker repository.

[kvch]

There was a minor hiccup when publishing the Journalbeat image. It is resolved now.

[jalaziz]

Are there any plans to support complete unit matching as was done with the community beat or should I file a separate issue? (see mheese/journalbeat@2989e18)

It seems that matching on systemd.unit doesn't necessarily capture all the logs that journalctl -u does.

[kvch]

Yes, it is planned. But first, a minimal matching is coming then the support for the full filtering functionality of the community Beat.
There is one PR in progress with the next step towards the goal: #10985

[urso]

@kvch is this meta ticket up to date?

[kvch]

Now yes. I have added the last Journalbeat PR: #10985

[sepich]

Is there any date for multiline: support?
Right now (7.4.0) it just silently does not work, should there be at least some not implemented warning in the logs?
Thanks

[kvch]

@sepich What do you mean by silently not working? Is this option documented somewhere?

[jbguerraz]

@kvch we're ready to basically port (understand, mostly copy/paste :D) https://github.com/elastic/beats/pull/570/files to journalbeat
Is that the way you had (or, you would have) in mind ? if so, we can tackle it, otherwhise, could you provide some guidance ? :)

[kvch]

Our plan in the long term is to merge Journalbeat into Filebeat as an input. When merging we would like to provide the same options for the new input as log. This mostly depends on the PR #12908.
@jbguerraz I will get back to you in a few days about short term plans. Thank you for your patience.

[jbguerraz]

Hello @kvch :)
Is there a way to help you move ahead ?
Have a great day!

[kvch]

@jbguerraz I am working on a proof of concept for what I have in mind for addressing multiline. However, it involves a somewhat bigger refactoring, so it might take a few days. I will ping you in the PR.

[jbguerraz]

Hello @kvch :)
What's up ? any way to provide some help on the matter ?
Thank you!

[kvch]

I have started a POC but I ended up taking a step back because it Filebeat handles events differently than other Beats. I have opened a proposal for refactoring the pipelines: #16137

But there are a few open questions.

[q2dg]

Hello.
I see you keep "support more boolean expressions in case of include_matches" task as pending.
Currently, there's an implicit OR between several elements of include_matches array, right?
What do you mean is adding some sort of AND? Anything more specific?
Thanks!!

[kvch]

ATM you can only link conditional expressions with OR when reading from the journal. My plan is to add support for AND.

In the meantime, you can use the processors of Journalbeat to drop events if it does not fit your requirements just like in case of other Beats. For example, you can drop events from the dbus-daemon with the following processor configuration:

processors:
- drop_event:
    when:
       equals:
           journald.process.name: dbus-daemon

[yupm]

May I know if there is any progress on the support for multiline in Journalbeats?
Currently we are writing kubernetes/docker logs to journald. However, some of the stack traces are multi line and this gets split up.
It seems that Filebeat has the ability to manage multiline https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html. Might it be easy to port over this feature to Journalbeat?

I have looked into using Logstash to process file events, but the document states that:

If you are using a Logstash input plugin that supports multiple hosts, such as the beats input plugin, you should not use the multiline codec to handle multiline events. Doing so may result in the mixing of streams and corrupted event data. In this situation, you need to handle multiline events before sending the event data to Logstash.

This request was also raised here: #10114

[kvch]

Closing this issue because we already release journald input in Filebeat. The input comes with several improvements over the Beat, including better filtering and complete support for parsers (multiline, json, etc.). The outstanding issues are described in other tickets.