Journald input in Filebeat

[kvch]

Add new input to Filebeat to collect entries from journald journals. The feature's already been under development. But now it's blocked.

Input files

If paths is empty, the default journal is opened. It's possible to provide directories and single journal files as inputs.

Filtering

It is possible to filter entries at journald level by providing key-value pairs. Thus, Filebeat does not need to filter at all or needs to filter less incoming events. Filter expressions needs to be match exactly the values of fields.

Example configuration

- type: journald
  paths:
    - /dev/log
    - /var/log/messages/my-journal-file
  filters:
    unit: nginx.service
    level: error

Why is it blocked?

The way journald tracks its offsets is not yet supported by Filebeat registry. Handling and saving positions if Filebeat needs a refactoring, so it becomes possible to save journald state info.

[kvch]

Journalbeat issue: #8323

[ksemaev]

@kvch the current situation is - we have both systemd and syslog type of logs at the same time on most of operating systems. That is not truely desirable to have two type of beats running on each instance just because of different log types. As I see - journalbeat is like fluentd - it just supports journald logs. Do we have any beat that can handle both systemd and syslog types of logs?

[kvch]

Unfortunately, right now there is no Beat which supports both inputs at the same time. However, we are still planning to add journald input to Filebeat. The necessary refactoring are in progress, but we don't know exactly when the new input going to be added.

For users who does not mind running a separate Beat to collect journald entries we would like to provide a new Journalbeat in a future release.

[jain108shubhamtbt]

hi,

when will we get journald input support in filebeat ? Please update.

[kvch]

Unfortunately, there hasn't been any notable updates since my last post. The registry refactoring is still in progress. In the meantime, Journalbeat is being developed, so when the time comes, you are getting a mature input.

[earlpotter0]

Any update?

[kvch]

Unfortunately, there is no update. For future reference, when there is an update with the Journald input, it will be added to this ticket. So if one subscribes, he/she can get notified.

[zgfh]

Any update?

[stevehorsfield]

This might be useful for people here: https://medium.com/@stevehorsfield/send-your-systemd-journal-logs-to-graylog-a2cbcd982cb4?source=friends_link&sk=e6801624a3fa2be715c31af98750cab4

[earlpotter0]

While the above is cool, it would be nice to have an Elastic supported tool.

1. Journalbeat works but is experimental and not supported by Elastic.
2. Beats and the ECS make a lot of sense, but having 3 or 4 (File, Metric, Audit, Journal) beats running on a machine starts to chew up a lot of resources. It does make a lot of sense to have these all as a module in one beat that are enabled as needed.

[M0rdecay]

Any update here?
The need for journald support in Fliebeat is really important right now...

[colttt]

I guess there is no update(3years later),but as @earlpotter0 said, its annoying to have a lot of beats running on one machine.

[odinho]

We migrated to Vector, which supports journald, instead.

[morganchristiansson]

After apt install rsyslogd the expected logfiles are created under /var/log and filebeat ingests them by default and it works with the filebeat system module...

I thought maybe the filebeat syslog input could also work but haven't tried. It wouldn't work with default modules which expect logfiles tho.

[nimarezainia]

Completed in 7.16

[earlpotter0]

Still experimental, but it's a start!! https://www.elastic.co/guide/en/beats/filebeat/7.16/filebeat-input-journald.html

[morganchristiansson]

Works for me. This is fantastic work. I can now remove journalbeat and use filebeat only.

Next can the filebeat system module and dashboards work with input type: journald? I've currently installed rsyslogd as workaround which logs to all expected files in /var/log that system module expects. It would be better if journald is used which is default in Ubuntu Server. (Tested on Ubuntu 21.04).

The current system module inputs are hardcoded to input: log:

• https://github.com/elastic/beats/blob/master/filebeat/module/system/auth/config/auth.yml
• https://github.com/elastic/beats/blob/master/filebeat/module/system/syslog/config/syslog.yml

(this should be it's own issue...)