Journald support for System module

[belimawr]

Proposed commit message

This commit adds journald support for the System module, both filesets now have a use_journald variable
that can be set to force using Journald to ingest syslog and auth logs.

The ingest pipelines are updated, now there is an entrypoint pipeline that selects the correct one according to the field
input.type.

System tests are also added.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact

Author's Checklist

• Ensure the journald input is ingesting data correctly

How to test this PR locally

1. Package Filebeat from this PR
2. Configure the ES output and Kibana credentials
3. Enable the system module (./filebeat modules enable system), set var.use_journald: true for both filesets (edit modules.d/system.yml)
4. Setup the assets: ./filebeat setup --modules system (this requires Kibana credentials correctly set)
5. Run Filebeat as root
6. Look at the logs in the filebeat-* data view, filter by event.dataset: system.syslog or event.dataset: system.auth, ensure the logs are correctly ingested
7. Look at the system module dashboards, ensure they're working/show data.

Related issues

• Closes Investigate the best way to decide when to read system logs from files or journald #40526

## Use cases

Screenshots

Dashboards, journald and logs side by side

Events, journald and logs side by side

~~## Logs~~

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[belimawr]

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):

root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 

The dashboards seem correctly populated (screenshots in the PR description).

I'm still investigating this difference.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[AndersonQ]

any reason for not placing it inside a testdata folder?

[belimawr]

Yes, I placed it where the other test files are. Those are used by the python system tests.

[AndersonQ]

any reason for not placing it inside a testdata folder?

[belimawr]

Yes, I placed it where the other test files are. Those are used by the python system tests.

[AndersonQ]

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):

root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 

The dashboards seem correctly populated (screenshots in the PR description).

I'm still investigating this difference.

so, that means the PR isn't yet ready to be merged, right? We don't wanna it to go in unless we're sure we're not loosing logs.

Perhaps it should go back to draft? What do you think?

[AndersonQ]

Anyway I'll test it

[AndersonQ]

Anyway I'll test it

it works. I don't see the map SSH failed login attempts source locations [Filebeat System] ECS populated. @belimawr do you know where the date to populate this map comes from?

[AndersonQ]

I'm not sure where it needs to be placed, but if there is a difference between he number of logs collected between journald and the log input it might be good to have it documented somewhere.

[belimawr]

We're already collecting all the journald logs, to me the difference in the number of logs from journald and the files is a OS detail, also some OSes only use jouranld, which makes comparing impossible unless something like rsyslog is installed and setup to generate the log files.

[AndersonQ]

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):

root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 

The dashboards seem correctly populated (screenshots in the PR description).
I'm still investigating this difference.

so, that means the PR isn't yet ready to be merged, right? We don't wanna it to go in unless we're sure we're not loosing logs.

Perhaps it should go back to draft? What do you think?

we had a call, the journald input consumes all the logs on journald, so I don't see it as a blocker here

[belimawr]

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):

root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 

The dashboards seem correctly populated (screenshots in the PR description).
I'm still investigating this difference.

so, that means the PR isn't yet ready to be merged, right? We don't wanna it to go in unless we're sure we're not loosing logs.

Perhaps it should go back to draft? What do you think?

It's ready for review. I've investigated more and the system module can ingest without errors all logs in the journal, so any difference in the amount of logs available is at the OS level, outside of the scope of this PR.

If needed, I could dig deeper to understand the difference. Let me know if you, or any other reviewer, believes it's necessary.

[AndersonQ]

I don't see https://github.com/elastic/beats/pull/41555/files#r1842419453 as a blocker. We just need to check if we need to add any doc explaining the difference between the log input and journald. Also the whole point is to be able to have the system module on systems where there is only journald, so, again not a blocker

[belimawr]

Anyway I'll test it

it works. I don't see the map SSH failed login attempts source locations [Filebeat System] ECS populated. @belimawr do you know where the date to populate this map comes from?

No, I don't.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 40526-system-module-debian-12-V2 upstream/40526-system-module-debian-12-V2
git merge upstream/main
git push upstream 40526-system-module-debian-12-V2