[receiver/journald] Add more filtering options

[andrzej-stencel]

Component(s)

receiver/journald

Is your feature request related to a problem? Please describe.

The Journald receiver currently offers the following options to filter the journald logs: unit and priority. Sometimes these options are not enough. I have a scenario where a user is running iptables on a host with systemd, and the logs from iptables are written to journald as kernel logs. They do not have any unit assigned, so you cannot filter them by unit, and filtering by priority doesn't help either. It seems that in this case filtering by kernel messages (-k/--dmesg as described in the man page) would help.

Describe the solution you'd like

The filtering options in the Journald receiver are under the hood translated into the options of the journalctl binary. Looking at the command's man page, there are a couple other filtering options that sound like a good candidate to be exposed in the receiver:

• -k, --dmesg - Show only kernel messages.
• -g, --grep= - Filter output to entries where the MESSAGE field matches the specified regular expression.
• -t, --identifier= - Show messages for the specified SYSLOG_IDENTIFIER

Describe alternatives you've considered

An alternative is to read all logs without filtering and apply a filter operator, for example like this to get only kernel logs:

receivers:
  journald:
    operators:
    - type: filter
      expr: body._TRANSPORT != "kernel"

It's easy to trip on the filter operator as you need to remember to invert the condition 😅, also this is probably a lot less performant, especially if you have a lot of logs in systemd.

Additional context

No response

[andrzej-stencel]

I missed the "obvious" way to filter journald logs by any property with journalctl FIELD=value, as described in the man page's initial DESCRIPTION section and in the EXAMPLES section. This is apparently what Fluent Bit and Fluentd are exposing, see Fluent Bit systemd input plugin and Fluentd systemd input plugin.

[andrzej-stencel]

Apparently the Journald receiver doesn't have any specific code owners 🙀

opentelemetry-collector-contrib/.github/CODEOWNERS

Line 189 in ca50a98

receiver/journaldreceiver/ @open-telemetry/collector-contrib-approvers

@djaglowski do you have an opinion or do you know who might? 🙂

[djaglowski]

@astencel-sumo, I'd be willing to be a code owner. I'm more familiar with the stanza framework than with journald in particular. Any chance you'd want to co-own this with me?

[andrzej-stencel]

I'd be happy to co-own this component with you @djaglowski, but I'm thinking perhaps @sumo-drosiek would be an even better candidate for this component. What do you think @sumo-drosiek and @djaglowski?

[sumo-drosiek]

I would be very honored and happy to co-own the receiver

[djaglowski]

Great, thanks @sumo-drosiek and @astencel-sumo!

[djaglowski]

@sumo-drosiek, it appears you are not a member of the OpenTelemetry org. Typically we need code owners to be members so that we can assign issues, etc. If you want to apply I'm happy to be one of your sponsors.

[sumo-drosiek]

@sumo-drosiek, it appears you are not a member of the OpenTelemetry org. Typically we need code owners to be members so that we can assign issues, etc. If you want to apply I'm happy to be one of your sponsors.

Yes, it seems I need to take care of formal part. I appreciate it and going to start the process

[atoulme]

@sumo-drosiek if you are looking for a second sponsor I would be happy to be one of them.

[sumo-drosiek]

@djaglowski @atoulme @astencel-sumo here is my application then: open-telemetry/community#1399

[andrzej-stencel]

OK, back to the topic of this issue then 😅 I think perhaps it's best to start with adding the matches property as described in the man page in a manner like this?

receivers:
  journald:
    matches:
    - _SYSTEMD_UNIT: avahi-daemon.service
      _PID: 28097
    - _SYSTEMD_UNIT: dbus.service

The matches property is a list of maps.. The above example is equivalent to journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.service, as described in the EXAMPLES section of the man page.

This is similar to Fluentd's implementation, which I believe to be feature complete. Compared to that, the Fluent Bit's implementation seems to be not as feature complete, if I'm understanding correctly, as it only allows for a single list of conditions, which are all joined with either an OR or an AND.

[sumo-drosiek]

OK, back to the topic of this issue then sweat_smile I think perhaps it's best to start with adding the matches property as described in the man page in a manner like this?

receivers:
  journald:
    matches:
    - _SYSTEMD_UNIT: avahi-daemon.service
      _PID: 28097
    - _SYSTEMD_UNIT: dbus.service

The matches property is a list of maps.. The above example is equivalent to journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.service, as described in the EXAMPLES section of the man page.

This is similar to Fluentd's implementation, which I believe to be feature complete. Compared to that, the Fluent Bit's implementation seems to be not as feature complete, if I'm understanding correctly, as it only allows for a single list of conditions, which are all joined with either an OR or an AND.

It may work, as we are able to cover the following examples:

• journalctl _SYSTEMD_UNIT=avahi-daemon.service

  receivers:
    journald:
      matches:
      - _SYSTEMD_UNIT: avahi-daemon.service
  

  or

  receivers:
    journald:
      units:
      - avahi-daemon.service
  

• journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=28097

  receivers:
    journald:
      matches:
      - _SYSTEMD_UNIT: avahi-daemon.service
        _PID: 28097
  

• journalctl _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=dbus.service

    journald:
      matches:
      - _SYSTEMD_UNIT: avahi-daemon.service
      - _SYSTEMD_UNIT: dbus.service
  

  or

  receivers:
    journald:
      units:
      - avahi-daemon.service
      - dbus.service
  

• journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.service

  receivers:
    journald:
      matches:
      - _SYSTEMD_UNIT: avahi-daemon.service
        _PID: 28097
      - _SYSTEMD_UNIT: dbus.service

The only thing which I don't really like is duplication of units functionality,
but we can treat matches are extension of units (with OR approach)

For example for the following configuration:

receivers:
  journald:
    units:
    - avahi-daemon.service
    - dbus.service
    matches:
    - _PID: 28097

The behavior will be:

journalctl _SYSTEMD_UNIT=avahi-daemon.service + _SYSTEMD_UNIT=dbus.service + _PID=28097

and another example:

receivers:
  journald:
    units:
    - avahi-daemon.service
    - dbus.service
    matches:
    - _SYSTEMD_UNIT: dbus.service
      _PID: 28097

converts to

journalctl _SYSTEMD_UNIT=avahi-daemon.service + _SYSTEMD_UNIT=dbus.service _PID=28097

[sumo-drosiek]

Also until there is no decision if we are going to migrate to journald-gateway in the future I believe all features should be compatible for both approaches journalctl and journald-gateway

[djaglowski]

The matches proposal makes sense to me. If it makes units redundant, we can eventually deprecate that setting.

[andrzej-stencel]

Thanks a lot folks for your thoughts. 🙏

@sumo-drosiek about the units option, it is currently translated into journalctl's --unit option. This option, according to the journalctl man page, does a bit more than just adding _SYSTEMD_UNIT=value match:

journalctl -u name expands to a complex filter similar to

        _SYSTEMD_UNIT=name.service
             + UNIT=name.service _PID=1
             + OBJECT_SYSTEMD_UNIT=name.service _UID=0
             + COREDUMP_UNIT=name.service _UID=0 MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1

In light of this, it seems to me that the unit option has its reason to live beside the newly added matches option. I don't think it should be deprecated after adding the matches. What do you think?

[andrzej-stencel]

Also until there is no decision if we are going to migrate to journald-gateway in the future I believe all features should be compatible for both approaches journalctl and journald-gateway

@sumo-drosiek Am I looking at the right thing? https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html

This one does seem to support the matches in the form of the KEY=match URL parameter, but only as a single flat list of conditions that I'm not even sure if they're joined with an AND or an OR? 🤔

[sumo-drosiek]

Also until there is no decision if we are going to migrate to journald-gateway in the future I believe all features should be compatible for both approaches journalctl and journald-gateway

@sumo-drosiek Am I looking at the right thing? https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html

Yes, this is correct manual.

This one does seem to support the matches in the form of the KEY=match URL parameter, but only as a single flat list of conditions that I'm not even sure if they're joined with an AND or an OR? thinking

It is ANDed:

➜  ~ curl -H "Accept: application/json" '192.168.56.6:19531/[email protected]' -s | jq '.CODE_LINE' | sort | uniq
"5553"
"557"
"713"
➜  ~ curl -H "Accept: application/json" '192.168.56.6:19531/[email protected]&CODE_LINE=713' -s | jq '.CODE_LINE' | sort | uniq
"713"

In order to achieve OR seems that separate requests have to be send.

[sumo-drosiek]

@astencel-sumo, @djaglowski With matches configuration do we need to implement remaining options?

    -k, --dmesg - Show only kernel messages.
    -g, --grep= - Filter output to entries where the MESSAGE field matches the specified regular expression.
    -t, --identifier= - Show messages for the specified SYSLOG_IDENTIFIER

Looking at the behavior I would say that they can be worth to have it, especially grep

[djaglowski]

They all seem worthwhile to me.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/journald: @sumo-drosiek @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sumo-drosiek]

@astencel-sumo @djaglowski I think we can finally close this issue :)