logvm(s)

[marmarek]

Reported by joanna on 25 Apr 2014 16:41 UTC
A VM to collect logs from other VMs via qrexec service. Allow tools to easy copy logs to other VMs, as well as do log processing (logs are untrusted, but the logvm should be considered not-sensitive).

Potentially allow more than one logvm for paranoid configurations, cofigurable via qrexec policy.

Migrated-From: https://wiki.qubes-os.org/ticket/830

[Rudd-O]

This is EXCELLENT.

Plus, with journald log replication, it should be trivial to do.

Care must be taken that the files received by the logvm do not compromise the logvm upon using journalctl to read those logs.

[andrewdavidwong]

@Rudd-O: Would you be interested in taking this on?

[Rudd-O]

I wish I could, but I am extremely busy.

Is this intended to send/receive arbitrary logs, or specific structured logs like systemd ones? systemd has a good protocol for incremental send and receive of logs, and it is quite possible that it can be adapted to this use case with a minimal shim.

What's the idea w.r.t. who initiates the copy? Is it the logvm that extracts logs from other machines? Or is it the VM producing the log entries which sends it to the logvm?

[marmarek]

Is this intended to send/receive arbitrary logs, or specific structured logs like systemd ones? systemd has a good protocol for incremental send and receive of logs, and it is quite possible that it can be adapted to this use case with a minimal shim.

I think it's fair assumption to have systemd (or generally syslog-like) logs. But this shouldn't use any complex protocol, to not expose too much code for attacks. It looks like a simple "one log message per line" should be enough.

What's the idea w.r.t. who initiates the copy? Is it the logvm that extracts logs from other machines? Or is it the VM producing the log entries which sends it to the logvm?

Logically VM producing the logs should send them to logvm. After all it know when there is anything for sending, so no need for polling. Maybe even send logs to logvm instead of writing them to /var/log (or /var/log/journal).

[Rudd-O]

On 06/12/2016 06:09 PM, Marek Marczykowski-Górecki wrote:

Is this intended to send/receive arbitrary logs, or specific
structured logs like systemd ones? systemd has a good protocol for
incremental send and receive of logs, and it is quite possible
that it can be adapted to this use case with a minimal shim.

I think it's fair assumption to have systemd (or generally
syslog-like) logs. But this shouldn't use any complex protocol, to not
expose too much code for attacks. It looks like a simple "one log
message per line" should be enough.

You would lose a LOT of useful and certified metadata if you only
ingested "one message per line" syslog-style logs. Perhaps the format
used by systemd-journald log forwarding is not suitable for what you
want to do, or perhaps it is.

Look into how it works.

What's the idea w.r.t. who initiates the copy? Is it the logvm
that extracts logs from other machines? Or is it the VM producing
the log entries which sends it to the logvm?

Logically VM producing the logs should send them to logvm. After all
it know when there is anything for sending, so no need for polling.
Maybe even send logs to logvm /instead of/ writing them to |/var/log|
(or |/var/log/journal|).

Send instead of write sounds like a sensible thing.

It sounds like what you want is a systemd-journald log forwarder on each
client VM, continually sending log data over a vchan connection to the
log VM. That should not be very difficult to do.

Rudd-O
http://rudd-o.com/

[marmarek]

It sounds like what you want is a systemd-journald log forwarder on each client VM, continually sending log data over a vchan connection to the log VM.

Yes, something like this. But the priority is to not allow such logvm being compromised with some mis-formatted message. If that requirement means dropping some metadata, so be it.

[Rudd-O]

On 06/12/2016 06:31 PM, Marek Marczykowski-Górecki wrote:

It sounds like what you want is a systemd-journald log forwarder
on each client VM, continually sending log data over a vchan
connection to the log VM.

Yes, something like this. But the priority is to not allow such logvm
being compromised with some mis-formatted message. If that requirement
means dropping some metadata, so be it.

This is why the receiving code in the systemd log forwarder needs to be
reviewed. I expect it to be generally better than other stuff, if only
because it is meant to be exposed as a service on a network. If, for
some reason, the code does not meet your security standards, then the
two sides of the vchan connection can do serialization and deserialization.

If you reuse the log forwarder system, you also get forward secure
sealing of logs from VMs. That means a VM cannot tamper with old log
entries (without it becoming evident) no matter what it tries to do.

Rudd-O
http://rudd-o.com/

[donob4n]

So one option would be redirect journald using a version of systemd-journal-remote with vchan connections but it could be security problem due complex protocol and potentially a VM could compromise it.
As benefit logVM would have a powerful tool for monitoring, filter, search....

Another option I see, doing a very simple logVM wich just receives messages and save thems, is using an unikernel for it. It could dump the received lines in xen console so you can easily see all logs events in realtime and save them in files (separed by VM, dates, ...), if you need more detailed analysis of logs you could then bypass them (maybe with readonly access) to guiVM, dom0 or another trusted VM.

[Rudd-O]

I like the unikernel idea to save RAM in the VM that receives logs, but then querying the log would be tricky. I'd very much like to aggregate logs from several Qubes servers as well.

[donob4n]

Yes the idea is querying the log with another trusted VM which direct access to log files, it will not be as powerful as journald but the unikernel logic would be remain very minimal.

By Qubes servers do you mean like AppVM's?

[donob4n]

Another option could be using the unikernel for saving raw json output from journald without parsing it. Maybe with a very basic parsing logic so it could dump something readable in the console. Then, if you want a detailed log analysys, you create a offline disposableVM which loads the desired logs from the read-only data (or a copy of it) in logVM.

This way the only vulnerable part is the disposableVM, if later appears some bug in journald you could load the unmodified original logs with a patched version. If some VM tried to exploit this bug you could detect it later.

[marmarek]

See also related discussion from last year here

[DemiMarie]

After reading it, I assume that journald support should be optional.

For an initial proof of concept what do you think about using 'journalctl -f' output and the @HW42 'qubes.AppendLog' service?

Without the metadata, the logs would be FAR less useful to me. I use -u and --user a LOT.

[DemiMarie]

As far as security is concerned, would a sanitizer written in Rust be sufficient @marmarek @HW42?

[DemiMarie]

Update: The export format used is extremely simple and should be very easy to validate. There is a bunch of complicated indexing logic, but the indexes are not included in the exports, so we do not need to validate them. They will be regenerated by the receiving systemd-journald instance.

To compromise the LogVM, an attacker would need to either find a vulnerability in parsing the incoming log stream, find a vulnerability in the indexing process, or find a way to cause systemd-journald to emit index entries that compromise it when they are read back. The journal export format is extremely similar to the format used to submit entries to systemd-journald, and so it is already a security boundary. While I have not looked at the code myself, a quick look at the systemd GitHub indicates that it is continuously fuzzed under various sanitizers. I do not expect it to have any critical vulnerabilities, but would be willing to perform a security audit of at least the parsers.

We do need to prevent spoofing and injection attacks: one qube must not be able to inject logs that claim to be from another qube. That can be done by either overriding the _HOSTNAME field, or by logging to separate files.

[3hhh]

It is fairly simple to implement this via e.g. rsyslog omprog | qrexec-client-vm | logger. The rsyslog configuration in the target VM is then up to the user. A default could be one rotating file per VM.

I'd recommend using rsyslog over systemd as it's higher level and provides way more features (e.g. it would also work with custom log files from whatever user apps etc.).

[DemiMarie]

I'd recommend using rsyslog over systemd as it's higher level and provides way more features (e.g. it would also work with custom log files from whatever user apps etc.).

rsyslog has a history of vulnerabilities, so something simpler is to be preferred.

[3hhh]

rsyslog has a history of vulnerabilities, so something simpler is to be preferred.

Prove it please.

A random search shows me 19 CVEs since 2005 most of which would have been irrelevant for the setup I proposed.
Moreover paranoid users could simply not install a logvm (it's not needed by most users anyway).

[DemiMarie]

rsyslog has a history of vulnerabilities, so something simpler is to be preferred.

Prove it please.

A random search shows me 19 CVEs since 2005 most of which would have been irrelevant for the setup I proposed.
Moreover paranoid users could simply not install a logvm (it's not needed by most users anyway).

rsyslog is a large project with a lot of C code, and we would prefer to use something simpler by default. Users should be able to install rsyslog if they want more functionality.

[unman]

On Thu, Jul 08, 2021 at 12:44:57PM -0700, Demi Marie Obenour wrote: > > `rsyslog` has a history of vulnerabilities, so something simpler is to be preferred. > > Prove it please. > > A [random search](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=rsyslog) shows me 19 CVEs since 2005 most of which would have been irrelevant for the setup I proposed. > Moreover paranoid users could simply not install a logvm (it's not needed by most users anyway). rsyslog is a large project with a lot of C code, and we would prefer to use something simpler by default. Users should be able to install rsyslog if they want more functionality.

We've had this discussion at #5722 - rsyslog is installed by default in a standard Debian install. I doubt that systemd is substantially simpler than rsyslog.

[DemiMarie]

I doubt that systemd is substantially simpler than rsyslog.

systemd as a whole, no, but I suspect that systemd-journald is.

[3hhh]

I doubt that systemd is substantially simpler than rsyslog.

systemd as a whole, no, but I suspect that systemd-journald is.

Well, and I think the same about the relatively small part of rsyslog that would be used in the target logvm. Reading log data from stdin and writing to a Unix socket (logger) and reading from a Unix socket and writing to a file (rsyslog) should not be too complex. I don't care about the sending part as that would be the attacker VM anyway.

[DemiMarie]

I doubt that systemd is substantially simpler than rsyslog.

systemd as a whole, no, but I suspect that systemd-journald is.

Well, and I think the same about the relatively small part of rsyslog that would be used in the target logvm. Reading log data from stdin and writing to a Unix socket (logger) and reading from a Unix socket and writing to a file (rsyslog) should not be too complex. I don't care about the sending part as that would be the attacker VM anyway.

Is it possible to preserve structured logging metadata this way? systemd-journald can.

[unman]

On Fri, Jul 09, 2021 at 06:11:55AM -0700, Demi Marie Obenour wrote: > > > I doubt that systemd is substantially simpler than rsyslog. > > > > > > systemd as a whole, no, but I suspect that systemd-journald is. > > Well, and I think the same about the relatively small part of `rsyslog` that would be used in the target `logvm`. Reading log data from stdin and writing to a Unix socket (`logger`) and reading from a Unix socket and writing to a file (`rsyslog`) should not be too complex. I don't care about the sending part as that would be the attacker VM anyway. Is it possible to preserve structured logging metadata this way? systemd-journald can.

Of course. But seems there's a fair bit of mission creep going on here, blocking a 7 year old problem.

[marmarek]

Well, and I think the same about the relatively small part of rsyslog that would be used in the target logvm. Reading log data from stdin and writing to a Unix socket (logger) and reading from a Unix socket and writing to a file (rsyslog) should not be too complex. I don't care about the sending part as that would be the attacker VM anyway.

The important part while doing so, is to clearly mark which log line comes from where. VM should not be able to spoof the log origin, and probably timestamp too.

[DemiMarie]

Well, and I think the same about the relatively small part of rsyslog that would be used in the target logvm. Reading log data from stdin and writing to a Unix socket (logger) and reading from a Unix socket and writing to a file (rsyslog) should not be too complex. I don't care about the sending part as that would be the attacker VM anyway.

The important part while doing so, is to clearly mark which log line comes from where. VM should not be able to spoof the log origin, and probably timestamp too.

Should we try to get a patch to upstream systemd-journald, or should we implement a custom proxy in Rust? It is worth noting that events form systemd-journald already have a timestamp, which is the time that the log was generated in the VM. I don’t want to throw that information away.

[Rudd-O]

I would strongly recommend against using rsyslog for any of this. rsyslog logs are text, rather than structured. They are much lower fidelity -- we will lose important data when transferring data this way. This suggestion I oppose would also impose a requirement that we run yet another daemon which is neither default nor required to run Fedora or Debian these days.

Use the journal's export format, and import it, perhaps adding an attribute _SOURCE_VM or something if really necessary (I think it won't be). The journal has utilities to ingest said logs, index said logs, and query said logs, which vastly outstrip what rsyslog offers.

[3hhh]

On 10/26/21 1:24 PM, Rudd-O wrote: I would strongly recommend against using `rsyslog` for any of this. `rsyslog` logs are text, rather than structured. They are much lower fidelity -- we will lose important data when transferring data this way. This suggestion I oppose would also impose a requirement that we run yet another daemon which is neither default nor required to run Fedora or Debian these days.

The PR doesn't impose any restrictions on the data sent or with what software it is sent or received, but implements the method of transport. `rsyslog` is just an example for the sender side. If you find some journal config more suitable for you, use that. If you wish to write your own tool, do that. Also experience shows that the data you mention is not so important after all.

Use the journal's export format, and import it, perhaps adding an attribute `_SOURCE_VM` or something if really necessary (I think it won't be). The journal has utilities to ingest said logs, index said logs, and query said logs, which vastly outstrip what `rsyslog` offers.

The last statement is mostly wrong. The journal doesn't do log sending/receiving/parsing/modifying as its main job. `rsyslog` does and is the standard utility for that purpose for the last ~15 years. The journal job is mostly storing and indexing for local viewing. There are some tools like `systemd-journal-remote` for very special log forwarding (i.e. journal logs only), but nothing general-purpose (aka any kind of log).

[Rudd-O]

You yourself admit that there's systemd-journal-remote, therefore invalidating the claim that the journal doesn't do log sending -- it explicitly includes an utility for the purpose. This is what should be used. rsyslog is nice if your log server permissibly receives arbitrary unstructured junk on an UDP port. For structured logging, potentially even supporting forward secure sealing -- a desirable property of any reasonable secure system -- systemd-journal-remote is the right design choice.

[3hhh]

On 10/26/21 11:44 PM, Rudd-O wrote: You yourself admit that there's `systemd-journal-remote`. This is what should be used. `rsyslog` is nice if your log server permissibly receives arbitrary unstructured junk on an UDP port. For structured logging, `systemd-journal-remote` is the right design choice.

Then use it. It should already work via `journalctl -f -o export | qrexec-client-vm [logvm] qubes.ConnectTCP [port]` with `[port]` being a listener for a `systemd-journal-remote` instance inside `[logvm]` since ~2019 or so, not sure why people are still complaining it's not there. Probably they were too lazy to look or the log content was not so important after all? The `qubes.Syslog` PR supports _arbitrarily_ formatted single line logs (you called it "unstructured junk") even from sources that don't run systemd (such as Windows systems, Unixes, ...). Only the target system needs to be _some_ Linux (with or without systemd), Unix might work as well (assuming qrexec runs without systemd and on Unix). To stick with the systemd example, it would e.g. be `journalctl -f -o json | qrexec-client-vm [logvm] qubes.Syslog`. So in total all systemd fans should be happy with what already exists and the others might put the PR to use in the rare case when it's needed. Decentralized logs still have the best security features IMHO.