proxyvm with resolv.conf pointing at systemd-resolved doesn't forward DNS

[marmarek]

Observation

openQA test in scenario qubesos-4.1-update-x86_64-system_tests_network@64bit fails in
VmNetworking_fedora-35

# test_020_simple_proxyvm_nm
# failure: 

# timestamp 2022-04-26T10:07:36.749208
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/qubes/tests/__init__.py", line 220, in wrapper
    func(self, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/qubes/tests/integ/network.py", line 237, in test_020_simple_proxyvm_nm
    self.assertEqual(self.run_cmd(self.testvm1, self.ping_name), 0,
AssertionError: 2 != 0 : Ping by name failed (after NM reconnection)

Test suite description

netvm -> proxyvm (with NM) -> testvm1

test checks (among other things) name resolution in testvm1

Reproducible

Fails since (at least) Build 2022042606-4.1 (current job)

This affects Fedora 35 (#6969 ) template, and likely Arch too.

Expected result

Last good: 2022041906-4.1 (or more recent)

Further details

Always latest result in this scenario: latest

When NM is enabled, /etc/resolv.conf is not overriden by QubesOS/qubes-core-agent-linux#373 - it still symlinks to systemd-resolved. It includes nameserver 127.0.0.53 line. Qubes tries to redirect DNS traffic to anything found in /etc/resolv.conf, but redirecting (DNAT) to services listening on lo interface fails (because net.ipv4.conf.all.route_localnet = 0).

Possible solutions:

1. make /usr/lib/qubes/qubes-setup-dnat-to-ns script aware of resolved and get DNS addresses from it (it may not be obvious which DNS server to use - there can be both global and per-interface settings. especially if VPN is in use).
2. Make redirect to systemd-resolved work (likely by enabling DNSStubListenerExtra= option in its config).

I'm slightly in favor of the second option, because then systemd-resolved can route DNS queries to different servers based on the query itself (for example if VPN declares only some specific domain). But I'd like to hear some opinions from others.

[DemiMarie]

I'm slightly in favor of the second option, because then systemd-resolved can route DNS queries to different servers based on the query itself (for example if VPN declares only some specific domain). But I'd like to hear some opinions from others.

That’s my preferred option too, if systemd-resolved is enabled. The default should likely be disabled outside of sys-net and maybe VPN qubes.

[marmarek]

Yes, this all applies only to the case of /etc/resolv.conf pointing at systemd-resolved (so, not even just systemd-resolved running).

[3hhh]

I think the idea of systemd-resolved is to locally cache DNS requests in AppVMs. If you start serving DNS from it to upstream VMs, that may open up security holes.

I'd therefore suggest a two-fold approach:

1. Make sure systemd-resolved is disabled by default as it was in the past.
2. Let the user enable it manually, if s/he decides to. Once that happens, it starts serving DNS for upstream VMs as well (unless it's an AppVM).

I use 2. in some firewall VMs for a while to get DNS caching for upstream VMs and to implement #5225 as a side effect (didn't really need it, but thought it was cool). The code is here, but probably not really useful for you guys upstream apart from maybe the idea. The resolved.conf I use is there, too.

Essentially I just run the following in /rw/config/qubes-firewall.d/

#!/bin/bash
exec_first='
set -e -o pipefail
#allow local DNS server on firewall
#NOTE: vif+ should be the upstream interfaces
iptables -I INPUT -i vif+ -p udp --dport 53 -j ACCEPT
iptables -I INPUT -i vif+ -p tcp --dport 53 -j ACCEPT

#start the DNS server
#NOTE: systemd-resolved monitors /etc/hosts for changes
systemctl start systemd-resolved'

dns_pin2hosts "daemon" --exec-first "$exec_first" &
disown

The other scripts aren't involved for that setup.

EDIT: I'm on 4.1.

[marmarek]

1. Make sure systemd-resolved is disabled by default as it was in the past.

I'd rather not change this per https://www.qubes-os.org/faq/#what-is-qubes-attitude-toward-changing-guest-distros. If given distro decides to enable systemd-resolved by default, let it be this way. But note that using systemd-resolved for local names resolution is (mostly) independent of using it for forwarded traffic. The former is handled by /etc/nsswitch.conf - resolve is before dns, so most local cases don't even reach looking at /etc/resolv.conf.

QubesOS/qubes-core-agent-linux#373 overrides /etc/resolv.conf symlink only when configuring DNS servers directly (without network-manager involvement). Honestly, I'm not sure if that's the right thing to do if systemd-resolved is running, but according to the long discussion in systemd/systemd#21317, it is supported configuration.

If NM is enabled, it will set DNS addresses itself (likely only in systemd-resolved, leaving /etc/resolv.conf managed by systemd-resolved). This is exactly the case where things currently break.

I think the idea of systemd-resolved is to locally cache DNS requests in AppVMs. If you start serving DNS from it to upstream VMs, that may open up security holes.

This is the main reason why I'm not so sure about forwarding traffic to it from downstream AppVMs. But the existence of DNSStubListenerExtra= option, suggests that handling external requests is supported too.

[DemiMarie]

1. Make sure systemd-resolved is disabled by default as it was in the past.

I'd rather not change this per https://www.qubes-os.org/faq/#what-is-qubes-attitude-toward-changing-guest-distros. If given distro decides to enable systemd-resolved by default, let it be this way.

I actually disagree with this. The reason is that systemd-resolved is a caching recursive resolver, and it is usually best to have only one or two of those on a system. Any more wastes resources, which contradicts our goal of reducing qube resource requirements. Furthermore, the appropriate configuration for systemd-resolved is very often qube-specific.

On my system, for example, systemd-resolved is configured to use DNS over TLS. Needless to say, that will not work in an AppVM that is pointed at a stub listener! I think a much more reasonable approach is to run systemd-resolved if NetworkManager or systemd-networkd is enabled, or if the user explicitly requests it. Otherwise, it is better to disable systemd-resolved by default in qubes other than sys-net, just like we disable NetworkManager by default for such qubes.

But note that using systemd-resolved for local names resolution is (mostly) independent of using it for forwarded traffic. The former is handled by /etc/nsswitch.conf - resolve is before dns, so most local cases don't even reach looking at /etc/resolv.conf.

QubesOS/qubes-core-agent-linux#373 overrides /etc/resolv.conf symlink only when configuring DNS servers directly (without network-manager involvement). Honestly, I'm not sure if that's the right thing to do if systemd-resolved is running, but according to the long discussion in systemd/systemd#21317, it is supported configuration.

I suggest using resolvectl if systemd-resolved is running.

If NM is enabled, it will set DNS addresses itself (likely only in systemd-resolved, leaving /etc/resolv.conf managed by systemd-resolved). This is exactly the case where things currently break.

I think the idea of systemd-resolved is to locally cache DNS requests in AppVMs. If you start serving DNS from it to upstream VMs, that may open up security holes.

This is the main reason why I'm not so sure about forwarding traffic to it from downstream AppVMs. But the existence of DNSStubListenerExtra= option, suggests that handling external requests is supported too.

From a performance and resource-consumption perspective, I would prefer to have a single instance of systemd-resolved in sys-net, and have all of the other VMs use that instance by default.

Also, we need to make sure that mDNS and LLMNR are disabled by default; both have security problems.

[dylangerdaly]

How can I revert this? Currently I'm having to add systemctl stop systemd-resolved to rc.local because this change broke everytihng.

[marmarek]

How can I revert this? Currently I'm having to add systemctl stop systemd-resolved to rc.local because this change broke everytihng.

If you want to disable systemd-resolved - do just that: call systemctl disable systemd-resolved in the template.

[3hhh]

Make redirect to systemd-resolved work (likely by enabling DNSStubListenerExtra= option in its config).

I don't get that part then. Doesn't it mean that you are essentially considering to redirect DNS traffic from upstream VMs to systemd-resolved inside a netvm? If Qubes dnats DNS traffic to whatever is inside /etc/resolv.conf incl. DNS traffic from upstream VMs and resolv.conf contains systemd-resolved, that means it's going to systemd-resolved for me.

Since Qubes OS networking then starts to use DNS over systemd-resolved, I don't see why it should leave control over it to the installed template distro. In fact the latter should be even more of a maintenance hazard (distros setting & changing configs as they wish, i.e. Qubes having to work with a multitude of configs) than the former.

If you don't want to incorporate systemd-resolved into Qubes OS networking, I'd still suggest to make sure it stays out of the way of Qubes OS networking (disabled or configure NM to ignore it / grab resolv.conf) and leave it to the user to configure whatever DNS tricks s/he wants to have.

Anyway some attacks I came up with for systemd-resolved serving upstream VMs:

1. From a compromised VM attack a vulnerability inside systemd-resolved (special DNS request), compromise that downstream VM and then start resolving requests from upstream VMs to servers controlled by the attacker or just run other attacks on the traffic passed through. This way, attempt to compromise all VMs sharing the same systemd-resolved server.
2. Check from DNS response times inside a compromised VM whether another VM already requested a domain and use that info for de-anonymization (side channel attack).
3. Cache poisining and other DNS attacks more likely to affect > 1 VM.

(Btw downstream = closer to the Internet "sea" for me.)

Also, I belive that attacking systemd-resolved might not be too hard as the authors originally assumed it to be used locally only (= relatively trusted environment) and added the option to listen on non-local interfaces only later.
Those are the reasons why I think that with redirect support it would be better to have systemd-resolved disabled by default and leave further decisions to a hopefully well-informed user.

[marmarek]

Ok, makes sense. The side channel attack is probably the most realistic (doesn't rely on any other bug). And also, allows one VM to basically spy (just used domain names, but still) on user actions done in all the other VMs.

So, I'll go with the first option - forward DNS queries directly to upstream DNS, bypassing systemd-resolved at all (regardless of if it's running or not). I'll keep it enabled (in distributions doing so) for local queries, though.

[3hhh]

So, I'll go with the first option - forward DNS queries directly to upstream DNS, bypassing systemd-resolved at all (regardless of if it's running or not). I'll keep it enabled (in distributions doing so) for local queries, though.

Yes, that's probably the better choice for a short term update.
Incorporating systemd-resolved into the Qubes networking stack - even if optional - would be more suitable for a point release.

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.1 testing repository for the CentOS centos-stream8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.1-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.1.35-1 has been pushed to the r4.1 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.1.35-1+deb10u1 has been pushed to the r4.1 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.65-1.fc32) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.1 stable repository for the CentOS centos-stream8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update