No DNS in inheriting VMs after enabling fedora-35 template's fedora-updates-testing repo and updating

[brendanhoar]

How to file a helpful issue

Qubes OS release

R4.0

Brief summary

If you change the flag from
0 to 1 and shut down the template, then update the template, any newly started VMs begin reporting "temporary" dns errors.

https://forum.qubes-os.org/t/r4-0-fedora-35-update-no-dns/10669/7

As per forum post, likely due to a systemd/resolved update.

Steps to reproduce

Expected behavior

Actual behavior

[crat0z]

4.1 too. Stopping/disabling systemd-resolved fixes this. Nevermind, it is not that trivial. Disabling systemd-resolved in template stops the creation of /etc/resolv.conf. However, after VM startup a temporary fix is to stop systemd-resolved, and it seems to work because qubes has overwritten /etc/resolv.conf at that point.

EDIT: After disabling systemd-resolved in template, and deleting /etc/resolv.conf, systemd still makes the symlink to /run/systemd/resolve/stub-resolv.conf... Not sure if that is how it was before but that is strange. Disabling systemd-resolved service means /run/systemd/resolve directory is never created, so when qubes writes to /etc/resolv.conf the file cannot be created.

[lubellier]

With 4.1, I also saw this problem with the today update of my fedora-35 template (qubes-os notification for update available, update via qubes update, only stable repositories). The today update includes systemd.

From my test, impact on fedora-35 based AppVM and DispVM.

Example for disp6558, first details about this DispVM:

user@dom0 ~]$ qvm-ls disp6558
NAME      STATE    CLASS   LABEL  TEMPLATE       NETVM
disp6558  Running  DispVM  red    fedora-35-dvm  sys-firewall
[user@dom0 ~]$ qvm-ls fedora-35-dvm
NAME           STATE   CLASS  LABEL  TEMPLATE  NETVM
fedora-35-dvm  Halted  AppVM  red    tpl-f35   sys-firewall
[user@dom0 ~]$ qvm-ls tpl-f35
NAME     STATE   CLASS       LABEL  TEMPLATE  NETVM
tpl-f35  Halted  TemplateVM  black  -         -

Tests on disp6558:

[user@disp6558 ~]$ cat /etc/resolv.conf 
nameserver 10.139.1.1
nameserver 10.139.1.2
[user@disp6558 ~]$ ls -al /etc/resolv.conf 
lrwxrwxrwx 1 root root 39 Apr 11 11:19 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
[user@disp6558 ~]$ rpm -qa systemd-resolved
systemd-resolved-249.11-1.fc35.x86_64
[user@disp6558 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:5e:6c:00 brd ff:ff:ff:ff:ff:ff
    inet 10.138.25.158/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe5e:6c00/64 scope link 
       valid_lft forever preferred_lft forever
[user@disp6558 ~]$ ping -c1 www.qubes-os.org
ping: www.qubes-os.org: Temporary failure in name resolution

Workaround from the forum DNS broken today after latest update topic:

[user@disp6558 ~]$ resolvectl dns eth0 10.139.1.1
[user@disp6558 ~]$ ping -c1 www.qubes-os.org
PING qubesos.github.io (185.199.109.153) 56(84) bytes of data.
64 bytes from cdn-185-199-109-153.github.com (185.199.109.153): icmp_seq=1 ttl=57 time=11.6 ms

--- qubesos.github.io ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.573/11.573/11.573/0.000 ms
[user@disp6558 ~]$ cat /etc/resolv.conf 
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

[crat0z]

OK, I've found a somewhat better short term solution. Perform the steps below inside of a template to fix this issue for now

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
sudo rm /etc/resolv.conf
sudo touch /etc/resolv.conf

Some links to look at:
https://src.fedoraproject.org/rpms/systemd/commits/f35
systemd/systemd#21317

As previously mentioned, with systemd-resolved disabled, for some reason systemd still creates the symlink to stub-resolv.conf, which is why the touch command is necessary in the above.

[Geblaat]

Note that enabling the updates-testing repo is no longer required to be impacted by this, I got the systemd update through stable repo already 2 days ago (Fedora 35). A swift response from Qubes Team would be great before the majority of users update their Fedora templates and manual fixing will be necessary. @marmarek

As previously mentioned, with systemd-resolved disabled, for some reason systemd still creates the symlink to stub-resolv.conf, which is why the touch command is necessary in the above.

Thank you for the workaround!

[lubellier]

@andrewdavidwong please, see the @Geblaat 's above comment, and add the relevant tags (4.1 stable and P:critical or P:major ?).
Thank you.

[andrewdavidwong]

@andrewdavidwong please, see the @Geblaat 's above comment, and add the relevant tags (4.1 stable and P:critical or P:major ?).

Thank you.

The milestone reflects the earliest affected supported release.

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.64-1+deb10u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.1 stable repository for the CentOS centos-stream8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.1.34-1+deb10u1 has been pushed to the r4.1 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update