Updating Fedora templates results in loss of internet connection (only in the templates) #9025

Nurmagoz · 2024-03-08T12:41:57Z

Qubes OS release

4.2

Steps to reproduce

Upgraded Fedora templates (both full and minimal), then shut them down and restarted sys-net/firewall. Now, there is no possibility to upgrade any templates (neither Fedora nor Debian).

Expected behavior

Upgrades should proceed as usual.

Actual behavior

Fedora 39 template:

[user@fedora-39 ~]$ sudo dnf upgrade --refresh && sudo dnf clean all
Fedora 39 - x86_64                              0.0  B/s |   0  B     00:01    
Errors during downloading metadata for repository 'fedora':
  - Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-39&arch=x86_64 [CONNECT tunnel failed, response 403]
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-39&arch=x86_64 [CONNECT tunnel failed, response 403]
[user@fedora-39 ~]$

Debian 12 template:

user@debian-12:~$ sudo apt update && sudo apt full-upgrade && sudo apt autoremove --purge && sudo apt autoclean
Ign:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
Ign:2 https://deb.debian.org/debian bookworm InRelease
Ign:3 https://deb.debian.org/debian-security bookworm-security InRelease
Ign:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
Ign:2 https://deb.debian.org/debian bookworm InRelease
Ign:3 https://deb.debian.org/debian-security bookworm-security InRelease
Ign:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
Ign:2 https://deb.debian.org/debian bookworm InRelease
Ign:3 https://deb.debian.org/debian-security bookworm-security InRelease
Err:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
  Invalid response from proxy: HTTP/1.0 403 Access denied  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
Err:2 https://deb.debian.org/debian bookworm InRelease
  Invalid response from proxy: HTTP/1.0 403 Access denied  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
Err:3 https://deb.debian.org/debian-security bookworm-security InRelease
  Invalid response from proxy: HTTP/1.0 403 Access denied  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
Reading package lists... Done
E: Failed to fetch https://deb.debian.org/debian/dists/bookworm/InRelease  Invalid response from proxy: HTTP/1.0 403 Access denied  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
E: Failed to fetch https://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Invalid response from proxy: HTTP/1.0 403 Access denied  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
E: Failed to fetch https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease  Invalid response from proxy: HTTP/1.0 403 Access denied  Server: tinyproxy/1.10.0  Content-Type: text/html  Connection: close     [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
user@debian-12:~$

Fedora curl 127.0.0.1:8082

[user@fedora-39 ~]$ curl 127.0.0.1:8082
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
<title>403 Access denied</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>

<body>

<h1>Access denied</h1>

<p>The administrator of this proxy has not configured it to service requests from your host.</p>

<hr />

<p><em>Generated by <a href="https://tinyproxy.github.io/">tinyproxy</a> version 1.10.0.</em></p>

</body>

</html>
[user@fedora-39 ~]$

Debian curl 127.0.0.1:8082

user@debian-12-minimal:~$ curl 127.0.0.1:8082
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">

<head>
<title>403 Access denied</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>

<body>

<h1>Access denied</h1>

<p>The administrator of this proxy has not configured it to service requests from your host.</p>

<hr />

<p><em>Generated by <a href="https://tinyproxy.github.io/">tinyproxy</a> version 1.10.0.</em></p>

</body>

</html>
user@debian-12-minimal:~$

Workaround

Shut down sys-net/firewall/usb.. and switch the DVM template on which these DispVMs are based to Debian-12. The internet connection will return to the templates.

Note

The internet connection is lost only in the templates, not in DispVMs or AppVMs.

The text was updated successfully, but these errors were encountered:

adrelanos · 2024-03-08T15:12:40Z

Confirmed. Happens with fedora-39 Template after it was been upgraded. Does not happen before the Template has been upgraded. Therefore some upgrade must have broken this.

I would call this a critical issue because it will prevent users from upgrading again. Should be fixed in the stable repository as quickly as possible so as few users as possible will receive this faulty upgrade.

Attempting to provide further hopefully useful debug information...

In sys-net:

sudo systemctl status qubes-updates-proxy | cat

● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
     Loaded: loaded (/usr/lib/systemd/system/qubes-updates-proxy.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Fri 2024-03-08 10:07:56 EST; 1min 40s ago
   Main PID: 1704 (tinyproxy)
      Tasks: 3 (limit: 370)
     Memory: 1.0M
        CPU: 12ms
     CGroup: /system.slice/qubes-updates-proxy.service
             ├─1704 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
             ├─1705 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
             └─1706 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

Mar 08 10:08:02 sys-net tinyproxy[1705]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:03 sys-net tinyproxy[1706]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:03 sys-net tinyproxy[1705]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:03 sys-net tinyproxy[1706]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:05 sys-net tinyproxy[1705]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:05 sys-net tinyproxy[1705]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:05 sys-net tinyproxy[1706]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:09 sys-net tinyproxy[1706]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:09 sys-net tinyproxy[1706]: Unauthorized connection from "localhost" [::1].
Mar 08 10:08:09 sys-net tinyproxy[1705]: Unauthorized connection from "localhost" [::1].

adrelanos · 2024-03-08T15:19:04Z

I found a workaround for this issue...

In sys-net,

Open file /etc/tinyproxy/tinyproxy-updates.conf with root rights.

sudoedit /etc/tinyproxy/tinyproxy-updates.conf

Add.

Allow ::1

Save.

sudo systemctl restart qubes-updates-proxy

I am pretty sure the same file modification could be applied in the fedora-39 Template.

Minimalist73 · 2024-03-08T16:43:48Z

Can also confirm this.
Here are the package upgraded before it happened:

flexiblas-netlib 3.4.11.fc39 -> 3.4.21.fc39
flexiblas-openblas-openmp 3.4.11.fc39 -> 3.4.21.fc39
flexiblas 3.4.11.fc39 -> 3.4.21.fc39
systemd-libs 254.91.fc39 -> 254.101.fc39
systemd-networkd 254.91.fc39 -> 254.101.fc39
systemd-pam 254.91.fc39 -> 254.101.fc39
systemd-resolved 254.91.fc39 -> 254.101.fc39
systemd 254.91.fc39 -> 254.101.fc39
systemd-udev 254.91.fc39 -> 254.101.fc39
thunderbird-librnp-rnp 115.8.01.fc39 -> 115.8.11.fc39
thunderbird 115.8.01.fc39 -> 115.8.11.fc39
socat 1.7.4.43.fc39 -> 1.8.0.02.fc39

Seem to be related to socat and its upgrade to 1.8.0.
From the changelog:

TCP client now automatically tries all addresses (IPv4 and IPv6) provided by nameserver until success

Editing qubes.UpdatesProxy from TCP to TCP4 fix it for me. So for some reason, it tries IPv6 and fails even when IPv4 is a working solution.

adrelanos · 2024-03-09T10:21:01Z

dom0 updates seem to be unaffected. (Because it does not use Qubes UpdatesProxy. It uses a different mechanism, Qubes UpdateVM.)

So it seems this can and should be fixed in both levels:

A) in the Template (add Allow ::1 to tinyproxy config). This is for correctness, completeness sake. And,
B) in dom0. This is to unbreak the broken update mechanism of the Template. Does Qubes already have a mechanism for dom0 to apply hot fixes to Templates?

GWeck · 2024-03-09T11:31:46Z

A) fixed it for me with Fedora 39. I didn't try B).

marmarek · 2024-03-09T15:27:05Z

B) in dom0. This is to unbreak the broken update mechanism of the Template. Does Qubes already have a mechanism for dom0 to apply hot fixes to Templates?

Yes, that's exactly why we have our update tool instead calling apt/dnf directly.
One unfortunate thing is the fix will require updating twice:

run template update once - this will apply the fix, but the update itself will fail, since the fix isn't in sys-net at this point yet
restart sys-net
update again - now it will work

Since the issue breaks updates, apply the fix via updater wrapper QubesOS/qubes-issues#9025

Recent change in socat made it prefer ::1 address for localhost. This breaks updates proxy, because it allowed only 127.0.0.1. Force the IPv4 address. An alternative could be allowing IPv6 address in the proxy instead, but since IPv6 can be disabled (Whonix does it), it would mean having different paths depending on IP settings, which could be confusing at times. Fixes QubesOS/qubes-issues#9025 (cherry picked from commit 8acd68c)

qubesos-bot · 2024-03-15T16:39:06Z

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot · 2024-03-15T16:39:32Z

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot · 2024-03-15T16:52:42Z

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot · 2024-03-15T16:53:18Z

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot · 2024-03-15T16:54:00Z

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot · 2024-03-15T16:54:51Z

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

probablyodd · 2024-03-17T02:52:53Z

A) fixed it for me with Fedora 39. I didn't try B).

Likewise, fixed with A.

kaie · 2024-04-11T17:09:00Z

This is still an issue, if sys-net is based on the debian-12-xfce template, despite having using workaround A) to manually update, and having the latest packages installed.
(I probably have this, because I selected that as the default template on Qubes 4.2 install.)

After switching sys-net to use template fedora-39-xfce, updating works.

andrewdavidwong · 2024-04-12T08:29:34Z

This is still an issue, if sys-net is based on the debian-12-xfce template, despite having using workaround A) to manually update, and having the latest packages installed. (I probably have this, because I selected that as the default template on Qubes 4.2 install.)

After switching sys-net to use template fedora-39-xfce, updating works.

My understanding is that issue is about the case in which sys-net and sys-firewall are based on a Fedora template. It sounds like your situation is different, since your sys-net is based on a Debian template. Since you also said that switching sys-net to a Fedora template works, that seems to be further evidence that this issue has indeed been fixed. If this understanding is correct, then you should open a separate issue for your problem (unless one already exists, in which case you should comment on that existing issue instead).

Nurmagoz added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug labels Mar 8, 2024

andrewdavidwong pinned this issue Mar 9, 2024

andrewdavidwong changed the title ~~Upgrading Fedora templates results in loss of internet connection (only in the templates)~~ Updating Fedora templates results in loss of internet connection (only in the templates) Mar 9, 2024

marmarek self-assigned this Mar 9, 2024

marmarek added this to Current team tasks Mar 9, 2024

marmarek moved this to In progress in Current team tasks Mar 9, 2024

marmarek mentioned this issue Mar 9, 2024

Force using 127.0.0.1 when connecting to updates proxy QubesOS/qubes-core-agent-linux#489

Merged

marmarek added a commit to marmarek/qubes-mgmt-salt-dom0-update that referenced this issue Mar 9, 2024

Apply hotfix for #9025

de3220e

Since the issue breaks updates, apply the fix via updater wrapper QubesOS/qubes-issues#9025

marmarek mentioned this issue Mar 9, 2024

Apply hotfix for #9025 QubesOS/qubes-mgmt-salt-dom0-update#27

Merged

marmarek added a commit to marmarek/qubes-core-admin-linux that referenced this issue Mar 9, 2024

Apply hotfix for #9025

35b9efe

Since the issue breaks updates, apply the fix via updater wrapper QubesOS/qubes-issues#9025

marmarek mentioned this issue Mar 9, 2024

Apply hotfix for #9025 QubesOS/qubes-core-admin-linux#145

Merged

This was referenced Mar 9, 2024

mgmt-salt-dom0-update v4.1.13 (r4.2) QubesOS/updates-status#4428

Closed

mgmt-salt-dom0-update v4.1.13 (r4.1) QubesOS/updates-status#4429

Closed

This was referenced Mar 9, 2024

core-agent-linux v4.2.29 (r4.2) QubesOS/updates-status#4430

Closed

core-agent-linux v4.2.29 (r4.2) QubesOS/updates-status#4431

Closed

qubesos-bot added r4.1-fc39-stable and removed r4.1-fc39-cur-test labels Mar 15, 2024

qubesos-bot added r4.2-vm-bookworm-stable and removed r4.2-vm-bookworm-cur-test labels Mar 15, 2024

qubesos-bot added r4.2-vm-trixie-stable and removed r4.2-vm-trixie-cur-test labels Mar 15, 2024

qubesos-bot added r4.2-vm-fc37-stable and removed r4.2-vm-fc37-cur-test labels Mar 15, 2024

qubesos-bot added r4.2-vm-fc38-stable and removed r4.2-vm-fc38-cur-test labels Mar 15, 2024

qubesos-bot added r4.2-vm-fc39-stable and removed r4.2-vm-fc39-cur-test labels Mar 15, 2024

qubesos-bot added r4.2-vm-fc40-stable and removed r4.2-vm-fc40-cur-test labels Mar 15, 2024

marmarek mentioned this issue Mar 16, 2024

Support TCP socket qrexec services #9037

Closed

3 tasks

viq mentioned this issue Mar 16, 2024

Enabling IPv6 breaks updates and whonix #9042

Closed

andrewdavidwong unpinned this issue Mar 30, 2024

kaie mentioned this issue Apr 12, 2024

qubes-updates-proxy: Start request repeated too quickly #9110

Closed

This was referenced Apr 17, 2024

core-agent-linux v4.2.29 (r4.3) QubesOS/updates-status#4517

Closed

mgmt-salt-dom0-update v4.1.14 (r4.3) QubesOS/updates-status#4551

Closed

qubesos-secondary-bot mentioned this issue May 2, 2024

core-agent-linux v4.1.49 (r4.1) QubesOS/updates-status#4660

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating Fedora templates results in loss of internet connection (only in the templates) #9025

Updating Fedora templates results in loss of internet connection (only in the templates) #9025

Nurmagoz commented Mar 8, 2024 •

edited

Loading

adrelanos commented Mar 8, 2024

adrelanos commented Mar 8, 2024

Minimalist73 commented Mar 8, 2024

adrelanos commented Mar 9, 2024

GWeck commented Mar 9, 2024

marmarek commented Mar 9, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

probablyodd commented Mar 17, 2024

kaie commented Apr 11, 2024

andrewdavidwong commented Apr 12, 2024

Updating Fedora templates results in loss of internet connection (only in the templates) #9025

Updating Fedora templates results in loss of internet connection (only in the templates) #9025

Comments

Nurmagoz commented Mar 8, 2024 • edited Loading

Qubes OS release

Steps to reproduce

Expected behavior

Actual behavior

Workaround

Note

adrelanos commented Mar 8, 2024

adrelanos commented Mar 8, 2024

Minimalist73 commented Mar 8, 2024

adrelanos commented Mar 9, 2024

GWeck commented Mar 9, 2024

marmarek commented Mar 9, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

qubesos-bot commented Mar 15, 2024

probablyodd commented Mar 17, 2024

kaie commented Apr 11, 2024

andrewdavidwong commented Apr 12, 2024

Nurmagoz commented Mar 8, 2024 •

edited

Loading