Fedora 37 fails to update via salt.

[marmarek]

    Failed to update via salt.

sudo qubesctl --skip-dom0 --targets=fedora-37 --show-output state.sls update.qubes-vm

fedora-37:
  ----------
            ID: dnf-and-rpm
      Function: pkg.installed
        Result: True
       Comment: All specified packages are already installed and are at the desired version
       Started: 03:08:41.188802
      Duration: 565.189 ms
       Changes:   
  ----------
            ID: /usr/lib/rpm/macros.d/macros.qubes
      Function: file.absent
        Result: True
       Comment: File /usr/lib/rpm/macros.d/macros.qubes is not present
       Started: 03:08:41.757306
      Duration: 0.429 ms
       Changes:   
  ----------
            ID: dnf-makecache
      Function: cmd.script
        Result: True
       Comment: DNF cache successfully created
       Started: 03:08:41.758476
      Duration: 24450.779 ms
       Changes:   
  ----------
            ID: Disable deltarpm
      Function: file.append
          Name: /etc/dnf/dnf.conf
        Result: False
       Comment: An exception occurred in this state: Traceback (most recent call last):
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/state.py", line 2276, in call
                    ret = self.states[cdata["full"]](
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 149, in __call__
                    return self.loader.run(run_func, *args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 1228, in run
                    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 1243, in _run_as
                    return _func_or_method(*args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 1276, in wrapper
                    return f(*args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/states/file.py", line 6431, in append
                    if __salt__["file.search"](
                       ^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 149, in __call__
                    return self.loader.run(run_func, *args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 1228, in run
                    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/lazy.py", line 1243, in _run_as
                    return _func_or_method(*args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/modules/file.py", line 3141, in search
                    return replace(
                           ^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/modules/file.py", line 2516, in replace
                    flags_num = _get_flags(flags)
                                ^^^^^^^^^^^^^^^^^
                  File "/var/tmp/.root_dd8a91_salt/pyall/salt/modules/file.py", line 1643, in _get_flags
                    _flag = getattr(re, str(flag).upper())
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                AttributeError: module 're' has no attribute 'RE.MULTILINE'
       Started: 03:09:06.209454
      Duration: 18.838 ms
       Changes:   
  ----------
            ID: update
      Function: pkg.uptodate
        Result: False
       Comment: One or more requisite failed: update.qubes-vm.Disable deltarpm
       Started: 03:09:06.228807
      Duration: 0.004 ms
       Changes:   
  ----------
            ID: notify-updates
      Function: cmd.run
          Name: /usr/lib/qubes/upgrades-status-notify
        Result: True
       Comment: Command "/usr/lib/qubes/upgrades-status-notify" run
       Started: 03:09:06.228851
      Duration: 4085.96 ms
       Changes:   
                ----------
                pid:
                    980
                retcode:
                    0
                stderr:
                stdout:
Summary for fedora-37
Succeeded: 4 (changed=1)

Failed:    2
Total states run:     6

Total run time:  29.121 s

Originally posted by @noskb in QubesOS/updates-status#3189 (comment)

[marmarek]

Related to #7807

[marmarek]

Salt upstream bug: saltstack/salt#62676

[marmarek]

It should be rather easy to avoid using file salt module for the time being (however silly that sounds).

[Solomon1732]

It also failes for Fedora 36 and dom0. The former returns a python exception related to salt. The latter It fails on some shell script inside the update-downloading qube before it even reaches dom0 for verification. As a result I can not neither update fedora templates nor dom0. Something broke after a recent update. I don't know which one and when. When I'll return to said comouter I'll share the errors. error. Right know I am not around it.

Edit: Fedora update did throw an exception, but it worked when I tried again. Now *only dom0 can't update.

[Solomon1732]

Here's the relevant log of qubesctl: dom0-update.log. I piped the result to a file and removed the color code characters. It's clear to me that the /usr/lib/qubes/qubes-download-dom0-updates.sh shell executable had trouble with something, but I don't know with what. If there's a way for me to get relevant logs I will be more than happy to provide them.

Note that this happens to me in all qubes used for update, both Whonix based and Fedora 36 based.

Edit: I forgot to share the command I ran: sudo qubesctl --show-output state.sls update.qubes-dom0 > dom0-update.log. It also happens to me through the GUI.

[marmarek]

Check journalctl --user in sys-whonix (the one set as updatevm), it should contain extra details.
If still nothing there, you may edit /usr/lib/qubes/qubes-download-dom0-updates.sh to add set -x at the beginning.

[Solomon1732]

I ran journalctl --user -e

Nov 19 09:51:18 host qubes.VMShell-dom0[11602]: Unable to detect release version (use '--releasever' to specify release version)
Nov 19 09:51:19 host qubes.VMShell-dom0[11602]: No packages downloaded
Nov 19 09:51:26 host qubes.VMShell-dom0[11797]: Unable to detect release version (use '--releasever' to specify release version)
Nov 19 09:51:31 host systemd[832]: run-user-0.mount: Succeeded.
Nov 19 09:58:28 host qubes.VMShell-dom0[11797]: No packages downloaded
Nov 19 09:58:35 host qubes.VMShell-dom0[12044]: Unable to detect release version (use '--releasever' to specify release version)
Nov 19 09:58:40 host systemd[832]: run-user-0.mount: Succeeded.
Nov 19 10:13:07 host qubes.VMShell-dom0[12044]: Error: Error downloading packages:
Nov 19 10:13:07 host qubes.VMShell-dom0[12044]:   Cannot download rpm/kernel-qubes-vm-5.15.76-1.fc32.qubes.x86_64.rpm: All mirrors were tried

[marmarek]

Hmm, that's weird. Anyway, does adding --releasever=4.1 help?

[Solomon1732]

It may be because I use the onion addresses. But through Whonix it should work. I'll try.

[Solomon1732]

To which part of the command do I add it? I forgot it's qubesctl and not qubes-dom0-update 😅

[marmarek]

Ah, right. For this, try calling qubes-dom0-update directly.

[Solomon1732]

It spits its usual "Unable to detect release version (use --releasever to specify release version)". Then it proceeded to download packages as usual. And it fact it looks like it skipped over a good chunk of already downloaded ones. I'll share the results when it finishes.

As a side note, that warning is why I switched to a dedicated static dispvm qube for dom0. I have no idea why it's there. It persisted even after I reinstalled to update from 4.0 to 4.1.

[brendanhoar]

I switched to a dedicated static dispvm qube for dom0

How is this done?

B

[Solomon1732]

@marmarek Ah, I see the issue. It says this:

The downloaded packages were saved in cache until next successful transaction.
You can remove cache packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Cannot download rpm/kernel-5.15-76-1.f32.qubes.x86_64.rpm; All mirrors were tried

It may be a connection issue over tor 😅 It threw the error partway into downloading it.

---

@brendanhoar this is the tl;dr. I'd share more, but I'd rather this be continued in a separate discussion (or issue?) to not clog this issue for the devs more than I already have 😅 If you find any relevant place, and have questions, just ping me there and link this message or issue 😄

How is this done?

B
I use the GUI:

1. Create an appVM, say, Foo-dvm > open the qube's settings > Advanced tab > check Disposable template.
2. Then I create a new VM (say, Foo) as TemplateVM type, set Foo-dvm as its template. If you wish to update over Tor just make sys-whonix or your equivalent as its Net qube.
3. Applications button in Qubes > Qubes Tools > Qubes Global Setting > switch Dom0 update qube to your static dispvm (say, Foo).

The pros:

• If the update gets screwed up somehow, it's still a dispVM. Just close it
• No need to delete cache files if it stops in the middle
• Usual benefits of dispVMs

The cons:

• salt isn't configured for this. It means that you have to manually close it after it finishes. I'm sure user salt files can fix it, I just find it too byzantine to finish learning how to do it. Edit: A few months ago I tried to learn how to create them. I abandoned my attempt partway
• Edit 3: it also renders it impossible for the update checker to check updates for dom0. I suspect it's because it needs the dom0 update qube to run continuously (such as sys-firewall and sys-whonix do).

Edit 2: salt still knows to start said qube up. It simply doesn't shut it down. I wager that the relevant system (salt) files were made for sys-firewall and sys-whonix, hence the need to keep them running.

[Solomon1732]

Yep, it's the connection speed. It's just super slow. So the package manager disconnects, tries another mirror, sees it is slow as well, repeats until its out of mirrors, then inevitably fails. So it wasn't salt after all, just the usual connection issues. Thanks for helping me figure it out.

It would be helpful if salt could tell me if an update failed owing to connection speed or some other issue. Is it possible to add it to Qubes in the future?

[DemiMarie]

It would be helpful if salt could tell me if an update failed owing to connection speed or some other issue. Is it possible to add it to Qubes in the future?

This is going to be very difficult. It would require using the DNF API instead of the CLI, for one. That said, @marmarek this is a good reason for users to be able to view (properly sanitized!) log messages in the future non-interactive updater, which would make this obvious.

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt-dom0-update (including package qubes-mgmt-salt-dom0-update-4.1.10-1.fc32) has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[3hhh]

As #8784 was closed as duplicate of this one, please re-open this one.

[marmarek]

As #8784 was closed as duplicate of this one, please re-open this one.

That ticket says using debian-11 mgmt qube for managing debian-12 fails. I don't think that's supportable configuration - mgmt qube (default-mgmt-dvm by default) should be at least the version of the target qube (regarding salt version). So, using debian-12 to manage debian-12 can work. Using sufficiently new fedora to manage debian-12 should also work. But not the other way around. That's why several of Fedora template announcements reminds to switch default-mgmt-dvm too, and why it's explicitly reminded at https://www.qubes-os.org/doc/templates/#switching