Custom iso building: make fails at step "get-sources" #7977

Jauchi · 2023-01-08T13:55:36Z

Qubes OS release

Tried in a fedora 32 AppVM (as per documentation) and on "vanilla" fedora 32.

Brief summary

I tried following the docs, on the get-sources step, it fails with the following output:

--> Downloading additional sources for linux-kernel...
make[1]: Entering directory '/home/user/qubes-builder/qubes-src/linux-kernel'
if [ -f /usr/bin/qvm-run-vm ]; \
        then qvm-run-vm --dispvm 2>/dev/null xzcat <linux-5.15.85.tar.xz.UNTRUSTED > linux-5.15.85.tar.UNTRUSTED; \
else xzcat <linux-5.15.85.tar.xz.UNTRUSTED > linux-5.15.85.tar.UNTRUSTED; fi
cat kernel.org-1-key.asc kernel.org-2-key.asc | gpg --dearmor >linux-keyring.gpg
gpgv --keyring ./linux-keyring.gpg linux-5.15.85.tar.sign linux-5.15.85.tar.UNTRUSTED || \
  { echo "Wrong signature on linux-5.15.85.tar.UNTRUSTED!"; exit 1; }
gpgv: Signature made Wed 21 Dec 2022 05:37:35 PM CET
gpgv:                using RSA key 647F28654894E3BD457199BE38DBBDC86092693E
gpgv: Good signature from "Greg Kroah-Hartman (Linux kernel stable release signing key) <[email protected]>"
mv linux-5.15.85.tar.UNTRUSTED linux-5.15.85.tar
if [ -f /usr/bin/qvm-run-vm ]; \
        then qvm-run-vm --dispvm 2>/dev/null zcat <macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.gz.UNTRUSTED > macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.UNTRUSTED; \
else zcat <macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.gz.UNTRUSTED > macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.UNTRUSTED; fi
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `sha256sum --status --strict -c <(printf "92bb7cbeb8df5a57d57ffffc193f621d20b336eb57af0aa81ce30404225c7425  -\n") <macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.UNTRUSTED || \'
make[1]: *** [Makefile:100: macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar] Error 1
rm macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.UNTRUSTED linux-keyring.gpg
make[1]: Leaving directory '/home/user/qubes-builder/qubes-src/linux-kernel'
make: *** [Makefile:226: linux-kernel.get-sources-extra] Error 2

Steps to reproduce

Get a fedora AppVM
Follow instructions to build an ISO file up to get-sources
it fails

Expected behavior

get-sources does not fail and I can continue to compile.

Actual behavior

Fails with (presumably) syntax error

Thanks for all the great work, this is my first issue and I have been using qubes for a lot of weird things already, all of these work perfectly fine!

The text was updated successfully, but these errors were encountered:

Jauchi · 2023-01-08T14:38:49Z

Quick update: renaming the files seems to fix the issue:

[user@localhost-live linux-kernel]$ pwd
/home/user/qubes-builder/qubes-src/linux-kernel
[user@localhost-live linux-kernel]$ mv macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.sha256 macbook12-spi-driver.tar.sha256
[user@localhost-live linux-kernel]$ mv macbook12-spi-driver-f85d028a41fbc928048c8ad248735501e76a2ce9.tar.gz.UNTRUSTED macbook12-spi-driver.tar.gz.UNTRUSTED

And now, I get a 404 error;

--> Downloading additional sources for zlib...
make[1]: Entering directory '/home/user/qubes-builder/qubes-src/zlib'
curl: (22) The requested URL returned error: 404 Not Found
make[1]: *** [Makefile:27: zlib-1.2.12.tar.xz] Error 22
make[1]: Leaving directory '/home/user/qubes-builder/qubes-src/zlib'
make: *** [Makefile:226: zlib.get-sources-extra] Error 2

If you want, I can make a seperate issue for the 404 problem...

andrewdavidwong · 2023-01-09T15:38:11Z

And now, I get a 404 error

I wonder if that might be a temporary network problem.

xaki23 · 2023-01-09T19:59:00Z

i can confirm the 404, and my guess is thats intentional because of
https://nvd.nist.gov/vuln/detail/CVE-2022-37434

trying right now whether it is enough to bump the version to 1.2.13

edit:
it is not, because the patches also contain references to VERSION and one is even about CVE-2022-37434.
so either some adjustment for 1.2.13 is required, or some alternate url for the 1.2.12 tarball.
a gz version can be found here: https://www.zlib.net/fossils/zlib-1.2.12.tar.gz
when repackaging it as xz, the pkg build works

xaki23 · 2023-01-09T22:42:45Z

after working around the zlib issue, the next thing that fails is ...

make[1]: Entering directory '/home/user/qubes-builder'
git -C /home/user/qubes-builder/chroot-dom0-fc32/home/user/qubes-src/remote-support clean -f -d -X
/home/user/qubes-builder/scripts/create-archive /home/user/qubes-builder/chroot-dom0-fc32/home/user/qubes-src/remote-support qubes-remote-support-receiver-dom0-1.0.1.tar.gz
~/qubes-builder/chroot-dom0-fc32/home/user/qubes-src/remote-support ~/qubes-builder
~/qubes-builder
-> Building remote-support (rpm_spec/qubes-remote-support-receiver-dom0.spec) for fc32 dom0 (logfile: build-logs/remote-support-dom0-fc32.log)
--> Done:
      qubes-src/remote-support/pkgs/dom0-fc32/noarch/qubes-remote-support-receiver-dom0-1.0.1-1.fc32.noarch.rpm
make[1]: Leaving directory '/home/user/qubes-builder'
make[1]: Entering directory '/home/user/qubes-builder'
/home/user/qubes-builder/qubes-src/builder-rpm/Makefile-mock.rpmbuilder:18: *** package 'mock' must be installed.  Stop.
make[1]: Leaving directory '/home/user/qubes-builder'
make: *** [Makefile:273: windows-tools-cross-dom0] Error 1

mock is a USE_DIST_BUILD_TOOLS conditional dep in builder-rpm: https://github.com/QubesOS/qubes-builder-rpm/blob/main/builder.conf
any reason to not just make it unconditional?

xaki23 · 2023-01-09T23:55:43Z

and the next thing that fails (on "make iso") is

-> Building installer-qubes-os iso for fc32 (logfile: build-logs/installer-qubes-os-iso-fc32.log)...
--> build failed!
fedora                                          8.9 MB/s |  70 MB     00:07
fedora-updates                                  8.0 MB/s |  30 MB     00:03
installer                                        14 kB/s | 257  B     00:00
qubes-dom0                                      7.1 MB/s | 523 kB     00:00
dom0-updates                                    9.0 kB/s | 257  B     00:00
No match for argument: kernel-latest
No match for argument: kernel-latest-qubes-vm
Error: Unable to find a match: kernel-latest kernel-latest-qubes-vm
make[1]: *** [Makefile:128: iso-installer-gather] Error 1
make[1]: Leaving directory '/home/user/qubes-src/installer-qubes-os'
make: *** [Makefile:575: iso] Error 1

... which is most likely fallout from #5900

xaki23 · 2023-01-10T00:22:07Z

and after manually building kernel-latest-* with

make get-sources COMPONENTS=linux-kernel BRANCH_linux_kernel=main
make qubes-dom0 COMPONENTS=linux-kernel BRANCH_linux_kernel=main

the next "make iso" fails in a very unexpected way:

-> Building installer-qubes-os iso for fc32 (logfile: build-logs/installer-qubes-os-iso-fc32.log)...
--> build failed!
No match for group package "qubes-template-debian-11"
No match for group package "kernel-qubes-vm"
Error: 
 Problem: package kernel-latest-qubes-vm-1000:6.1.3-1.fc32.qubes.x86_64 requires kernel-devel = 6.1.3-1.fc32.qubes.x86_64, but none of the providers can be installed
  - package kernel-latest-devel-1000:6.1.3-1.fc32.qubes.x86_64 requires perl(Math::BigInt), but none of the providers can be installed
  - conflicting requests
  - package perl-Math-BigInt-1:1.9998.18-2.fc32.noarch is filtered out by exclude filtering
(try to add '--skip-broken' to skip uninstallable packages)
make[1]: *** [Makefile:128: iso-installer-gather] Error 1
make[1]: Leaving directory '/home/user/qubes-src/installer-qubes-os'
make: *** [Makefile:575: iso] Error 1

marmarek · 2023-01-10T01:10:11Z

package kernel-latest-devel-1000:6.1.3-1.fc32.qubes.x86_64 requires perl(Math::BigInt)

For some reason, kernel built with USE_DIST_BUILD_TOOLS=0 has this dependency, while with USE_DIST_BUILD_TOOLS=1 does not. Likely some extra packages installed in chroot cause that. Build log of official package is at QubesOS/updates-status#3366 (comment), you can compare with your local one (and grep for BigInt).

xaki23 · 2023-01-10T23:33:43Z

with the zlib problem fixed, i have now been able to build ISOs for both 4.1 and 4.2/fc32:

buildvm is a fedora 37, vcpus 12, memory 2000, maxmem 16000, private 100 GiB

$ git clone https://github.com/QubesOS/qubes-builder
$ cd qubes-builder

$ cat > override.conf << EOF
USE_DIST_BUILD_TOOLS=1
COMPONENTS += \\
                          linux-kernel-latest
GIT_URL_linux_kernel_latest = \$(GIT_BASEURL)/\$(GIT_PREFIX)linux-kernel
BRANCH_linux_kernel_latest = master
EOF

$ ./setup
Y (install deps)
yes (master key)
yes (signing key)
4.1
stable
yes (faster)
ok (no prebuilts)
no (ssh access)
no (not just templates)
deselect all but fc36
deselect builder-debian
no (dont download just yet)

$ make install-deps
$ time make get-sources
real    13m54.178s
user    0m35.928s
sys     0m32.313s

$ make install-deps
$ make remount
$ time make qubes
real    238m46.418s
user    551m28.608s
sys     84m25.268s

$ time make iso
real    22m13.532s
user    39m57.585s
sys     2m42.559s

3.5G    iso/Qubes-20230110-x86_64.iso

xaki23 · 2023-03-30T19:39:46Z

afaict this can be closed as resolved with the zlib fix merged.

andrewdavidwong · 2023-03-30T20:47:43Z

Closing as resolved. If anyone believes this issue is not yet resolved, or if anyone is still affected by this issue, please leave a comment, and we'll be happy to reopen it. Thank you.

kennethrrosen · 2023-04-30T12:31:15Z

Per this thread I am running into similar issues.

if [ -f /usr/bin/qvm-run-vm ]; \
        then qvm-run-vm --no-gui --dispvm 2>/dev/null zcat <macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar.gz.UNTRUSTED > macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar.UNTRUSTED; \
else zcat <macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar.gz.UNTRUSTED > macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar.UNTRUSTED; fi
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `sha256sum --status --strict -c <(printf "1641a09e8ae4fc494b8e44f1bc86d19cefcdc5ad74722ce058148b35a194aeb6  -\n") <macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar.UNTRUSTED || \'
make[1]: *** [Makefile:100: macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar] Error 1
rm macbook12-spi-driver-2905d318d1a3ee1a227052490bf20eddef2592f9.tar.UNTRUSTED linux-keyring.gpg
make[1]: Leaving directory '/home/user/qubes-builder/qubes-src/linux-kernel-latest'
make: *** [Makefile:226: linux-kernel-latest.get-sources-extra] Error 2

According to this commit, macbook12-spi-driver should be disabled, but it still persists in /linux-kernel/Makefile and /linux-kernel/kernel.spec.in

Additionally, if I pass the -k flag to bypass the error exit on the macbook12-spi-driver, there are several more errors in creating vnm-xen-stubdom-legacy, and a series of zen packages, and for which I can create a separate issues.

After passing the -k flag:

make[1]: Target 'get-sources' not remade because of errors.
make[1]: Leaving director '/home/user/qubes-builder/qubes-src/windows-tool-cross'
make: *** [Makefile:226: windows-tools-cross.get-sources-extra] Error 2
make: Target 'get-sources' not remade because of errors.

Thanks in advance to everyone for their insight and help.

marmarek · 2023-04-30T21:11:27Z

While the macbook driver indeed isn't really enabled in the build right now, the download error suggests some more generic issue. What your /bin/sh points to, not bash, right?

kennethrrosen · 2023-05-01T07:22:16Z

It is pointing to /usr/bin/bash as my test system is a vanilla latest-release. The AppVM is F32

xaki23 · 2023-05-01T07:32:09Z

It is pointing to /usr/bin/bash as my test system is a vanilla latest-release. The AppVM is F32

fedora 32 has been EOL since 2021.
if you are still using f32 based appvms, you are not anywhere near "latest-release".
currently supported fedora versions are 36+37, and 38 actually works too (but is prerel).

xaki23 · 2023-05-01T07:48:46Z

By latest release, I mean the entire QubesOS system. These issues also occurred in F36 and F37

worked fine here just now on a fedora 37 buildvm.
are you sure you tried it on an actual full f37 buildvm and not something like a -minimal?

kennethrrosen · 2023-05-01T07:52:07Z

Yes, you're right. Can close; I'll issue a pull request to change ISO build documentation

xaki23 · 2023-05-01T08:05:55Z

fwiw, i was able to repro the problem on a full f32 buildvm.
seems the <() thing got added somewhere between bash 5.0 and 5.2?

thanks for the report, and thanks for filing an update to the documentation!

Jauchi added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug labels Jan 8, 2023

andrewdavidwong added C: builder Qubes Builder needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels Jan 9, 2023

andrewdavidwong added this to the Release 4.1 updates milestone Jan 9, 2023

xaki23 mentioned this issue Jan 10, 2023

update to v1.2.13 QubesOS/qubes-zlib#1

Merged

andrewdavidwong closed this as completed Mar 30, 2023

andrewdavidwong added diagnosed Technical diagnosis has been performed (see issue comments). and removed needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels Mar 30, 2023

andrewdavidwong reopened this Apr 30, 2023

andrewdavidwong added needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. and removed diagnosed Technical diagnosis has been performed (see issue comments). labels Apr 30, 2023

andrewdavidwong closed this as completed May 1, 2023

andrewdavidwong added diagnosed Technical diagnosis has been performed (see issue comments). and removed needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels May 1, 2023

andrewdavidwong added the affects-4.1 This issue affects Qubes OS 4.1. label Aug 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom iso building: make fails at step "get-sources" #7977

Custom iso building: make fails at step "get-sources" #7977

Jauchi commented Jan 8, 2023

Jauchi commented Jan 8, 2023

andrewdavidwong commented Jan 9, 2023

xaki23 commented Jan 9, 2023 •

edited

Loading

xaki23 commented Jan 9, 2023

xaki23 commented Jan 9, 2023

xaki23 commented Jan 10, 2023

marmarek commented Jan 10, 2023

xaki23 commented Jan 10, 2023

xaki23 commented Mar 30, 2023

andrewdavidwong commented Mar 30, 2023

kennethrrosen commented Apr 30, 2023 •

edited

Loading

marmarek commented Apr 30, 2023

kennethrrosen commented May 1, 2023

xaki23 commented May 1, 2023

xaki23 commented May 1, 2023

kennethrrosen commented May 1, 2023

xaki23 commented May 1, 2023

Custom iso building: make fails at step "get-sources" #7977

Custom iso building: make fails at step "get-sources" #7977

Comments

Jauchi commented Jan 8, 2023

Qubes OS release

Brief summary

Steps to reproduce

Expected behavior

Actual behavior

Jauchi commented Jan 8, 2023

andrewdavidwong commented Jan 9, 2023

xaki23 commented Jan 9, 2023 • edited Loading

xaki23 commented Jan 9, 2023

xaki23 commented Jan 9, 2023

xaki23 commented Jan 10, 2023

marmarek commented Jan 10, 2023

xaki23 commented Jan 10, 2023

xaki23 commented Mar 30, 2023

andrewdavidwong commented Mar 30, 2023

kennethrrosen commented Apr 30, 2023 • edited Loading

marmarek commented Apr 30, 2023

kennethrrosen commented May 1, 2023

xaki23 commented May 1, 2023

xaki23 commented May 1, 2023

kennethrrosen commented May 1, 2023

xaki23 commented May 1, 2023

xaki23 commented Jan 9, 2023 •

edited

Loading

kennethrrosen commented Apr 30, 2023 •

edited

Loading