Warning: systemd-boot failure with newer nixpkgs version

[oliverbestmann]

Not an actual issue for this repository, just a warning for everyone who updates their nixpkgs regulary:

systemd-boot: Assertion "Bs->AllocatePages(...) == EFI_SUCCESS" failed at ...

I've just reverted an update of nixpkgs on my fork, as it led to a totally unbootable system: oliverbestmann@5d723ce

I needed to do a recovery nixos-install as described here: https://github.com/tpwrules/nixos-apple-silicon/blob/main/docs/uefi-standalone.md#rescue

[andre4ik3]

Yep, found this out the hard way after doing a nixos-rebuild ... && reboot overnight. The full error is this, coming from this file (but this is just a helper function -- not sure what the root cause is, will investigate later):

systemd-boot: Assertion 'BS->AllocatePages(type, memory_type, n_pages, &addr) == EFI_SUCCESS' failed at ../src/boot/efi/util.h:77@xmalloc_pages, halting.

To fix it, downloading full installer and doing a full nixos-install isn't actually needed, you can do it using nothing but UEFI shell:

1. Fetch UEFI shell and a working systemd-boot binaries:

   nix build nixpkgs/nixos-24.05#legacyPackages.aarch64-linux.edk2-uefi-shell
   cp ./result/shell.efi /tmp/BOOTAA64.efi
   nix build nixpkgs/nixos-24.05#legacyPackages.aarch64-linux.systemd
   cp ./result/lib/systemd/boot/efi/systemd-bootaa64.efi /tmp/systemd-boot.efi
   

2. Format a FAT32 USB stick, copy over the two files to it in this layout:

   /EFI/BOOT/BOOTAA64.efi   <= this should be the shell.efi from edk2-uefi-shell
   /systemd-boot.efi        <= and this should be a working systemd-boot.efi from systemd
   

3. Eject USB stick, plug into broken Mac and boot it. It should immediately boot into UEFI shell without pausing, if the USB stick is recognized and set up correctly.

4. Find which FSX (where X is some number) partition is the EFI partition of NixOS and which is the USB stick. For me, FS0 was my NixOS EFI, and FS2 was my USB stick. You can check using ls FSX:\, the NixOS partition will have asahi and m1n1 folders, the USB stick will have systemd-boot.efi.

5. Now just copy over the working systemd-boot.efi from the USB stick to the EFI partition:

   Shell> cp FS2:\systemd-boot.efi FS0:\EFI\BOOT\BOOTAA64.efi
   Copying FS2:\systemd-boot.efi -> FS0:\EFI\BOOT\BOOTAA64.efi
   Destination file already exists.  Overwrite? Yes, No, All, Cancel Y
   - [ok]
   

6. Exit the shell by typing exit, this should make it immediately boot to NixOS.

7. Once back in NixOS, downgrade/pin nixpkgs to a safe version. In this case, stable 24.05 isn't affected, it's only unstable/24.11 that has the broken bootloader version.

Edit: made an issue in systemd: systemd/systemd#35026

[mariesavch]

same DO NOT UPDATE nixpkgs

[foldfree]

Thank you so much for providing a detailed fix guide. This update effectively broke ALL my builds and rolling back had no effect.

[andre4ik3]

Here is a TEMPORARY WORKAROUND patch that reverts the problematic commit, while the issue is fixed upstream, if anyone wants to use latest nixpkgs and can't/doesn't want to pin systemd to stable (e.g. because it would require a whole separate flake input and overlays and stuff).

Whilst I tested it on my personal systems, it's important to stress that it's a temporary workaround made in like 30 minutes, so please backup everything before applying it.

You can apply it to systemd.package like any other patch, then nixos-rebuild, and make sure the hash of the bootloader on your partition is 39393daafe18c44ba2ea708f834214b752e29f67691449d176a997e1686778a2. If it's 008189810420279126ed083d9f2f52c28eb912f7d1c5ccdac9f294e86f181249, then DO NOT REBOOT as you'll have a bricked system. Instead run sudo bootctl install --no-variables and check the hash again, that should fix it.

I would still strongly recommend pinning systemd.package or your whole configuration entirely to nixpkgs stable or an earlier unstable commit if at all possible.

[oliverbestmann]

Thank you for handling this so well!

[oliverbestmann]

I've pinned systemd to a working version in my fork now. Thanks for the suggestion!

Commit is here: oliverbestmann@97ff3d5 It might not be the most beautiful way to do it, but it works ;)

[tpwrules]

Sorry for being quiet on this, I have been short on time recently. I will ship a revert of the problematic patch soon, and get back to package upgrading in a bit. Due to where we are in the NixOS release cycle upstream Nixpkgs likely can't do much.

[andre4ik3]

Sorry for being quiet on this, I have been short on time recently. I will ship a revert of the problematic patch soon, and get back to package upgrading in a bit. Due to where we are in the NixOS release cycle upstream Nixpkgs likely can't do much.

A revert patch just landed in Nixpkgs upstream, maybe this can be integrated here until it's in unstable/24.11? NixOS/nixpkgs#355290

[tpwrules]

This is fixed (with low overhead) in the latest release. Thanks all for the work on this.

[psyolia]

So its safe to update again if I am on nixpkgs-unstable?

[oliverbestmann]

Yes. A patch to systemd will be applied, so systemd will be compiled from source, but it should boot again.

[benmkw]

Is there a way to copy only the two relevant files (BOOTAA64.efi and systemd-boot.efi) from a bootet ISO image over to repair a broken install?

Context:
I bricked my system by upgrading while still being on stable ... i think. Because I don't have another nix system I bootet from the latest ISO via USB and tried to copy over BOOTAA64.efi but did not find systemd-boot.efi so this did not work out.
When trying nixos-install --no-root-password --no-channel-copy I could not run it because I had no LAN access/ could not connect to wifi from the bootet ISO (would need to try again elsewhere).

Not an actual issue for this repository, just a warning for everyone who updates their nixpkgs regulary:

systemd-boot: Assertion "Bs->AllocatePages(...) == EFI_SUCCESS" failed at ...

I've just reverted an update of nixpkgs on my fork, as it led to a totally unbootable system: oliverbestmann@5d723ce

I needed to do a recovery nixos-install as described here: https://github.com/tpwrules/nixos-apple-silicon/blob/main/docs/uefi-standalone.md#rescue

and

Yep, found this out the hard way after doing a nixos-rebuild ... && reboot overnight. The full error is this, coming from this file (but this is just a helper function -- not sure what the root cause is, will investigate later):

systemd-boot: Assertion 'BS->AllocatePages(type, memory_type, n_pages, &addr) == EFI_SUCCESS' failed at ../src/boot/efi/util.h:77@xmalloc_pages, halting.

To fix it, downloading full installer and doing a full nixos-install isn't actually needed, you can do it using nothing but UEFI shell:

1. Fetch UEFI shell and a working `systemd-boot` binaries:
   ```
   nix build nixpkgs/nixos-24.05#legacyPackages.aarch64-linux.edk2-uefi-shell
   cp ./result/shell.efi /tmp/BOOTAA64.efi
   nix build nixpkgs/nixos-24.05#legacyPackages.aarch64-linux.systemd
   cp ./result/lib/systemd/boot/efi/systemd-bootaa64.efi /tmp/systemd-boot.efi
   ```

2. Format a FAT32 USB stick, copy over the two files to it in this layout:
   ```
   /EFI/BOOT/BOOTAA64.efi   <= this should be the shell.efi from edk2-uefi-shell
   /systemd-boot.efi        <= and this should be a working systemd-boot.efi from systemd
   ```

3. Eject USB stick, plug into broken Mac and boot it. It should immediately boot into UEFI shell without pausing, if the USB stick is recognized and set up correctly.

4. Find which `FSX` (where `X` is some number) partition is the EFI partition of NixOS and which is the USB stick. For me, `FS0` was my NixOS EFI, and `FS2` was my USB stick. You can check using `ls FSX:\`, the NixOS partition will have `asahi` and `m1n1` folders, the USB stick will have `systemd-boot.efi`.

5. Now just copy over the working `systemd-boot.efi` from the USB stick to the EFI partition:
   ```
   Shell> cp FS2:\systemd-boot.efi FS0:\EFI\BOOT\BOOTAA64.efi
   Copying FS2:\systemd-boot.efi -> FS0:\EFI\BOOT\BOOTAA64.efi
   Destination file already exists.  Overwrite? Yes, No, All, Cancel Y
   - [ok]
   ```

6. Exit the shell by typing `exit`, this should make it immediately boot to NixOS.

7. Once back in NixOS, downgrade/pin nixpkgs to a safe version. In this case, stable 24.05 isn't affected, it's only unstable/24.11 that has the broken bootloader version.

Edit: made an issue in systemd: systemd/systemd#35026

[foldfree]

Is there a way to copy only the two relevant files (BOOTAA64.efi and systemd-boot.efi) from a bootet ISO image over to repair a broken install?

You still have the macOS partition right?
Just install the nix package manager there, and from there follow #248 (comment)

[benmkw]

Is there a way to copy only the two relevant files (BOOTAA64.efi and systemd-boot.efi) from a bootet ISO image over to repair a broken install?

You still have the macOS partition right? Just install the nix package manager there, and from there follow #248 (comment)

thanks yeah that might work

[lloeki]

Just did a nix flake update (with nixpkgs-24.11) and nixos-rebuild build, this now fails at patching:

$ nix log /nix/store/s33w9gdsz42a3rzkk7pk6dlyaxh26x80-systemd-256.8.drv
[...]
applying patch /nix/store/xvkm9kfll35kp52nwbig4qrdg748v0zc-0019-Revert-boot-Make-initrd_prepare-semantically-equival.patch
patching file src/boot/efi/boot.c
Hunk #4 FAILED at 2292.
Hunk #5 succeeded at 2382 (offset 9 lines).
Hunk #6 succeeded at 2402 (offset 9 lines).
1 out of 6 hunks FAILED -- saving rejects to file src/boot/efi/boot.c.rej

Note that this is 256.8, while the affected systems version was 256.7.

Looks like Revert-boot-Make-initrd_prepare-semantically-equival.patch is the patch in question from https://github.com/NixOS/nixpkgs/pull/355290/files

Here's the 256.7..256.8 log: systemd/systemd@v256.7...v256.8 which includes systemd/systemd@a9d9db7 from systemd/systemd#35149 which fixes systemd/systemd#35026

So 256.8 has a fix, which causes the revert patch to not apply.

@tpwrules Maybe this repo should be updated to assert systemd is 256.7 before applying the revert patch.

[oliverbestmann]

There is already a pr by @flokli. You can point your flake to that commit or try out my fork of this repo until @tpwrules merges the pr.

[lloeki]

Oh thanks @oliverbestmann, I missed #252 during my digging 😓