Raspberry Pi 4 USB boot support

[samueldr]

Boots on Rpi4 via an SD card (USB didn't boot). I do get an error for bluetooth:

Since that's better than not booting, let's merge this.

Originally posted by @domenkozar in #121720 (comment)

---

Let's track USB boot issues here.

First, please describe the steps you have tried exhaustively. It would be helpful if you could either provide the Nixpkgs revision used to build the image, or an Hydra link to the job for the image.

I have not yet taken the time to fully USB boot a Raspberry Pi 4 NixOS image. USB support in #121720 was tested when the initial boot firmware was present on an SD card. It was tested with the iso image and not the SD image.

Initial Boot Firmware here means U-Boot.

Known working

• UEFI iso image when the initial boot firmware is provided through an SD card (verified by samueldr)

Not known

• SD image burnt on a USB drive
  • As-Is
  • With updated firmware files (?)
  • 8GiB variant vs. 2, 4
  • Pi 400?

Is the EEPROM firmware important? (I believe it is.)

Testing information

When testing, please describe, if possible:

• The variant and board revision.
• The EEPROM firmware version in use
• The USB storage type / model
• Whether Raspberry Pi OS worked under the conditions
• Whether U-Boot starts
• Whether the system image stays stuck in Stage-1 or goes to Stage-2

Note that it is possible that not all USB drives work with the Raspberry Pi. Testing the official Raspberry Pi OS should be done to verify the board should or should not be able to boot from the drive you're using.

---

(cc #63720)

[bbqsrc]

@samueldr I will do this with a 2GB and 8GB model right now and get back to you with all the specifics you have asked for. 😄

[proofconstruction]

Came across this issue after having the same problem shown here #121720 (comment) , and described here #121834 (comment).

To recap for other readers, the problem there is that the latest successful build (nixos-sd-image-21.05pre287596.ae1c8ede09b-aarch64-linux.img) of the “new kernel” variant SD card image here https://hydra.nixos.org/build/142588923 boots from SD card (SanDisk Ultra 32GB A1) correctly to the console, but then USB peripherals cannot be used. By comparison, the most recent successful “raspberrypi4” image here https://hydra.nixos.org/build/134720986 correctly boots and has working USB.

Here’s what I found.

Device info:
Raspberry Pi 4B 8GB, board revision 1.4
EEPROM bootloader version: Thu 29 Apr 16:11:25 UTC 2021 (1619712685)
EEPROM VL805 firmware version: 000138a1

First, baseline tests with the latest Raspberry Pi OS image to establish that the device correctly boots from USB:

---

Test 1, Raspberry Pi OS
USB storage type: flash drive, HP PMAP v125w 16GB
Raspberry Pi OS (2021-03-04-raspios-buster-armhf-lite.img) boots correctly with full functionality.

---

Test 2, Raspberry Pi OS
USB storage type: Samsung 860 Evo m.2 SATA 256GB connected via ASMedia ASM1153E (ArgonOne case)
Raspberry Pi OS (2021-03-04-raspios-buster-armhf-lite.img) boots correctly with full functionality.

---

Tests 3 and 4, NixOS “new kernel” with the build from https://hydra.nixos.org/build/142588923 and the same USB devices:
In both cases, progress gets stuck at the “starting kernel…” screen.

This is the same image that, from an SD card, booted correctly to console but did not have USB.

---

Out of curiosity, I also tested the last successful “raspberrypi4” image on both USB devices, and in both cases get a different failure, which reboots and cycles indefinitely, seen here.

[bbqsrc]

• Variant and board revision: Raspberry Pi 4 Model B Rev 1.1 (2GB)
• EEPROM firmware version: tested with "Sep 3 2020" and "Feb 16 2021" (the eeprom in their 'latest' channel)
• USB storage type/model: Kingston DataTraveler SE9 G2 (64GB), some old Lexar 8GB USB 2.0, some old Verbatim "Store N Go" 8GB USB 2.0.

I used the nixos-sd-image-21.05pre287596.ae1c8ede09b-aarch64-linux.img image for all tests.

Started at eeprom version "Sep 3 2020", booted the Raspberry Pi OS from sd card to confirm everything was working and it was. Put the same image on USB, tried to boot from that. It would get reasonably far but then complain of EXT4 errors, which was strange. Reset the image and tried again, same behaviour.

Updated to "Feb 16 2021" via the sdcard raspbian, reset the image on the USB stick again, it booted cleanly this time. So can confirm USB boot works with their distro but did require updating the eeprom for loading that USB stick.

Attempted to boot with nixos-sd-image-21.05pre287596.ae1c8ede09b-aarch64-linux.img from sd card, which works, albeit with the same Bluetooth errors as listed in the bug description. Attempting to boot with the USB stick however results in being stuck at the loading kernel screen forever:

This occurs to all three USB sticks with no difference other than the printed current device information.

---

• Variant and board revision: Raspberry Pi 4 Model B Rev 1.4 (8GB)
• EEPROM firmware version: tested with "Sep 3 2020"

Using any of the USB sticks as before has the exact same "Starting kernel ..." stopping position.

Via sd card, it will boot all the way to bash, and show the bluetooth errors same as the 2GB model, but also USB doesn't work at all (as reported in another issue).

I haven't changed the eeprom version on this one yet because I don't want to interfere with testing of the USB fixes yet, but tested this 8GB model just for differential comparison.

[proofconstruction]

This issue #114157 (comment) seems to indicate that USB boot has been accomplished, though I had no luck following the same steps.

@LucaFulchir could you post your device & firmware info here as another data point?

[samueldr]

It is known that "board revision 1.4", introduced with the 8GB variant changes how USB works. This is why we can see an SD boot work, and USB does not work once booted in the Linux system. That is another issue to be investigated, but is nice having confirmation that on the equivalent setup with another board variant USB works in Linux.

---

Attempting to boot with the USB stick however results in being stuck at the loading kernel screen forever:

This is pretty much expected, as required modules are not present in the built image; the PR adding them has been merged only yesterday. I probably should have stated in the initial post to wait a bit for testing "full" USB boot, or build your own images 😄.

[bbqsrc]

Happy to build my own image, though I do not know how. 😄 If you can provide instructions, I can help as needed with the current rpi4 related issues.

[samueldr]

In a Nixpkgs checkout, on an aarch64 machine. (E.g. your raspberry pi)

 $ nix-build nixos/release.nix -A sd_image_new_kernel.aarch64-linux

If there's not much to build, only assembling the final bits, it won't be as long. It probably will still take a while if it's building on an I/O starved storage media (e.g. SD cards).

[bbqsrc]

@samueldr rather a bit of a chicken and egg problem here, as I was trying to set up an aarch64 build system on the rpi... haha. I do have an amd64 nixos system though and can cross-compile if you can explain what I need to do :D

[samueldr]

Since cross-compilation doesn't work with workarounds still, I wouldn't recommend it for "general" testing purposes. It is usable, though, when you compare in a before/after situation.

Right now when I cross-compile an SD image or ISO image I use my cross-system repository. It assumes either --arg pkgs to be used, or <nixpkgs> to resolve to the Nixpkgs you want to test.

I assume that, for now, waiting on a build will be easier, and more valuable.

[proofconstruction]

#121720 this PR made it into a build here https://hydra.nixos.org/build/142755927

I have the image (nixos-sd-image-21.05pre287941.54fa457b258-aarch64-linux.img) from that build running on USB with everything working as expected.

[samueldr]

@proofconstruction sorry if I'm a bit pedantic, but "running on USB" on what hardware/eeprom combination exactly?

[proofconstruction]

@samueldr Pedantic is good! My apologies - same configuration as my first post in the thread:
Raspberry Pi 4B 8GB, board revision 1.4
EEPROM bootloader version: Thu 29 Apr 16:11:25 UTC 2021 (1619712685)
USB storage type: Samsung 860 Evo m.2 SATA 256GB connected via ASMedia ASM1153E bridge

So, the latest change works as intended on the 8gb model pi4.

[bbqsrc]

Can confirm USB boot works with nixos-sd-image-21.05pre287941.54fa457b258-aarch64-linux on 2GB and 8GB rpi4. No eeprom update was necessary, surprisingly. :)

[samueldr]

I'm closing this as solved booting entirely from USB should be working for at the very least the main boards from the Raspberry Pi 4 family. Status for CM4 and 400 are unknown still, but it is also unclear generally speaking what the current state is.

[natto1784]

The SD image works with my USB stick on Raspberry PI 4 4GB but after installing nixos and rebooting, it gets stuck on the "Starting Kernel".
Also some error "mmc1 timeout waiting for hardware interrupt" keeps getting spammed in the the first time as well as after rebooting (during the "Starting Kernel" screen), I think it has something to do with my USB stick.
However the former issue still remains unsolved

[samueldr]

@natto1784 while it won't be "optimal", try adding <nixpkgs/nixos/modules/profiles/all-hardware.nix> to your imports.

This is highly likely because of two reasons that it "stays stuck" at "Starting Kernel...".

1. The video driver is not part of stage-1, so there's no way to show anything on the display
2. Some drivers required for USB are not part of stage-1, so it cannot mount the rootfs to continue into stage2

So I suspect that right now it's kind of technically working, but getting stuck unable to continue booting.

As a general debugging note "Stuck at Starting kernel..." generally means the kernel has started.

[natto1784]

Yeah that fixes it, I don't think it is a problem about video drivers as i don't have any display servers enabled. I'll try pinning down what drivers I exactly need when i get time. Thanks

[samueldr]

I don't think it is a problem about video drivers as i don't have any display servers enabled

On the Raspberry Pi family of hardware, with the mainline kernel, the console VTs require the vc4 kernel to be loaded IIRC. So the console won't be shown without that kernel driver.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/install-nixos-on-raspberry-pi-4-ssd/22788/2

[LookoutHill]

@nixos-discourse Thanks for the link back! Looking at the screenshots above my issue is surely related.

I'll try one of the build's listed above as working. I guess I could also look at getting another USB/SATA3 adapter, but my Startech USB3S2SAT3CB was purchased because it enabled the Ubuntu aarch64 image to boot properly from an SSD and run stably.

I could use access to someone who knows where to find the u-boot source package that's included in the SD image so I can compare it with the one used by Ubuntu. And it would be helpful to know if there's anything in the /boot/config.txt file that I could add to increase the logging verbosity.

[n8henrie]

while it won't be "optimal", try adding <nixpkgs/nixos/modules/profiles/all-hardware.nix> to your imports.

@samueldr

I did a somewhat painful manual bisect process of all the kernel modules in all-hardware.nix and found that I only require these 3 in availableKernelModules to get a successful USB boot on my Pi4:

boot.initrd.availableKernelModules = [
        "uas"
        "pcie-brcmstb"
        "reset-raspberrypi"
      ];

EDIT: Not sure if this makes a difference of the top of my head, but I'm booting from a compressed BTRFS subvolume, so this could also be pulling in other kernel modules

boot.initrd.kernelModules = ["zstd" "btrfs"];

EDIT2: And for some reason it stops working if I use the rpi 4 kernel (boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4 or what have you), and a more recent kernel than default is required by the nixos-hardware rpi4 module. ¯\_(ツ)_/¯

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rpi4-cant-usb-boot-from-btrfs-with-rpi-kernel/28668/1