The Device Tree isnt passed from start4.elf to the UEFI firmware RPI_EFI.fd.

[Schiddik]

Hello everyone,

currently Iam doing my Masters in Darmstadt, Germany.

Please believe me when I say that i DEEPLY appreciate any answer to my questions!!! Thanks to You in advance!
I will enumerate my questions so that they can be referenced easily.

My task is to deploy Kubernetes and Ceph onto a cluster of 60 Raspberry Pi Compute Blades.
The Compute Blades use the Compute Module 4 (CM4). Most of the Compute Blades dont have persistent storage.

Thus, my task is to boot the CM4s via PXE.
I have already booted the CM4 via the "normal" PXE bootflow.
I will now share my understanding of the "normal" PXE bootflow.
I still have a lot of gaps in my understanding. I would highly appreciate if You could fill these gaps with Your knowledge.

The initial rootfilesystem was built by Buildroot. It has a custom /init script.
The kernel (vmlinuz.gz) doesnt have an UEFI stub and is generated by the Raspberry PI Imager (Ubuntu 24.04).
Eventhough somewhere on the internet I read that every Ubuntu kernel has a UEFI stub by default. But more on that later.

1. My first question is if the L2 cache is located in the GPU or in the ARM chip.
2. How does the second stage bootloader (SSB) (formerly bootcode.bin) know about the onboard NIC? Are the MMIO regions and the interrupt lines and the capabilities compiled into the SSB?
3. How are the files like meta-data and network-config passed into the kernel?
4. How does the kernel accept this information or these files? Doesnt the kernel need an argument in its entry point for this kind of information? Or are these files copied into the initial rootfilesystem by chance?
5. We all know the standard signatures of programs: int main(argc, argv). Which parameters does the entry point of a NON-UEFI kernel take as arguments? As far as i know one of the needs to be the address of the device tree. Is that correct?
6. What are the program signatures of start4.elf, RPI_EFI.fd, GRUB, a NON-EFI kernel, and a kernel with an EFI stub?
7. What is the fixup4.dat file? What does it contain? When booting via UEFI, the start4.elf "fails to apply fixups". What could be the cause?
8. What is the memory layout of the CM4? Does the start4.elf file need to be loaded to a specific address? Does the kernel need to be loaded to a specific address?
9. How does the kernel know which memory areas it is allowed to use?
10. Is start4.elf unloaded from memory when it passes control to the kernel? How is the stack of the start4.elf program removed?

In this "normal" PXE boot flow the kernel recognizes the NIC and creates an eth0 interface, which I then use to connect via SSH.

10.1) This question is not related to question 10.
My current root file system in the "normal" PXE bootflow doesnt have the apt package manager. Thus, iam unable to install new software (eg. the dependencies for kubernetes and ceph).
Would You have an idea on how to add apt to the root file system? The root filesystem was built with buildroot and they only provice the opkg package manager.
My final goal is to install kubernetes and ceph via ansible.

Now coming to the UEFI bootflow:

Additionally I was able to boot the Raspberry Pi CM4 (on the Compute Blade) via an UEFI compliant firmware.
To get a hold of the UEFI compliant firmware that would run on the Compute Blade I used this repository: compute-blade-cm4-uefi (https://github.com/uptime-industries/compute-blade-cm4-uefi)
That repository is a fork of the pftf/RPi4 (https://github.com/pftf/RPi4) repository.
Apparently the pftf/RPI4 repository "contains installable builds of the "EDK2 Raspberry Pi 4 UEFI firmware" (https://github.com/tianocore/edk2-platforms/tree/master/Platform/RaspberryPi/RPi4)

This "EDK2 Raspberry Pi 4 UEFI firmware" repo contains the RPI4.dsc and RPI4.fdf files.
11. What do these files mean and are they standardized in the UEFI specification?

The generated folder has the /firmware/brcm folder which contains files like brcmfmac43455-sdio.Raspberry.
12. What are these files used for? When are they used? By whom?

This is how the directory structure of the UEFI bootflow looks like:

The generated config.txt files contains the armstub=RPI_EFI.fd parameter.

13.1 According to my understanding, an armstub is the software that the start4.elf program executes after it is done with all its setup work.
start4.elf then passes control to the armstub.
I have read on the internet that an armstub is a small executable that initializes an ARM CPU.
Some of its tasks are writing specific values to predefined (by the ARM spec) control registers.
This could result in enabling the branch predictor or the MMU.
13.2 Is this understanding correct?
13.3 The input parameters of the armstub are three registers r0,r1,r2. One of these should get the address of the device tree. Is this correct?

The trick of using UEFI as firmware is that start4.elf is "tricked" into calling an armstub, which internally is a UEFI compliant firmware (RPI_EFI.fd) that was generated by https://github.com/tianocore/edk2-platforms/tree/master/Platform/RaspberryPi/RPi4

14. Is this behaviour also implemented in RPI_EFI.fd? Does it also expect the device tree in one of these parameters? And does it read the device tree?

The RPI4 repo states that:
Device Tree boot of OSes such as Linux may not work at all.
For this reason, the provision of a Device Tree is disabled by default in the user settings, in order to enforce ACPI boot.

15. Does this mean that RPI_EFI.fd (despite being an armstub) doesnt read the Device Tree that it doesnt supply the DTB Table in the SystemTable (which is passed to the Grub2 Bootloader) ?

16.How can RPI_EFI.fd use the NIC if the information about the NIC (MMIO regions, interrupt lines etc.) are not passed via the device tree? What information does RPI_EFI.fd need to have to communicate with the NIC? Is that information compiled into the RPI_EFI.fd firmware?

The UEFI boot flow is described below:

17. What consitutes to hardware initialization? Could You give some examples of what You classify under hardware initialization? Does arm have specific registers that have to have initial values at start up? Are internal components (eg. branch predictors, MMU, TLB, Interrupt Vector Table etc...) always configured via control registers?
    17.1 What hardware initialization steps do the first stage bootloader, the second stage bootloader, and start4.elf really do?
18. In the picture above: How does the grub2 bootloader pass the device tree, the kernel parameters and the initial rootfilesystem to the kernel?
    Apparently, the SystemTable has an entry for the DTB Table.
    So a possible flow for the device tree could be this one: start4.elf reads the device tree and adds the overlays (based on the information in config.txt). Then it passes the address of the device tree to the RPI_EFI.fd armstub via the "r2" register for example. RPI_EFI.fd creates a DTB Table out of that information an puts that table in the SystemTable. Then grub2 can access the DTB Table. grub2 can then pass the DTB Table to the Kernel via the SystemTable too because the kernel has an EFI stub.
    Are the kernel parameters read by RPI_EFI.fd from the cmdline.txt and passed to the grub2 bootloader via the UEFI Load_Options?
    This references back to the question 6.
    I would be highly interested in the signature of the entry points of start4.elf, RPI_EFI.fd, grub2, and the kernel (vmlinuz)
19. A remaining question in this context is: How does grub2 pass the location of the rootfilesystem (initrd) to the kernel? Is this information located in the SystemTable or the ImageHandle of the EFI entry point?

However, I do have a problem!

In this UEFI boot flow the fixups are not applied properly. This wasnt the case in the "normal" PXE boot flow (described above).
20. What could be the reason for this and does this pose a problem? What is the actual task of the fixup4.dat file?

HUGE problem: The RPI_EFI.fd firmware doesnt recognize the MAC address of the internal NIC. However, it is able to use the NIC. The source address of the outgoing ethernet frames is 00:00:00:00:00:00. Iam really surprised that the RPI_EFI.fd firmware "knows" about the NIC and has the correct information about the MMIO regions and the Interrupt Lines etc... However, it cant read the MAC address of the NIC?! This seems obscure to me.
21. Does the NIC store its MAC address in a persistent hardware register? And how would the RPI_EFI.fd firmware access this information?

I would highly appreciate any solutions/ideas to eliminate this problem!

The start4.elf program can properly access the NIC and it also knows the MAC address of the NIC.

23. The pictures above show the SystemTable of the RPI_EFI.fd firmware. The DTB Table points to address 0. Does this mean that start4.elf didnt pass the address of the device tree to the RPI_EFI.fd armstub?

The RPI_EFI.fd firmware download the grub2 bootloader via TFTP. grub2 then also downloads the Linux kernel and the initial rootfilesystem via TFTP. grub2 uses the networking capabilities of the RPI_EFI.fd firmware via the supplied boot services in the SystemTable.
24. Is this correct?

25. What does the message "Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path mean"?

26. What does "Using DTB from configuration table" mean? I thought that the DTB in the SystemTable was a Null pointer. How can that table be used as a Device Tree?

As You can see in the picture above, the kernel parameters from the cmdline.txt are not passed to the kernel. In my setup the cmdline.txt file contains a kernel parameter.
This can mean many things. start4.elf didnt pass these kernel parameters to RPI_EFI.fd OR RPI_EFI.fd didnt pass them to the grub2 bootloader OR the grub2 bootloader didnt pass them to the kernel.

27. Does start4.elf pass the kernel parameters to RPI_EFI.fd or does RPI_EFI.fd read those values by itself? And if it reads them by itself, how does it know the location of the cmdline.txt file?

28. What could be the possible cause for the fact, that the kernel parameters are not passed properly?

29. The picture above also shows that there is only the loopback (lo) interface created by the kernel. Does this mean that the kernel didnt find any ethernet devices and thus didnt bother creating an eth0/enp0s3 interface (because he couldnt bind these interfaces to any physical device) ?

30. Does this mean for certain, that the kernel didnt receive a device tree?

I have tried adding this line to the grub.cfg file: linux (tftp)/vmlinuz dtb=/bcm2711-rpi-cm4.dtb
31. Would this allow me to put the device tree into the initial root filesystem, which could then be used by the kernel?

Optional questions:
32. How does star4.elf pass the address of the device tree to an armstub?
33. Does start4.elf always call an armstub even if the armstub parameter in config.txt isnt specified? If not specified, does start4.elf call a default stub?
34. In the UEFI boot flow, the armstub RPI_EFI.fd downloads the grub2 bootloader, which then downloads the kernel and intird. How would a different kind of armstub know the address of kernel that it needs to execute? Is the address of the kernel standardized and thus doesnt need to be passed as a parameter into the armstub?
35. How does the second stage bootloader know the MAC of the internal NIC? Does the NIC have persistent storage that stores the NIC? I heard somewhere that the MAC is computed based on the serial number. If thats so, the where is the serial number of the CM4 board stored? Possibly in the EEPROM or the internal ROM in the GPU?
36. How does RPI_EFI.fd know about the NIC? Are the MMIO regions and interrupt numbers compiled into it?
37. This is a very vague question whose scope is propably way too big: Which memory regions are used by the 1st, 2nd stage bootloader, start4.elf, RPI_EFI.fd, grub, the kernel?
38. How would start4.elf pass control to a “normal” armstub? Is it the responsibility of the armstub to clean the leftover memory that the start4.elf used? How does it know where the start4.elf was loaded to and how much stack space it required?
39. Does the RPI_EFI.fd firmware “claim” all memory? Meaning, that it acts as if all of the SDRAM belongs to it. And does RPI_EFI.fd only keep the region where itself is located (the location where the RPI_EFI.fd firmware was loaded needs to be known beforehand in this case).

Iam DEEPLY grateful for every information I can get! Thank You in advance:)

[einsteinagogo]

This is excellent, let me have a full read and catchup.

I use TFTP to obtain the UEFI files from tftp server, by changing the bootloader order - these files can be configured to PXE UEFI BOOT from tftp server.

I then use iSCSI and install in this case ESXi ARM to an iSCSI LUN,

Does this help ?

[Schiddik]

@einsteinagogo Thank You for Your answer!!!

Unfortunately, I do not exactly understand what You're referring to.

Do You use the UEFI files generated by this Repository?

And is Your kernel able to detect the NIC? Are You able to communicate over the network?

Are you using these values as the location for the device tree?

[einsteinagogo]

This repo here, is a Fork (copy) and been modified from the original here

https://github.com/pftf/RPi4

I'm using this repo above link provided, and I have no issues with communications via the network.

[Schiddik]

Are You using a bootlaoder as an intermediary between the kernel and the UEFI firmware?
Are You using grub2 for example?

[einsteinagogo]

The steps I use are

1. Bootloader, (reprogrammed to default Network Boot) uses TFTP to obtain the "firmware"

e.g. the config.txt, start4.elf, RPI_EFI.fd

2. UEFI "BIOS" code loads configured for iSCSI, I then use PXE in UEFI again to start the installation from the same TFTP server, this boots the ESXi ARM install and installs to the iSCSI LUN.

3. RPi CM4 re-boots and boots from iSCSI LUN

[Schiddik]

@einsteinagogo Were You able to go through my questions? :)

[cabeljunky]

Hello @Schiddik ,

You have a nice detailed explanation on the boot process.
For my situation I have use TFTP server like described here: https://www.sidero.dev/v0.6/guides/rpi4-as-servers/#persisting-changes.

I have used netboot.xyz for the PIXE boot and the TFTP

The main issue that I have is that when using this approach I need for every RB Pi a own RPI_EFI.fd. As it contains the mac address of the device. (described here: pftf/RPi4#59) ) as this might explain how it gets the mac address

[einsteinagogo]

I did read somewhere, there are tools which can write to the RPI_EFI.fd, because it's the variables which write back to the RPI_EFI.fd which corrupts it, this would then be possible to create RPI_EFI.fd files for individual RPi, I'm going to look at these tools later next month, and see if they work as described.

[cabeljunky]

@einsteinagogo that would be a nice option.

Do you refer to this topic: pftf#6 where you then can store the value outside the RPI_EFI.fd

[einsteinagogo]

@cabeljunky it wasn't that one, I'll have to find the references.