[Raspberry Pi 4] Add an option to disable "Low Peripheral" mode

[hvenev]

According to the BCM2711 datasheet, the ARM core can use one of two physical address layouts:

• Full 35-bit Address Map
• Low Peripheral mode

In "Low Peripheral" mode, peripherals are mapped over the last 64M of RAM. If that RAM can otherwise be used, a firmware update that lets people disable "Low Peripheral" mode would literally let them "download more RAM".

[pelwell]

This is something which would probably already have been done were it not for the fact that the 32-bit ARM kernels don't support the "spin-table" mechanism for waking the secondary CPU cores (although with enough effort I'm sure it could be added). Currently we use some mailbox registers to synchronise the starting of the secondary cores, and in high-peripherals mode these would no longer have a 32-bit physical address. We (I) don't want to have to add LPAE support to the 32-bit stub (<100 instructions that run before entering the kernel), meaning that the mailboxes would be out of reach of the stub.

LPAE is unnecessary on a 64-bit ARM, so we could reserve the high-peri support for 64-bit builds - wait until the 5.4 switchover is complete and this may happen.

[andreiw]

Is there a way to switch the layout dynamically from the Arm cores? e.g. something that low level kernel/bootloader/firmware entry code could do?

Or could it become a config.txt option?

[pelwell]

The controlling register requires a secure write, which only the VPU can do. However, there is a way you can try it.

Firmware releases since the 26th March 2020 have supported a new config.txt setting - arm_peri_high. If non-zero it writes the appropriate value to the magic register and the peripherals move to the high location. It will also set it to 1 automatically if it detects a DTB where the "/soc" DT node has a "ranges" property that is longer than 4 cells (16 bytes) and where the second cell/word is non-zero. Using two words for ARM addresses, the high word is 0 in low-peri mode and 4 in high-peri mode.

Three important caveats:

1. This currently only works on a 64-bit kernel, for the reasons outlined above.
2. If you get the DTB wrong it will not boot.
3. This feature comes with no warranty - it's in there for testing purposes.

[hvenev]

What about the other DT nodes like scb, v3dbus, and emmc2bus? They also seem use fc000000-ffffffff in "ranges". I suppose they also need to be fixed.

Also what about the 64-bit stub? It appears to write to hardcoded ARM-local addresses (LOCAL_CONTROL, LOCAL_PRESCALER, and GIC stuff).

[pelwell]

All those things will need changing, but the source code and tools are available. This is not for the faint-hearted, and it is not supported.

[hvenev]

Is there anything other than the stubs that is executed before the kernel and relies on fixed addresses?

[pelwell]

No - the stubs are by definition the first instructions executed by the ARMs.

[hvenev]

OK, it seems to work. I tested ethernet, USB 3, and display output.

However, the firmware still doesn't report the last 64M of RAM as usable:

$ od -t x4 --endian=big /sys/firmware/devicetree/base/memory@0/reg 
0000000 00000000 00000000 28000000 00000000
0000020 40000000 bc000000
0000030

$ cat /proc/iomem
00000000-27ffffff : System RAM
  00000000-00000fff : reserved
  00080000-00beffff : Kernel code
  00bf0000-00d4ffff : reserved
  00d50000-00e7bfff : Kernel data
  13c00000-27bfffff : reserved
  27fe4000-27feffff : reserved
40000000-fbffffff : System RAM
  f7000000-faffffff : reserved
  fb902000-fb961fff : reserved
  fb962000-fbf62fff : reserved
  fbf63000-fbfb6fff : reserved
  fbfb9000-fbfbafff : reserved
  fbfbb000-fbfbdfff : reserved
  fbfbe000-fbffffff : reserved
47d500000-47d50930f : 47d500000.pcie
47d580000-47d58ffff : 47d580000.ethernet
  47d580e14-47d580e1c : unimac-mdio.-19
47e007000-47e007aff : 47e007000.dma
47e00a000-47e00a023 : 47e100000.watchdog
47e00b840-47e00b87b : 47e00b840.mailbox
47e00b880-47e00b8bf : 47e00b880.mailbox
47e100000-47e100113 : 47e100000.watchdog
47e101000-47e102fff : 47e101000.cprman
47e104000-47e104027 : 47e104000.rng
47e200000-47e2000b3 : 47e200000.gpio
47e201000-47e2011ff : serial@7e201000
  47e201000-47e2011ff : 47e201000.serial
47e204000-47e2041ff : 47e204000.spi
47e215000-47e215007 : 47e215000.aux
47e300000-47e3000ff : 47e300000.mmcnr
47e340000-47e3400ff : 47e340000.emmc2
47e600000-47e6000ff : 47e600000.firmwarekms
47e804000-47e804fff : 47e804000.i2c
47ec00000-47ec03fff : 47ec00000.v3d
47ec04000-47ec07fff : 47ec00000.v3d
47ec11000-47ec1101f : 47e100000.watchdog
47ef00000-47ef0000f : 47ef00000.clock
600000000-603ffffff : pcie@7d500000
  600000000-6000fffff : PCI Bus 0000:01
    600000000-600000fff : 0000:01:00.0
      600000000-600000fff : xhci-hcd

[pelwell]

That makes sense. The code that generates the contents of the memory node is unaware of the arm_peri_high flag, so is always carving out the final 64MB for the peripherals. That's an easy fix.

[pelwell]

I have a patch for internal testing which is simple enough to be correct by inspection, but if you share your .dts/.dtb file I can test it (along with your stub PR).

[hvenev]

I'm working on making it usable. Give me 15 minutes.

Now that you've explained how high peripheral mode can be enabled, should I close this issue and open another one for the remaining memory and one for the stub?

[pelwell]

You've got the PR for the stub which doubles as an issue, and the memory patch is already well under way, so at this point I think more issues would just get in the way. Thanks for asking, though.

[pelwell]

I've also got it booting now. The firmware is auto-detecting the high-peri DTB and setting the config flag:

pi@raspberrypi:~$ vcgencmd get_config arm_peri_high
arm_peri_high=1

It's also declaring the memory correctly:

pi@raspberrypi:~$ od -t x4 --endian=big /sys/firmware/devicetree/base/memory@0/reg
0000000 00000000 00000000 37000000 00000000
0000020 40000000 c0000000
0000030

The iomem info also looks good (N.B. it was all zeroes until I randomly sudo'd it):

pi@raspberrypi:~$ sudo cat /proc/iomem
00000000-36ffffff : System RAM
  00000000-00000fff : reserved
  00080000-00dbffff : Kernel code
  00dc0000-00eaffff : reserved
  00eb0000-0111cfff : Kernel data
  1ec00000-2ebfffff : reserved
  2eff4000-2effffff : reserved
  33000000-36ffffff : reserved
40000000-ffffffff : System RAM
  fb000000-feffffff : reserved
  ff127000-ff186fff : reserved
  ff187000-ff787fff : reserved
  ff788000-ff803fff : reserved
  ff806000-ff807fff : reserved
  ff808000-ff80afff : reserved
  ff80b000-ffffffff : reserved
47d500000-47d50930f : 47d500000.pcie
47d580000-47d58ffff : 47d580000.ethernet
  47d580e14-47d580e1c : unimac-mdio.-19
47e007000-47e007aff : 47e007000.dma
47e00a000-47e00a023 : 47e100000.watchdog
47e00b840-47e00b87b : 47e00b840.mailbox
47e00b880-47e00b8bf : 47e00b880.mailbox
47e100000-47e100113 : 47e100000.watchdog
47e101000-47e102fff : 47e101000.cprman
47e104000-47e104027 : 47e104000.rng
47e200000-47e2000b3 : 47e200000.gpio
47e201000-47e2011ff : serial@7e201000
  47e201000-47e2011ff : 47e201000.serial
47e215000-47e215007 : 47e215000.aux
47e215040-47e21507f : serial
47e300000-47e3000ff : 47e300000.mmcnr
47e340000-47e3400ff : 47e340000.emmc2
47ec00000-47ec03fff : 47ec00000.v3d
47ec04000-47ec07fff : 47ec00000.v3d
47ec11000-47ec1101f : 47e100000.watchdog
47ef00000-47ef0000f : 47ef00000.clock
600000000-603ffffff : pcie@7d500000
  600000000-6000fffff : PCI Bus 0000:01
    600000000-600000fff : 0000:01:00.0
      600000000-600000fff : xhci-hcd

At this point I need to:

1. Merge the armstubs patch.
2. Get the firmware to load the -highperi variant if arm_peri_high=1 on a 2711.
3. Refactor High peripheral mode linux#3566 to put the highperi diffs in an overlay - I'd prefer to avoid having two Pi 4 .dtbs.

[hvenev]

It's also declaring the memory correctly:

pi@raspberrypi:~$ od -t x4 --endian=big /sys/firmware/devicetree/base/memory@0/reg
0000000 00000000 00000000 37000000 00000000
0000020 40000000 c0000000
0000030

Is that with the firmware fix you talked about?

...

At this point I need to:
...
3. Refactor raspberrypi/linux#3566 to put the highperi diffs in an overlay - I'd prefer to avoid having two Pi 4 .dtbs.

I can do that if you prefer.

[pelwell]

Is that with the firmware fix you talked about?

Yes. The patch is merged now, so will be in the next release.

I can do that if you prefer.

That would be great, if you have the time.

[pelwell]

With 3 taken care of (thanks), the list is now:

1. Merge the armstubs patch.
2. Get the firmware to load the -highperi variant if arm_peri_high=1 on a 2711 in 64-bit mode.
3. Done.
4. Get the firmware to load the highperi overlay automatically if arm_peri_high=1.

All are fairly small tasks, and should be done in the next few days.

[hvenev]

I tested the latest firmware update and it fixes the remaining items on the list.

Note that for me pcie works (a USB 3 HDD is probed and mounted) even if I keep the commit that removes one of the ranges, and it appears that the inbound window is placed at address 0:

[    0.222770] brcm-pcie 47d500000.pcie:   IB MEM 0x0000000000..0x00bfffffff -> 0x0000000000

If I set total_mem=2048, it still works:

[    0.222735] brcm-pcie 47d500000.pcie:   IB MEM 0x0000000000..0x007fffffff -> 0x0000000000

I think the reason is that the inbound window setup uses dma-ranges, not ranges.