Several shared libraries on 32-bit Bookworm are linked with 4K page alignment producing fails in systemd-nspawn

[gitbls]

While many of the shared libraries in 32-bit Bookworm are linked with 16K page alignment, several seem to be linked with 4K page alignment. This causes issues, for instance, when run in systemd-nspawn on a Pi5 with 16K pages.

I did not check the whole system, but found several here:

cd  /usr/lib/arm-linux-gnueabihf/
for i in *.so* ; do echo $i ; readelf -l $i | grep LOAD ; done | less

When programs needing these 4K page-aligned libraries are run inside a systemd-nspawn, they fail. For instance

p51/ssd/work$ uname -a
Linux p51 6.1.0-rpi4-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.54-1+rpt2 (2023-10-05) aarch64 GNU/Linux
p51/ssd/work$ sudo sdm --explore 2023-10-10-raspios-bookworm-armhf-lite.img 
* Mount IMG '2023-10-10-raspios-bookworm-armhf-lite.img'
mount: /dev/loop0p2 mounted on /mnt/sdm.
mount: /dev/loop0p1 mounted on /mnt/sdm/boot.
* Enter IMG '2023-10-10-raspios-bookworm-armhf-lite.img'
-bash: error while loading shared libraries: libtinfo.so.6: ELF load command address/offset not page-aligned
umount: /mnt/sdm/boot unmounted
umount: /mnt/sdm unmounted

This can be easily demonstrated with the following program:

#!/bin/bash
#
# $1: name of IMG to mount and systemd-nspawn into
# $2: mount point (/mnt/$2)
#

function cleanup() {
    if [ "$mpt" != "" ]
    then
        umount -v /mnt/$mpt/{boot,}
        losetup -d $loopdev
        [ "$mpt" != "sdm" ] && rm -rf /mnt/$mpt
        mpt=""
    fi
}

function errexit() {
    echo -e "$1"
    cleanup
    exit 1
}

function doexit() {
    cleanup
    exit 0
}

img="$1"
mpt="$2"
[ "$img" == "" ] && errexit "? Missing IMG filename"
[ "$mpt" == "" ] && mpt="sdm"
[ -f $img ] || errexit "? File '$img' cannot be accessed"
sdm --ppart $img
trap "doexit" EXIT
mkdir -p /mnt/$mpt/boot
loopdev=$(losetup --show -P -f $img)
mount -v ${loopdev}p2 /mnt/$mpt
mount -v ${loopdev}p1 /mnt/$mpt/boot
systemd-nspawn --directory /mnt/$mpt < $(tty)
trap EXIT
umount /mnt/$mpt/{boot,}
losetup -d $loopdev

[pelwell]

This is a known problem with 16KB pages. If your example is an important use case, put kernel=kernel8.img to revert to 4KB pages.

[gitbls]

Thanks, Phil. I'm aware of the kernel=kernel8.img fix and am already working out a way to inform my users of this workaround should they bump into it. I don't have any control over what IMGs they work on, of course.

Just noting for possible future consideration that Docker users could run into this issue as well if they try to use a 32-bit Docker container on a 64-bit system. I don't know if Docker always uses it, but it appears that there are people that want to use systemd-nspawn with Docker. See, for example, section 5.8 here: https://wiki.archlinux.org/title/Systemd-nspawn

Also, out of curiosity, why are 2/3 of the shared libraries in that directory linked with 16K alignment and 1/3 of them are not? Are there issues with the "not" ones if they are 16K linked, or is it a matter of scarce resources not being available to update the remaining 1/3 of the library build files?

Thanks for any insights you are able to share on this.

[charles2910]

It is a pretty broad issue, I couldn't even execute the 32bit ls on raspbian after dpkg --add-architecture armhf and installing its dependencies (plus libc6:armhf).

After setting kernel=kernel8.img things start working.

(as a note, it was pretty difficult to find this issue, took my a few days to find this answer)

[CendioOssman]

Any update on this? This is preventing us from running programs that are only available in armhf packaging:

error while loading shared libraries: libpthread.so.0: ELF load command address/offset not page-aligned

[popcornmix]

The solution is to use kernel=kernel8.img if you want to run 32-bit binaries.

There is no other solution.