Cross-compilation to aarch64 fails because of __arm__ ends up also defined.

[cveilleux]

System information

Type	Version/Name
Distribution	Yocto
Distribution Version	Warrior (2.7)
Linux Kernel	4.19.80
Architecture	aarch64
ZFS Version	0.8.4
SPL Version	0.8.4

Describe the problem you're observing

Cross-compilation using the Yocto build system works for arm, but fails when targetting aarch64.

Describe how to reproduce the problem

The autotools class of yocto will produce what looks like a reasonable setup for cross-compiling zfs.

The ./configure step is called with:

./configure --build=x86_64-linux --host=aarch64-poky-linux --target=aarch64-poky-linux \
 --prefix=/usr --exec_prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin \
 --libexecdir=/usr/libexec --datadir=/usr/share --sysconfdir=/etc \
 --sharedstatedir=/com --localstatedir=/var --libdir=/usr/lib --includedir=/usr/include \
 --oldincludedir=/usr/include --infodir=/usr/share/info --mandir=/usr/share/man \
 --disable-silent-rules \
 --with-libtool-ysroot=/home/cveilleux/git/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot \
 --disable-static \
 --with-linux=/home/cveilleux/git/os/poky/build/tmp/work-shared/raspberrypi4-64/kernel-source \
 --with-linux-obj=/home/cveilleux/git/os/poky/build/tmp/work-shared/raspberrypi4-64/kernel-build-artifacts

This sets both the host and target and the libtool root if fully setup with all the requirements compiled for aarch64.

You can see my bitbake recipe here:

https://gist.github.com/cveilleux/54961ccc41071e8aee8c19b69fcba78f

Include any warning/errors/backtraces from the system logs

The compilation on aarch64 will crash on the first input file with:

 | ../../aarch64-poky-linux-libtool  --tag=CC --silent  --mode=compile aarch64-poky-linux-gcc  -mcpu=cortex-a72+crc+crypto -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -fstack-protector-strong  -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot -DHAVE_CONFIG_H -include ../.ERROR: Task (/builds/sentinel-c/os/meta-chinook/recipes-kernel/zfs/zfs_0.8.4.bb:do_compile) failed with exit code '1'
 ./zfs_config.h -I../../include -I../../lib/libspl/include  -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_LARGE_STACKS=1 -DTEXT_DOMAIN=\"zfs-linux-user\" -DLIBEXECDIR=\"/usr/libexec\" -DRUNSTATEDIR=\"/var/run\" -DSBINDIR=\"/usr/sbin\" -DSYSCONFDIR=\"/etc\" -UDEBUG -DNDEBUG   -std=gnu99 -Wall -Wstrict-prototypes -fno-strict-aliasing -fno-omit-frame-pointer    -Wframe-larger-than=4096 -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0=/usr/src/debug/zfs/0.8.4-r0                      -fdebug-prefix-map=/builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0=/usr/src/debug/zfs/0.8.4-r0                      -fdebug-prefix-map=/builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot=                      -fdebug-prefix-map=/builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot-native=  -c -o avl.lo ../../module/avl/avl.c
 | In file included from /builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot/usr/include/endian.h:36,
 |                  from /builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot/usr/include/sys/types.h:193,
 |                  from ../../lib/libspl/include/sys/types.h:38,
 |                  from ../../module/avl/avl.c:101:
 | /builds/sentinel-c/os/poky/build/tmp/work/raspberrypi4_64-poky-linux/zfs/0.8.4-r0/recipe-sysroot/usr/include/bits/endian.h:40:10: fatal error: bits/endian-32.h: No such file or directory
 |  #include <bits/endian-32.h>
 |           ^~~~~~~~~~~~~~~~~~
 | compilation terminated.
 | Makefile:560: recipe for target 'avl.lo' failed
 | make[3]: *** [avl.lo] Error 1

As you can see, libc's bits/endian.h is trying to import 32 bits headers, which do not exist on aarch64. Something is wrong.

Investigation

I've looked into the ifdefs of endian.h. The relevant part:

#if defined (__bpf__)
#define __MHWORDSIZE                    64
#elif defined (__arm__)
#define __MHWORDSIZE                    32
#elif defined (__aarch64__) && defined ( __LP64__)
#define __MHWORDSIZE                    64
#elif defined (__aarch64__)
#define __MHWORDSIZE                    32
#else
#include <bits/wordsize.h>
#if defined (__WORDSIZE)
#define __MHWORDSIZE                    __WORDSIZE
#else
#error "__WORDSIZE is not defined"
#endif
#endif

The ifdefs looks for arm first, then for arch64.

With some debugging I've concluded that zfs ends up defining both arm and aarch64, and the ifdef orders ends-up wrongly selected 32 bit wordsize.

Crappy hack to get it to compile

I found that include/spl/sys/isa_defs.h and lib/libspl/include/sys/isa_defs.h in zfs source ends up defining both arm and aarch64 here:

https://github.com/openzfs/zfs/blob/master/lib/libspl/include/sys/isa_defs.h#L113-L115

and here:

zfs/include/os/linux/spl/sys/isa_defs.h

Lines 101 to 103 in 4254e40

	#if !defined(__arm__)
	#define __arm__
	#endif

By removing these defs, I was able to cross-compile for aarch64.

I so far tested that the produced builds from this hack work on arm and amd64, but I am waiting on hardware to confirm the fix also works on aarch64. I will report back on success.

I am not confident enough that this change won't break anything to submit a pull-request.

I've searched the zfs code-base and there seems to be only very few cases where there is an ifdef on arm and not also a specific one on aarch64.

[cveilleux]

I can confirm the produced aarch64 binaries with arm undefined seem to work fine on aarch64 (raspberrypi4 in 64 bit mode).

Linux test1 4.19.88 #1 SMP PREEMPT Mon May 11 13:24:57 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

[c-goes]

I tried your recipe on hardknott and found it helpful.

But I needed to add

LDFLAGS = "-O1"

to make it compile. (There was an error aarch64-poky-linux-ld: unrecognized option '-Wl,-O1')

However I can't confirm this issue here for building for Odroid device that is also aarch64. Maybe a problem with rasberrypi recipe? Nonetheless thanks for this recipe. I hope we will find more people testing this recipe on aarch64 devices.

[c-goes]

Using Dunfell I could reproduce @cveilleux's issue.
I'm not sure what has changed between Dunfell and Hardknott.