Unable to cross-compile CoreCLR for Linux ARM vfpv4-d16 #83170

hopix · 2023-03-09T00:32:59Z

Description

Tring to cross compile .net 7 for Linux ARM with VFP configuration for vfpv4-d16. Build succeeds, but running simple console app on target HW fails with Illegal instruction. After checking build logs it seems that wrong -mfpu=vfpv3 switch is used compilation.

I am able to cross compile .net 6 for ARM vfpv4-d16 and it works well on target HW! So now I am trying to migrate my existing solution to .net 7

Using docs here: https://github.com/dotnet/runtime/blob/release/7.0/docs/workflow/building/coreclr/cross-building.md

Reproduction Steps

.net 7.x* build in Docker:
docker run --rm -v /lab/runtime7:/runtime -w /runtime -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-arm-20220426130400-6e40d49 ./build.sh -subset clr+host.native+libs.native -runtimeFlavor CoreCLR -cross -arch arm -clang9 -configuration Release -librariesConfiguration Release -runtimeConfiguration Release -verbosity detailed -cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 -cmakeargs -DCLR_ARM_FPU_TYPE=vfpv4-d16
And check build logs (native parts) for correct -mfpu and CLR_ARM_FPU_CAPABILITY switch value.

I am using source from tag: https://github.com/dotnet/runtime/tree/v7.0.3

Expected behavior

Build logs (native parts) should contain correct -mfpu=vfpv4-d16 and -DCLR_ARM_FPU_CAPABILITY=0x3 switch values used for native compilation.

Actual behavior

Wrong -mfpu=vfpv3 and -DCLR_ARM_FPU_CAPABILITY=0x7 is currently used.

Regression?

No response

Known Workarounds

No response

Configuration

My build host is Docker container (running on Windows 11 + WSL2)
My target is Linux on ARM

Other information

I have .net 6.x build using Docker which works on target HW OK:
docker run --rm -v /lab/runtime6:/runtime -w /runtime -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-20210719121212-8a8d3be ./build.sh -subset clr+host.native+libs.native -clang9 --arch arm -c release --cross -cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 -cmakeargs -DCLR_ARM_FPU_TYPE=vfpv4-d16

The text was updated successfully, but these errors were encountered:

ghost · 2023-03-09T00:33:05Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Tring to cross compile .net 7 for Linux ARM with VFP configuration for vfpv4-d16. Build succeeds, but running simple console app on target HW fails with Illegal instruction. After checking build logs it seems that wrong -mfpu=vfpv3 switch is used compilation.

I am able to cross compile .net 6 for ARM vfpv4-d16 and it works well on target HW! So now I am trying to migrate my existing solution to .net 7

Using docs here: https://github.com/dotnet/runtime/blob/release/7.0/docs/workflow/building/coreclr/cross-building.md

Reproduction Steps

.net 7.x* build in Docker:
docker run --rm -v /lab/runtime7:/runtime -w /runtime -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-arm-20220426130400-6e40d49 ./build.sh -subset clr+host.native+libs.native -runtimeFlavor CoreCLR -cross -arch arm -clang9 -configuration Release -librariesConfiguration Release -runtimeConfiguration Release -verbosity detailed -cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 -cmakeargs -DCLR_ARM_FPU_TYPE=vfpv4-d16
And check build logs (native parts) for correct -mfpu and CLR_ARM_FPU_CAPABILITY switch value.

I am using source from tag: https://github.com/dotnet/runtime/tree/v7.0.3

Expected behavior

Build logs (native parts) should contain correct -mfpu=vfpv4-d16 and -DCLR_ARM_FPU_CAPABILITY=0x3 switch values used for native compilation.

Actual behavior

Wrong -mfpu=vfpv3 and -DCLR_ARM_FPU_CAPABILITY=0x7 is currently used.

Regression?

No response

Known Workarounds

No response

Configuration

My build host is Docker container (running on Windows 11 + WSL2)
My target is Linux on ARM

Other information

I have .net 6.x build using Docker which works on target HW OK:
docker run --rm -v /lab/runtime6:/runtime -w /runtime -e ROOTFS_DIR=/crossrootfs/arm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-20210719121212-8a8d3be ./build.sh -subset clr+host.native+libs.native -clang9 --arch arm -c release --cross -cmakeargs -DCLR_ARM_FPU_CAPABILITY=0x3 -cmakeargs -DCLR_ARM_FPU_TYPE=vfpv4-d16

Author:	hopix
Assignees:	-
Labels:	`area-Infrastructure-coreclr`
Milestone:	-

am11 · 2023-03-09T03:14:37Z

Could you try applying this patch on v7.0.3:

--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -294,6 +294,9 @@ if(TARGET_ARCH_NAME MATCHES "^(arm|armel)$")
 
   add_definitions (-DCLR_ARM_FPU_CAPABILITY=${CLR_ARM_FPU_CAPABILITY})
 
+  # persist variables across multiple try_compile passes
+  list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES CLR_ARM_FPU_TYPE CLR_ARM_FPU_CAPABILITY)
+
   if(TARGET_ARCH_NAME STREQUAL "armel")
     add_compile_options(-mfloat-abi=softfp)
   endif()

I am not sure how it would be working on v6.0.14 since they both have this issue in cross-compile mode.

hopix · 2023-03-09T11:27:37Z

Thanks, but even with your patch I still getting:
Program received signal SIGILL, Illegal instruction. 0xb6c7848e in ?? () from /usr/share/dotnet/host/fxr/7.0.3/libhostfxr.so

am11 · 2023-03-09T13:19:38Z

How about:

diff --git a/eng/common/cross/toolchain.cmake b/eng/common/cross/toolchain.cmake
index 964610524..2c3d50da7 100644
--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -294,6 +294,9 @@ if(TARGET_ARCH_NAME MATCHES "^(arm|armel)$")
 
   add_definitions (-DCLR_ARM_FPU_CAPABILITY=${CLR_ARM_FPU_CAPABILITY})
 
+  # persist variables across multiple try_compile passes
+  list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES CLR_ARM_FPU_TYPE CLR_ARM_FPU_CAPABILITY)
+
   if(TARGET_ARCH_NAME STREQUAL "armel")
     add_compile_options(-mfloat-abi=softfp)
   endif()
diff --git a/src/native/corehost/corehost.proj b/src/native/corehost/corehost.proj
index a320ab4d8..5a653423f 100644
--- a/src/native/corehost/corehost.proj
+++ b/src/native/corehost/corehost.proj
@@ -77,7 +77,7 @@
       <BuildArgs Condition="'$(KeepNativeSymbols)' != 'false'">$(BuildArgs) -keepnativesymbols</BuildArgs>
       <BuildArgs Condition="'$(CrossBuild)' == 'true'">$(BuildArgs) -cross</BuildArgs>
       <BuildArgs Condition="'$(Compiler)' != ''">$(BuildArgs) $(Compiler)</BuildArgs>
-      <BuildArgs Condition="'$(CMakeArgs)' != ''">$(BuildArgs) $(CMakeArgs)</BuildArgs>
+      <BuildArgs Condition="'$(CMakeArgs)' != ''">$(BuildArgs) -cmakeargs "$(CMakeArgs)"</BuildArgs>
       <BuildArgs Condition="'$(Ninja)' == 'true'">$(BuildArgs) -ninja</BuildArgs>
       <BuildArgs>$(BuildArgs) -runtimeflavor $(RuntimeFlavor)</BuildArgs>
       <BuildArgs Condition="'$(OfficialBuildId)' != ''">$(BuildArgs) /p:OfficialBuildId="$(OfficialBuildId)"</BuildArgs>
diff --git a/src/native/libs/build-native.proj b/src/native/libs/build-native.proj
index b102a6b84..aef78b0dd 100644
--- a/src/native/libs/build-native.proj
+++ b/src/native/libs/build-native.proj
@@ -28,7 +28,7 @@
       <_PortableBuildArg Condition="'$(PortableBuild)' != 'true'"> -portablebuild=false</_PortableBuildArg>
       <_CrossBuildArg Condition="'$(CrossBuild)' == 'true'"> -cross</_CrossBuildArg>
       <_KeepNativeSymbolsBuildArg Condition="'$(KeepNativeSymbols)' != 'false'"> -keepnativesymbols</_KeepNativeSymbolsBuildArg>
-      <_CMakeArgs Condition="'$(CMakeArgs)' != ''"> $(CMakeArgs)</_CMakeArgs>
+      <_CMakeArgs Condition="'$(CMakeArgs)' != ''"> -cmakeargs "$(CMakeArgs)"</_CMakeArgs>
 
       <!--
         BuildNativeCompiler is a pass-through argument, to pass an argument to build-native.sh. It is intended to be

hopix · 2023-03-09T15:03:44Z

I did quick recompile libhostfxr.so only with 2nd patch and app starts ok. 👍
Now I am going to do full build and deploy to confirm all works.

Just for reference originally it was failing here:

Program received signal SIGILL, Illegal instruction.
0xb6c78166 in std::__detail::_Prime_rehash_policy::_M_bkt_for_elements (this=0x4150e0, __n=1) at /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/7.5.0/../../../../include/c++/7.5.0/bits/hashtable_policy.h:478
478     /crossrootfs/arm/usr/lib/gcc/arm-linux-gnueabihf/7.5.0/../../../../include/c++/7.5.0/bits/hashtable_policy.h: No such file or directory.
(gdb) disasm
Undefined command: "disasm".  Try "help".
(gdb) disass
Dump of assembler code for function _ZNKSt8__detail20_Prime_rehash_policy19_M_bkt_for_elementsEj:
   0xb6c78156 <+0>:     push    {r7, lr}
   0xb6c78158 <+2>:     mov     r7, sp
   0xb6c7815a <+4>:     sub     sp, #8
   0xb6c7815c <+6>:     str     r0, [sp, #4]
   0xb6c7815e <+8>:     str     r1, [sp, #0]
   0xb6c78160 <+10>:    ldr     r0, [sp, #4]
   0xb6c78162 <+12>:    vldr    s0, [sp]
=> 0xb6c78166 <+16>:    vcvt.f64.u32    d16, s0
   0xb6c7816a <+20>:    vldr    s0, [r0]
   0xb6c7816e <+24>:    vcvt.f64.f32    d17, s0
   0xb6c78172 <+28>:    vdiv.f64        d0, d16, d17
   0xb6c78176 <+32>:    blx     0xb6c7d6f0
   0xb6c7817a <+36>:    vcvt.u32.f64    s2, d0
   0xb6c7817e <+40>:    vmov    r0, s2
   0xb6c78182 <+44>:    add     sp, #8
   0xb6c78184 <+46>:    pop     {r7, pc}
End of assembler dump.

hopix · 2023-03-09T20:53:14Z

Did some more testing and looks OK with the patch. Thanks.

dotnet-issue-labeler bot added the area-Infrastructure-coreclr label Mar 9, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Mar 9, 2023

This was referenced Mar 9, 2023

Persist FPU variables across try_compile passes dotnet/arcade#12817

Merged

Persist FPU variables across try_compile passes #83206

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Mar 9, 2023

jkotas closed this as completed in #83206 Mar 25, 2023

ghost removed in-pr There is an active PR which will close this issue when it is merged untriaged New issue has not been triaged by the area owner labels Mar 25, 2023

ghost locked as resolved and limited conversation to collaborators Apr 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to cross-compile CoreCLR for Linux ARM vfpv4-d16 #83170

Unable to cross-compile CoreCLR for Linux ARM vfpv4-d16 #83170

hopix commented Mar 9, 2023

ghost commented Mar 9, 2023

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

am11 commented Mar 9, 2023

hopix commented Mar 9, 2023

am11 commented Mar 9, 2023

hopix commented Mar 9, 2023

hopix commented Mar 9, 2023

Unable to cross-compile CoreCLR for Linux ARM vfpv4-d16 #83170

Unable to cross-compile CoreCLR for Linux ARM vfpv4-d16 #83170

Comments

hopix commented Mar 9, 2023

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

ghost commented Mar 9, 2023

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

am11 commented Mar 9, 2023

hopix commented Mar 9, 2023

am11 commented Mar 9, 2023

hopix commented Mar 9, 2023

hopix commented Mar 9, 2023