ROCm 3.6.0 ICE

[cwsmith]

We are trying to build a C++ code for MI60 using ROCm 3.6.0 on the Corona system at LLNL and hit the following error in LLVM:

LLVM ERROR: Unsupported expression in static initializer: addrspacecast ([3 x i32] addrspace(4)* getelementptr inbounds ([1 x [3 x i32]], [1 x [3 x i32]] addrspace(4)* @_ZN7Omega_h6swap3dL11triangles_3E, i64 0, i64 0) to [3 x i32]*)

Below are instructions to reproduce the error.

Download

git clone [email protected]:SCOREC/omega_h.git
cd omega_h
git checkout clang11_hip

Create envCoronaRocm.sh with the following contents:

module load opt
module load rocm/3.6.0
module unload intel

Build for host using Clang11

source envCoronaClang11.sh
mkdir build-omega-clang11
cd !$
cmake ../omega_h \
  -DCMAKE_INSTALL_PREFIX=$PWD/install \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_HIP=OFF \
  -DOmega_h_USE_Kokkos=OFF \
  -DOmega_h_USE_MPI=OFF \
  -DCMAKE_CXX_COMPILER=/opt/rocm-3.6.0/llvm/bin/clang++ \
  -DOmega_h_CXX_WARNINGS=OFF \
  -DBUILD_TESTING=ON

make -j4
ctest # all tests pass

Build for MI60 using hipcc

source envCoronaClang11.sh
mkdir build-omega-rocm36
cd !$
rocm=/opt/rocm-3.6.0
export HIP_PATH=$rocm/hip
export CMAKE_PREFIX_PATH=$rocm:$CMAKE_PREFIX_PATH
cmake ../omega_h \
  -DCMAKE_INSTALL_PREFIX=$PWD/install \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_HIP=ON \
  -DOmega_h_USE_Kokkos=OFF \
  -DCMAKE_CXX_FLAGS="--amdgpu-target=gfx906" \
  -DHIP_PATH=${rocm}/hip \
  -DOmega_h_USE_MPI=OFF \
  -DCMAKE_CXX_COMPILER=hipcc \
  -DOmega_h_CXX_WARNINGS=OFF \
  -DBUILD_TESTING=ON

make VERBOSE=1 # this takes several minutes

this produces the following LLVM error:

[  1%] Building CXX object src/CMakeFiles/omega_h.dir/Omega_h_swap3d_qualities.cpp.o
cd /g/g19/smith516/develop/build-omega-rocm35/src && /opt/rocm-3.6.0/bin/hipcc   -I/g/g19/smith516/develop/omega_h/src -I/g/g19/smith516/develop/build-omega-rocm35/src -I/g/g19/smith516/develop/omega_h/tpl  --amdgpu-target=gfx906   -std=c++11 -o CMakeFiles/omega_h.dir/Omega_h_swap3d_qualities.cpp.o -c /g/g19/smith516/develop/omega_h/src/Omega_h_swap3d_qualities.cpp
LLVM ERROR: Unsupported expression in static initializer: addrspacecast ([3 x i32] addrspace(4)* getelementptr inbounds ([1 x [3 x i32]], [1 x [3 x i32]] addrspace(4)* @_ZN7Omega_h6swap3dL11triangles_3E, i64 0, i64 0) to [3 x i32]*)
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /opt/rocm-3.6.0/llvm/bin/llc /var/tmp/smith516/Omega_h_swap3d_qualities-0a954b-gfx906-optimized-a887e8.bc -O3 -mtriple=amdgcn-amd-amdhsa -mcpu=gfx906 -filetype=obj -amdgpu-early-inline-all=true -amdgpu-function-calls=false -o /var/tmp/smith516/Omega_h_swap3d_qualities-0a954b-gfx906-6404ed.o 
1.      Running pass 'CallGraph Pass Manager' on module '/var/tmp/smith516/Omega_h_swap3d_qualities-0a954b-gfx906-optimized-a887e8.bc'.
 #0 0x000000000199f16a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/opt/rocm-3.6.0/llvm/bin/llc+0x199f16a)
 #1 0x000000000199d064 llvm::sys::RunSignalHandlers() (/opt/rocm-3.6.0/llvm/bin/llc+0x199d064)
 #2 0x000000000199d1a3 SignalHandler(int) (/opt/rocm-3.6.0/llvm/bin/llc+0x199d1a3)
 #3 0x00002aaaaacde630 __restore_rt (/lib64/libpthread.so.0+0xf630)
 #4 0x00002aaaabf8c387 raise /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0
 #5 0x00002aaaabf8da78 abort /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0
 #6 0x000000000192a340 llvm::report_fatal_error(llvm::Twine const&, bool) (/opt/rocm-3.6.0/llvm/bin/llc+0x192a340)
 #7 0x000000000192a4ae (/opt/rocm-3.6.0/llvm/bin/llc+0x192a4ae)
 #8 0x0000000000db2d6a llvm::AsmPrinter::lowerConstant(llvm::Constant const*) (/opt/rocm-3.6.0/llvm/bin/llc+0xdb2d6a)
 #9 0x0000000000db2ade llvm::AsmPrinter::lowerConstant(llvm::Constant const*) (/opt/rocm-3.6.0/llvm/bin/llc+0xdb2ade)
#10 0x0000000000db86e3 emitGlobalConstantImpl(llvm::DataLayout const&, llvm::Constant const*, llvm::AsmPrinter&, llvm::Constant const*, unsigned long) (/opt/rocm-3.6.0/llvm/bin/llc+0xdb86e3)
#11 0x0000000000db8cfe emitGlobalConstantImpl(llvm::DataLayout const&, llvm::Constant const*, llvm::AsmPrinter&, llvm::Constant const*, unsigned long) (/opt/rocm-3.6.0/llvm/bin/llc+0xdb8cfe)
#12 0x0000000000dbb08a llvm::AsmPrinter::emitGlobalVariable(llvm::GlobalVariable const*) (/opt/rocm-3.6.0/llvm/bin/llc+0xdbb08a)
#13 0x00000000006f08bf llvm::AMDGPUAsmPrinter::emitGlobalVariable(llvm::GlobalVariable const*) (/opt/rocm-3.6.0/llvm/bin/llc+0x6f08bf)
#14 0x0000000000db6c7a llvm::AsmPrinter::doFinalization(llvm::Module&) (/opt/rocm-3.6.0/llvm/bin/llc+0xdb6c7a)
#15 0x0000000001317fac llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias.492) (/opt/rocm-3.6.0/llvm/bin/llc+0x1317fac)
#16 0x0000000000c2fbbc (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) (/opt/rocm-3.6.0/llvm/bin/llc+0xc2fbbc)
#17 0x00000000013237d6 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/rocm-3.6.0/llvm/bin/llc+0x13237d6)
#18 0x00000000006b25c3 compileModule(char**, llvm::LLVMContext&) (/opt/rocm-3.6.0/llvm/bin/llc+0x6b25c3)
#19 0x0000000000647b15 main (/opt/rocm-3.6.0/llvm/bin/llc+0x647b15)
#20 0x00002aaaabf78555 __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:300:0
#21 0x00000000006ac4f5 _start (/opt/rocm-3.6.0/llvm/bin/llc+0x6ac4f5)
clang-11: error: unable to execute command: Aborted (core dumped)
clang-11: error: amdgcn-link command failed due to signal (use -v to see invocation)
clang version 11.0.0 (/data/jenkins_workspace/centos_pipeline_job_3.6/rocm-rel-3.6/rocm-3.6-43-20200704/7.5/external/llvm-project/clang f7b7e21a21d08df6971d2c77315a0e41b7639334)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-3.6.0/llvm/bin
clang-11: note: diagnostic msg: Error generating preprocessed source(s).
make[2]: *** [src/CMakeFiles/omega_h.dir/Omega_h_swap3d_qualities.cpp.o] Error 254
make[2]: Leaving directory `/g/g19/smith516/develop/build-omega-rocm35'
make[1]: *** [src/CMakeFiles/omega_h.dir/all] Error 2
make[1]: Leaving directory `/g/g19/smith516/develop/build-omega-rocm35'
make: *** [all] Error 2

[arsenm]

57bd64ff8434aa55aeb1c7a1035f4b5b7468b809 should have fixed this

[jlgreathouse]

Looking through our build systems, I believe that commit did not make it into ROCm 3.8. I believe the fix will come in ROCm 3.9.

[cwsmith]

This is resolved with rocm 3.9.