Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/usr/include leaking into conda-build with openmp runtimes enabled. #230

Open
DrTodd13 opened this issue Jul 25, 2023 · 24 comments
Open

/usr/include leaking into conda-build with openmp runtimes enabled. #230

DrTodd13 opened this issue Jul 25, 2023 · 24 comments
Labels

Comments

@DrTodd13
Copy link

Comment:

I'm not using the conda-forge recipes (instead basing what I'm doing on the Numba recipe) but I thought this group might have expertise to answer my question. In build.sh, I'm building clang/lld/libunwind and the openmp runtimes as below.

_cmake_config+=(-DLLVM_ENABLE_PROJECTS:STRING="clang;lld;libunwind")
_cmake_config+=(-DLLVM_ENABLE_RUNTIMES:STRING=openmp)

I'm doing this in a Centos 7 VM. The build gets into building the runtime and for some of the compilation steps I get the following errors.

[ 12%] Building LLVM bitcode Reduction.cpp-sm_52.bc
cd $SRC_DIR/llvm/build/runtimes/runtimes-bins/openmp/libomptarget/DeviceRTL && $SRC_DIR/llvm/build/bin/clang -S -x c++ -std=c++17 -fvisibility=hidden -O1 -mllvm -openmp-opt-disable -DSHARED_SCRATCHPAD_SIZE=512 -Xclang -emit-llvm-bc -Xclang -aux-triple -Xclang x86_64-unknown-linux-gnu -fopenmp -fopenmp-cuda-mode -Xclang -fopenmp-is-device -I$SRC_DIR/openmp/libomptarget/DeviceRTL/include -I$SRC_DIR/openmp/libomptarget/DeviceRTL/../include -I$SRC_DIR/llvm/include -I$SRC_DIR/llvm/build/include -DOMPTARGET_DEBUG=0 -Xclang -target-cpu -Xclang sm_52 -target nvptx64-nvidia-cuda -Xclang -target-feature -Xclang +ptx61 -D__CUDA_ARCH__=520 $SRC_DIR/openmp/libomptarget/DeviceRTL/src/Reduction.cpp -o Reduction.cpp-sm_52.bc
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/openmp/libomptarget/plugins/common/elf_common/elf_common.cpp:16:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/Object/Binary.h:17:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Triple.h:12:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Twine.h:12:
/home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/SmallVector.h:191:12: error: no template named 'enable_if_t' in namespace 'std'; did you mean 'enable_if'?
      std::enable_if_t<!std::is_same<std::remove_const_t<ItTy>, T *>::value,
      ~~~~~^~~~~~~~~~~
           enable_if
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1766:12: note: 'enable_if' declared here
    struct enable_if
           ^
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/openmp/libomptarget/plugins/common/elf_common/elf_common.cpp:16:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/Object/Binary.h:17:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Triple.h:12:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Twine.h:12:
/home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/SmallVector.h:191:43: error: no template named 'remove_const_t' in namespace 'std'; did you mean 'remove_const'?
      std::enable_if_t<!std::is_same<std::remove_const_t<ItTy>, T *>::value,
                                     ~~~~~^~~~~~~~~~~~~~
                                          remove_const
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1327:12: note: 'remove_const' declared here
    struct remove_const
           ^
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/openmp/libomptarget/plugins/common/elf_common/elf_common.cpp:16:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/Object/Binary.h:17:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Triple.h:12:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Twine.h:12:
/home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/SmallVector.h:204:12: error: no template named 'enable_if_t' in namespace 'std'; did you mean 'enable_if'?
      std::enable_if_t<!std::is_same<std::remove_const_t<ItTy>, T *>::value,
      ~~~~~^~~~~~~~~~~
           enable_if
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1766:12: note: 'enable_if' declared here
    struct enable_if
           ^
...

It seems to be trying to get the C++ include files from /usr/include and failing because enable_if_t is not available in the base gcc available on centos 7 (gcc 4.8.5). I don't know why it is trying to use this gcc at all since the conda environment created at the beginning has gcc 7.5 in it as follows:

## Package Plan ##

  environment location: /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/_build_env


The following NEW packages will be INSTALLED:

    _libgcc_mutex:                        0.1-main
    _openmp_mutex:                        5.1-1_gnu
    _sysroot_linux-64_curr_repodata_hack: 3-haa98f57_10
    binutils_impl_linux-64:               2.35.1-h27ae35d_9
    binutils_linux-64:                    2.35.1-h454624a_30
    bzip2:                                1.0.8-h7b6447c_0
    c-ares:                               1.19.0-h5eee18b_0
    ca-certificates:                      2023.05.30-h06a4308_0
    cmake:                                3.26.4-h96355d8_0
    curl:                                 7.88.1-h5eee18b_1
    expat:                                2.4.9-h6a678d5_0
    gcc_impl_linux-64:                    7.5.0-h7105cf2_17
    gcc_linux-64:                         7.5.0-h8f34230_30
    gdbm:                                 1.18-hd4cb3f1_4
    gettext:                              0.21.0-h39681ba_1
    git:                                  2.40.1-pl5340h36fbf9e_1
    gxx_impl_linux-64:                    7.5.0-h0a5bf11_17
    gxx_linux-64:                         7.5.0-hffc177d_30
    icu:                                  58.2-he6710b0_3
    kernel-headers_linux-64:              3.10.0-h57e8cba_10
    krb5:                                 1.20.1-h143b758_1
    ld_impl_linux-64:                     2.35.1-h7274673_9
    libcurl:                              7.88.1-h251f7ec_1
    libedit:                              3.1.20221030-h5eee18b_0
    libev:                                4.33-h7f8727e_1
    libffi:                               3.4.4-h6a678d5_0
    libgcc-devel_linux-64:                7.5.0-hbbeae57_17
    libgcc-ng:                            11.2.0-h1234567_1
    libgomp:                              11.2.0-h1234567_1
    libnghttp2:                           1.52.0-h2d74bed_1
    libssh2:                              1.10.0-hdbd6064_2
    libstdcxx-devel_linux-64:             7.5.0-hf0c5c8d_17
    libstdcxx-ng:                         11.2.0-h1234567_1
    libuuid:                              1.41.5-h5eee18b_0
    libuv:                                1.44.2-h5eee18b_0
    libxml2:                              2.10.3-hcbfbd50_0
    lz4-c:                                1.9.4-h6a678d5_0
    make:                                 4.2.1-h1bed415_1
    ncurses:                              6.4-h6a678d5_0
    openssl:                              3.0.9-h7f8727e_0
    pcre2:                                10.42-hebb0a14_0
    perl:                                 5.34.0-h5eee18b_2
    pip:                                  23.1.2-py311h06a4308_0
    python:                               3.11.4-h955ad1f_0
    readline:                             8.2-h5eee18b_0
    rhash:                                1.4.3-hdbd6064_0
    setuptools:                           67.8.0-py311h06a4308_0
    sqlite:                               3.41.2-h5eee18b_0
    sysroot_linux-64:                     2.17-h57e8cba_10
    tk:                                   8.6.12-h1ccaba5_0
    tzdata:                               2023c-h04d1e81_0
    wheel:                                0.38.4-py311h06a4308_0
    xz:                                   5.4.2-h5eee18b_0
    zlib:                                 1.2.13-h5eee18b_0
    zstd:                                 1.5.5-hc292b87_0

Any suggestions on how to proceed? Thanks!

@isuruf
Copy link
Member

isuruf commented Jul 25, 2023

Is it using the just built clang to build openmp?

@DrTodd13
Copy link
Author

In the second line of the error report above you can see "$SRC_DIR/llvm/build/bin/clang" as the exe of the build command so, yes, I believe that it is using the just built clang. If I try to build the runtime without including clang as a project to build then it gives me an error so I assume that is a requirement of building the runtime.

@isuruf
Copy link
Member

isuruf commented Jul 25, 2023

Yeah, you need some patches to clang to automatically find the sysroot. (Or set --sysroot to make clang aware of the conda sysroot)

@h-vetinari
Copy link
Member

Should we backport those patches to LLVM/clang 15?

@DrTodd13
Copy link
Author

DrTodd13 commented Jul 27, 2023

@isuruf It seems that for the runtime build portion, when cmake is trying to use the newly built clang to check for the compiler capabilities, that all those checks are failing because of the following:

(/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env) [username@localhost _build_env]$ $SRC_DIR/llvm/build/bin/clang ~/openmp/pthread_test_cmake.c
/usr/bin/ld: cannot find crtbegin.o: No such file or directory
/usr/bin/ld: cannot find -lgcc
/usr/bin/ld: cannot find -lgcc
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

Suggestions? Thanks!

@isuruf
Copy link
Member

isuruf commented Jul 27, 2023

Did you use my suggestions above?

@isuruf
Copy link
Member

isuruf commented Jul 27, 2023

Should we backport those patches to LLVM/clang 15?

Don't see why not

@DrTodd13
Copy link
Author

Did you use my suggestions above?

Yes, I used the second one.

@isuruf
Copy link
Member

isuruf commented Jul 27, 2023

CONDA_BUILD_SYSROOT is set in your env?

@DrTodd13
Copy link
Author

DrTodd13 commented Jul 27, 2023

(/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env) [username@localhost llvmdev_1690419527432]$ echo $CONDA_BUILD_SYSROOT
/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env/x86_64-conda-linux-gnu/sysroot

@DrTodd13
Copy link
Author

DrTodd13 commented Jul 27, 2023

(/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env) [username@localhost llvmdev_1690419527432]$ find . -name "libgcc*" -print
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-10/lib/gcc/aarch64-unknown-linux-gnu/10/libgcc.a
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-7.5.0/lib/gcc/aarch64-unknown-linux-gnu/7.5.0/libgcc.a
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-9.3.0/lib/gcc/aarch64-unknown-linux-gnu/9.3.0/libgcc.a
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-9.3.1/lib/gcc/aarch64-unknown-linux-gnu/9.3.1/libgcc.a
./work/clang/test/Driver/Inputs/basic_avr_tree/usr/lib/gcc/avr/5.4.0/libgcc.a
./work/clang/test/Driver/Inputs/basic_avr_tree_2/opt/local/lib/gcc/avr/10.3.0/libgcc.a
./work/clang/test/Driver/Inputs/montavista_i686_tree/usr/lib/gcc/i686-montavista-linux/4.2.0/libgcc.a
./_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/conda-meta/libgcc-ng-11.2.0-h1234567_1.json
./_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libgcc_s.so
./_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libgcc_s.so.1
./_build_env/conda-meta/libgcc-devel_linux-64-7.5.0-hbbeae57_17.json
./_build_env/conda-meta/libgcc-ng-11.2.0-h1234567_1.json
./_build_env/x86_64-conda-linux-gnu/sysroot/lib64/libgcc_s.so
./_build_env/x86_64-conda-linux-gnu/sysroot/lib64/libgcc_s.so.1
./_build_env/lib/gcc/x86_64-conda-linux-gnu/7.5.0/libgcc.a
./_build_env/lib/gcc/x86_64-conda-linux-gnu/7.5.0/libgcc_eh.a
./_build_env/lib/libgcc_s.so
./_build_env/lib/libgcc_s.so.1

@isuruf
Copy link
Member

isuruf commented Jul 27, 2023

Hmm, you might need the other two patches too.

@DrTodd13
Copy link
Author

Hmm, you might need the other two patches too.

I thought the same and I tried it. After the rebuild, activating the build environment and rerunning the pthread test program, It changed the linker from /usr/bin/ld to /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/_build_env/bin/x86_64-conda-linux-gnu-ld but otherwise the errors are the same about crtbegin.o and -lgcc not being found.

@h-vetinari
Copy link
Member

Sidenote: Please post code & shell output enclosed in triple backticks.

@h-vetinari
Copy link
Member

Should we backport those patches to LLVM/clang 15?

Don't see why not

#232 & conda-forge/clangdev-feedstock#234

@h-vetinari
Copy link
Member

Though I note that the errors mention clang-14:

clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

so the backports to the 15.x branch won't have an effect. How are you installing clang @DrTodd13?

@DrTodd13
Copy link
Author

Though I note that the errors mention clang-14:

clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

so the backports to the 15.x branch won't have an effect. How are you installing clang @DrTodd13?

Yes indeed, I am backporting to llvm 14.0.6. You can see what I'm doing in my first post where I show a part of build.sh where I am enabling the clang, lld, and libunwind projects along with the openmp runtime. I'm not "installing" clang...it is just being built as part of the llvm build process and then a later part of the build process that is doing the runtime is using the clang that was built earlier in the build.

@DrTodd13
Copy link
Author

@isuruf @h-vetinari I found https://discourse.llvm.org/t/cannot-find-lgcc-with-new-clang-worked-with-clang-7/4232/7.

Running clang --verbose on the clang that is being built in the conda build environment finds no candidate gcc implementations.

$SRC_DIR/llvm/build/bin/clang --verbose
clang version 14.0.6 (/home/username/miniconda3/envs/pyomp39/conda-bld/git_cache/github.com/Python-for-HPC/llvm-project.git 3f24beb247fa4f527be15ba155d8eff338744c75)
Target: x86_64-conda-linux-gnu
Thread model: posix
InstalledDir: /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/work/llvm/build/bin

@isuruf
Copy link
Member

isuruf commented Jul 28, 2023

Try

ln -sf  /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/x86_64-conda-linux-gnu /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/work/llvm/build/x86_64-conda-linux-gnu

@DrTodd13
Copy link
Author

Try

ln -sf  /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/x86_64-conda-linux-gnu /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/work/llvm/build/x86_64-conda-linux-gnu

That directory didn't exist directly under llvmdev_1690494537428 but does exist in the _build_env subdir. I tried doing the ln from that directory to work/llvm/build but it had no effect.

@h-vetinari
Copy link
Member

The following NEW packages will be INSTALLED:

From what I can tell you are not using conda-forge packages for the build environment. Can you do

conda config --add channels conda-forge
conda config --set channel_priority strict

and try again with a fresh build environment?

@DrTodd13
Copy link
Author

The following NEW packages will be INSTALLED:

From what I can tell you are not using conda-forge packages for the build environment. Can you do

conda config --add channels conda-forge
conda config --set channel_priority strict

and try again with a fresh build environment?

Did a completely clean environment and got everything from conda-forge and it still gives me the same error.

@h-vetinari
Copy link
Member

Checking in on this old issue again. AFAIU everything should be fixed for v15+. Is there still a problem @DrTodd13?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants