Cannot compile custom extensions due to missing header files

[nikhilweee]

Declaration

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue

The post-link recipe suggests that conda install cudatoolkit-dev will symlink all libraries, binaries and header files to $CONDA_PREFIX/lib, $CONDA_PREFIX/bin and $CONDA_PREFIX/include respectively. That's great, because now I can just use these directories instead of individually specifying each and every directory in $CONDA_PREFIX/pkgs/cuda-toolkit.

However, the issue still remains that I have to manually add these paths to the respective environment variables (CUDA_HOME, LD_LIBRARY_PATH, etc) to be able to fully use them. conda install cudatoolkit-dev doesn't do that for me. As an example, if I wish to compile a custom CUDA extension with pytorch, I can't do this out of the box because GCC will complain about missing header files (cublas_v2.h, etc). I either have to manually specify -I$(CONDA_PREFIX)/include while using GCC, or set the environment variable CPATH (export CPATH=$CONDA_PREFIX/include).

Since the post install script already consolidates all libs, bins and include files, I wonder whether it also intends to modify environment variables? If not, what's the recommended way? Do I have to set environment variables manually, or should conda install cudatoolkit-dev automatically do that for me?


Details about conda environment (conda list):

$ conda list
# packages in environment at /project/venv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
blas                      1.0                         mkl  
ca-certificates           2020.12.8            h06a4308_0  
certifi                   2020.12.5        py37h06a4308_0  
cffi                      1.14.4           py37h261ae71_0  
cudatoolkit               10.0.130                      0  
cudatoolkit-dev           10.1.243             h516909a_3    conda-forge
cudnn                     7.6.5                cuda10.0_0  
intel-openmp              2020.2                      254  
ld_impl_linux-64          2.33.1               h53a641e_7  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py37he8ac12f_0  
mkl_fft                   1.2.0            py37h23d657b_0  
mkl_random                1.1.1            py37h0573a6f_0  
ncurses                   6.2                  he6710b0_1  
ninja                     1.10.2           py37hff7bd54_0  
numpy                     1.19.2           py37h54aff64_0  
numpy-base                1.19.2           py37hfa32c7d_0  
openssl                   1.1.1i               h27cfd23_0  
pip                       20.3.3           py37h06a4308_0  
pycparser                 2.20                       py_2  
python                    3.7.9                h7579374_0  
pytorch                   1.1.0           cuda100py37he554f03_0  
readline                  8.0                  h7b6447c_0  
setuptools                51.1.2           py37h06a4308_4  
six                       1.15.0           py37h06a4308_0  
sqlite                    3.33.0               h62c20be_0  
tk                        8.6.10               hbc83047_0  
wheel                     0.36.2             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7b6447c_3  

Details about conda and system ( conda info ):

$ conda info
     active environment : /project/venv
    active env location : /project/venv
            shell level : 1
       user config file : /home/me/.condarc
 populated config files : /home/me/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __cuda=11.1=0
                          __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /home/me/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/me/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.5 Linux/4.15.0-123-generic ubuntu/18.04.5 glibc/2.27
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False