How to compile and run the code

Setup the environment

Step 1: create a new conda environment

conda create --name join
conda activate join

Step 2: Install cuDF

We have two options for installing cuDF.

Either installing directly through conda,

conda install -c rapidsai -c nvidia -c conda-forge cudf=0.19 python=3.8 cudatoolkit=<your CUDA version>

or compile from source by following these instructions.

Step 3: Install UCX and NCCL through conda

conda install -c rapidsai -c nvidia -c conda-forge ucx ucx-proc=*=gpu nccl

Step 4: Install nvcomp

Clone the nvcomp repo and build nvcomp

git clone https://github.com/NVIDIA/nvcomp && cd nvcomp && git checkout branch-2.0
mkdir -p build && cd build && cmake ..
make -j

Step 5: Setup the environment variables

export CUDA_ROOT=$(dirname $(dirname $(which nvcc)))
export CUDF_ROOT=$CONDA_PREFIX
export UCX_ROOT=$CUDF_HOME
export NCCL_ROOT=$CUDF_HOME
export MPI_ROOT=$(dirname $(dirname $(which mpirun)))
export NVCOMP_ROOT=<nvcomp build directory in step 4>
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDF_ROOT/lib:$NVCOMP_ROOT/lib

Step 6: Compile the distributed join code

Get the code through

git clone https://github.com/rapidsai/distributed-join

Then compile the code

mkdir build && cd build
cmake ..
make -j

Run the code

UCX_MEMTYPE_CACHE=n UCX_TLS=sm,cuda_copy,cuda_ipc mpirun -n 4 --cpus-per-rank 2 bin/benchmark/distributed_join