Taichi GPU Functions Automatic Degradation onto CPUs on Supercomputers

[Ruoyu66666]

Describe the bug
I wrote a small program to compare Taichi and numpy, where Taichi function is supposed to run on Nvidia GPUs on TACC, the supercomputer maintained by the University of Texas System. However, the Taichi kernel degrades automatically to CPU. When I check GPU working status as program run, no job is running. My doubt was confirmed by setting (arch = ti.cuda and arch=ti.cpu). The run times are equal. Can someone with experience of using GPUs on supercomputers help? I appreciate your help ahead.

Update: the TACC staff said he think Taichi developers may help after I showed him my code and the error file. He said the problem should not be the CUDA installation on the TACC side.

The simple code is shown here:

taichi_computation_trial.txt

I attached the errors here:

slurm-733695.txt

[ailzhang]

@Ruoyu66666 it looks like taichi failed to find libcuda.so in the system path. Do you need to load addition modules for CUDA as well.

Currently Loaded Modules:
  1) intel/19.1.1   3) python3/3.9.7   5) pmix/3.2.3     7) TACC
  2) impi/19.0.9    4) cmake/3.24.2    6) xalt/2.10.32

[Ruoyu66666]

@ailzhang Great thanks for your reply! I am contacting TACC staff with your reply. ---Ruoyu

[Ruoyu66666]

@ailzhang Thanks again for your look! The issue was resolved by setting a different library path for Taichi to find CUDA.