Replies: 4 comments 5 replies
-
|
Which ROCm version are you using? |
Beta Was this translation helpful? Give feedback.
-
|
Jax: 0.4.18 |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, during build time it should have automatically detected if hipblas-lt is installed and include/not include the header file. Can you please share the build error message you see? |
Beta Was this translation helpful? Give feedback.
-
|
The error is: See attached log: Sorry for the delay on this. |
Beta Was this translation helpful? Give feedback.
-
|
I am able to build just fine in a container without hipblas-lt installed using this code level for XLA and JAX
Can you try building with above and see if that makes the problem go away? If it does, then most likely you don't have right code level for ROCm xla. Correct enablement support for hipblas lt was added recently in XLA. |
Beta Was this translation helpful? Give feedback.
-
|
No luck. I feel like it did get further in before complaining though, but maybe that’s just me. I am trying again with a clean chroot. Any idea how to troubleshoot the bazel configuration? I am wondering if something is causing it to toggle the hipblas-lt flags, but I am not familiar enough with this build system to know where to look. [5,984 / 7,253] Compiling mlir/lib/Dialect/Index/IR/IndexOps.cpp; 12s local ... (18 actions, 15 running)
ERROR: /home/user/.cache/bazel/_bazel_user/70b60264865517de98a7c354d9faab82/external/xla/xla/stream_executor/rocm/BUILD:463:11: Compiling xla/stream_executor/rocm/hip_blas_utils.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command (from target @xla//xla/stream_executor/rocm:hip_blas_utils)
(cd /home/user/.cache/bazel/_bazel_user/70b60264865517de98a7c354d9faab82/execroot/__main__ && \
exec env - \
PATH=/home/user/.roswell/bin:/home/user/.config/emacs/bin:/home/user/.nimble/bin:/home/user/.luarocks/bin:/home/user/.cargo/bin:/home/user/.local/jpm_tree/bin:/home/user/.local/bin:/home/user/.roswell/bin:/home/user/.config/emacs/bin:/home/user/.nimble/bin:/home/user/.luarocks/bin:/home/user/.cargo/bin:/home/user/.local/jpm_tree/bin:/home/user/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
PWD=/proc/self/cwd \
TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx908,gfx90a,gfx1030,gfx1100,gfx1101,gfx1102 \
external/local_config_rocm/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++14' -MD -MF bazel-out/k8-opt/bin/external/xla/xla/stream_executor/rocm/_objs/hip_blas_utils/hip_blas_utils.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/xla/xla/stream_executor/rocm/_objs/hip_blas_utils/hip_blas_utils.pic.o' -fPIC -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY '-DBAZEL_CURRENT_REPOSITORY="xla"' -iquote external/xla -iquote bazel-out/k8-opt/bin/external/xla -iquote external/local_config_rocm -iquote bazel-out/k8-opt/bin/external/local_config_rocm -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/tsl -iquote bazel-out/k8-opt/bin/external/tsl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/ml_dtypes -iquote bazel-out/k8-opt/bin/external/ml_dtypes -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/local_config_nccl -iquote bazel-out/k8-opt/bin/external/local_config_nccl -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/float8 -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/int4 -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers_virtual -Ibazel-out/k8-opt/bin/external/local_config_nccl/_virtual_includes/nccl_config -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -isystem external/local_config_rocm/rocm -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm -isystem external/local_config_rocm/rocm/rocm/include -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include -isystem external/tsl/third_party/eigen3/mkl_include -isystem bazel-out/k8-opt/bin/external/tsl/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes -isystem external/ml_dtypes/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes/ml_dtypes -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/local_config_rocm/rocm/rocm/include/rocrand -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocrand -isystem external/local_config_rocm/rocm/rocm/include/roctracer -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/roctracer -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src '-fvisibility=hidden' -Wno-sign-compare -Wno-unknown-warning-option -Wno-stringop-truncation -Wno-array-parameter '-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.' -mavx '-std=c++17' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-DTENSORFLOW_USE_ROCM=1' -D__HIP_PLATFORM_HCC__ -DEIGEN_USE_HIP -no-canonical-prefixes -fno-canonical-system-headers -c external/xla/xla/stream_executor/rocm/hip_blas_utils.cc -o bazel-out/k8-opt/bin/external/xla/xla/stream_executor/rocm/_objs/hip_blas_utils/hip_blas_utils.pic.o)
# Configuration: 5bf9781fb7147e36d5e78ae642473b510d8022add5a53218e0ed4f2bdc3b6cbf
# Execution platform: @local_execution_config_platform//:platform
/home/user/.cache/bazel/_bazel_user/70b60264865517de98a7c354d9faab82/execroot/__main__/external/local_config_rocm/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc:23: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
import pipes
In file included from external/xla/xla/stream_executor/rocm/hip_blas_utils.h:22,
from external/xla/xla/stream_executor/rocm/hip_blas_utils.cc:16:
external/xla/xla/stream_executor/rocm/hipblaslt_wrapper.h:24:10: fatal error: rocm/include/hipblaslt/hipblaslt.h: No such file or directory
24 | #include "rocm/include/hipblaslt/hipblaslt.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Target //jaxlib/tools:build_wheel failed to build
INFO: Elapsed time: 976.003s, Critical Path: 101.13s
INFO: 6004 processes: 771 internal, 5233 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
Traceback (most recent call last):
File "/data/tmp/jax-stuff/python-jax-rocm/src/jax/build/build.py", line 598, in <module>
main()
File "/data/tmp/jax-stuff/python-jax-rocm/src/jax/build/build.py", line 580, in main
shell(command)
File "/data/tmp/jax-stuff/python-jax-rocm/src/jax/build/build.py", line 44, in shell
output = subprocess.check_output(cmd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, |
Beta Was this translation helpful? Give feedback.
-
|
Could you provide some details on how to reproduce that container? I am still getting this issue even with a clean chroot and the branches listed. |
Beta Was this translation helpful? Give feedback.
-
|
@iicurtis Hey, so sorry for the confusion. Actually, I am seeing the build error in my local environment. Will get back to you when I have an update on this issue. |
Beta Was this translation helpful? Give feedback.
-
|
At the moment, the pre-built wheels for JAX don't work without hipBLASLt installed, and I get link errors when trying to perform matrix multiplication on device. I assume that if I have an unsupported device for hipBLASLt, XLA is going to fall back on a different codepath anyway? |
Beta Was this translation helpful? Give feedback.
-
When building with rocm XLA and —enable-rocm, the build fails due to a missing
hipblaslt.hinclude. Arch linux does not have a supported build path for this (yet), and it appears to only support a single device. Could this be put behind a build flag to enable builds without hipblaslt? I am sorry if this is not the forum for this. @rahulbatra85Beta Was this translation helpful? Give feedback.
All reactions