Add magma conda package install support via MAGMA_PACKAGE_SOURCE #30

jataylo · 2023-05-03T09:31:53Z

Add condition to install_rocm_magma.sh which will either install magma via a package source if supplied or fallback to building from source.

Corresponding rocAutomation PR:
https://github.com/ROCmSoftwarePlatform/rocAutomation/pull/161

build_manylinux_image.sh snippet:

#64 [rocm_final 7/7] RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh
#64 sha256:1b8cb3c1d37d22ee3a84c0912d1f5aadcf5c92d1fcd78d0e83d96cf3a3e33a6f
#64 1.169 MAGMA_PACKAGE_SOURCE is set, installing magma from source
#64 1.169 + '[' -z .../magma-rocm-2.6.2-+RC5_44_35.tar.bz2 ']'
#64 1.169 + echo 'MAGMA_PACKAGE_SOURCE is set, installing magma from source'
#64 1.169 + install_magma_package
#64 1.169 + MAGMA_VERSION=2.6.2
#64 1.169 + rocm_path=/opt/rocm
#64 1.170 ++ mktemp -d
#64 1.174 + tmp_dir=/tmp/tmp.9OOq5zKOo8
#64 1.174 + pushd /tmp/tmp.9OOq5zKOo8
#64 1.174 /tmp/tmp.9OOq5zKOo8 /
#64 1.174 + wget --no-check-certificate -q ...magma-rocm-2.6.2-+RC5_44_35.tar.bz2
#64 4.581 + tar -xf magma-rocm-2.6.2-+RC5_44_35.tar.bz2
#64 26.13 + mkdir -p /opt/rocm/magma
#64 26.13 + '[' -e /tmp/tmp.9OOq5zKOo8/magma/include ']'
#64 26.13 + mv /tmp/tmp.9OOq5zKOo8/magma/include /opt/rocm/magma/include
#64 26.14 Successfully installed MAGMA include files to /opt/rocm/magma/include
#64 26.14 + echo 'Successfully installed MAGMA include files to /opt/rocm/magma/include'
#64 26.14 + '[' -e /tmp/tmp.9OOq5zKOo8/magma/lib/ ']'
#64 26.14 + mv /tmp/tmp.9OOq5zKOo8/magma/lib/ /opt/rocm/magma/lib
#64 26.14 Successfully installed MAGMA library files to /opt/rocm/magma/lib
#64 26.14 /
#64 26.14 + echo 'Successfully installed MAGMA library files to /opt/rocm/magma/lib'
#64 26.14 + popd
#64 26.14 + rm -rf /tmp/tmp.9OOq5zKOo8
#64 26.71 + '[' '!' -d /opt/rocm/magma ']'
#64 DONE 34.0s

* add dependencies for rocm5.1 * install miopen dependencies using cmake * change lib path in build_rocm.sh for rocm5.1 * change amdgpu_version for rocm5.1 * remove 4.5.2 and add 5.1

Libdrm rocm fix

IFU 2022-05-16

[ROCm] add dependencies for rocm5.2

* change the MAYBE_LIB64 path for just for 5.2

* Changes to support ROCm 5.3 * Updated as per comments

- In ROCm 5.3 libtorch build are failing during magma build due to to missing python binary so added install statement

* Updating the condition for noRCCL build * Updated changes as per comments

IFU-main-2022-09-15

…rocm_fork Disable MLIR backend when building MIOpen

Use ROCm5.3 branch for MIOpen

…taging_branch Use staging branch of MIOpen for ROCm5.3

…release (#14)

* Update to so patching for ROCm Wildcard used in grep to grab the actual numbered so file referenced in patchelf. This allows the removal of specifying the so number in DEPS_LIST & DEPS_SONAME This commit also adds the functionality for trimming so names to build_libtorch.sh from build_common.sh * Refactor to remove switch statement in build_rocm.sh This commit refactors build_rocm.sh and brings in a few major updates: - No longer required to specify the full .so name (with number) for ROCm libraries - The .so versions are copied and the patching code will fix the links to point to this version - No longer required to specify paths for ROCm libraries allowing the removal of the large switch - Paths are acquired programmatically with find - No longer required to specify both the path and filename for the OS specific libraries - Programatically extract file name from the path - Automatically extract Tensile/Kernels files for the architectures specified in PYTORCH_ROCM_ARCH and any non-arch specific files e.g. TensileLibrary.dat

* Remove miopen custom build step * Bundle MIOpen db files in wheel * Correct path

* Use libtinfo.so.6 for Ubuntu 2004 * Fix to origname grep * Condition on ROCM_VERSION for libtinfo6

We require the same fix that was made on upstream pytorch pytorch/pytorch#91371 ROCm/pytorch@b72ec7c Without this change install_conda.sh stage fails ``` #21 6.254 CondaFileIOError: '/opt/conda/pkgs/envs/*/env.txt'. [Errno 2] No such file or directory: '/opt/conda/pkgs/envs/*/env.txt' #21 6.254 #21 ERROR: executor failed running [/bin/sh -c bash ./install_conda.sh && rm install_conda.sh]: exit code: 1 ------ > [conda 2/3] RUN bash ./install_conda.sh && rm install_conda.sh: ------ executor failed running [/bin/sh -c bash ./install_conda.sh && rm install_conda.sh]: exit code: 1 ``` Locally tested with the `/builder/libtorch/build_docker.sh`

Co-authored-by: Jack Taylor <[email protected]>

…m_fork_main

KyleCZH and others added 30 commits March 22, 2022 19:07

[ROCm] add dependencies for rocm5.1 (#1)

df95cf7

* add dependencies for rocm5.1 * install miopen dependencies using cmake * change lib path in build_rocm.sh for rocm5.1 * change amdgpu_version for rocm5.1 * remove 4.5.2 and add 5.1

[ROCm] install custom libdrm to silence missing amdgpu.ids warning

c44dfc0

copy aux file amdgpu.ids

102ca3a

begin search for amdgpu.ids from install root, not lib

628d2ef

Merge pull request #2 from KyleCZH/libdrm_rocm_fix

671b088

Libdrm rocm fix

Merge branch 'main' into rocm_ifu_20220516

4eb0de6

Merge pull request #3 from ROCmSoftwarePlatform/rocm_ifu_20220516

96c3fbd

IFU 2022-05-16

refactor manywheel/build_rocm.sh

558113c

update manywheel/build_rocm.sh for ROCm 5.2

952e042

update scripts and Dockerfile for ROCm 5.2

3474f5d

update rocblas aux file list

87e5cef

Merge pull request #4 from ROCmSoftwarePlatform/rocm_52_fork

7ee79d2

[ROCm] add dependencies for rocm5.2

git224-core no longer available, use git236-core

156f24a

add ROCm 5.1.3 AMDGPU support

d211ae1

Add 5.1.3 support for libtorch

af4f26f

update libamd_comgr.so path for ROCm5.2 (#5)

0134ed6

* change the MAYBE_LIB64 path for just for 5.2

Add AMDGPU for ROCm5.2

36a5684

Merge branch 'main' into main_rocm

0238135

reconcile common/install_rocm_drm.sh with upstream

b086528

Updates to support rocm5.3 wheel builds (#6)

68bacb6

* Changes to support ROCm 5.3 * Updated as per comments

Installing python before magma build

fe0f987

- In ROCm 5.3 libtorch build are failing during magma build due to to missing python binary so added install statement

Move python install to libtorch/Dockerfile (#8)

02688de

Merge remote-tracking branch 'origin/main' into IFU-main-2022-09-15

5b83342

Updating the condition for noRCCL build (#9)

6b5b73e

* Updating the condition for noRCCL build * Updated changes as per comments

Merge pull request #10 from ROCmSoftwarePlatform/IFU-main-2022-09-15

8fc92e6

IFU-main-2022-09-15

Disable MLIR when building MIOpen

5a0ae80

Merge pull request #11 from ROCmSoftwarePlatform/disable_miopen_mlir_…

bc6fe73

…rocm_fork Disable MLIR backend when building MIOpen

Use MIOpen branch for ROCm5.3; Change all conditions to -eq

42f213f

Merge pull request #12 from ROCmSoftwarePlatform/rocm5.3

0468db5

Use ROCm5.3 branch for MIOpen

Use staging branch of MIOpen for ROCm5.3

c4d81fe

jithunnair-amd and others added 20 commits October 5, 2022 02:07

Merge pull request #13 from ROCmSoftwarePlatform/use_miopen_rocm5.3_s…

55db5ba

…taging_branch Use staging branch of MIOpen for ROCm5.3

Allow ROCm minor releases to use the same MIOpen branch as the major …

594e241

…release (#14)

correct logic to ensure rocm5.4 doesn't fall in wrong condition

9b4cdd0

Update for rocm5.4 branch

39f492a

Update amdgpu repo url for ROCm5.3

eac0e77

rocfft/hipfft link to libhiprtc.so in ROCm5.4 (#15)

6dd80e4

Disable MIOpen build from source for PyTorch wheels (#16)

1cffc92

* Remove miopen custom build step * Bundle MIOpen db files in wheel * Correct path

libtinfo.so version update and logic fix (#19)

7a44689

* Use libtinfo.so.6 for Ubuntu 2004 * Fix to origname grep * Condition on ROCM_VERSION for libtinfo6

Conditionalise librocfft-device so's out of rocm5.5 (#22)

21cc230

Co-authored-by: Jack Taylor <[email protected]>

Exit on missing file in build_rocm.sh (#23)

c0632f3

Merge remote-tracking branch 'upstream/main' into IFU-main-2023-03-03

ae59a10

Remove building magma from source

24f9208

Revert

23be9d1

Remove rocm5.1 rocm5.2 from libtorch Dockerfile

17b7478

Add miopen install rpm script (#26)

82ccc1c

Merge branch 'upstream_main' into IFU-main-2023-03-24

009ed17

Merge remote-tracking branch 'rocm_fork/IFU-main-2023-03-24' into roc…

be17939

…m_fork_main

Add magma conda package install support via MAGMA_PACKAGE_SOURCE

9f1167f

jithunnair-amd force-pushed the main branch from b2960e3 to af4827c Compare March 12, 2024 06:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add magma conda package install support via MAGMA_PACKAGE_SOURCE #30

Add magma conda package install support via MAGMA_PACKAGE_SOURCE #30

jataylo commented May 3, 2023 •

edited

Loading

Add magma conda package install support via MAGMA_PACKAGE_SOURCE #30

Are you sure you want to change the base?

Add magma conda package install support via MAGMA_PACKAGE_SOURCE #30

Conversation

jataylo commented May 3, 2023 • edited Loading

jataylo commented May 3, 2023 •

edited

Loading