RFC: Use `compiler('cuda')` #121

conda-forge-admin · 2023-03-28T10:15:26Z

Update recipe to use compiler('cuda') to configure CUDA builds. Adds content needed to conda-build-config.yaml as pulled from conda-forge-pinning for the specific CUDA & CentOS versions used on the different architectures.

Hi! This is the friendly automated conda-forge-webservice.

I've rerendered the recipe as instructed in #120.

Here's a checklist to do before merging.

Bump the build number if needed.

Fixes #120

conda-forge-webservices · 2023-03-28T10:15:32Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

…nda-forge-pinning 2023.03.28.09.32.54

Can drop `enable_cuda` conditioning of `ucx` as the package works with or without CUDA now.

As `enable_cuda` is no longer used in `meta.yaml`, drop it from `conda_build_config.yaml`.

jakirkham · 2023-03-28T10:22:07Z

@conda-forge-admin, please re-render.

…nda-forge-pinning 2023.03.28.09.32.54

Since there is no `ucx` for `osx`, only require it on `linux`.

jakirkham · 2023-03-28T10:33:17Z

@conda-forge-admin, please re-render.

…nda-forge-pinning 2023.03.28.09.32.54

Transfer `cudatoolkit` constrain to `cuda` compiler. Also pull in associated `zip_keys` so zipping still works with these changes.

Equivalent to the previous behavior where `cuda` builds were always done on `linux`. This just skips non-`cuda` builds to restore that behavior with the `cuda` compiler.

jakirkham · 2023-03-28T10:54:46Z

@conda-forge-admin, please re-render.

…nda-forge-pinning 2023.03.28.09.32.54

Restores the behavior of `cudatoolkit` being optional when using the `cuda` compiler.

jakirkham · 2023-03-28T11:02:21Z

@conda-forge-admin, please re-render.

github-actions · 2023-03-28T11:04:01Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/openmpi-feedstock/actions/runs/4542296552.

…nda-forge-pinning 2024.01.05.22.51.09

When `{{ compiler("cuda") }}` is used in a package, a matrix of CUDA versions is considered including a non-CUDA build. As a result, there is always a non-CUDA build. This means that we can just check whether or not the current build uses CUDA and select the packages to include accordingly. In other words there is no need for a separate `enable_cuda` value. So this rewrites the recipe logic to obviate and drop `enable_cuda`. Should make it easier to follow for those familiar with other CUDA recipes.

jakirkham · 2024-01-06T07:39:42Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2024.01.05.22.51.09

This wouldn't be added as `run_exports_from` `{{ compiler('cuda') }}` are ignored. So add it manually.

jakirkham

Have commented on a few items of note below to provide any clarifications that might be useful

jakirkham · 2024-01-06T07:57:30Z

recipe/meta.yaml

+      ignore_run_exports_from:
+         - {{ compiler('cuda') }}  # [cuda_compiler != "None"]


Since we handle our CUDA dependencies via run_constrained, this disables any exports the compiler would have added

jakirkham · 2024-01-06T08:12:03Z

...ccuda_compiler_version11.8cxx_compiler_version11fortran_compiler_version11mpi_typeconda.yaml

 c_compiler:
 - gcc
 c_compiler_version:
- '12'
+- '11'


This was using GCC 12 before. Am guessing that is just because 12 is the default GCC version used if not otherwise overridden

Whereas CUDA 11.8, is configured with GCC 11

Not sure whether this matters for OpenMPI in practice. So just mentioning for clarity

jakirkham · 2024-01-06T08:12:34Z

...ccuda_compiler_version11.8cxx_compiler_version11fortran_compiler_version11mpi_typeconda.yaml

 cdt_name:
- cos6
+- cos7


The CUDA 11.8 Docker image is on CentOS 7. So this got bumped from CentOS 6 to 7

Also note that linux_aarch64 and linux_ppc64le already are using CentOS 7. So this really only affects linux_64 (bringing it inline with the others)

Am guessing the long tail of CentOS 6 is diminishing. Though wouldn't be surprised if there are still some CentOS 6 users out there

Anyways just mentioning it for awareness

jakirkham · 2024-01-06T08:12:56Z

...ccuda_compiler_version11.8cxx_compiler_version11fortran_compiler_version11mpi_typeconda.yaml

-cuda_version:
- '11.2'
+cuda_compiler:
+- nvcc
+cuda_compiler_version:
+- '11.8'


Instead of cuda_version, we now have cuda_compiler_version. Also moved to CUDA 11.8 as requested

Though could do 11.2 or another version just as easily. Just need to update the skip in the recipe

jakirkham · 2024-01-06T08:13:23Z

recipe/conda_build_config.yaml

-docker_image:  # [linux]
-  - quay.io/condaforge/linux-anvil-cuda:11.2           # [linux64]
-  - quay.io/condaforge/linux-anvil-ppc64le-cuda:11.2   # [ppc64le]
-  - quay.io/condaforge/linux-anvil-aarch64-cuda:11.2   # [aarch64]


This is now handled for us thanks to {{ compiler("cuda") }} and the global pinnings

jakirkham · 2024-01-06T08:13:41Z

recipe/conda_build_config.yaml

-cuda_version:  # [linux]
-  - 11.2       # [linux64]
-  - 11.2       # [ppc64le]
-  - 11.2       # [aarch64]


This is superseded by cuda_compiler_version, which is set for us

jakirkham · 2024-01-06T08:14:01Z

recipe/conda_build_config.yaml

-enable_ucx:
-  - True       # [linux]
-  - False      # [not linux]


Inlined this as it seems to always be used with Linux. Please let me know if I've missed something here

jakirkham · 2024-01-06T08:14:24Z

recipe/conda_build_config.yaml

-enable_cuda:
-  - True       # [linux]
-  - False      # [not linux]


When {{ compiler("cuda") }} is used, there are CUDA and non-CUDA builds. So we can check whether CUDA is used via other variables (like cuda_compiler_version)

jakirkham · 2024-01-06T08:14:38Z

recipe/meta.yaml

@@ -1,6 +1,7 @@
 {% set version = "5.0.1" %}
 {% set major = version.rpartition('.')[0] %}
-{% set build = 0 %}
+{% set cuda_major = (cuda_compiler_version|default("11.8")).rpartition('.')[0] %}


Short-cut determine the CUDA major version. Needed a fallback during conda-build's rendering passes (hence default). Though the default is overridden in the final result

jakirkham · 2024-01-06T08:14:52Z

recipe/meta.yaml

@@ -20,6 +21,8 @@ source:
 build:
  number: {{ build }}
  skip: true  # [win]
+  skip: true  # [linux and cuda_compiler_version != "11.8"]


This ensures Linux only builds with CUDA and only used CUDA 11.8 (no other versions)

jakirkham · 2024-01-06T08:58:09Z

It looks like we are using cross-compilation for linux_aarch64 and emulation for linux_ppc64le. Is that expected?

openmpi-feedstock/conda-forge.yml

Lines 1 to 3 in c577e78

    
           build_platform: 
        
             linux_aarch64: linux_64 
        
             osx_arm64: osx_64

openmpi-feedstock/conda-forge.yml

Lines 10 to 12 in c577e78

    
           provider: 
        
             linux_aarch64: default 
        
             linux_ppc64le: azure

leofang

Thanks, John, LGTM, let's let @dalcinl take a look too.

recipe/meta.yaml

leofang · 2024-01-08T00:22:07Z

It looks like we are using cross-compilation for linux_aarch64 and emulation for linux_ppc64le. Is that expected?

openmpi-feedstock/conda-forge.yml

Lines 1 to 3 in c577e78

build_platform:

linux_aarch64: linux_64

osx_arm64: osx_64

It is surprising. It seems the migration bot added the cross compiling line above in commit 2783362. Not sure why it'd do that?

leofang · 2024-01-08T00:34:34Z

It looks like we are using cross-compilation for linux_aarch64 and emulation for linux_ppc64le. Is that expected?

It is surprising. It seems the migration bot added the cross compiling line above in commit 2783362. Not sure why it'd do that?

Did a bit more digging, it seems up to the last PR that we merged (#135) we still ran emulation for linux_aarch64. Also, I recalled I didn't quite figure out how to cross-compile Open MPI (#86), so it shouldn't be the default. However, the bot commit was merged 10 months ago, so the only difference that I can think of is ~~someone~~ somehow switching to the CUDA compiler triggers this issue (that should have surfaced earlier). Let me force using QEMU and retry.

leofang · 2024-01-08T00:38:17Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2024.01.07.10.43.57

jakirkham · 2024-01-09T23:16:29Z

Thanks Leo! 🙏

Figured some wires were getting crossed in linux_aarch64. Just wasn't sure which way to untangle them. Thanks for sorting that out 🙂

jakirkham · 2024-01-09T23:17:22Z

@dalcinl what do you think about this PR?

leofang · 2024-01-10T16:55:24Z

Thanks John & Lisandro! Let's merge.

jakirkham · 2024-01-10T20:06:44Z

Thank you both! 🙏

dummy commit for rerendering

6145879

conda-forge-admin requested review from astrofrog-conda-forge, beckermr, bekozi and dalcinl as code owners March 28, 2023 10:15

conda-forge-webservices bot mentioned this pull request Mar 28, 2023

@conda-forge-admin , please re-render #120

Closed

conda-forge-admin requested review from leofang, minrk, msarahan and ocefpaf as code owners March 28, 2023 10:15

conda-forge-webservices[bot] and others added 3 commits March 28, 2023 10:17

MNT: Re-rendered with conda-build 3.24.0, conda-smithy 3.23.1, and co…

a77e1ba

…nda-forge-pinning 2023.03.28.09.32.54

Always use ucx

abf743b

Can drop `enable_cuda` conditioning of `ucx` as the package works with or without CUDA now.

Drop enable_cuda from conda_build_config.yaml

c8694e2

As `enable_cuda` is no longer used in `meta.yaml`, drop it from `conda_build_config.yaml`.

conda-forge-webservices[bot] and others added 2 commits March 28, 2023 10:24

MNT: Re-rendered with conda-build 3.24.0, conda-smithy 3.23.1, and co…

283c764

…nda-forge-pinning 2023.03.28.09.32.54

Require ucx on linux only

da85c82

Since there is no `ucx` for `osx`, only require it on `linux`.

jakirkham marked this pull request as draft March 28, 2023 10:29

Require cuda compiler on linux

5d92942

conda-forge-webservices[bot] and others added 3 commits March 28, 2023 10:35

MNT: Re-rendered with conda-build 3.24.0, conda-smithy 3.23.1, and co…

8c4a830

…nda-forge-pinning 2023.03.28.09.32.54

Constraint cuda compiler version

efaa5cd

Transfer `cudatoolkit` constrain to `cuda` compiler. Also pull in associated `zip_keys` so zipping still works with these changes.

Skip non-cuda builds on linux

b175e33

Equivalent to the previous behavior where `cuda` builds were always done on `linux`. This just skips non-`cuda` builds to restore that behavior with the `cuda` compiler.

conda-forge-webservices[bot] and others added 3 commits March 28, 2023 10:56

MNT: Re-rendered with conda-build 3.24.0, conda-smithy 3.23.1, and co…

497f255

…nda-forge-pinning 2023.03.28.09.32.54

Make cudatoolkit optional (again)

930c16e

Restores the behavior of `cudatoolkit` being optional when using the `cuda` compiler.

Drop extra <space>

1566c0b

Bump build/number to 2

0092d50

conda-forge-webservices[bot] and others added 2 commits January 6, 2024 07:30

MNT: Re-rendered with conda-build 3.28.2, conda-smithy 3.30.2, and co…

5668b7d

…nda-forge-pinning 2024.01.05.22.51.09

conda-forge-webservices[bot] and others added 3 commits January 6, 2024 07:41

MNT: Re-rendered with conda-build 3.28.2, conda-smithy 3.30.2, and co…

9a2268c

…nda-forge-pinning 2024.01.05.22.51.09

Readd cuda-version

f9d9512

This wouldn't be added as `run_exports_from` `{{ compiler('cuda') }}` are ignored. So add it manually.

Drop unneeded skip

7d4671a

jakirkham reviewed Jan 6, 2024

View reviewed changes

jakirkham changed the title ~~WIP, RFC: Use compiler('cuda')~~ RFC: Use compiler('cuda') Jan 6, 2024

jakirkham marked this pull request as ready for review January 6, 2024 08:18

jakirkham requested a review from astrofrog as a code owner January 6, 2024 08:18

leofang reviewed Jan 6, 2024

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

avoid cross compiling for linux

2abb406

conda-forge-webservices[bot] and others added 2 commits January 8, 2024 00:40

MNT: Re-rendered with conda-build 3.28.2, conda-smithy 3.30.2, and co…

605490c

…nda-forge-pinning 2024.01.07.10.43.57

remove erroneous addition of cudatoolkit

67ca1a7

leofang approved these changes Jan 8, 2024

View reviewed changes

leofang mentioned this pull request Jan 8, 2024

Port CUDA changes to v4.x branch #138

Open

dalcinl approved these changes Jan 10, 2024

View reviewed changes

leofang merged commit bea25a2 into conda-forge:main Jan 10, 2024
8 checks passed

This was referenced Jan 10, 2024

Rebuild for CUDA 12 w/arch + Windows support #139

Closed

Rebuild for CUDA 11.8 w/arch support #140

Closed

leofang mentioned this pull request Jan 25, 2024

Rebuild for libhwloc 2.9.3 #141

Merged

jakirkham mentioned this pull request Feb 7, 2024

Add sysroot to requirements/build #145

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Use `compiler('cuda')` #121

RFC: Use `compiler('cuda')` #121

conda-forge-admin commented Mar 28, 2023 •

edited by jakirkham

Loading

conda-forge-webservices bot commented Mar 28, 2023

jakirkham commented Mar 28, 2023

jakirkham commented Mar 28, 2023

jakirkham commented Mar 28, 2023

jakirkham commented Mar 28, 2023

github-actions bot commented Mar 28, 2023

jakirkham commented Jan 6, 2024

jakirkham left a comment

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham Jan 6, 2024

jakirkham commented Jan 6, 2024

leofang left a comment

leofang commented Jan 8, 2024

leofang commented Jan 8, 2024 •

edited

Loading

leofang commented Jan 8, 2024

jakirkham commented Jan 9, 2024

jakirkham commented Jan 9, 2024

leofang commented Jan 10, 2024

jakirkham commented Jan 10, 2024

		ignore_run_exports_from:
		- {{ compiler('cuda') }} # [cuda_compiler != "None"]

RFC: Use compiler('cuda') #121

RFC: Use compiler('cuda') #121

Conversation

conda-forge-admin commented Mar 28, 2023 • edited by jakirkham Loading

conda-forge-webservices bot commented Mar 28, 2023

jakirkham commented Mar 28, 2023

jakirkham commented Mar 28, 2023

jakirkham commented Mar 28, 2023

jakirkham commented Mar 28, 2023

github-actions bot commented Mar 28, 2023

jakirkham commented Jan 6, 2024

jakirkham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakirkham commented Jan 6, 2024

leofang left a comment

Choose a reason for hiding this comment

leofang commented Jan 8, 2024

leofang commented Jan 8, 2024 • edited Loading

leofang commented Jan 8, 2024

jakirkham commented Jan 9, 2024

jakirkham commented Jan 9, 2024

leofang commented Jan 10, 2024

jakirkham commented Jan 10, 2024

RFC: Use `compiler('cuda')` #121

RFC: Use `compiler('cuda')` #121

conda-forge-admin commented Mar 28, 2023 •

edited by jakirkham

Loading

leofang commented Jan 8, 2024 •

edited

Loading