Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix pytorch-cpu/gpu build only for single python version #282

Merged
merged 4 commits into from
Nov 5, 2024

Conversation

jeongseok-meta
Copy link
Contributor

@jeongseok-meta jeongseok-meta commented Nov 3, 2024

Resolves #281

Checklist

  • Used a personal fork of the feedstock to propose changes
  • Bumped the build number (if the version is unchanged)
  • Reset the build number to 0 (if the version changed)
  • Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
  • Ensured the license file is being packaged.

@conda-forge-admin
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@jeongseok-meta jeongseok-meta force-pushed the fix_pytorch_cpu_strings branch from ce32d55 to 35cba95 Compare November 4, 2024 00:26
@jslee02
Copy link

jslee02 commented Nov 4, 2024

aarch64 + cuda 12.0 build failed: linux_aarch64_c_compiler_version12c_stdlib_version2.28cuda_compilercuda-nvcccuda_compiler_version12.0cxx_compiler_version12-log.zip

Could not solve for environment specs
The following packages are incompatible
├─ �[31mpytorch-gpu 2.5.1.* cuda120hf733e3a_202�[0m is not installable because it requires
│  └─ �[31mpytorch 2.5.1 cuda_generic*202�[0m, which does not exist (perhaps a missing channel);
└─ �[31mpytorch 2.5.1.* cuda_generic*202�[0m does not exist (perhaps a typo or a missing channel).

recipe/meta.yaml Outdated Show resolved Hide resolved
Co-authored-by: Mark Harfouche <[email protected]>
@hmaarrfk
Copy link
Contributor

hmaarrfk commented Nov 4, 2024

here goes an other 2 hours.....

@jslee02
Copy link

jslee02 commented Nov 4, 2024

Still building, but it seems to work! Spoken too early. It's pytorch:

image

and pytorch-cpu still outputs for only one python version:

image

Build log: log_linux64_mkl_cpu.zip

@jeongseok-meta
Copy link
Contributor Author

Wait, perhaps this is the expected behavior (not specific to any python version). If so, it appears to be working as intended. I'll continue building...

@jeongseok-meta jeongseok-meta marked this pull request as ready for review November 4, 2024 14:32
@jeongseok-meta
Copy link
Contributor Author

Still building, but the estimated time to completion is 4-5 hours.

Packages that have been built so far:

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Nov 4, 2024

Wait, perhaps this is the expected behavior (not specific to any python version). If so, it appears to be working as intended. I'll continue building...

Correct, this is the expected behavior.

When the megabuild happens a single pytorch-cpu will be made that can point to any pytorch for any version of python.

The old used exact which doesn't work well for "splayed" builds where we have:

  1. C++ build.
  2. Many python builds

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Nov 4, 2024

I'm sorry to say this, but I think it might be good to include this patch:
pytorch/pytorch@a00faf4

It is manifesting as a failure in conda-forge/torchvision-feedstock#102

@jeongseok-meta jeongseok-meta mentioned this pull request Nov 4, 2024
5 tasks
- {{ pin_subpackage("pytorch", exact=True) }}
- pytorch {{ version }}=cuda*{{ PKG_BUILDNUM }} # [megabuild and cuda_compiler_version != "None"]
- pytorch {{ version }}=cpu_{{ blas_impl }}*{{ PKG_BUILDNUM }} # [megabuild and cuda_compiler_version == "None"]
- {{ pin_subpackage("pytorch", exact=True) }} # [not megabuild]
Copy link

@danpetry danpetry Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you might want to remove this line, otherwise you'll fix to a python version for non-megabuild variants?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The non megabuillds should have multiple packags created. Is that not the case?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok. I'm sure you're right, I need to check more carefully I think.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still getting familiar with your recipe.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The megabuilds were created since before the large runner from quantsigh hour builds, we used to HAVE to build locally. And it would effectively save me a 12 hours of rebuilding libtorch 4 times.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I guess "megabuild" = 1 x libtorch and 5 x python bindings/code, as opposed to 5 x normal builds (libtorch + bindings + python code)?

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Nov 5, 2024

anaconda copy --from-label main --to-label main --to-owner conda-forge jslee02/libtorch/2.5.1
anaconda copy --from-label main --to-label main --to-owner conda-forge jslee02/pytorch/2.5.1
anaconda copy --from-label main --to-label main --to-owner conda-forge jslee02/pytorch-gpu/2.5.1
anaconda copy --from-label main --to-label main --to-owner conda-forge jslee02/pytorch-cpu/2.5.1

@hmaarrfk hmaarrfk merged commit b1c36f0 into conda-forge:main Nov 5, 2024
18 of 30 checks passed
@jeongseok-meta jeongseok-meta deleted the fix_pytorch_cpu_strings branch November 5, 2024 04:07
@jeongseok-meta jeongseok-meta restored the fix_pytorch_cpu_strings branch November 7, 2024 01:52
@jeongseok-meta jeongseok-meta deleted the fix_pytorch_cpu_strings branch November 7, 2024 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix pytorch-{cpu,gpu} packages are built only for single python version
5 participants