[BE]: Update cudnn to 8.9.7.29

[Skylion007]

Update cudnn to 8.9.7.29 . We just updated the cudnn frontend, might as well. Mostly has improvements for the cudnn flash attention implementation which we are interested in exploring. Such as in #115663

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/120642

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 4 Unrelated Failures

As of commit ece0cce with merge base 8bf9e99 ():

NEW FAILURES - The following jobs have failed:

• Lint / workflow-checks / linux-job (gh)
  ##[error]The operation was canceled.
• periodic / parallelnative-linux-jammy-py3.8-gcc11 / build (gh)
  Process completed with exit code 1.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• periodic / linux-focal-cuda11.8-py3.9-gcc9 / test (multigpu, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh) (similar failure)
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
• periodic / linux-focal-rocm6.1-py3.8 / test (distributed, 1, 2, linux.rocm.gpu) (gh) (similar failure)
  distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16_size1
• periodic / linux-focal-rocm6.1-py3.8 / test (distributed, 2, 2, linux.rocm.gpu) (gh) (similar failure)
  distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_compile_attention_fn1
• periodic / linux-vulkan-focal-py3.11-clang10 / test (default, 1, 1, linux.2xlarge) (gh) (similar failure)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Skylion007]

@pytorchbot merge

[pytorchmergebot]

Merge failed

Reason: Approvers from one of the following sets are needed:

• OSS CI (alband, dagitses, pytorch/pytorch-dev-infra)
• superuser (pytorch/metamates)
• Core Reviewers (mruberry, lezcano, Skylion007, ngimel, peterbell10)
• Core Maintainers (soumith, gchanan, ezyang, dzhulgakov, malfet)

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

[Skylion007]

@pytorchbot merge

[malfet]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

Merge failed

Reason: 3 jobs have failed, first few of them are: trunk, linux-binary-manywheel, linux-binary-libtorch-cxx11-abi

Details for Dev Infra team

Raised by workflow job

[Skylion007]

@pytorchbot merge

[malfet]

Ok, the sad thing is: there are no nvidia-cudnn-cu11==8.9.7.29 package on pypi , only 8.9.6 at the time of the writing:

[eqy]

Let me ping cuDNN about it as 8.9.7 is among the more well-tested released and should be hosted at this point

[ptrblck]

Yes, @malfet is correct and nvidia-cudnn-cu11==8.9.7.29 is not available as cuDNN is running into the project size limitations.
RFE to increase the size limit for nvidia-cudnn-cu11: pypi/support#3402
RFE to increase the size limit for nvidia-cudnn-cu12: pypi/support#3408

Both were opened in ~Dec. 2023 and we haven't received any updates yet. Unsure if @rgommers would know more

[rgommers]

Both were opened in ~Dec. 2023 and we haven't received any updates yet. Unsure if @rgommers would know more

Not much more. There is an active discussion on Discourse about issues with responses to PyPI support requests in general. I just commented a few days ago on that thread about issues with limit size requests - no response or activity yet.

I'll note that the situation for JAX that I linked to in that comment seems even worse; they have been deleting their old releases for a while now to make space for new releases (which of course broke some users who pinned those old versions).

For PyTorch itself I think we've avoided hard release blockers so far (correct me if I'm wrong @malfet), but it really isn't a good situation that important projects like cuDNN cannot upload releases.

The canonical discussion on this is "What to do about GPUs? (and the built distributions that support them)"; that never really got resolved.

I am considering writing a larger new post about it later this week, depending on the urgency, but I really don't know if it'll help given how politicized that issue has become. There aren't many PyPI admins, and most are volunteers. And the one active PSF staff member doesn't have much time allocated to this; it looks like they're trying to get a new paid support engineer funded by the PSF, but that may take a while to materialize.

[Skylion007]

It would be nice to update to the latest CUDNN that we can support for 2.3. What's the latest version that is feasible to support?

[malfet]

I can update our index to 8.9.7.29 from NVIDIA's pypi that should unblock both this change and the release

[atalman]

Please update only 11.8 cudnn version. At this point we want to upgrade only cuda 11.8 cudnn.

[ptrblck]

@atalman why do we not want to update cuDNN for the CUDA 12 builds? This would cause a divergence.

[atalman]

@ptrblck Sorry yes, issue exist in cudnn 11 but not 12, my bad https://pypi.org/project/nvidia-cudnn-cu12/#files. I do see cudnn 8.9.7.29 available.

[atalman]

@Skylion007 @malfet Here are additional work required for cudnn : https://github.com/pytorch/builder/blob/main/CUDA_UPGRADE_GUIDE.MD#upgrade-cudnn-version-only
For Windows we would need to rebuild the AMI.

[atalman]

@pytorchmergebot rebase -b main

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

[pytorchmergebot]

Successfully rebased skylion007/update-cudnn-8-9-7-29 onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout skylion007/update-cudnn-8-9-7-29 && git pull --rebase)

[Skylion007]

Closing in favor of #123475

[Skylion007]

@pytorchbot rebase

[pytorchmergebot]

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[pytorchmergebot]

Successfully rebased skylion007/update-cudnn-8-9-7-29 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout skylion007/update-cudnn-8-9-7-29 && git pull --rebase)

[nWEIdia]

Closing as v9 is in :)