update RAPIDS dependencies to 24.4, refactor dependencies.yaml

[jameslamb]

Proposes updating all remaining 24.2.* dependencies to 24.4.*.

I believe that's the desired behavior for the branch-24.04 branch and that these were just missed in 6e76da4#diff-5475a6d76de4c506ee92cf6f941bba30a6a07b5881cfea531d42a6ec035095a6.

This also pulls in some dependency refactoring originally added in #5711, which allows greater use of dependencies.yaml globs (and therefore less maintenance effort to support new CUDA versions).

[bdice]

@jameslamb You uncovered a real issue, I just found the same thing with rapidsai/ucxx#171 and it affects a bunch of repos across RAPIDS.

Context: We've been migrating our dependencies.yaml to specify RAPIDS packages in separate depends_on_X lists like this with separate matrix entries for the -cu11 and -cu12 packages. This separation of lists for RAPIDS packages helps with making devcontainers / conda / pip all happy, and centralizes some of the things like extra index URLs that need to be added. However, we haven't aligned ci/release/update-version.sh to accommodate those suffixed entries. We probably need to find a strategy for unifying this across RAPIDS. I would suggest that we experiment with this sed command to ensure it covers the suffixed entries. This is probably a good candidate for an issue in https://github.com/rapidsai/build-planning so we can determine a strategy and sweep through all of RAPIDS.

[bdice]

This PR needs some updates to ci/release/update-version.sh to prevent this from reoccurring. We might also want to port over some of the dependency refactoring in #5711 (some issues in dependencies.yaml were fixed there) so we only have to fix update-version.sh once, but I'll leave that choice to you.

[raydouglass]

@jameslamb You uncovered a real issue, I just found the same thing with rapidsai/ucxx#171 and it affects a bunch of repos across RAPIDS.

Looks like just cudf & cuml were affected during creating branch-24.04. Un-merged PRs could be affected too.

I opened rapidsai/cudf#14825 to fix it in cudf. But something more robust across all of RAPIDS is a good idea.

[jameslamb]

This is probably a good candidate for an issue in https://github.com/rapidsai/build-planning so we can determine a strategy and sweep through all of RAPIDS.
...
... something more robust across all of RAPIDS is a good idea.

I'll open an issue in build-planning @bdice @raydouglass .

This PR needs some updates to ci/release/update-version.sh to prevent this from reoccurring. We might also want to port over some of the dependency refactoring in #5711

I just pushed commits that do both these things, and re-requested a review.

[jameslamb]

This devcontainer job failure looks like a permissions issue to me.

 -- CPM: Adding package [email protected] (branch-24.04)
  
  [1/9] Creating directories for 'cumlprims_mg-populate'
  [1/9] Performing download step (git clone) for 'cumlprims_mg-populate'
  Cloning into 'cumlprims_mg-src'...
  Host key verification failed.
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
  Cloning into 'cumlprims_mg-src'...
  Host key verification failed.
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
  Cloning into 'cumlprims_mg-src'...
  Host key verification failed.
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
  -- Had to git clone more than once: 3 times.
  CMake Error at cumlprims_mg-subbuild/cumlprims_mg-populate-prefix/tmp/cumlprims_mg-populate-gitclone.cmake:39 (message):
    Failed to clone repository: '[email protected]:rapidsai/cumlprims_mg.git'

(build link)

[jameslamb]

Maybe this deploy key has been removed?

cuml/.github/workflows/pr.yaml

Line 117 in a93040c

extra-repo-deploy-key: CUMLPRIMS_SSH_PRIVATE_DEPLOY_KEY

@raydouglass is that something you could check?

[jameslamb]

Discussed offline w/ @raydouglass

I misinterpreted that erorr "Host key verification failed." It's not about this repo's SSH keys... it's about the key fingerprint of the github.com host.

We suspect this was a temporary disruption on GitHub's side and that a re-run of CI here will resolve the failing devcontainer build (build link).

[bdice]

Do we know why were these missed by update-version.sh previously? They're not -cu11 or -cu12 names. Was it due to a forward-merge after branch-24.04 was created?

[raydouglass]

The &rmm_conda part prevented being updated. That's why the line in update-version.sh added the .* as in -.* ${DEP}...

[jameslamb]

Makes sense to me!

The presence of - in these commands also explains why some projects have separate lines for updating pyproject.toml files, like this:

https://github.com/rapidsai/cugraph/blob/f20219ed62aaade9ff21e5867ea828ee20baef9b/ci/release/update-version.sh#L97-L107

[jameslamb]

Looks like the devcontainers build is now passing thanks to the changes in rapidsai/devcontainers#214.

Thanks @trxcllnt !!!

[AyodeAwe]

/merge

[AyodeAwe]

Admin merged on author's request

[jakirkham]

Thanks all! 🙏