[BUG] Conda recipe improvements

[jakirkham]

Am documenting some observations and suggestions after working a little bit with the recipes in RAPIDS. There are some things that could be improved and simplified by leveraging features from conda-build 3. Also there are some build practices from defaults and conda-forge that would be good to adopt to avoid some common issues. My hope is this makes things easier to maintain in the long run. May update this list as things come up. Thoughts on this list would be welcome.

• host for linked dependencies and build for CLI build tools (e.g. Split build and host custrings#155 )
• Build variants for python instead of x.x.
• Build matrices of different versions using build variants.
• Require pip and setuptools to be explicitly included (set add_pip_as_python_dependency to False)
• Disable dependency downloads by setuptools or pip during build
• Look into building with pip ( Use pip conda-forge/conda-forge.github.io#518 )
• Avoid requiring setuptools in run unless pkg_resources is used
• Using the new compiler syntax (can be combined with legacy compilers in the near term)
• Migrate to the new compilers in defaults
• Add a compiler (wrapper?) package for nvcc ( Shim package to interact with an **existing** NVCC install conda-forge/staged-recipes#8229 )
• Split packages for packages sharing content (e.g. cudf)
• run_exports for libraries that need to be pinned (e.g. libcudf)
• Spacing out version spec from package name ( conda-build 3 allows spec without space between name and version spec conda/conda-build#2828 )
• Dynamically linking cudatoolkit libraries
• Preserve other build string content (like the package hash) when altering it (as an example).

Note: I think these can be saved until things cool down a bit.

[kkraus14]

@jakirkham can a subpackage depend on another subpackage? I.E. can the cudf subpackage depend on the libcudf subpackage? From what I'm seeing it doesn't look possible yet.

[jakirkham]

Yes, this should be possible. Here's an example.

Were there specific issues that you encountered? Is there a recipe I can look at?

[jakirkham]

Have updated the nvcc compiler wrapper item (as there is a proposed solution) and added dynamically linking to cudatoolkit. Though am happy to discuss these further.

[jakirkham]

I'd be curious what the status is here. Also would be willing to help. In particular I think it would be interesting to try and get Conda compilers (and the NVCC wrapper compiler) working on one RAPIDS package (maybe cuDF?) and then see how things go.

[harrism]

Moving this old bug to 0.14 and assigning @jakirkham to decide if or how to proceed.

[jakirkham]

Sounds good. Thanks Mark! 😄

[vyasr]

@jakirkham any idea where we stand on this issue now? At least a few of the above issues no longer seem entirely relevant (e.g. using pip) while others have been addressed (the questions around compilers).

[jakirkham]

Certainly things have gotten better, but it looks like there is still more to do here. It will probably require someone to go through this in detail, which I have not yet done, but here are some cursory observations.

From a quick look, it doesn't look like adding pip to requirements/host is still needed. Setting add_pip_as_python_dependency to False would happen in either build scripts or Docker images as we decide (though am not seeing that from a quick search). It appears python x.x is still used in raft (cc @cjnolet), which we should drop. This example is using setuptools in run, but am unclear on whether it is needed. The disabling pip/setuptools downloads was mostly handled upstream (though there may be edge cases like split packages where we need to be mindful conda/conda-build#3993 , which is easy to do). We are still doing python setup.py install, which should be moved to pip install. The package hash is still missing from the build string. We may still want to consider using a split package to build libcudf & cudf and split the pieces out.

Investigating the usage of variants in a build matrix is still an interesting piece of work, but may involve other changes around handling global pinnings. This may require scoping.

So things have definitely improved (particularly on the compiler front), but there is still some room for improvement. Much of this is lower hanging fruit (especially given other changes that have occurred).

[vyasr]

Is the split not what was done in #10326?

I'm happy to help work through some of these, given all the ongoing build system work I think we'd benefit a lot from cleaning up some of these issues as we go.

[jakirkham]

That's a different kind of splitting. The splitting here is suggesting folding Python & C++ into the same recipe

Yep that's the approach we have taken so far and that seems reasonable

On the variant piece it may be productive to have a meeting with OPS. There has been some discussion about revisiting how pinnings are handled and this fits in with that work

[kkraus14]

FWIW, I raised an issue (#11119) about my struggles with using multi output recipes and folding cudf and libcudf into the same recipe will just further exacerbate that.

If the conda recipes are treated purely as an implementation detail and user consumption of them is explicitly out of scope, folding them all into a single recipe is somewhat reasonable, though it does limit the ability to parallelize the build process in building conda packages.

[jakirkham]

Thanks for surfacing that. Hadn't seen that issue. Agree that is something we want to take into account. Preferably before doing more splitting (since we will want to extend that to any added outputs)

[vyasr]

I just went through the above checklist and the bulk of the items that were previously unchecked were either completed or obsoleted by more recent changes (including the broad shift towards scikit-build/scikit-build-core, the requirement to invoke pip rather than setup.py, the various conda recipe improvements during the CUDA 12 work, etc). The remaining unchecked items are no longer clearly improvements for a variety of reasons. I think we can safely close this issue for now and consider any future improvements in light of the current state of the recipes. Also, we will want to coordinate such work across RAPIDS so that it's not specific to just cudf.