Use stricter pattern for reading data from other files in conda recipes

[jameslamb]

Description

conda-build supports a function load_file_data() which can be used to load data from a file that is then referenced in other Jinja2 template statements throughout the recipes.

See "Loading data from other files" (conda-build docs).

Across RAPIDS, this is used in a few places to read data out of pyproject.toml files and re-use it in conda recipes. For example, in dask-cuda:

{% set data = load_file_data("pyproject.toml") %}

build:
  ...
  entry_points:
    {% for e in data.get("project", {}).get("scripts", {}).items() %}
    - {{ e|join(" = ") }}
    {% endfor %}

(dask-cuda/conda/recipes/dask-cuda/meta.yaml)

That function returns a Python dictionary. Because of the use of .get() in those statements, it's possible for data that should be present to be silently ignored.

Those should be switched to a stricter access pattern that'd result in a loud build-time error if any expected data is missing, like this:

build:
  ...
  entry_points:
    {% for e in data["project")["scripts"].items() %}
    - {{ e|join(" = ") }}
    {% endfor %}

Benefits of this work

• reduces the risk of conda packages having silently-broken medata
  • *(e.g., accidentally dropping a required dependency from the package metadata)
• makes the meta.yaml files slightly easier to read

Acceptance Criteria

• all RAPIDS conda recipes using conda's load_*() functions access those functions' data with a strict pattern that'll fail if any expected data is missing

Approach

The supported functions are:

• load_setup_py_data
• load_file_data
• load_str_data

Use GitHub search to double-check that the task list attached to this issue captures all uses of them across RAPIDS.

Whenever those are used, switch to a stricter pattern using [] subsetting instead of .get().

Notes

Inspired by my experience in rapidsai/ucx-py#1044, moving dependency data around in pyproject.toml and needing to adjust a corresponding conda recipe.

Tasks

Preview Give feedback

1. dask-cuda (make conda recipe data-loading stricter dask-cuda#1349)
2. numba-cuda (make conda recipe data-loading stricter NVIDIA/numba-cuda#3)
3. pynvjitlink (make conda recipe data-loading stricter pynvjitlink#90)
4. rapids-build-backend (make conda recipe data-loading stricter rapids-build-backend#44)
5. ucx-py (use rapids-build-backend ucx-py#1044)

[jameslamb]

"always use .get()" might be too strict / not possible.

See these discussions:

• Add Conda packages and CI workflows pynvjitlink#33 (comment)
• use rapids-build-backend ucx-py#1044 (comment)

And this failing PR: rapidsai/dask-cuda#1349

[jameslamb]

There was an interesting case I ran into in dask-cuda, where it seems like this is not supported:

{% for e in data.get("project", {}).get("scripts", {}).items() %}
  - {{ e|join(" = ") }}
{% endfor %}

but this is:

{% for k in data["project"]["scripts"] %}
  - {{ k ~ " = " ~ data["project"]["scripts"][k] }}
{% endfor %}

ref: rapidsai/dask-cuda#1349 (comment)

@jakirkham if you look at that and think "that's unexpected and looks like a bug in conda-build", let me know and I'd be happy to put together a bug report. I'm just not sure if it's a bug or just a gap in my understanding of Jinja2.

[jameslamb]

This is complete 🎉