Option to generate run_exports.json file

[jaimergp]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

In this issue I am asking for a CLI flag / option to generate a run_exports.json file, next to repodata.json and friends, much like channeldata.json is handled.

Once available, this flag could be enabled on Anaconda.org (at least for some channels).

Why is this needed?

Building infrastructure (such as some conda-forge bots) use run_exports to calculate which packages need rebuilding as part of an upgrade. Right now, conda-forge needs to maintain their own JSON database, which involves downloading and extracting the new artifacts as available.

What should happen?

When invoked with the new flag (e.g. --run-exports), conda-index would also generate a
run_exports.json file, which would be placed next to repodata.json, containing the run_exports metadata from each artifact:

{
  "info": {
    "subdir": "osx-64"
  },
  "packages": {
    ...
    "assimp-5.2.5-h276577b_0.tar.bz2": {
      "run_exports": {
        "weak": ["assimp >=5.2.5,<5.2.6.0a0"],
        "strong": [],
      }
    },
    ...
  }
}

To be discussed: run_exports is not a very common field (most packages do not have it), so it might make sense to only generate entries for those artifacts whose run_exports are populated.

Additional Context

channeldata.json already offers some run_exports metadata. However, this is insufficient because for some reason it is flatted on a per-version basis, which doesn't cover build updates.

Having a separate file for run_exports would also allow conda-forge to patch it like it is done with repodata.json, saving the hassle (and CI resources) of rebuilding a package just to amend some metadata.

Note: adding this field to repodata.json is also an option, but conda clients do not really use this at install time. It would only be used by conda-build and the surrounding infrastructure.

[jaimergp]

Gentle ping @conda/conda-index and @conda/conda-core :)

[chenghlee]

I've added this as a topic of discussion for this week's community meeting. I like the idea of providing per-artifact run_exports, but my brain is still post-PyCon mush mode to properly think through the pros & cons of having a new file vs. an extension in repodata.json.

[jaimergp]

Thanks @chenghlee! Let me start a list we can iterate on:

repodata.json	run_exports.json
✅ Already existing file	✅ Better separation of concerns
✅ Less changes in conda-index, CDN infra?	✅ conda clients do not need to worry about an extra file
❌ Bigger repodata files for data not used by conda clients (although this is minimized by JLAP)	❌ Might require more efforts in conda-index, CDN infra and other projects
❌ Might require updates in tools that load repodatas with schemas

[dholth]

Only needed by conda-build? Only need to generate new file then update a few upload scripts. Do we hotfix this field?

…

On Tue, Apr 25, 2023, 11:09 AM Cheng H. Lee ***@***.***> wrote: I've added this as a topic of discussion for this week's community meeting. I like the idea of providing per-artifact run_exports, but my brain is still post-PyCon mush mode to properly think through the pros & cons of having a new file vs. an extension in repodata.json. — Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABSZEQOL55GWNOBU4RURJLXC7SK3ANCNFSM6AAAAAAW6P4V6M> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[jaimergp]

We have not been able to hotfix or repodata-patch this field in the past, and it has been a limitation in conda-forge too. Enabling that would allow us to skip some unnecessary rebuilding (sometimes we have to rebuild just to patch run_exports).

[jaimergp]

CEP-12 is now a thing and the community approved this feature.

The gist is:

• run_exports.json will be served next to repodata.json; aka one file per subdir.
• All files need to be listed, even if no run_exports data is available for them.
• This file won't be patched for now.

I'm happy to start a PR targeting this, but let me know how you'd like to proceed or if you have any pointers, @dholth! Thanks!

[dholth]

Why list all empty files?

…

On Mon, Aug 14, 2023, 12:24 PM jaimergp ***@***.***> wrote: CEP-12 <https://github.com/conda-incubator/ceps/blob/main/cep-12.md> is now a thing and the community approved this feature. The gist is: - run_exports.json will be served next to repodata.json; aka one file per subdir. - All files need to be listed, even if no run_exports data is available for them. - This file won't be patched for now. I'm happy to start a PR targeting this, but let me know how you'd like to proceed or if you have any pointers, @dholth <https://github.com/dholth>! Thanks! — Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABSZEQEU7ZRKJQKVOJZX2DXVJGLTANCNFSM6AAAAAAW6P4V6M> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jaimergp]

To ensure that all files indexed are indeed contained in the run_exports.json. Otherwise we could run into questions like "is this file out of date or is there a bug in the processing pipeline?". This also allows us to use it without having to check repodata.json for the full list.

[jaimergp]

Closed by #110