Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make feasible for recipes to get a serialized subgraph of the current node #16173

Open
memsharded opened this issue Apr 29, 2024 · 2 comments · May be fixed by #17166
Open

Make feasible for recipes to get a serialized subgraph of the current node #16173

memsharded opened this issue Apr 29, 2024 · 2 comments · May be fixed by #17166
Milestone

Comments

@memsharded
Copy link
Member

Recipes can store in their own package binaries or in metadata the json (or other format)

@memsharded memsharded added this to the 2.4.0 milestone Apr 29, 2024
@stackfun
Copy link

stackfun commented Apr 29, 2024

Two use cases related to this:

  1. we want to quickly visualize the dependency graph of any package given its full package reference without having to download the recipes into the cache.

  2. Given a package reference and only the package reference (no additional settings/options), we'd like to load the full dependency graph into a Conan graph object in memory. Once we have the full dependency graph, we can download transitive dependencies, generate the packages list format, generate a lockfile, etc.

In Conan v1, we read the [full_settings] and [full_options] from conaninfo.txt, and use that configuration to load the graph from the conanfile.py. The [settings] and [options] is not enough to load the graph in a lot of situations as the settings and options could have been removed from the package_id calculation. For example, compiler.libcxx could be missing which is required in some platforms, or options values could be missing which don't specify a default which prevents us from loading the graph.

In Conan v2, the full_settings / full_options metadata hs been removed so we'll need a different way to load the graph given the package reference.

@memsharded
Copy link
Member Author

Given a package reference and only the package reference (no additional settings/options), we'd like to load the full dependency graph into a Conan graph object in memory. Once we have the full dependency graph, we can download transitive dependencies, generate the packages list format, generate a lockfile, etc.

In Conan v2, the full_settings / full_options metadata hs been removed so we'll need a different way to load the graph given the package reference.

Please note that it was removed because this is not how this is supposed to work. Conan can already compute the lockfiles, the dependency graphs with inputs the profiles. The package_id is a result, a function of a profile, but it is not intended that package_id are the inputs to resolve a dependency graph.
Whatever is stored in a package binary can be a "snapshot" of the dependency graph that was used at the moment of building that binary. But the same binary might be actually used with different dependencies. For example the same app/1.0:package_id1 binary for an executable will be exactly the same for different versions of the dependencies when those are shared libraries and do not bump the minor versions. So computing a dependency graph from whatever is stored inside the package will only give an outdated version of the dependencies, kind of a lockfile.

But if the intention is to have reproducible dependencies, then the recommendation is to use the built-in lockfiles, not having to do a custom creation of lockfiles from inside that. In the case of the executable application above, you can have 3 different lockfiles, for 3 different releases when some dependencies shared libraries have got new versions, but the executable is always app/1.0:package_id.

This feature is intended for the other use cases, like the visualization case you were describing above, or more like a logging, traceability. But this wouldn't be intended to be used for actual dependency resolution or reproducibility, Conan can achieve that without having to rely on those files.

@memsharded memsharded modified the milestones: 2.4.0, 2.X Jun 2, 2024
@memsharded memsharded modified the milestones: 2.X, 2.9.0 Oct 2, 2024
@memsharded memsharded linked a pull request Oct 15, 2024 that will close this issue
@memsharded memsharded modified the milestones: 2.9.0, 2.10.0 Oct 28, 2024
@memsharded memsharded modified the milestones: 2.10.0, 2.11 Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants