[RFC 0083] Common interface package sets

[FRidh]

rendered

[FRidh]

cc @adisbladis @doronbehar @jtojnar @Ericson2314 because of your various involvements with package sets.

[shlevy]

Possible interaction with flakes: I'd like to be able to have the functions to build a package set in a flake, as well as the ability to declaratively overlay package sets in my flake config.

[Ma27]

Even though I like flakes, I'm not so sure if it's a good idea to design any RFC around any feature that's not yet stabilized (i.e. the APIs can all change and it's not in a stable Nix release) or even documented.

[jtojnar]

I though we were moving towards foo.pkgs as it makes it harder to have a mismatch of foo versions.

[FRidh]

I don't follow you here. Do you refer to how we pass in say python or pythonPackages instead of the individual Python packages to a non-Python package? That is indeed to prevent mismatches, however, it does not matter whether it is python or pythonPackages.

[jtojnar]

I meant that if we have two top-level attributes, overriding just one in an overlay can cause mismatches (unless they refer to each other through scoped names but that has its own downsides). Stuffing everything under the interpreter allows us greater control over references thanks to encapsulation (but maybe at performance cost compared to just bare scopes?).

[doronbehar]

I thought we do: fooPackages = lib.recurseIntoAttrs foo.pkgs;.

[jtojnar]

recurseIntoAttrs only has effect on whether tools will try to find packages under that attribute. How coupled the two attributes are depends on which scope does foo come from.

[Ericson2314]

I've always been wary of foo.pkgs because one compiler/interpreter can build multiple package sets. Having the package set of a tool is philosophically backwards, it should be the compiler/interpreter of the package set.

[adisbladis]

it should be the compiler/interpreter of the package set

In theory, I would agree. In practice however this makes it so that you have to take care to not pass an incompatible Python version along with it's set.
I think the practicality of this outweighs any philosophical notion.

[sternenseemann]

But in practice both python3.pkgs and python2.pkgs would be the same sets, but with different behavior depending on the python versions which dynamically determine the value of disabled and similar stuff. In that sense it isn't even possible to pass an incompatible python interpreter if I'm not mistaken.

Also in other language-specific package sets there is much less breakage between compiler versions than between python2 and python3 where it would make even less sense.

[jtojnar]

For PHP, we create the php.pkgs internally in the PHP derivation “template” function. That is, a package set for the new variant is just created automatically when the variant is created (e.g. using import "${nixpkgs}/pkgs/development/interpreters/php/generic.nix") { version = "..."; sha256 = "..."; }.

[FRidh]

From the looks of it is somewhat similar to the passthru for Python.

[jtojnar]

Do we even want nesting? For example, for PHP, we could have two clearly separated package sets:

• tools – those are just scripts written in PHP, they would probably not even need to be under PHP, just like we do not put Python applications to python.pkgs.
• extensions – shared libraries linked against PHP that PHP needs to load at startup

PHP does not really need something like python.withPackages since PHP packages are just text and it is the application, not PHP runtime, responsible for loading them. (AFAIK unlike in Python, anything binary needs to be an extension, not a regular autoloaded package.)

[FRidh]

Do we even want nesting?

I don't know actually. So far it seems we have not, so we can just leave it out of the RFC.

[jtojnar]

For PHP, withPackages does not make sense either, since we do not actually package any packages other than applications.

For the extensions, I think that it should be php.withExtensions since that does not create an environment but creates a config file listing the extensions and wraps php with a path to the config file.

[FRidh]

I think this shows why withPackages is often enough, but not always and it hence should be a recommendation, but not a rule. In the end, we need a more powerful buildEnv that can compose an environment given a mixture of types of packages using certain rules.

[doronbehar]

"powerful buildEnv" -> #75 ...

[fare]

In some package sets, you might want to combine multiple aspects: which version of the compiler should I use with which version of the libraries?

Plus some libraries are only available on some versions of the compiler, and I may want to pick and choose: "oh, I'd like these stable libraries with version 3.4 of the compiler, except I want override these particular libraries with newer versions I need—plus their dependencies."

I don't think the job of nixpkgs would be to solve complex sets of dependency constraints, but I believe its job should be to check and enforce the declared constraints (unless explicitly lifted / overridden by the user).

For all these reasons, I think it would be nicer to standardize on some extension mechanism when there are so many incompatible variants of the same thing.

At some point, I would offer POP as a better such extension mechanism to standardize on, that allows for a DAG of dependencies between aspects of your package set. But POP is not quite ready for that yet (the API needs to be made nicer, and I ought to use the C3 algorithm for linearization). [See indeed #82 for a proposal that allows to introduce POP smoothly into nixpkgs, by first tagging it as experimental.]

[fare]

Also, another thing I'm trying to do is allow override of "source" settings before libraries are turned into derivations. Either there should be a standard for such settings to be included in passthru, or for otherwise organizing collections of pre-packages that allow such overrides.

[sternenseemann]

Somewhat related NixOS/nixpkgs#75485

[adisbladis]

Suggested change

	appliceable. An example of where it would not make sense to include such a
	applicable. An example of where it would not make sense to include such a

[FRidh]

Referring to the incorrect set NixOS/nixpkgs#114538