Reorg Nixpkgs --- rip off bandaid approach

[Ericson2314]

Issue description

Nixpkgs' architecture is not the best. Issues:

• Leaks memory: too many cycles mean nothing can be GC'd
• overriding is gross: We should compose small parts into the final thing we want. "Modifying" the whole post composition is fundamentally anti-declarative

Potential solution

• I think we need to expose all the functions we callPackage on their own.
• As a middle ground, also expose the function to be fixed ("all packages") but no fixed point "yet"
• derivations shouldn't leak either inputs, but just return string (single output) or record of strings. Anything we want to expose should be done with passthru
• No .override .overrideAttrs .extend or similar. If you want something different, you need to rebuild the fixed point manually.

This is a radical shift that would make many "minor" overrides quite a bit more annoying, in addition to other breakage. But I think it would also put us on a much firmer, more sustainable foundation. Any idea how we get there incrementally?

[roberth]

I have more questions than answers. Here we go :)

If you want something different, you need to rebuild the fixed point manually.

We'll want to help with this in some way. Also there's real information in the glue, that we can't ignore.

• package arguments have generic names, because otherwise the user can't know where to inject their version of things
• packages need to specify versions and variants of their dependencies somehow

The current pattern is to write an abstract package function somewhere in pkgs/, then callPackage it with the right dependency versions/variants in pkgs/top-level/all-packages.nix.

I suppose this information can be made first class as a function from something like pkgs to the package function arguments?

How will packaging functions like runCommand and buildGoModule be replaced?

rip off bandaid [...] Any idea how we get there incrementally?

This seems like a good idea because incremental changes would be too numerous and disorganized. For this change to be viable, we need a plan. I'd suggest experimenting in a fork, adding to Nixpkgs only, so that it can be merged without breaking continuity for users and maintainers. Packages can migrate individually. The two solutions should coexist in the same monorepo to make the migration feasible; incrementality not by making small changes, but by interoperability during the migration. The risk is that the migration gets stuck because the new solution isn't good enough, so that's why we need to validate the new solution in a fork first.

derivations shouldn't leak either inputs, but just return string (single output) or record of strings. Anything we want to expose should be done with passthru

The new mkDerivation equivalent must accept only declared arguments like name and env : attrsOf stringLike.

If it doesn't have overriding, we don't need passthru. // on the result will do.

nix-repl> builtins.length (builtins.attrNames hello)
42

I kid you not.

nix-repl> hello.exe
error: attribute 'exe' missing

😭

---

Some tangential(?) questions

• can we split stdenv into parts? There's the basics like mkDerivation and setup.sh and the hooks, and then there's toolchain packages, which we want to be able to override. stdenvNoCC is an abomination. I think we should have setupenv and toolchain in the buildInputs etc. "Standard" is an illusion, just like the FHS.
• can we split the mkDerivation logic into composable parts? There's a lot of logic that gets added by the language infrastructures. If the mkDerivation logic was composed, those language infrastructures can integrate a lot better. stdenv.mkDerivation: overlay style overridable recursive attributes #119942 allows defining such logic, but is restricted to "prototype inheritance" rather than something more composable like the module system.

[roberth]

Wrote NixOS/nix#6507 to discuss the definition of "package" as the interface between Nix itself and expressions. It needs to be far, far more conservative for compatibility reasons of course and I think these should be largely distinct conversations, but the connection is present.

[SuperSandro2000]

• No .override .overrideAttrs .extend or similar. If you want something different, you need to rebuild the fixed point manually.

This is a radical shift that would make many "minor" overrides quite a bit more annoying, in addition to other breakage. But I think it would also put us on a much firmer, more sustainable foundation. Any idea how we get there incrementally?

That's one of the main selling points of nix/nixpkgs.

[roberth]

[overriding is] one of the main selling points of nix/nixpkgs.

It is important to me too.

Overriding, as it exists, is inconsistent and a bit buggy at times because of the complexity of implementing it correctly while trying to work around the fact that multiple levels of overriding don't compose well.
For this reason alone, it's worth looking into alternatives.

The risk is that the migration gets stuck because the new solution isn't good enough, so that's why we need to validate the new solution in a fork first.

Let me add that reverting to the status quo is a perfectly acceptable outcome at any point before deprecation.

[Ericson2314]

Some tangential(?) questions

I think those are both excellent points, but it is useful to separate the steps causing mass rebuilds (those) from things that are more purely nix expr orgnaization and so should not.

[fogti]

An important thing to consider is improving how e.g. python packaging works, because it is really hard to "just" add a new python package in an overlay, because it must be overridden for each python version, instead of having e.g. a python fixpoint extender, like:

final: prev: {
  python-all-pkgs = python-fix: python-interp: (prev.pyrthon-all-pkgs python-fix python-interp) // {
    my-awesome-package = python-interp.mkPythonModule /* etc */;
  };
}

[tomberek]

I think we need to expose all the functions we callPackage on their own.

I completely agree. More and more power has been added to make manipulation of the post-callPackage'd derivation and package sets convenient, but it has drawbacks that make large-scale composition difficult.

I've been calling the function that can be callPackage'd a "proto-derivation" - the thing that can produce a derivation. I've been using this as the base unit of composition, as opposed to override/overlay approaches. This tends to produce easier to understand behavior and easier composition. The outcome can still use overrides, but that becomes far less common. Re-use is as easy as thing = callPackage "${some-other-src}/some-dir" {a-override = 42;};

Using scopes allows nice unification of various package sets enabling this to work:

using pkgs {
          hello-go = ./pkgs/hello-go;
          hello-perl = ./pkgs/hello-perl;

          haskellPackages = using pkgs.haskellPackages {
            hello-haskell-library = ./pkgs/haskellPackages/hello-haskell-library;
          };
          hello-haskell = ./pkgs/hello-haskell; # has access to above library and rest of pkgs.haskellPackages

          python3Packages = using pkgs.python3Packages {
            hello-python-library = ./pkgs/python3Packages/hello-python-library;
          };
          hello-python = ./pkgs/hello-python;

          # Escape-hatch into full nixpkgs overrides
          hello-python-override = callPackage: (callPackage ./pkgs/hello-python {})
                 .overrideAttrs (_: {name="hello-python-override";});

          # Example: Use the nix expresssions in an external project
          project-hello-rust = "${project}/some-dir";
        }

[roberth]

@tomberek Looks nice. Some questions.

• What do you mean with "scopes"?
• What will using be responsible for?
• How are the per package default arguments declared?

---

Doesn't seem like a bad idea to start listing requirements. I'll make a start.

Must have:

• Packages can declare which (variants of) dependencies that want to inject into their generic parameters
• Cross compilation keeps working
• Dependencies are memoized (multiple uses of the same derivation cause only one evaluation of the package function)
• Package sets can be customized without forking Nixpkgs
• A transition path: sufficient compatibility with existing Nixpkgs patterns, back and forth
• ...

Nice to have (maybe):

• Other packages / package sets can be introspected while constructing the overrides
  • borrowing other packages' choices of dependencies: nix.inputs.boehmgc (not entirely unlike input propagation, but at the eval level, and precise)
  • omitting packages from a package set based on a condition, previously: self: super: optionalAttrs super.stdenv.isLinux { ..... })
• ...

[infinisil]

The Nixpkgs Architecture Team has now been formed to discuss and develop ideas like this.