Flakes: add system-independent outputs?

[SomeoneSerge]

As far as I know an "appropriate" flakes interface for mach-nix is still more or less an open question, and it has already been discussed in: #153

As a resolution of that discussion, the mk(Python|...) functions have been added as lib.mk... flake outputs, which made them, well, accessible in the importing flakes.

However, those functions are currently extracted from default-mach-nix which has already been populated with a concrete version of nixpkgs, imported in a let in statement in mach-nix's flake.nix. This means that one can't pass those functions their own version of nixpkgs (e.g. final or prev in an overlay) without re-importing mach-nix manually as a non-flake.

One resolution could be to make lib functions system-independent (I thought they are, by-convention, supposed to be, but I'm not sure and I can't find references for that)

The other (and probably optimal, at the moment) resolution would be to provide a flake .overlay output, which would be a function final: prev: adding a mach-nix attrset to nixpkgs possessing all the appropriate functions. and derivations. If I understand correctly, this is what poetry2nix does.

P.S. I'm unsure, again, if it's a convention, but I saw in some repos that with an overlay output it might be convenient to refactor packages of flake.nix to just inherit the respective outputs from pkgs, where pkgs would be imported with this exact overlay. Thus we'd have a system-independent overlay output and the packages that refer to nixpkgs version pinned in the flake.lock, addressing the questions raised in the previous issue in a rather un-confusing way

P.P.S. I still don't know anything about whether pypy-deps-db and nixpkgs versions need to be matched in any way. E.g. if one uses a mach-nix overlay on their own revision of nixpkgs, would they ever need to pass a custom pypy-deps-db too? Should we make pypy-deps-db an argument of something, or perhaps pass it through the pkgs as well, so as to leave an option to override it in another overlay?

[DavHau]

Thanks for that. I'm not using mach-nix much with flakes for myself. Your input is useful.
Providing an overlay might be a reasonable approach to deal with the dependency on nixpkgs.
I'm open to experiment with the overlay. We can just add it and see how it goes. Are you motivated to work on this?

Concerning the import arguments which are unpractical to use for flake users:
One solution would be to add all arguments to the individual lib functions. There would be a default but it can be overridden on each function call.
The problem with this is, that it opens the door for bugs which are hard to debug. If for example a buildPythonPackage result is used inside mkPython, then they both must be executed on the same pypi-deps-db, otherwise collisions might happen.
It would be better if those arguments could be passed once via some init method to lower the risk of stuff being mixed the wrong way.
An alternative solution could be, instead of providing flakes users with a readily imported mach-nix, offer just a mach-nix initialization function that requires a set of attributes being passed (can be empty), similar to the current import function of mach-nix.

P.P.S. I still don't know anything about whether pypy-deps-db and nixpkgs versions need to be matched in any way. E.g. if one uses a mach-nix overlay on their own revision of nixpkgs, would they ever need to pass a custom pypy-deps-db too? Should we make pypy-deps-db an argument of something, or perhaps pass it through the pkgs as well, so as to leave an option to override it in another overlay?

pypi-deps-db should always be newer than nixpkgs. If used via flakes, it will raise an error if it is not. So that isn't an issue. The user will be informed if action needs to be traken. And yes, it definitely needs to be an argument either for the init functions and also for the lib functions.

The next question which arises, do we want to keep the API downward compatible or not? I vote for no, to reduce the amount of possible ways to call mach-nix.

[DavHau]

P.S. I'm unsure, again, if it's a convention, but I saw in some repos that with an overlay output it might be convenient to refactor packages of flake.nix to just inherit the respective outputs from pkgs, where pkgs would be imported with this exact overlay. Thus we'd have a system-independent overlay output and the packages that refer to nixpkgs version pinned in the flake.lock, addressing the questions raised in the previous issue in a rather un-confusing way

I bet that there isn't a convention. The flakes feature of nix is in experimental status. Everything could still be changed in a way that is not backward compatible. Also the lib output is not listed by flake show, so I very much assume that there is no specific structure expected by any nix tooling. It's just a random output. There aren't even official docs for flakes outputs at all AFAIK.

[SomeoneSerge]

We can just add it and see how it goes. Are you motivated to work on this?

Yes, I'd love to prepare a PR, hopefully soon

pypi-deps-db should always be newer than nixpkgs.

Oh, I wonder if the conflicts I reported in the other issue may be due to this...

P.S. My fingers seem to autocomplete system to systemd whenever something breaks...

[SomeoneSerge]

I poked about moving the dataOutdated check into the overlay as well and so far couldn't come up with anything reasonable. The nixpkgs attrset coming into the overlay has no lastModified. I couldn't find any way to recover the sources from it either. On #nix:nixos.org I was made aware of lib.version (which resolves to the semver), and I also found lib.trivial.revisionWithDefault which is supposed to readFile [...]/.git-revision but in the overlay the path information is already erased

On a meta-level, it doesn't look very compositional that the user is constrained to pin nixpkgs below mach-nix's

[DavHau]

The nixpkgs attrset coming into the overlay has no lastModified

Yep, that check will only work with the flakes of nixpkgs. We could just remove/ignore it for the overlay.

On a meta-level, it doesn't look very compositional that the user is constrained to pin nixpkgs below mach-nix's

It is not constrained to be below the mach-nix nixpkgs. It should just be older than the pypi-deps-db. And the user should be able to disable that restriction by setting ignoreDataOutdated when initializing mach-nix.