Allow plain Nix expressions as flake inputs

[thufschmitt]

(This is mostly a reformulation and extension of #3843 (comment). I’ve been meaning to post this as its own issue for a long time as it’s a potential solution for a bunch of different issues)

As a possible way out of the combinatorial explosion of flake outputs required by the impossibility to pass arguments to flakes (the system issue, dirty things like #4996 required to accomodate everyone’s need, etc..), we could add a new type of flake input nixExpr which would inline a Nix expression.

Motivations

Fix #3843

I’m not sure which one would be best, but this would enable several solutions to #3843:

• Flakes could effectively take the current system as input by having inputs.system = { type = "nixExpr"; value = "x86_64-linux"; } which could be changed with --override-input system nixExpr:x86_64-darwin. Probably needs a better UX, but the gist of the mechanism would be here.
• Less invasive (and more eval-cache-friendly), we could keep the packages.{system}.foo structure, but have inputs.allowedSystems = { type = "nixExpr"; value = [ "x86_64-linux", "x86_64-darwin" ]; }, so that if someone wants to build the flake for another system, it’s possible to override this list

More generally, allow for parametrized matrix builds

• I’m not really familiar with all the details of cross-compilation, but the system approach is probably pretty limited if one wants to express all the possible cross-compilation strategies between a few different system types
• Restore the ability to build with clang on linux #4129 also wants to add a new dimension to the list of possible builds of the Nix flake. Currently the values of this new dimension have to be hardcoded, but with this approach, we could just have inputs.stdenv.value = "stdenv" (and define the actually used stdenv as stdenv = nixpkgs.${inputs.stdenv}), so that ppl wanting to use clang can just do --override-input stdenv "nixExpr://clangStdenv" (or whatever)

Design proposal

(This is only a first draft, could certainly be refined).

• Flakes would have a new input type nixExpr.
• This input type would take as only argument value, which is a json-style Nix value. So not an arbitrary expression, but something arbitrarily (though finitely, rec shouldn’t be allowed) nested
• Like the other input types, it would also have a url-like form: value:[expr]
• Like any other input, it could be overriden from the cli using --override-input. Which would be similar to passing arguments to the flake

Design considerations

• Having to use --override-input foo value:xyz to pass an argument isn’t the nicest syntax. Maybe we could have a --arg foo xyz flag that would be syntactic sugar over this
• Overriding string inputs wouldn’t be very nice. If I want to override an input such as inputs.system.value = "x86_64-linux", I need to do --override-input system value:\"x86_64-darwin\" (with the escaped quotations, because it has to be a Nix string).
  • Maybe we could also add a --arg-str flag like the old CLI had for this (arguably common) use-case
  • Alternatively, we could have --arg do some magic that would make --arg foo xyz equivalent to --arg foo \"xyz\" if xyz is alphanumeric++ − that would work because it wouldn’t be valid otherwise as arg can’t contain variables or keywords.

Purity concerns

The main reason for not allowing arguments to flakes is that they’re supposed to be pure self-contained entities.
This doesn’t break this property as these “arguments” are just a special type of flake inputs − meaning that for example they are reflected in the lockfile and taken into account when computing the hash of the flake. Besides, this is already possible to emulate in a dirty − and arguably more impure − way using relative paths as inputs and overwriting them on the fly)

/cc @tomberek since you suggested the same idea on matrix
/cc @Mic92 @domenkozar

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/passing-options-to-flakes/7579/7

[Kha]

Another benefit over the old --arg is that by consolidating all arguments at the flake.nix toplevel, perhaps with an optional docstring, it should be easy to implement cmdline completion for --arg. One could even go one step further with this and allow --debug true as a convenient shorthand for --arg debug true (where true definitely shouldn't be stringified btw /cc @regnat). Or even --debug and --no-debug, but that would be cleaner if arguments also had an associated type.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/passing-options-to-flakes/7579/8

[edolstra]

I'm not convinced this is a good idea, and it could lead to a lot of anti-patterns.

Conceptually, changing flake.lock is not different from changing flake.nix - they both dirty the flake. So using it to pass parameters like system is a non-starter: passing --arg system bla dirties the flake, so I either have to commit flake.lock every time I pass a different system, or accept that my flake doesn't have a Git revision anymore (so no more provenance).

In addition, such inputs are global and can't be overriden outside of the flake input mechanism. So for instance, if I need a flake output for two different values of system, I would have to clone the corresponding flake input, e.g.

{
  inputs.nixpkgs_x86_64-linux = { url = github:NixOS/nixpkgs; args.system = "x86_64-linux"; };
  inputs.nixpkgs_x86_64-darwin = { url = github:NixOS/nixpkgs; args.system = "x86_64-darwin"; };

  outputs = { self, nixpkgs_x86_64-linux, nixpkgs_x86_64-darwin, ... }: { ... };
}

P.S. the approach in dc4a280 was to implement --arg by generating a temporary flake that depends on the original flake and passes new Nix module options. But this requires some type of generic override mechanism (like Nix modules) which we currently don't have.

[Mic92]

Does dc4a280 require flakes to be in toml format for overrides to work?

[thufschmitt]

Conceptually, changing flake.lock is not different from changing flake.nix - they both dirty the flake

That’s right for toplevel flakes (though I’m not sure it’s the best possible choice), and that indeed makes passing system as an argument that way not a good option (I new I shouldn’t have included this example 😒 ). But that doesn’t invalidate the idea in general. In particular just being able to have an extensible lists of supportedSystems (or however it would be called) so that it can be overridden for dependent flakes would be incredibly useful

In addition, such inputs are global and can't be overriden outside of the flake input mechanism. So for instance, if I need a flake output for two different values of system, I would have to clone the corresponding flake input

That’s also right, but it doesn’t seem like a fundamental issue to me. At least not if we don’t take system as example.
Otoh, a nice thing with this is that we get the possibility to pass arguments to the dependent flakes. Like

{
    inputs.debugMode.value = false;
    inputs.foo = {
        url = "...";
        inputs.debugMode.follows = "debugMode";
    };
}

(granted it’s not utterly pretty though)

[tomberek]

I'm not convinced this is a good idea, and it could lead to a lot of anti-patterns.

For the record: some anti-patterns that currently work. Warning: ugly hacks.

My thought is that you can currently declare a file with a Nix expression as an input and import it. It is not a big step from that to allowing a nix expression directly. The default can be declared in the flake.nix and tracked and locked in the same way that a file would be. Overriding on the CLI is like overriding an input, which disables writing the lock file anyway. The only concern is that doing this may encourage more usage of the underlying mechanism of override-input and reduce overall user experience.

[tomberek]

PoC that only allows string values: tomberek@a41ee9e

{
  description = "A string as an input argument";
  inputs.thing.value = "mething233";

  outputs = { self, nixpkgs, thing }: with nixpkgs.legacyPackages.x86_64-linux; {
    defaultPackage.x86_64-linux = runCommand "something" {} ''
      echo xxx
      echo value is: ${builtins.readFile thing}
      echo xxx
      '';
  };
}

spaces are not allowed in urls, so can't be used here, will need to modify fetchers.cc to understand the modification of a value input.

nix build -L --override-input thing 'value:anotherValue'

[kamadorueda]

Summary of the discussion as of 2022-03-10

#5663

• Motivation:

  • Having a generic mechanism for passing arguments to the Flake.

• Use cases:

  • Reducing boilerplate when doing matrix-type configuration of the same package
  • Reducing boilerplate of boolean flags configuration of the same package
  • Reducing boilerplate when trying to use different python versions
  • And more generically, using less code for expressing configurability

• Criticism of current design:

  • Requires boilerplate to enumerate all possible configurations

• Pros of current design:

  • It's easily discoverable

[jakubgs]

Lack of arguments/options for flakes makes it impossible to use Flakes for builds of our mobile application.

We have various arguments that we pass to the builds like build number, extra Gradle arguments, or architectures. The issue with architectures is that an Android APK can include multiple architectures, so using the system argument would have include a product of every architecture combination. For example: armeabi-v7a;arm64-v8a, armeabi-v7a;x86, arm64-v8a;x86, and so on. Which isn't the prettiest solution.

Combined with other arguments - like build number - makes using separate build targets not feasible, since it's just a number.

[LHurttila]

PoC that only allows string values: tomberek@a41ee9e

{
  description = "A string as an input argument";
  inputs.thing.value = "mething233";

  outputs = { self, nixpkgs, thing }: with nixpkgs.legacyPackages.x86_64-linux; {
    defaultPackage.x86_64-linux = runCommand "something" {} ''
      echo xxx
      echo value is: ${builtins.readFile thing}
      echo xxx
      '';
  };
}

spaces are not allowed in urls, so can't be used here, will need to modify fetchers.cc to understand the modification of a value input.

nix build -L --override-input thing 'value:anotherValue'

Looks great but can't figure out how to use that PoC. Seems like it adds the funtionality to nix directly so can I somehow patch it in the same nixos flake that would use it in its inputs? Any example would be really welcome.

[jhol]

@tomberek this looks very promising. Did you have any plans to submit the PoC?

[tomberek]

@tomberek this looks very promising. Did you have any plans to submit the PoC?

I was not planning on it as it was more of a quick hack to explore design and learn about the underlying code. I'm convinced it's possible to implement in a better way, but I'm not yet convinced of what a good design would be or if it would be a net gain.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-migrating-from-channels-to-flakes-include-input-packages-in-output-modules/22407/2

[Atry]

I think the easiest way to pass a nix expression to a flake is just to generate some files and let the flake read from the generated files.

You might need git add generated.nix to build with the dirty work tree.

[Atry]

For enumerable arguments, the best practice is to let them be attributes, similar to the packages.x86_64-linux convention.

[Atry]

Another approach might be to use builtins.getFlake to call a function in a flake.

nix build --expr "(builtins.getFlake ''git+file://$PWD'').functions.x86_64-linux.myFunction { buildNumber = 999; }"

[MangoIV]

Hi, I just wanted to chime in and ask whether there are plans to progress on this issue, this is a serious backwards incompatibility with the "legacy" nix commands and I have seen several workflows breaking because of this, most recently in the ghc.nix repo.

Thank you so much for consideration of this.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-can-we-pass-argument-to-flake-nix/30833/3

[Freed-Wu]

Any progress related to tomberek@a41ee9e?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-how-to-have-a-literal-value-input-no-url-file-path/36035/2

[theoparis]

Any progress related to tomberek@a41ee9e?

It seems like #9325 broke these changes. I need to be able to pass options to my custom LLVM flake.

Edit: Upon trying to update these changes, I realized clang-format was not ran on the nix codebase, causing reformatting due to clang-format running on save in my IDE.

[peacememories]

I also wanted to chime in and say that allowing inputs to flake builds would be important to us.
One use case I imagine many people will run into is building websites with SSGs. These often have features where new posts are only generated after a certain date, so they rebuild the same configuration regularly and after some time the new post shows up.

This does not work when building the website in Nix Flakes. Sure, we can depend on the current time since that's where our abstraction leaks a bit, but rebuilding an hour later will not cause a rebuild since the derivation is already cached.

I think the cleanest way to fix this would be to pass the current time as an argument to a derivation function, but as we established, this is not really possible in pure flakes right now. I might try the getFlake workaround mentioned above since all other workarounds which involve creating files or commits have the obvious drawback of changing self.rev, which we depend on.

This highlights another option for me, though. I understand the worry about invalidating the flake when allowing flake inputs to be overridden, but allowing function inputs to be passed through nix build seems less dangerous? This would allow evaluating the flake definition in a fully cached way, and the passed arguments could contribute to the derivation hash, making it safe to cache? I do have to admit that I'm not that knowledgeable when it comes to nix internals and purity.

[peacememories]

Another approach might be to use builtins.getFlake to call a function in a flake.

nix build --expr "(builtins.getFlake ''git+file://$PWD'').functions.x86_64-linux.myFunction { buildNumber = 999; }"

Just as a heads-up, for me this needs --impure as an argument (or building with nix-build -E), because the current flake is not locked.

[nrdxp]

I had this same idea just a moment ago when trying to think of some way to pass config to the default nixpkgs flake. I agree with @tomberek's previous comment that this is essentially already possible anyway by simply specifying a normal file as an input which itself contains a nix expression which can then be imported. I already use this for divnix/nosys, for example, to demonstrate the "system as an input" concept.

All implenting this would mean is that users now have a fast and sane way to do the exact same thing in a less hacky and cumbersome way. I haven't seen it mentioned yet, but it would also allow for inputs of type nixExpr to be more conveniently expressed using default args, i.e:

{
  # other inputs ...
  outputs = { system ? "x86_64-linux", config ? { inherit system; }, self, ... }: {
    # rest of flake
  };
}

[Ma27]

As a reminder, there's still the idea of "configurable flakes" from @lheckemann: https://media.ccc.de/v/nixcon-2023-36413-what-flakes-needs-technically-#t=777

In contrast to allowing arbitrary Nix expressions as inputs, this provides a structured and validated input. In other words, the author of a flake doesn't need to validate a flake input on their own. In fact, letting Nix do that gives us consistent errors across all flakes.

cc @NixOS/nix-team I can't recall that this proposal has been considered, so I'd be interested if there are reasons in favor/against it (short of "needs somebody to implement that"). Not sure yet, but if we come up with a useful design for that, I'd probably be open to do exactly that :)

[peacememories]

Would definitely solve my problem and seems rather elegant. I am not at all opinionated on how to solve this configuration problem, I just believe it should be solved because it seems there are many valid usecases

[roberth]

With an over-reliance on input overriding like this, we'll have no place for actual configurations, in the sense that a configuration is an arrangement where everything is defined and no parameters are left. The outputs attribute is currently our best place for such things.
An input can affect anything in the flake, which is messy. It will be unclear which parameters can or should be provided, to affect a given output attribute.
After all this time, haven't we learned that global state is not so great? One could be mistaken to think that it's a core value of Nix...
Of course we're going to need to override and configure things, but it should not happen at the root of outputs.
How about we create a new concept "overridable", which exists inside outputs (so that multiple can exist side by side), each with their own set of inputs, and each returning only the attributes that are actually affected by the setting. And let's just call them functions. Functions with optional defaults. If it has defaults for everything, great, it can be promoted to the top level. (Maybe automatically, maybe not)

[Ma27]

@roberth just to avoid we're talking past each other: is this a criticism on the original suggestion or on the alternative that I've brought up? Because "It will be unclear which parameters can or should be provided" doesn't sound like an issue about something that explicitly defines options and defaults for that (i.e. configurable flakes).

each with their own set of inputs, and each returning only the attributes that are actually affected by the setting

But you'd still need to be able to pass arbitrary Nix values to these overridables, correct? Or how else would you pass e.g. allowUnfree to an overridable which instantiates nixpkgs?

[roberth]

Both suffer from the same issue, being flake-wide. Documentation does help, but "global" settings are not a great design, and a separation between inputs and configuration is a bit artificial or unnecessary - why not have structured docs for the inputs?

But you'd still need to be able to pass arbitrary Nix values to these overridables, correct?

Yes, and these would be mostly normal functions (depending on caching, possibly), so that would be supported.

[Atry]

PoC that only allows string values: tomberek@a41ee9e

{
  description = "A string as an input argument";
  inputs.thing.value = "mething233";

  outputs = { self, nixpkgs, thing }: with nixpkgs.legacyPackages.x86_64-linux; {
    defaultPackage.x86_64-linux = runCommand "something" {} ''
      echo xxx
      echo value is: ${builtins.readFile thing}
      echo xxx
      '';
  };
}

spaces are not allowed in urls, so can't be used here, will need to modify fetchers.cc to understand the modification of a value input.

nix build -L --override-input thing 'value:anotherValue'

We don't need the patch. We can do this right now using file+file protocol:

{
  description = "A string as an input argument";
  inputs.thing.url = "file+file:///dev/null";
  inputs.thing.flake = false;

  outputs = { self, nixpkgs, thing }: with nixpkgs.legacyPackages.x86_64-linux; {
    packages.x86_64-linux.default = runCommand "something" {} ''
      echo xxx
      echo value is: ${builtins.readFile thing.outPath}
      echo xxx
      '';
  };
}

THING=anotherValue
nix build --override-input thing file+file://<(printf %s "$THING")

[Ma27]

How about we create a new concept "overridable", which exists inside outputs (so that multiple can exist side by side), each with their own set of inputs, and each returning only the attributes that are actually affected by the setting. And let's just call them functions. Functions with optional defaults. If it has defaults for everything, great, it can be promoted to the top level

@roberth I'm sorry, but I still have a hard time parsing this.

What does input mean here? Are we actually talking about flake inputs? And what does setting mean in the context?

I think I'll need an example here.

[roberth]

I was a little bit facetious because I find Flake's reinvention of the wheel frustrating. I'm sorry that that made my message unclear.

"Input" applied to both flake inputs and function arguments, as I was attempting to reconcile those two concepts.

I don't know if I have a particularly actionable suggestion or great example, because whenever I try to come up with something, I have an urge to first tear a whole bunch of things down. For instance, we should really have something that reconciles

• inputs
• "flake configuration"
• the system parameter
• bundler argument builders (in the sense of flakes; not in the sense of derivations; not in the sense of Nixpkgs; not in the sense of remote builds)

But this is basically impossible without also replacing them.

Finally if we want "discoverability" in "configuration", we also need to respect the interface segregation principle, because, e.g. inputs.nixpkgs must not apply to overlays (or if it does, that is very important to convey to users!), so even our familiar context of outputs = { nixpkgs, ... }: ... can't be kept in a good design of configurability.

Again, I'm sorry that I don't have good answers, but hopefully we can begin to clarify what's even the question.

[nikolaiser]

Based on the suggestion to use the file+file protocol by @Atry I created a small cli tool that allows to pass arguments to flakes with the syntax --arg key1=value1 --arg key2=value2

https://github.com/nikolaiser/purga

[Lindenk]

I also wanted to mention that passing inputs to a build is required for nix to be a viable build system for me as well. Even something as simple as setting the version number from CI needs the above hacky workaround at the moment

[daveman1010221]

I also wanted to mention that passing inputs to a build is required for nix to be a viable build system for me as well. Even something as simple as setting the version number from CI needs the above hacky workaround at the moment

This is exactly how I found this thread, trying to build a container from a flake, within a CI/CD server's configuration environment. We are currently using "impure" as our work-around here. This is a hard limitation on the usefulness of flakes, honestly.