Create a vision for Nix #7156
Create a vision for Nix #7156
Conversation
First very rough draft of a roadmap as discussed in last week's Nix team meeting. This is merely the output of a 45mins brainstorming session so far, so absolutely nothing is settled
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nix-team-meeting-minutes-oct-7-2022/22369/1 |
| ## Make Nix easy to adopt | ||
|
|
||
| - Well-defined target user base | ||
| - Well-defined core user stories |
There was a problem hiding this comment.
It's not really clear what the intent of this section is. E.g. we have ad hoc and declarative environments, so we should say here what we actually want to improve about them.
There was a problem hiding this comment.
Some of these are better not to be implemented in nix itself, but in places where the ecosystem can iterate on solutions. nix itself should be a source of stability. Places like nixpkgs and other repos are much more suitable for solving open-ended problems because those, unlike nix itself, can be pinned.
There was a problem hiding this comment.
Also these user stories should be clearly defined so we can prioritise tasks to improve those specifically instead of something else.
There was a problem hiding this comment.
we should say here what we actually want to improve about them.
I think the point of this item is that we need to know precisely how these can be used to solve concrete problems so that we know how we can meaningfully improve them
There was a problem hiding this comment.
Should there be an acknowledgement that this is at high risk (at least) to contradict the minimisation of opinionated decisions, or some idea how to manage this balance?
There was a problem hiding this comment.
this is at high risk to contradict the minimisation of opinionated decisions
@7c6f434c what would you charactarise as an opinionated decision in the context of determining first-class user stories? Maybe I'm misunderstanding what you're trying to say.
There was a problem hiding this comment.
Well, drawing a line between first-class and second-class usages is already an opinionated decision, and, for example, the «mass market» user stories probably don't use much package overrides so any story-based redesign will probably impair override facilities. (Without which facilities I probably find Nixpkgs much less useful). And look at unfree-predicate vs flakes to see how well things will go. Another thing is platform support: for first-class user stories you probably say that something either mostly works or not and this fact is recorded in the code base, for exotic cross-compilation with a unique set of tunables things might be broken by upstream changes and patched back to work too often for «broken» in a package to even make sense.
There was a problem hiding this comment.
@7c6f434c note that this is only about Nix itself, not Nixpkgs. The user stories would tell what Nix allows immediately, and Nixpkgs would use that for the extended use cases it enables in turn.
And look at unfree-predicate vs flakes to see how well things will go.
What do exactly you mean by that?
There was a problem hiding this comment.
With non-flake Nixpkgs, you can quickly and easily relax unfree/broken/platform checks for a single invocation. Flakes make it complicated with no clear way how to change Nixpkgs to have the same ease back. It's a Nix change that makes Nixpkgs less useful for some use cases.
| - One-liner setup ("templates") | ||
| - Easy modification/extension | ||
| - Easy and transparent usage ("direnv"/"lorri") | ||
| - Secret management as first-class citizen |
There was a problem hiding this comment.
| - Secret management as first-class citizen |
Not sure whether this is realistic in the roadmap time frame (except maybe via the ACL work).
There was a problem hiding this comment.
This is not a roadmap, this is a set of objectives. It does not have or imply an order or schedule. A roadmap is a sequence of sub-objectives to fulfill in order and may have a schedule.
There was a problem hiding this comment.
(eelco): Might be too low-level for the vision
(Robert): The ecosystem already has solutions for that, so probably not mandatory for Nix
There was a problem hiding this comment.
Hmm, is there any kind of a principle in the vision what should be allowed to evolve in the ecosystem and what ends up in the core?
There was a problem hiding this comment.
The ecosystem already has solutions for that, so probably not mandatory for Nix
@robeth Which are those solutions? There was a brief lunch break debate at NixCon that e.g. a primitive in the Nix language for dealing with secret values would be very helpful to enable more sophisticated solutions on top of that.
is there any kind of a principle in the vision what should be allowed to evolve in the ecosystem and what ends up in the core?
@7c6f434c not yet, but I think we absolutely have to answer that question, somehow. For example, it could be implied by the vision statement, i.e. with everything that is not specifically defined by it. In any case we have to make explicit that we draw such a boundary, and make perfectly clear where it is.
There was a problem hiding this comment.
What currently says why pinning part of flakes should not be external add-on on equal footing with niv that people used for many of the same aims? I think that any «define basic logic and derive from there» kind of boundary will end up demanding a large redrawing of lines, so you probably need to define at least a process for redrawing the boundary (which Nix currently lacks, judging from NixOS/rfcs#49)
There was a problem hiding this comment.
The ecosystem already has solutions for that, so probably not mandatory for Nix
@roberth Which are those solutions? There was a brief lunch break debate at NixCon that e.g. a primitive in the Nix language for dealing with secret values would be very helpful to enable more sophisticated solutions on top of that.
Existing solutions: NixOps deployment.keys, sops-nix, agenix.
Potential solutions:
- NixOS: make NixOps keys part of NixOS. For example
nixos-rebuildcould take secrets from/rootand install them. - Nix: extend the string context to allow marking of secret values so that they can not be written to the store.
- Nix: complicate the store concept. Not a fan of this.
| - Secret management as first-class citizen | ||
| - Configurations as first-class citizens | ||
| - Configuration/Modules/Nickel/etc. | ||
| - Language bindings |
There was a problem hiding this comment.
| - Language bindings |
Much more important to have JSON output on all relevant commands.
There was a problem hiding this comment.
I'd say equally important. A language binding can do strictly more than what JSON output achieves, but JSON is an easier and less invasive interface. We'll need both.
There was a problem hiding this comment.
Another thing could be implementation-agnostic conformance tests that allow reimplementing things in other languages, if C++ FFI is too hard.
There was a problem hiding this comment.
Ah, I see in the language case this is mentioned below.
There was a problem hiding this comment.
The relative importance of goals will be expressed by their ordering in the roadmap. We can say language bindings is something that's part of the vision or not, but will prioritise later.
There was a problem hiding this comment.
(roberth): A number of projects rely on or would benefit from unofficial bindings
There was a problem hiding this comment.
On the Nix side, I think that the important bit would be to provide a stable FFI interface that bindings can build against. Whether the different bindings live inside or outside of the Nix repo is more of a detail imho
| - ACLs | ||
| - rewriting | ||
| - ... | ||
| - Linux, MacOS and Windows support at feature parity |
There was a problem hiding this comment.
Don't think it's realistic to promise this, especially the Windows support.
There was a problem hiding this comment.
We'll want to manage expectations. This isn't quite a roadmap; see https://github.com/NixOS/nix/pull/7156/files#r992369142
There was a problem hiding this comment.
Agreed it is a lot for short term, but excellent for long term.
If get the Meson build system, we can iterate on Windows incrementally I think. E.g., goals like "libnixutil builds on Windows with both MinGW and LLVM in MSVC mode" can be tackled in isolation.
https://github.com/nix-windows/nix after all has things that are too messy right now, but do work. If we pick and choose when tackling those initial goals and clean up, it need not even be that hard.
There was a problem hiding this comment.
This is not a promise, but an ambition. The roadmap towards that ambition will be a promise.
There was a problem hiding this comment.
Windows is not even a Unix or *nix. Software is often build, distributed and developed very different and a lot of software on Windows is freeware or proprietary especially the entire foundation and core parts. I don't think it is realistic to get nix on feature parity on Windows.
| - Well-defined release process | ||
| - Feature support status | ||
| - Deprecation strategy | ||
| - LTS commitments |
There was a problem hiding this comment.
Not sure we should commit to this right now.
There was a problem hiding this comment.
I actually agree. "LTS" is but a means to an end, the end itself is is "stability policy" or, "support orgs that want to trade new features for more stability".
That Rust has so far gotten away with not having LTSs also makes me think judicious use of experimental features and cross-version testing might actually be sufficient.
There was a problem hiding this comment.
We already make long term commitments for reproducibility, Nixlang compatibility, a degree of worker protocol compatibility, etc. Have we made those commitments explicit?
There was a problem hiding this comment.
Dunno about "LTS" in particular, but the whole "compatibility guarantees" is a very important point in my opinion. As @roberth says, we already have most of these implicitly, and it's important to make them explicit so that users know what to expect
There was a problem hiding this comment.
Right now nixpkgs and so a bit distant also nix is the exact opposite of LTS and I know for a fact that many people I know that use nix choose nix because of the fast development cycle like Arch but with reproducibility and it if it falls apart you can rescue it without a reinstall. Converting nix(pkgs) to LTS would alienate a lot of users.
If LTS means compatibility guarantees (which is not the same thing) then the wording here must be improved.
There was a problem hiding this comment.
| - LTS commitments | |
| - Compatibility guarantees |
| - Well-defined architecture of isolated components | ||
| - Swappable store | ||
| - Formalize store protocol | ||
| - Swappable Nix language evaluator | ||
| - Swappable scheduler and remote-build system | ||
| - Integrate Hydra (modulo UI) into Nix | ||
| - Remote protocol speed and reliability improvements | ||
| - Binary cache protocol speed improvements |
There was a problem hiding this comment.
| - Well-defined architecture of isolated components | |
| - Swappable store | |
| - Formalize store protocol | |
| - Swappable Nix language evaluator | |
| - Swappable scheduler and remote-build system | |
| - Integrate Hydra (modulo UI) into Nix | |
| - Remote protocol speed and reliability improvements | |
| - Binary cache protocol speed improvements |
This is a lot of work with no benefit to end users.
There was a problem hiding this comment.
End users are not directly on the receiving end of this section, but by improving our ability to improve the product, we do improve the end user experience long term. (> Exemplary contributor and maintainer experience)
Componentization is good for local reasoning, for testing and it can improve build times.
(For example, having libstore and libexpr as a derivation-level dependencies helps when rebuilding (in Nix) a change to the CLI)
It also fosters other Nix implementation projects and specific use cases. For instance, hnix could more easily talk to the daemon, and an application of Nix in IoT could do store-level update distribution without ever shipping an evaluator to devices.
There was a problem hiding this comment.
This might not matter to beginners, but it absolutely does matter for end users / institutional users. Maybe chop up the section to indicate that split but please don't drop this!
There was a problem hiding this comment.
This is a lot of work with no benefit to end users
Not true, there are lots of ways in which this is beneficial to end users (this part of the doc is basically the Tvix architecture statement from a while ago, there are reasons people keep converging on these thoughts!).
As a concrete example for an end-user benefit, having pluggable store and substitution implementations lets users adapt to local constraints. If you're in a country like Egypt (where some of us live on and off), using Nix is currently prohibitively expensive due to the huge traffic wasted in central substitution and the price of internet in those countries. Having a switchable implementation there would let us plug in something like a P2P substituting stores with Trustix-verification without impacting the workflows of people that use cache.nixos.org or Cachix through the standard protocols.
Easy to come up with more examples like this for each of the points ... this really is an important topic!
There was a problem hiding this comment.
Whether (and when) in practice we'll go all the way down there is something for the roadmap (and I tend to agree with Eelco that given the state of things it might be too costly for the short/medium term), but I think that it is a desirable thing, at least as an internal distinction (a stronger one than the libstore/libexpr split which unfortunately has grown to be very leaky and I think isn't modular enough)
Having a switchable implementation there would let us plug in something like a P2P substituting stores with Trustix-verification without impacting the workflows of people that use cache.nixos.org or Cachix through the standard protocols.
Thinking about this (and most of the other use-cases that come to my mind), it feels like the most important part to properly distinguish from the rest (and that's kinda missing here) is an official and properly documented binary cache API. Does that seem right?
There was a problem hiding this comment.
No, the binary cache API itself won't be enough here. For example, for p2p substitution you need to solve problems like discovering local chunks, authenticating chunk identity and so on - something like this ("Nix for low external bandwidth") would likely replace the binary cache API completely. That means that what needs to be defined is what the expectations are for "communication" between e.g. eval and the store, or eval and a builder.
There was a problem hiding this comment.
@thufschmitt I think it is somewhat backwards to think of this as "tool costly". Fixing this is in fact how we pay for things like "Minimal custom code base"; only by cleaning up our own code can we figure out how to slot in off-the-shelf well.
This definitely has benefits on its own, but is is also necessary to tackle other goals without drowning in complexity. Not doing this is more costly if we are to tackle those other goals.
Also agreed with @tazjin. high quality network interfaces, whether for substituting or RPC, leverage domain-specific concerns in lots of ways. The current HTTP interface and the current RPC protocols are equally crummy in this regard.
I will say however that the "having a switchable implementation" would be fine to put under "Minimal custom code base" rather than this. I think that effectively clears up the gaps. The flow of benefits is: good layering -> easy swapping of bespoke current things for external things -> easy to experiment with P2P strategies.
There was a problem hiding this comment.
Well, purely as a user, and assuming only layer separation and not the further development enabled by it, I see layer separation as the only chance to close weird gaps in CLI such as impossible to find in docs (at least to me) operation «.drv + output name → output path» in a systematic way.
There was a problem hiding this comment.
If we can foster improvements (like already mentioned above) by splicing parts up into smaller pieces I am all for it but we should be careful that we not blindly start and end up with a thousand pieces in the end with no plan or goal and everything just got more complex.
| - Well-defined architecture of isolated components | |
| - Swappable store | |
| - Formalize store protocol | |
| - Swappable Nix language evaluator | |
| - Swappable scheduler and remote-build system | |
| - Integrate Hydra (modulo UI) into Nix | |
| - Remote protocol speed and reliability improvements | |
| - Binary cache protocol speed improvements | |
| - Well-defined architecture of isolated components | |
| - Swappable store and formalized store protocol | |
| - Swappable Nix language evaluator | |
| - Swappable remote-build system including it's scheduler |
I wouldn't merge hydra in the short term to keep complexity lower and the amount of code lower. Also improvements should go to a different section.
There was a problem hiding this comment.
| - Well-defined architecture of isolated components | |
| - Swappable store | |
| - Formalize store protocol | |
| - Swappable Nix language evaluator | |
| - Swappable scheduler and remote-build system | |
| - Integrate Hydra (modulo UI) into Nix | |
| - Remote protocol speed and reliability improvements | |
| - Binary cache protocol speed improvements | |
| - Well-defined architecture of isolated components | |
| - Swappable store and formalized store protocol | |
| - Swappable Nix language evaluator | |
| - Swappable remote build system and scheduler |
| - Minimal custom code base (proven off-the-shelf components where possible) | ||
| - Git file hashing | ||
| - Sandboxing, containers | ||
| - Capnproto for RPC | ||
| - Bazel RBE protocol | ||
| - ... |
There was a problem hiding this comment.
| - Minimal custom code base (proven off-the-shelf components where possible) | |
| - Git file hashing | |
| - Sandboxing, containers | |
| - Capnproto for RPC | |
| - Bazel RBE protocol | |
| - ... |
Likewise this doesn't really fit under "Make Nix a good investment for users".
There was a problem hiding this comment.
While the specific points should be subject to debate and were supposed to be random brainstorm ideas, the point here is to reduce maintenance surface. We discussed quite intensely that it needs a lot of deliberation what to depend on and what to purpose-build. But the general idea would be to do things the way outsiders would expect and only customize what is special.
There was a problem hiding this comment.
But the general idea would be to do things the way outsiders would expect and only customize what is special.
100% agree. I think there's three motivations here:
- By only focusing on what makes Nix Nix, we can reduce the maintenance burden (with the caveats that we must be very deliberate in choosing what we depend on as Valentin said)
- By reusing as much as possible off-the-shelf components/standards we remove as much accidental complexity from the face of new users as we can (because although the core of Nix would remain something new to learn, all the stuff around it would potentially be already boring technology). And we send the message that Nix isn't a weird exotic technology, but something firmly grounded in the state-of-the-art technological landscape
- In terms of pure expressiveness, speaking mainstream protocols means that Nix can integrate with existing tool in a much nicer way. Like what flakes do with the Git repositories for example
There was a problem hiding this comment.
| - Minimal custom code base (proven off-the-shelf components where possible) | |
| - Git file hashing | |
| - Sandboxing, containers | |
| - Capnproto for RPC | |
| - Bazel RBE protocol | |
| - ... | |
| - Try to use proven off-the-shelf components where possible like already done with: | |
| - editline | |
| - libarchive | |
| - lowdown | |
| - sqlite |
| - Memory safety validation (sanitizers, ...) | ||
| - Benchmarking infrastructure | ||
| - Test reports published and accessible | ||
| - Executable language specification |
There was a problem hiding this comment.
What exactly does this mean? Also curious who's working on it, as we're also working on a spec in TVL and it might be useful to collaborate.
There was a problem hiding this comment.
Think of it as a literate programming for the test suite. Goals:
- Improve the test suite
- Produce a detailed specification document (readable, linkable, etc)
- Interoperability; allow the test suite to be used in other implementations
- Potentially, dual-purpose the spec runner as a test framework for Nixlang libraries
I brought this up in the meeting.
We're not in a stage where we're planning implementation time frames and such. I'd describe this document as a detailed mission statement, rather than a roadmap. More on that in a separate thread.
There was a problem hiding this comment.
Oh this is great! I was already talking to @tazjin about these things too. "Literate implementation-agnostic conformance test suite" sounds amazing!!
There was a problem hiding this comment.
Hm, it does sound like a good idea but I have my doubts that it will be possible to create a literate spec like that which would also still be something a human would want to read.
Nix language has a huge amount of non-trivial edge cases (many of which are not covered by the C++ Nix test suite, but which have gained tests in Tvix that we run against both our interpreter and C++ Nix). While their behaviour is sometimes straightforward to describe (but not necessarily implement) in "spec-language", actually testing conformance is a verbose affair.
My gut feeling is that rather than trying specifically to make a literate spec, it would be more useful to have a spec and an accompanying conformance test suite (that can be read at one's leisure).
Either way, the hard part of this will be figuring out which things should go in the spec and which things are bugs in C++ Nix. In Tvix, our approach has been to consider "What does nixpkgs do?" as the primary guideline for this, which is the only way we see to actually cut down on some of the stuff that exists in the language. This is also the criteria by which we are working on our spec.
There was a problem hiding this comment.
I agree that not all parts of the spec will be fun to read, and the existence of executable test cases in the spec is not excuse not to write the necessary prose that belongs in a spec.
I don't have the illusion that turning a test suite into a literate program automatically creates a good spec, but it does help, and bringing the spec and the conformance suite closer together can only be a good thing. For example, some test cases can be in / near the main prose, whereas others are delegated to a detailed conformance suite section.
Either way, the hard part of this will be figuring out which things should go in the spec and which things are bugs in C++ Nix.
cut down on some of the stuff that exists in the language
I highly value Nix's ability to evaluate old expressions. Without this, what is the point of using Nix for reproducibility long term?
For me the choice seems obvious: put it in the spec. Now I don't have the experience of reimplementing Nix, so I'd like to know what behaviors you've considered to be bugs, but let's save that for later. We could have optional levels of conformance, to accommodate for other projects like Tvix.
There was a problem hiding this comment.
I highly value Nix's ability to evaluate old expressions. Without this, what is the point of using Nix for reproducibility long term?
For some behaviours of the C++ evaluator it'd be extremely surprising if you had any expressions depending on it, but unfortunately as an implementor (and doubly so as a spec writer) you need to investigate every little corner (including edge-cases). That means that there's likely some behaviour we can simplify classify as bugs in C++ Nix (and eventually fix) without including them in the spec.
I'd like to know what behaviors you've considered to be bugs, but let's save that for later
We've started a document on our end where we will collect a lot of these things, and we'll also be investigating (or have already done so) each of them to figure out whether expressions in nixpkgs or other large publicly known code-bases (e.g. home-manager, or TVL itself) depend on the behaviour. Note that this is very early form, with some things already added by @sternenseemann but lots of stuff we already know about missing. It might be useful to work on this in a shared Google Doc or similar!
There was a problem hiding this comment.
That means that there's likely some behaviour of X we can simplify classify as bugs (and eventually fix) without including them in the spec for X.
I've seen many instances where this principle could be applied. Documenting things with a bit of wishful thinking helps a lot with thinking straight in the first place.
|
Really love a lot of the items here, great work everyone! |
Co-authored-by: Robert Hensing <[email protected]>
| ## Make Nix easy to adopt | ||
|
|
||
| - Well-defined target user base | ||
| - Well-defined core user stories |
There was a problem hiding this comment.
we should say here what we actually want to improve about them.
I think the point of this item is that we need to know precisely how these can be used to solve concrete problems so that we know how we can meaningfully improve them
| - Test reports published and accessible | ||
| - Executable language specification | ||
|
|
||
| ## Make Nix a good investment for users |
There was a problem hiding this comment.
It looks like the intended meaning of “good investment” needs to be explained given the misunderstandings on that section
| ## Make Nix a good investment for users | |
| ## Make Nix a good investment for users | |
| _Give potential users the assurance that Nix will keep working reliably for them in the long run._ |
There was a problem hiding this comment.
How do we want to realize that other than giving our word? Sell service contracts and insurance?
There was a problem hiding this comment.
This is about the development model, e.g. determining how we deal with deprecation and sticking to that religiously. On the other hand, keeping Nix compatible with its surrounding hardware and software ecosystem, which continues to evolve, is indeed something that needs sustained maintenance.
There was a problem hiding this comment.
Keeping Nix compatible with ecosystem — would it include accepting a patch with an exploit for PlayStation 3 if somebody had provided it after working on OtherOS support there and Sony fraudulently disabling an advertised feature?
| - Well-defined release process | ||
| - Feature support status | ||
| - Deprecation strategy | ||
| - LTS commitments |
There was a problem hiding this comment.
Dunno about "LTS" in particular, but the whole "compatibility guarantees" is a very important point in my opinion. As @roberth says, we already have most of these implicitly, and it's important to make them explicit so that users know what to expect
| - Minimal custom code base (proven off-the-shelf components where possible) | ||
| - Git file hashing | ||
| - Sandboxing, containers | ||
| - Capnproto for RPC | ||
| - Bazel RBE protocol | ||
| - ... |
There was a problem hiding this comment.
But the general idea would be to do things the way outsiders would expect and only customize what is special.
100% agree. I think there's three motivations here:
- By only focusing on what makes Nix Nix, we can reduce the maintenance burden (with the caveats that we must be very deliberate in choosing what we depend on as Valentin said)
- By reusing as much as possible off-the-shelf components/standards we remove as much accidental complexity from the face of new users as we can (because although the core of Nix would remain something new to learn, all the stuff around it would potentially be already boring technology). And we send the message that Nix isn't a weird exotic technology, but something firmly grounded in the state-of-the-art technological landscape
- In terms of pure expressiveness, speaking mainstream protocols means that Nix can integrate with existing tool in a much nicer way. Like what flakes do with the Git repositories for example
| - Well-defined architecture of isolated components | ||
| - Swappable store | ||
| - Formalize store protocol | ||
| - Swappable Nix language evaluator | ||
| - Swappable scheduler and remote-build system | ||
| - Integrate Hydra (modulo UI) into Nix | ||
| - Remote protocol speed and reliability improvements | ||
| - Binary cache protocol speed improvements |
There was a problem hiding this comment.
Whether (and when) in practice we'll go all the way down there is something for the roadmap (and I tend to agree with Eelco that given the state of things it might be too costly for the short/medium term), but I think that it is a desirable thing, at least as an internal distinction (a stronger one than the libstore/libexpr split which unfortunately has grown to be very leaky and I think isn't modular enough)
Having a switchable implementation there would let us plug in something like a P2P substituting stores with Trustix-verification without impacting the workflows of people that use cache.nixos.org or Cachix through the standard protocols.
Thinking about this (and most of the other use-cases that come to my mind), it feels like the most important part to properly distinguish from the rest (and that's kinda missing here) is an official and properly documented binary cache API. Does that seem right?
|
(FYI to anyone reading this there are replies to @thufschmitt's comments but they are only showing up in the diff view, not conversations view.) |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/developing-a-system-that-replaces-nix-remote-build/22388/2 |
| - Executable language specification | ||
|
|
||
| ## Make Nix a good investment for users | ||
|
|
There was a problem hiding this comment.
I think the only way to enable effective development and maintenance to begin with, is to ruthlessly cut scope and abide to the single-responsibility principle. This is what the target audience and user stories play into.
This is to start a more focused discussion on what Nix is supposed to be for:
| - Nix is and stays policy-free |
That's the best thing we have so far. Nix should be invisible and just work, like plumbing - infrastructure.
There was a problem hiding this comment.
Does this intend to forbid any hardcoded reference to git in flakes code?
| - One-liner setup ("templates") | ||
| - Easy modification/extension | ||
| - Easy and transparent usage ("direnv"/"lorri") | ||
| - Secret management as first-class citizen |
There was a problem hiding this comment.
(eelco): Might be too low-level for the vision
(Robert): The ecosystem already has solutions for that, so probably not mandatory for Nix
| - Easy and transparent usage ("direnv"/"lorri") | ||
| - Secret management as first-class citizen | ||
| - Configurations as first-class citizens | ||
| - Configuration/Modules/Nickel/etc. |
There was a problem hiding this comment.
| - Configuration/Modules/Nickel/etc. | |
| - Have everything Nix be configurable, discoverable and introspectable | |
| - Ability to configure arbitrary systems (incl. secrets) |
There was a problem hiding this comment.
This strikes me as highly ambiguous. What is "everything in Nix" and what does it mean for that to have these properties? Do you mean the source code, or stuff at runtime?
There was a problem hiding this comment.
And even at runtime, is it in terms of Nix lib* as a library, or in terms of reflection available to Nix code?
| - Secret management as first-class citizen | ||
| - Configurations as first-class citizens | ||
| - Configuration/Modules/Nickel/etc. | ||
| - Language bindings |
There was a problem hiding this comment.
(roberth): A number of projects rely on or would benefit from unofficial bindings
| - Secret management as first-class citizen | ||
| - Configurations as first-class citizens | ||
| - Configuration/Modules/Nickel/etc. | ||
| - Language bindings |
There was a problem hiding this comment.
On the Nix side, I think that the important bit would be to provide a stable FFI interface that bindings can build against. Whether the different bindings live inside or outside of the Nix repo is more of a detail imho
| - Unprivileged installation and use | ||
| <!-- valentin: this needs clarification, I still don't know what rewriting is --> | ||
| - Portable store? | ||
| - restricted-root | ||
| - ACLs | ||
| - rewriting |
There was a problem hiding this comment.
Could be moved to a sub item. The important bit is that Nix runs everywhere
| - Swappable store | ||
| - Formalize store protocol | ||
| - Swappable Nix language evaluator | ||
| - Swappable scheduler and remote-build system |
There was a problem hiding this comment.
“swappable” might not be the right term here. I don't think we should care too much about it being swappable, but more about having clear and well-defined interfaces (the "swappable" thing can be a side-effect of this, but not the first goal)
There was a problem hiding this comment.
Swappable could also be a criterion of this, though.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2022-09-23-nix-team-meeting-minutes-1/22808/1 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2022-10-28-nix-team-meeting-minutes-5/22812/1 |
| - anyone who wants to manage the complexity of - and build - software | ||
| - transform bits in a declarative and reproducible way |
There was a problem hiding this comment.
Inspired by this sad Haiku from the Bazel world,
Yesterday it worked.
Today it is not working.
That's computers. Sigh.
I remembered this phrase I came up with when trying to describe what got me into Nix:
| - anyone who wants to manage the complexity of - and build - software | |
| - transform bits in a declarative and reproducible way | |
| - anyone who wants to make computers work reliably in the long run |
Advantages:
- not limited to software development, but still focused on making computers do something
- captures the reproducibility part without using the technical term
One could play with the "want to" part. To make it softer or broader:
| - anyone who wants to manage the complexity of - and build - software | |
| - transform bits in a declarative and reproducible way | |
| - anyone who cares for computers to work reliably in the long run |
| - anyone who wants to manage the complexity of - and build - software | |
| - transform bits in a declarative and reproducible way | |
| - anyone who has an interest in computers working reliably in the long run |
To make it stricter and narrower:
| - anyone who wants to manage the complexity of - and build - software | |
| - transform bits in a declarative and reproducible way | |
| - anyone who needs computers to work reliably in the long run |
| - anyone who wants to manage the complexity of - and build - software | |
| - transform bits in a declarative and reproducible way | |
| - anyone who requires computers to work reliably in the long run |
There was a problem hiding this comment.
Manage complexity of software is more fitting than making computers work reliably, the latter also needs stuff like easily swappable power supply components etc.
And indeede the value of Nix is in uniform handling of one aspect of complexity of managing software systems, not just reproducibility: I don't care about rebulds from scratch quite as much as about atomic updates and reduction of conflict surface.
(Moreover, for some notions of «run reliably» Nix does nothing with the problem that either you update and get new upstream bugs, or don't and could just as well boot a read-only Ubuntu live image or something)
There was a problem hiding this comment.
Discussed on the Nix team meeting:
- @fricklerhandwerk: "anyone who wants to make computers work reliably in the long run"
- "anyone who requires computers to work reliably in the long run"
- @edolstra: that's everybody. why not just say software engineers and DevOps people? this would be concrete
- @thufschmitt: data scientists are also part of our user base
- "anyone who depends on computers to work reliably in the long run"
- @thufschmitt: that's closer. have to determine the boundary where people make the trade-off of adopting Nix. the investment it's not worthwhile for everyone
- @thufschmitt: Different kind of audience: "People who just want things to work right here right now" (
nix run nixpkgs#hello). Kinda like part of the audience that docker got - "anyone who requires computers to work reliably both right now and in the long run, at scale"
- @tomberek: good distinction of communities: producers and consumers
- people who gravitate towards Nix are producers
- we make choices for the long-term instead of the short-term
- pros: reliability, experiment without fear
- cons: up-front investment
My proposal based on the discussion so far:
| - anyone who wants to manage the complexity of - and build - software | |
| - transform bits in a declarative and reproducible way | |
| - anyone who requires software to work reliably both right now *and* in the long run, at scale |
There was a problem hiding this comment.
I've thought about the problem some more and it seems that we (or at least me) are not clear on the purpose of defining the Nix audience.
Is it to attract people? That's better handled by the marketing team.
Is it to reject PRs? We should probably judge them on merit instead.
Is it to inform a roadmap? Then define one or two personas to focus on, which aren't the whole audience.
Is it something else? Then please enlighten me.
There was a problem hiding this comment.
Is it to reject PRs? We should probably judge them on merit instead.
Ideally, having a vision+roadmap makes it easier to decide which PRs are out of scope, i.e. should be closed regardless of their merit because they would take Nix into a direction we don't want.
So the audience shouldn't be too broad or too vague. I'd rather say "Nix is for software developers / DevOps" than "Nix is for anyone who want reliable software".
There was a problem hiding this comment.
I think it's fairly hard to write a vision statement for Nix as a whole, but much easier to rewrite ones for specific layers like libnixstore and libnixexpr. This should be a sign that divvying up responsibilities should make Nix project management easier and avoid clashing opinions or uncertainty.
There was a problem hiding this comment.
I agree with everything you wrote. Just one note, so we don't misunderstand each other.
I don't think picking favorites in public is something you do "simply".
I don't see how any of this is about picking favorites. The "simple" here was supposed to refer to what we talked about last week: It should be simple to answer for users or contributors if their intent to use or modify Nix is covered by the vision statement. That is primarily to communicate what kind of support to expect from maintainers.
We should by all means lead the discussion in public, to make sure we're on the right track. That is naturally not simple or easy, because it means saying "no" to some people in order to be able to clearly say "yes" to other people. It always hurts.
At the very least, the rationale behind that decision must be absolutely transparent, in my opinion.
I'm also fine with not determining this and instead skipping to user stories, but I suspect that will amount roughly to the same thing. Nix will not be able to solve everyone's problems, and we have to focus on something in the long term to be able to pick something to focus on in the short term.
There was a problem hiding this comment.
It should be simple to answer for users or contributors if their intent to use or modify Nix is covered by the vision statement.
Stated like this, it is «what Nix does» not «what use cases Nix serves» — which is of course better.
(And of course Nix does not help with reliability of software itself, but it is a good tool to reduce cross-contamination of different parts when such interactions are undesirable)
There was a problem hiding this comment.
Stated like this, it is «what Nix does» not «what use cases Nix serves»
@7c6f434c It sounds like we are thinking alike :) I started writing a bunch on what I am calling "Supply-" versus "demand-driven" project management.
There was a problem hiding this comment.
I mean, I myself am more or less Unix-maximalist (composable scope-able predictable tools, all that). Also I sincerely believe that there are many more tools useful despite their intended use cases but thanks to their predictability and friendliness to integrating into larger toolchains — than tools that are actually mainly useful as intended.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2022-11-04-nix-team-meeting-minutes-6/22998/1 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/tweag-nix-dev-update-39/23130/1 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2023-03-27-nix-team-meeting-minutes-44/26759/1 |
|
Closing since this isn't a viable approach. We'll focus on continue strengthening our processes instead. Might come back to this again in the future, but probably not. Ideally we should have something like that at the community level rather than restricted to just Nix. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nix-team-report-2022-10-2023-03/27486/1 |
For at least two years there were ongoing debates about the target audience of the Nix ecosystem. Nix maintainers did not manage to converge on a statement [0]. The marketing team is explicitly focusing on software developers [1], and the documentation team was supposed to align with that from the very beginning [2]. What was missing so far was what we mean by "software developers". Arguably, it encompasses a particular mind set about how to deal with computers and enough time to learn things. And while for documentation it matters most *what* the software is doing rather than *who* it is made for (or by), the ecosystem is a community effort, and people matter [3]. So far, we haven't really identified the (eventual) boundaries of this community (at least I don't know of any serious attempt), which also plays into the definition of a target audience. Yet, in practice, not every contribution, not every question or comment is treated equally, and this has reasons in our implicit assumptions about who belongs. This is an attempt to draw such a boundary that is not arbitrary, and neither to narrow nor too wide. The goal is, as always, to help Nix beginners set realistic expectations for their journey. The list of occupations/interests who may benefit from Nix is based on the 2022 [3] and 2023 [4] community surveys. [0]: NixOS/nix#7156 [1]: https://github.com/NixOS/nixos-homepage/blob/76d42a052dce7917007b5553dd08ea2569d04bc7/community/teams/marketing.tt#L89 [2]: https://discourse.nixos.org/t/2022-06-15-documentation-team-meeting-notes-1/20004 [3]: https://discourse.nixos.org/t/zurich-23-05-zhf-hackathon-and-workshop-report/29093#ux-workshop-21 [4]: https://discourse.nixos.org/t/nix-community-survey-2023-results/33124 [5]: https://discourse.nixos.org/t/2022-nix-survey-results/18983
For at least two years there were ongoing debates about the target audience of the Nix ecosystem. Nix maintainers did not manage to converge on a statement [0]. The marketing team is explicitly focusing on software developers [1], and the documentation team was supposed to align with that from the very beginning [2]. What was missing so far was what we mean by "software developers". Arguably, it encompasses a particular mind set about how to deal with computers and enough time to learn things. And while for documentation it matters most *what* the software is doing rather than *who* it is made for (or by), the ecosystem is a community effort, and people matter [3]. So far, we haven't really identified the (eventual) boundaries of this community (at least I don't know of any serious attempt), which also plays into the definition of a target audience. Yet, in practice, not every contribution, not every question or comment is treated equally, and this has reasons in our implicit assumptions about who belongs. This is an attempt to draw such a boundary that is not arbitrary, and neither to narrow nor too wide. The goal is, as always, to help Nix beginners set realistic expectations for their journey. The list of occupations/interests who may benefit from Nix is based on the 2022 [4] and 2023 [5] community surveys. [0]: NixOS/nix#7156 [1]: https://github.com/NixOS/nixos-homepage/blob/76d42a052dce7917007b5553dd08ea2569d04bc7/community/teams/marketing.tt#L89 [2]: https://discourse.nixos.org/t/2022-06-15-documentation-team-meeting-notes-1/20004 [3]: https://discourse.nixos.org/t/zurich-23-05-zhf-hackathon-and-workshop-report/29093#ux-workshop-21 [4]: https://discourse.nixos.org/t/2022-nix-survey-results/18983 [5]: https://discourse.nixos.org/t/nix-community-survey-2023-results/33124
For at least two years there were ongoing debates about the target audience of the Nix ecosystem. Nix maintainers did not manage to converge on a statement [0]. The marketing team is explicitly focusing on software developers [1], and the documentation team was supposed to align with that from the very beginning [2]. What was missing so far was what we mean by "software developers". Arguably, it encompasses a particular mind set about how to deal with computers and enough time to learn things. And while for documentation it matters most *what* the software is doing rather than *who* it is made for (or by), the ecosystem is a community effort, and people matter [3]. So far, we haven't really identified the (eventual) boundaries of this community (at least I don't know of any serious attempt), which also plays into the definition of a target audience. Yet, in practice, not every contribution, not every question or comment is treated equally, and this has reasons in our implicit assumptions about who belongs. This is an attempt to draw such a boundary that is not arbitrary, and neither to narrow nor too wide. The goal is, as always, to help Nix beginners set realistic expectations for their journey. The list of occupations/interests who may benefit from Nix is based on the 2022 [4] and 2023 [5] community surveys. [0]: NixOS/nix#7156 [1]: https://github.com/NixOS/nixos-homepage/blob/76d42a052dce7917007b5553dd08ea2569d04bc7/community/teams/marketing.tt#L89 [2]: https://discourse.nixos.org/t/2022-06-15-documentation-team-meeting-notes-1/20004 [3]: https://discourse.nixos.org/t/zurich-23-05-zhf-hackathon-and-workshop-report/29093#ux-workshop-21 [4]: https://discourse.nixos.org/t/2022-nix-survey-results/18983 [5]: https://discourse.nixos.org/t/nix-community-survey-2023-results/33124
* front page: Who is Nix for? For at least two years there were ongoing debates about the target audience of the Nix ecosystem. Nix maintainers did not manage to converge on a statement [0]. The marketing team is explicitly focusing on software developers [1], and the documentation team was supposed to align with that from the very beginning [2]. What was missing so far was what we mean by "software developers". Arguably, it encompasses a particular mind set about how to deal with computers and enough time to learn things. And while for documentation it matters most *what* the software is doing rather than *who* it is made for (or by), the ecosystem is a community effort, and people matter [3]. So far, we haven't really identified the (eventual) boundaries of this community (at least I don't know of any serious attempt), which also plays into the definition of a target audience. Yet, in practice, not every contribution, not every question or comment is treated equally, and this has reasons in our implicit assumptions about who belongs. This is an attempt to draw such a boundary that is not arbitrary, and neither to narrow nor too wide. The goal is, as always, to help Nix beginners set realistic expectations for their journey. The list of occupations/interests who may benefit from Nix is based on the 2022 [4] and 2023 [5] community surveys. [0]: NixOS/nix#7156 [1]: https://github.com/NixOS/nixos-homepage/blob/76d42a052dce7917007b5553dd08ea2569d04bc7/community/teams/marketing.tt#L89 [2]: https://discourse.nixos.org/t/2022-06-15-documentation-team-meeting-notes-1/20004 [3]: https://discourse.nixos.org/t/zurich-23-05-zhf-hackathon-and-workshop-report/29093#ux-workshop-21 [4]: https://discourse.nixos.org/t/2022-nix-survey-results/18983 [5]: https://discourse.nixos.org/t/nix-community-survey-2023-results/33124 Co-authored-by: Daniel Sidhion <[email protected]>
First very rough draft of a vision for Nix as discussed in last week's Nix team meeting.
This is merely the output of a 45mins brainstorming session so far, and a roadmap should follow once we have refined this.
cc @edolstra @fricklerhandwerk @roberth @tomberek