From 3a8338a6f78ff84b6f652c0784ba593c59a10bcd Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Fri, 1 Feb 2019 01:08:11 -0500
Subject: [PATCH 01/58] Initial draft "ret-cont" recursive Nix

---
 rfcs/0000-ret-cont-recursive-nix.md | 76 +++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 rfcs/0000-ret-cont-recursive-nix.md

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
new file mode 100644
index 000000000..27daf00d7
--- /dev/null
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -0,0 +1,76 @@
+---
+feature: ret-cont-recursive-nix
+start-date: 2019-02-01
+author: John Ericson (@Ericson2314)
+co-authors: (find a buddy later to help our with the RFC)
+related-issues: (will contain links to implementation PRs)
+---
+
+# Summary
+[summary]: #summary
+
+"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
+This avoids some platform-specific relating to nested sandboxing.
+More importantly, it prevents imperative and overly linear direct-style build scripts;
+easy to write but throwing the benefits of Nix.
+
+# Motivation
+[motivation]: #motivation
+
+The benefits of recursive Nix have been described in many places.
+One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
+The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+
+"Ret-cont" recursive nix is short for "return-continuation".
+Consider direct style recursive "nix-build":
+the outer build blocks while the inner one builds, and then continues.
+Just as we can CPS-transform programs, reifying the context of a function call as another function passed as an argument, so we can imagine splitting the derivation in two at this blocking point.
+This gives the "continuation" part of the name.
+
+But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
+Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
+Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
+This gives the "return" part of the name.
+
+I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
+This creates a linear chain of dependencies, which isn't terribly performent: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
+Building derivations is lot less convenient, but
+
+Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
+Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
+With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
+
+# Detailed design
+[design]: #detailed-design
+
+I defer to https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 with the exact details of setting up the daemon, etc.
+The meaningful distinction of this plan is how the derivation "handoff" works.
+A derivation can call `nix-instantiate` (or really communicate with the daemon however it wants) to create an arbitrary graph of derivations.
+It produces outputs which symlink to those derivations.
+Each of those outputs would then be replaced with the symlink's output of the same name in a downstream derivation.
+\[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
+We should muse on this point when implementing to reduce code and perhaps have good insights.]
+Those drvs are then built, perhaps building more derivations like this; it's possible to never terminate but that's the user's fault.
+We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation creating a cycle, we can error out instead of looping.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+We shouldn't be so worried about policing good taste in derivations, and allow full recursive Nix.
+
+# Alternatives
+[alternatives]: #alternatives
+
+Full recursive Nix (builds within builds), or keeping the status quo and use vendoring.
+Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I know longer consider the two in conflict.
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+The exact way the outputs refer to the replacement derivations / their outputs is subject to bikeshedding.
+
+# Future work
+[future]: #future-work
+
+A version of IFD that delays evaluation in derivation to keep evaluation non-blocking.
+This works on the same principle as this keeps all derivations non-blocking (be they higher order or not).

From 3b8422a054dde1ec5a22a80c2a095abf515756db Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Tue, 5 Feb 2019 01:59:28 -0500
Subject: [PATCH 02/58] Fix typos and finish trailing sentance

---
 rfcs/0000-ret-cont-recursive-nix.md | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 27daf00d7..3458e811d 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -10,9 +10,9 @@ related-issues: (will contain links to implementation PRs)
 [summary]: #summary
 
 "Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
-This avoids some platform-specific relating to nested sandboxing.
+This avoids some platform-specific contortions relating to nested sandboxing.
 More importantly, it prevents imperative and overly linear direct-style build scripts;
-easy to write but throwing the benefits of Nix.
+easy to write but throwing away the benefits of Nix.
 
 # Motivation
 [motivation]: #motivation
@@ -34,7 +34,8 @@ This gives the "return" part of the name.
 
 I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
 This creates a linear chain of dependencies, which isn't terribly performent: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
-Building derivations is lot less convenient, but
+Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
+And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
 
 Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
 Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
@@ -62,7 +63,7 @@ We shouldn't be so worried about policing good taste in derivations, and allow f
 [alternatives]: #alternatives
 
 Full recursive Nix (builds within builds), or keeping the status quo and use vendoring.
-Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I know longer consider the two in conflict.
+Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions

From 4da91936b0d5746b208f618bf8ad87d3f9c20a8d Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Tue, 5 Feb 2019 02:06:17 -0500
Subject: [PATCH 03/58] Switch to advocating temp store rather than daemon
 socket

Also:

 - Allow fixed output builds (in that temp store)

 - Clean up drawbacks and alternatives.
---
 rfcs/0000-ret-cont-recursive-nix.md | 45 ++++++++++++++++++++++-------
 1 file changed, 35 insertions(+), 10 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 3458e811d..2c1f51bef 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -44,26 +44,51 @@ With "ret-cont" recursive Nix, actual builds are never nested, so we don't need
 # Detailed design
 [design]: #detailed-design
 
-I defer to https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 with the exact details of setting up the daemon, etc.
-The meaningful distinction of this plan is how the derivation "handoff" works.
-A derivation can call `nix-instantiate` (or really communicate with the daemon however it wants) to create an arbitrary graph of derivations.
-It produces outputs which symlink to those derivations.
-Each of those outputs would then be replaced with the symlink's output of the same name in a downstream derivation.
+Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
+"store" would be a local Nix store limited to just drvs and fixed output builds.
+"drv" would contain a symlink to one the derivations in the store, the root.
+After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
+\[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
+Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
 \[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
 We should muse on this point when implementing to reduce code and perhaps have good insights.]
-Those drvs are then built, perhaps building more derivations like this; it's possible to never terminate but that's the user's fault.
-We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation creating a cycle, we can error out instead of looping.
+
+The *building* itself of derivations is unchanged:
+everything in the previous paragraph just describes marking derivations and post-processing their build results.
+Because drvs and produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate but that's the user's fault.
+We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
+\[Instead, the drv->drv substitution would never terminate.]
+Nix can should raise an error rather than looping, but either behavior is permissible.
 
 # Drawbacks
 [drawbacks]: #drawbacks
 
-We shouldn't be so worried about policing good taste in derivations, and allow full recursive Nix.
+ - The opinionated nature may put of those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
+
+ - If we ever want full recursive Nix, this doesn't really build in that direction.
+   It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.
+   To me though, this is a feature not a bug; I don't want to go in that direction just yet.
 
 # Alternatives
 [alternatives]: #alternatives
 
-Full recursive Nix (builds within builds), or keeping the status quo and use vendoring.
-Important from derivation has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
+ - Don't allow fixed-output builds.
+   All data can be stuck inside the drv file, so this can be cut without limiting expressive power.
+   But this is much less efficient, and more cumbersome for whatever produces the data.
+
+ - Use a socket to talk to the host daemon.
+   https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
+   This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
+   No direct access to the host daemon rules about a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
+   The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
+
+ - Full recursive Nix (builds within builds)
+
+ - Important from derivation.
+   This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
+
+ - Keeping the status quo and use vendoring.
+   But then Nix will never scale to bridging the package manager and build system divide.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions

From 800b5f3a76313aa91ef2b31829bae3c893aae876 Mon Sep 17 00:00:00 2001
From: Langston Barrett <langston.barrett@gmail.com>
Date: Wed, 6 Feb 2019 17:44:10 -0500
Subject: [PATCH 04/58] ret-cont-recursive-nix: Fix typo

Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 2c1f51bef..96123b8c3 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -33,7 +33,7 @@ Since the outer derivation evaluates (builds) the inner derivation rather than c
 This gives the "return" part of the name.
 
 I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
-This creates a linear chain of dependencies, which isn't terribly performent: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
+This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
 Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
 And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
 

From f7089838fb498213c7f478b877b9776194d10f90 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= <Mic92@users.noreply.github.com>
Date: Wed, 6 Feb 2019 18:58:01 -0500
Subject: [PATCH 05/58] ret-cont-recursive-nix: Fix typo

Thanks @Mic92

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 96123b8c3..2100cf15b 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -46,7 +46,7 @@ With "ret-cont" recursive Nix, actual builds are never nested, so we don't need
 
 Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
 "store" would be a local Nix store limited to just drvs and fixed output builds.
-"drv" would contain a symlink to one the derivations in the store, the root.
+"drv" would contain a symlink to one of the derivations in the store, the root.
 After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
 \[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
 Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.

From 6a87c1bc5f0d193cb883657d887359e576b598e9 Mon Sep 17 00:00:00 2001
From: Robin Gloster <mail@glob.in>
Date: Thu, 7 Feb 2019 15:48:42 -0500
Subject: [PATCH 06/58] ret-cont-recursive-nix: Fix typo

Thanks @globin

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 2100cf15b..5a7a0292e 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -63,7 +63,7 @@ Nix can should raise an error rather than looping, but either behavior is permis
 # Drawbacks
 [drawbacks]: #drawbacks
 
- - The opinionated nature may put of those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
+ - The opinionated nature may put off those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
 
  - If we ever want full recursive Nix, this doesn't really build in that direction.
    It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.

From 36193e5fe9651ab3657935ff2f6a61e80edb99ca Mon Sep 17 00:00:00 2001
From: Langston Barrett <langston.barrett@gmail.com>
Date: Fri, 8 Feb 2019 13:02:55 -0500
Subject: [PATCH 07/58] ret-cont-recursive-nix: Fix typo

Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 5a7a0292e..8d505dd81 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -84,7 +84,7 @@ Nix can should raise an error rather than looping, but either behavior is permis
 
  - Full recursive Nix (builds within builds)
 
- - Important from derivation.
+ - Import from derivation.
    This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
 
  - Keeping the status quo and use vendoring.

From ffb9203496abed07e63b264622c366724c4f5a96 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 10 Feb 2019 12:56:53 -0500
Subject: [PATCH 08/58] ret-cont-recursive-nix: Clean up motivation, adding
 examples

---
 rfcs/0000-ret-cont-recursive-nix.md | 89 ++++++++++++++++++++++++++---
 1 file changed, 81 insertions(+), 8 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 8d505dd81..9d6895fbd 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -19,27 +19,100 @@ easy to write but throwing away the benefits of Nix.
 
 The benefits of recursive Nix have been described in many places.
 One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
-The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+For this case, people usually imagine derivations like
+```nix
+{ stdenv, pkgs, nix }:
 
-"Ret-cont" recursive nix is short for "return-continuation".
-Consider direct style recursive "nix-build":
-the outer build blocks while the inner one builds, and then continues.
-Just as we can CPS-transform programs, reifying the context of a function call as another function passed as an argument, so we can imagine splitting the derivation in two at this blocking point.
-This gives the "continuation" part of the name.
+stdenv.mkDerivation {
+  name = "foo";
+  version = "1.2.3";
+
+  src = ...;
+
+  nativeBuildInputs = [ nix ];
+  NIX_PATH = "nixpkgs=${pkgs.path}";
 
+  outputs = [ "out" "dev" ];
+
+  doConfigure = false;
+  doBuild = false;
+
+  installPhase = ''
+    for o in $outputs; do
+      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+      cp -r $pkg ${!o}
+    done
+  '';
+}
+```
+The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
+For this, case, the one difference is we need to generate some Nix first.
+```nix
+installPhase = ''
+  bazel2nix # new bit
+  for o in $outputs; do
+    pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+    cp -r $pkg ${!o}
+  done
+'';
+```
+
+"Ret-cont" recursive Nix, short for "return-continuation" recursive Nix, is a different take on recursive Nix.
+The normal variant in the examples above might be termed "direct-style" recursive Nix.
+Consider what happens with the recursive "nix-build" in those examples:
+the outer build blocks while the inner one builds, and then the other one continues.
+Just as we can CPS-transform programs, reifying the context of a function call as another function (which is passed as an argument), so we can imagine splitting the derivation in two at this blocking point.
+This gives the "continuation" part of the name.
 But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
 Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
 Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
 This gives the "return" part of the name.
+Both differences together, the first example becomes something like:
+```nix
+{ stdenv, pkgs, nix }:
+
+stdenv.mkDerivation {
+  name = "foo";
+  version = "1.2.3";
+
+  src = ...;
+
+  nativeBuildInputs = [ nix ];
+  NIX_PATH = "nixpkgs=${pkgs.path}";
+
+  __recursive = true;
+
+  outputs = [ "drv" ];
+
+  doConfigure = false;
+  doBuild = false;
+
+  installPhase = ''
+    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+  '';
+}
+```
+Note how in this case we don't need to do any "post-processing" of the produced derivation.
+When the outer derivation can just "become" the inner derivation, explicitly copying the derivation outputs like before becomes unnecessary.
 
+So why prefer this variation of the standard design?
 I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
 This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
-Building derivations is lot less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
+Building derivations is less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
 And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
 
-Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restriction.
+Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restrictions that make it a Linux-only feature.
 Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
 With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
+Furthermore, we can skip needing to talk to the daemon by just producing a local store:
+```nix
+outputs = [ "drv" "store" ];
+installPhase = ''
+  mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+'';
+```
+This further simplifies the implementation.
+Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.
 
 # Detailed design
 [design]: #detailed-design

From 5564fdbd2b24199154f48dc5395e8c1bc32a3bed Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 10 Feb 2019 14:16:08 -0500
Subject: [PATCH 09/58] ret-cont-recursive-nix: Improve syntax highlighting

---
 rfcs/0000-ret-cont-recursive-nix.md | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 9d6895fbd..c958117ba 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -48,13 +48,16 @@ stdenv.mkDerivation {
 The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
 For this, case, the one difference is we need to generate some Nix first.
 ```nix
-installPhase = ''
-  bazel2nix # new bit
-  for o in $outputs; do
-    pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
-    cp -r $pkg ${!o}
-  done
-'';
+stdenv.mkDerivation {
+  # ...
+  installPhase = ''
+    bazel2nix # new bit
+    for o in $outputs; do
+      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
+      cp -r $pkg ${!o}
+    done
+  '';
+}
 ```
 
 "Ret-cont" recursive Nix, short for "return-continuation" recursive Nix, is a different take on recursive Nix.
@@ -106,10 +109,13 @@ Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice eithe
 With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
 Furthermore, we can skip needing to talk to the daemon by just producing a local store:
 ```nix
-outputs = [ "drv" "store" ];
-installPhase = ''
-  mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
-'';
+stdenv.mkDerivation {
+  # ...
+  outputs = [ "drv" "store" ];
+  installPhase = ''
+    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+  '';
+}
 ```
 This further simplifies the implementation.
 Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.

From 22f8322838a9f7dd0ad364cefc1c3fce0094b3c6 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 10 Feb 2019 19:27:59 -0500
Subject: [PATCH 10/58] Do a lousy job formalizing the detailed design

Break off some previously inline observations into their own subsection.
---
 rfcs/0000-ret-cont-recursive-nix.md | 90 ++++++++++++++++++++++++-----
 1 file changed, 77 insertions(+), 13 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index c958117ba..69231e1cc 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -123,21 +123,85 @@ Derivations remain built exactly as today, with only logic *between* building st
 # Detailed design
 [design]: #detailed-design
 
-Derivations building derivations should have some special attribute indicating this, and two outputs, "store" and "drv".
-"store" would be a local Nix store limited to just drvs and fixed output builds.
-"drv" would contain a symlink to one of the derivations in the store, the root.
+Derivations today build outputs, and are associated to those outputs.
+We extend the derivation language by allowing a derivation to indicate their output is more derivations, and ultimately be associated with one of *those* derivations's associated outputs.
+Derivations that that do so indicate this with some special attribute, say `__recursive`.
+Such derivations must have two outputs, `store` and `drv`.
+`store` would be a local Nix store limited to just drvs and fixed output builds.
+`drv` would contain a symlink to one of the derivations in the store, the root.
 After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
-\[This should be an untrusted operation because drvs and fixed-output builds are fully verifiable.]
 Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
-\[The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
-We should muse on this point when implementing to reduce code and perhaps have good insights.]
-
-The *building* itself of derivations is unchanged:
-everything in the previous paragraph just describes marking derivations and post-processing their build results.
-Because drvs and produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate but that's the user's fault.
-We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
-\[Instead, the drv->drv substitution would never terminate.]
-Nix can should raise an error rather than looping, but either behavior is permissible.
+
+To faux-formalize everything in the vein of a small-step semantics:
+```
+immediatelyDependsOn(drv0, drv1)
+immediatelyDependsOn(drv1, drv2)
+-------------------------------------------------- deps-trans
+transitivelyDependsOn(drv0, drv2)
+```
+```
+∀<d : transitivelyDependsOn(drv, -)>
+  d ? __recursive == false
+-------------------------------------------------- build-readiness
+isReadyToBuild(drv)
+```
+```
+drv0 : Drv
+∀<o : outputs(drv0)> build_o : StorePath
+∀<o : outputs(drv0)> build(drv0) = { ${o} = build_o; }
+isReadyToBuild(drv0)
+drv0 ? __recursive == false
+-------------------------------------------------- normal-build
+∀<o : outputs(drv0)> assoc(drv0, o, build_o)
+```
+```
+drv0, drv1 : Drv
+drv1path : RelativePath
+∀<o : outputs(drv0)> build0_o : StorePath
+isReadyToBuild(drv0)
+drv0 ? __recursive == true
+drv0.outputs = { "store" = ...; "drv" = ...; }
+build(drv0) = { succeeded = build0; }
+isTrustlessStore(build0_store)
+drv1 = read(build0_store + drv1path)
+readlink(build0_drv) = build0.store + drv1path
+-------------------------------------------------- immediate-drv-deligation
+reducesTo(drv0, drv1)
+```
+```
+drv0, drv1, drv2 : Drv
+reducesTo(drv1, drv2)
+immediatelyDependsOn(drv0, drv1)
+-------------------------------------------------- transitive-drv-deligation
+reducesTo(drv0, drv0[drv2/drv1])
+```
+```
+drv0, drv1 : Drv
+reducesTo(drv0, drv1)
+∀<o : outputs(drv0)> build0_o : StorePath
+∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
+-------------------------------------------------- delegative-build
+∀<o : outputs(drv0)> assoc(drv1, o, build1_o)
+```
+
+## Design Notes
+
+There's a few things we can call out from the faux-formalization.
+
+ - `isTrustlessStore` is called that because the restricted on the contents—fixed output builds / plain data and drvs—is fully and cheaply verifiabled.
+   This is in contrast to normal builds,
+   where the relationship between the derivation and build can only be verified by redoing the build,
+   and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.
+
+ - The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
+   We should muse on this point, and hopefully write a small-step semantics for both together that is more elegant than the above.
+
+ - The *building* itself of derivations is unchanged.
+   All the magic happens through the `reducesTo` relation.
+
+ - Because drvs can produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate (no `reducesTo` from a derivation to an `isReadyToBuild` derivation) but that's the user's fault.
+   We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
+   Nix should raise an error rather than looping, but either behavior is permissible.
 
 # Drawbacks
 [drawbacks]: #drawbacks

From 7f5f85492b7a1326c39b47fd6a3ef23cf9a10b64 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Mon, 11 Feb 2019 11:07:15 -0500
Subject: [PATCH 11/58] ret-cont-recursive-nix: Mention `builtins.exec` in
 alternatives

---
 rfcs/0000-ret-cont-recursive-nix.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 69231e1cc..63fd49fba 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -230,6 +230,11 @@ There's a few things we can call out from the faux-formalization.
  - Import from derivation.
    This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
 
+ - `builtins.exec` runs an arbitrary command at eval time as the user triggering evaluation.
+   This is highly impure; nothing at all tries to make the environment deterministic.
+   It is useful for writing fetchers that need the impurities to access the internet and secrets, while also managing their own caching.
+   But for everything else, I view it strictly worse than IFD.
+
  - Keeping the status quo and use vendoring.
    But then Nix will never scale to bridging the package manager and build system divide.
 

From 5c9f1fbb1149e9d188c157b905bf3bb563017308 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= <Mic92@users.noreply.github.com>
Date: Mon, 11 Feb 2019 11:11:11 -0500
Subject: [PATCH 12/58] ret-cont-recursive-nix: Fix typo

Thanks @Mic92!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 63fd49fba..cca238d12 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -125,7 +125,7 @@ Derivations remain built exactly as today, with only logic *between* building st
 
 Derivations today build outputs, and are associated to those outputs.
 We extend the derivation language by allowing a derivation to indicate their output is more derivations, and ultimately be associated with one of *those* derivations's associated outputs.
-Derivations that that do so indicate this with some special attribute, say `__recursive`.
+Derivations that do so indicate this with some special attribute, say `__recursive`.
 Such derivations must have two outputs, `store` and `drv`.
 `store` would be a local Nix store limited to just drvs and fixed output builds.
 `drv` would contain a symlink to one of the derivations in the store, the root.

From 5e56f21125faa6e92ff8269e8fd35655ad9ab958 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 24 Feb 2019 23:57:15 -0500
Subject: [PATCH 13/58] ret-cont-recursive-nix: Remove dangling "$o"

The later examples with `nix-instantiate` automatically get all the outputs of the final rewritten drv, so there's no output iteration needed. `"$o"` was mistakenly copied over from the earlier examples.

Thanks to @ocharles for asking me the question that led me to this. Hopefully this change answers that?
---
 rfcs/0000-ret-cont-recursive-nix.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index cca238d12..059cced12 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -91,7 +91,7 @@ stdenv.mkDerivation {
   doBuild = false;
 
   installPhase = ''
-    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {})') $drv
   '';
 }
 ```
@@ -113,7 +113,7 @@ stdenv.mkDerivation {
   # ...
   outputs = [ "drv" "store" ];
   installPhase = ''
-    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o") $drv
+    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {})') $drv
   '';
 }
 ```

From ba7dcce84a6acc6dad03d502ad95aab9cbc4e20c Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Thu, 15 Aug 2019 09:32:26 -0400
Subject: [PATCH 14/58] Update rfcs/0000-ret-cont-recursive-nix.md
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-Authored-By: Domen Kožar <domen@enlambda.com>
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 059cced12..d41d1899f 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -3,6 +3,8 @@ feature: ret-cont-recursive-nix
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
 co-authors: (find a buddy later to help our with the RFC)
+shepherd-leader: Franz Pletz
+shepherd-team: Franz Pletz, Eelco Dolstra, Shea Levy, Daniel Peebles
 related-issues: (will contain links to implementation PRs)
 ---
 

From 8bcb4e68d8a5aa69e6cad6f04eb88527d0c20813 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Sat, 2 Nov 2019 09:56:20 -0400
Subject: [PATCH 15/58] ret-cont-recursive: Fix typo

about -> out
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index d41d1899f..b9fa90a71 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -224,7 +224,7 @@ There's a few things we can call out from the faux-formalization.
  - Use a socket to talk to the host daemon.
    https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
    This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
-   No direct access to the host daemon rules about a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
+   No direct access to the host daemon rules out a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
    The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
 
  - Full recursive Nix (builds within builds)

From baae1e6b852a703a05997119acd49343717dac71 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 10:56:09 -0400
Subject: [PATCH 16/58] ret-cont: Add examples and expand future work

---
 rfcs/0000-ret-cont-recursive-nix.md | 149 +++++++++++++++++++++++++++-
 1 file changed, 147 insertions(+), 2 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index b9fa90a71..958058994 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -205,6 +205,97 @@ There's a few things we can call out from the faux-formalization.
    We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
    Nix should raise an error rather than looping, but either behavior is permissible.
 
+# Examples
+[examples]: #examples
+
+As a running example, I'll use @matthewbauer's [Reproducible résumé].
+(Do steal the method; don't steal Matt; he works with me!)
+Here's the original `default.nix`, which uses IFD:
+
+```nix
+{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
+let
+
+README = stdenv.mkDerivation {
+  name = "README";
+  unpackPhase = "true";
+  buildInputs = [ emacs ];
+  installPhase = ''
+    mkdir -p $out
+    cd $out
+    cp -r ${./fonts} fonts
+    cp ${./README.org} README.org
+    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
+    cp resume.nix default.nix
+  '';
+};
+
+in import README {inherit nixpkgs;}
+```
+
+The shortest way to make it instead use "Ret-cont" recursive Nix is this:
+
+```nix
+{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
+
+in stdenv.mkDerivation {
+  name = "README";
+  unpackPhase = "true";
+  outputs = [ "drv" "store" ];
+  buildInputs = [ emacs nix ];
+  __recursive = true;
+  installPhase = ''
+    mkdir -p $out
+    cd $out
+    cp -r ${./fonts} fonts
+    cp ${./README.org} README.org
+    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
+    mv $(nix-instantiate --store $store resume.nix --arg nixpkgs 'import ${nixpkgs.path}') > $drv
+  '';
+}
+```
+
+But note how this means we re-run emacs every time anything in Nixpkgs changes, no good!
+Here's a better version which is more incremental:
+
+```nix
+{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
+let
+
+# Just like original
+README = stdenv.mkDerivation {
+  name = "README";
+  unpackPhase = "true";
+  buildInputs = [ emacs ];
+  installPhase = ''
+    mkdir -p $out
+    cd $out
+    cp -r ${./fonts} fonts
+    cp ${./README.org} README.org
+    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
+    cp resume.nix default.nix
+  '';
+};
+
+in stdenv.mkDerivation {
+  name = "readme-outer";
+  unpackPhase = "true";
+  buildInputs = [ nix ];
+  installPhase = ''
+    mv $(nix-instantiate --store $store ${README} --arg nixpkgs 'import ${nixpkgs.path}') > $drv
+  '';
+}
+```
+
+Now only `readme-outer` is rebuilt when Nixpkgs and Nix changes.
+This may still seem wasteful, but remember we still need to reevaluate whenever those changes.
+Now some of that evaluation work is pushed into the derivations themselves.
+
+This is actually a crucial observation:
+A limit to scaling Nix today is that while our builds are very incremental, evaluation isn't.
+But if we can "push" some of the work of evaluation "deeper" into the derivaiton graph itself, we can be incremental with respect to both.
+This means we are incremental at all levels.
+
 # Drawbacks
 [drawbacks]: #drawbacks
 
@@ -248,5 +339,59 @@ The exact way the outputs refer to the replacement derivations / their outputs i
 # Future work
 [future]: #future-work
 
-A version of IFD that delays evaluation in derivation to keep evaluation non-blocking.
-This works on the same principle as this keeps all derivations non-blocking (be they higher order or not).
+1. As the example shows, we can push the work of evaluation into builds.
+   This unlocks lots of future work in Nixpkgs:
+
+   - Leveraging language-specific tools to generate plans Nix builds, rather than reimplementing much of those tools.
+     We do this with IFD today, but both due to Hydra's architecture, and concerns about eval times regardless of Hydra, we don't allow this in Nixpkgs.
+     This is a huge loss as we either do things entirely manually (python) Or vender tons of code (Haskell).
+     We will save valuable human time, and start to bridge the distribution / ops vs developer cultural divide by making it easier to work on your own packages.
+
+   - Simply fetching and importing packages which use Nix for their build system, like Nix itself and hydra, rather than vendoring that build system in.
+     This is an easier special case of the above, where the upstream developer knows and loves Nix, and their package has a Nix-based build system.
+     Flakes are supposed to help with this, as is `builtins.fetchTarball` skirting the IFD prohibition.
+     But, it's better if the hydra evaluator can avoid blocking on the download and/or evaluating the Nix expressions therein.
+     "Ret-cont" recursive Nix would allow this by just putting the "outer" derivation in Nixpkgs.
+
+2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
+   With `--pure` mode, Eelco has also talked about opening the door to caching builds.
+   "Ret-cont" recursive Nix is wonderful foundation for that.
+
+   I hope to at least later proposal automatically converting IFD into "Ret-cont":
+
+   IFD is also slow because the evaluator isn't concurrent and so imported derivations get built one at a time.
+   We can fix this somewhat by modifying the evaluator so that evaluation continues where the *value* of the previously-imported derivation isn't needed.
+   But, we still will inevitably get stuck somewhere shallower in the expression when the value being built is needed.
+
+   With "Ret-cont", we can cleverly avoid needing that value in certain common situations.
+   Quite often, IFD is creating a derivation, so we will have something like:
+
+   ```nix
+   "blah blah blah ${/*some expression ... is stuck because deep inside: */ (import /* ... */) /* ... */} blah blah blah"
+   ```
+
+   Without knowing what the value of that expression is, we may reasonably assume it's coercible to a string.
+   If it isn't, well, evaluation will fail anyways.
+   We can then make a "scratch" derivation, however, that reifies the evaluation of the stuck term, we can splice the scratch derivation's hash instead:
+
+   ```nix
+   "bash balh blash /nix/store/123asdf4sdf1g2dfgh34fg8h7fg69876-i-like-to-procrastinate-and-hope-for-the-best blah blah blah"
+   ```
+
+   Now, evaluation truly isn't stuck at all, and we are as free to continue as if there was no IFD at all!
+   The only failure mode would be if the import was a string but *wasn't* a derivation.
+   But, I imagine we can annotate things such that Nix knows when to speculate like this.
+   (c.f. Compiler hot and cold pragmas.)
+   The author of the Code almost always knows what *type* of thing they are splicing, so I would think we could so annotate quite faithfully.
+   I emphasize "type" because if we ever get a type system, this becomes much easier:
+   Specifying types for imported expressions (along with other explicit signature to guide inference) is wholly sufficient to derive the type of all such splices.
+
+  Another future project would be some speculative strictness to allow one round of evaluation to return *both* a partial build plan and stuck imported derivations.
+  Currently the plan must still be evaluated entirely before any building of "actually needed" derivations, i.e. those which *aren't* imported, begins.
+
+3. More broadly, we can get rid of a special notion of "evaluation" entirely.
+   We can think of evaluation today as basically a special case single layer of dynamism, where the outer work (evaluation) is impure.
+   If all `--pure` and IFD evaluation is done inside Nix builds, then the Nix daemon need not even know about the nix language at all.
+   We can have completely separate tools running inside sandboxes that deal with the Nix expression langauge.
+
+[Reproducible résumé]: https://github.com/matthewbauer/resume

From 9448a2a9330386caee38278b3062f2113b796003 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 11:23:00 -0400
Subject: [PATCH 17/58] ret-cont: Fix syntax error

No `let`, so don't need `in`.
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 958058994..86ae859cf 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -238,7 +238,7 @@ The shortest way to make it instead use "Ret-cont" recursive Nix is this:
 ```nix
 {nixpkgs ? <nixpkgs>}: with import nixpkgs {};
 
-in stdenv.mkDerivation {
+stdenv.mkDerivation {
   name = "README";
   unpackPhase = "true";
   outputs = [ "drv" "store" ];

From 37a643edb3bd507be7d082fbf22c5160e94e32a2 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 11:23:36 -0400
Subject: [PATCH 18/58] ret-cont: Mention Ninja's upcomming `dyndep` and C++
 oppertunity

---
 rfcs/0000-ret-cont-recursive-nix.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 86ae859cf..cab2a0c5d 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -353,6 +353,12 @@ The exact way the outputs refer to the replacement derivations / their outputs i
      But, it's better if the hydra evaluator can avoid blocking on the download and/or evaluating the Nix expressions therein.
      "Ret-cont" recursive Nix would allow this by just putting the "outer" derivation in Nixpkgs.
 
+   As an example of the former, Ninja (is getting)[Ninja Dyndeps] a very similar notion they call `dyndep`s in their upcoming release.
+   This is needed for C++ and Fortran modules.
+   If CMake and other tools use it as expected, we would need "Ret-cont" to automatically translate their build plans for fine-grained builds of large projects like LLVM or Chromium.
+   Shake, soon Ninja, and eventually [LLBuild], are the only general purpose build systems I know that do or aim to do dynamic dependencies, but none of them sandbox.
+   If we become the only way to both correctly and incrementally build modern C++, that will be a huge opportunity for further growth.
+
 2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
    With `--pure` mode, Eelco has also talked about opening the door to caching builds.
    "Ret-cont" recursive Nix is wonderful foundation for that.
@@ -395,3 +401,7 @@ The exact way the outputs refer to the replacement derivations / their outputs i
    We can have completely separate tools running inside sandboxes that deal with the Nix expression langauge.
 
 [Reproducible résumé]: https://github.com/matthewbauer/resume
+
+[Ninja Dyndeps]: https://github.com/ninja-build/ninja/pull/1521
+
+[LLBuild]: https://github.com/apple/swift-llbuild

From 14b134d683b67e45fc0b0e93ad91689721283ab3 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sat, 2 Nov 2019 11:24:54 -0400
Subject: [PATCH 19/58] ret-cont: Fix missing explicit `outputs` and
 `__recursive`

This was in the "wrapper" derivation example.
---
 rfcs/0000-ret-cont-recursive-nix.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index cab2a0c5d..6e729603c 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -242,8 +242,8 @@ stdenv.mkDerivation {
   name = "README";
   unpackPhase = "true";
   outputs = [ "drv" "store" ];
-  buildInputs = [ emacs nix ];
   __recursive = true;
+  buildInputs = [ emacs nix ];
   installPhase = ''
     mkdir -p $out
     cd $out
@@ -281,6 +281,8 @@ in stdenv.mkDerivation {
   name = "readme-outer";
   unpackPhase = "true";
   buildInputs = [ nix ];
+  outputs = [ "drv" "store" ];
+  __recursive = true;
   installPhase = ''
     mv $(nix-instantiate --store $store ${README} --arg nixpkgs 'import ${nixpkgs.path}') > $drv
   '';

From 1b0a6a1979d13179594e6422ad0a631a0179e4b8 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 5 Nov 2019 13:14:45 -0500
Subject: [PATCH 20/58] ret-cont: "caching builds" -> "caching evaluation"

We already cache builds just fine, thanks!

Thanks @globin for catching
---
 rfcs/0000-ret-cont-recursive-nix.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index 6e729603c..fc0b7099d 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -362,7 +362,7 @@ The exact way the outputs refer to the replacement derivations / their outputs i
    If we become the only way to both correctly and incrementally build modern C++, that will be a huge opportunity for further growth.
 
 2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
-   With `--pure` mode, Eelco has also talked about opening the door to caching builds.
+   With `--pure` mode, Eelco has also talked about opening the door to caching evaluation.
    "Ret-cont" recursive Nix is wonderful foundation for that.
 
    I hope to at least later proposal automatically converting IFD into "Ret-cont":

From 3fe1c3df6b69a3c9446efc495930f7e5f1714db5 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Thu, 12 Dec 2019 12:46:57 -0500
Subject: [PATCH 21/58] ret-cont: Improve formalism and reference #62

---
 rfcs/0000-ret-cont-recursive-nix.md | 42 ++++++++++++++++++-----------
 1 file changed, 27 insertions(+), 15 deletions(-)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-ret-cont-recursive-nix.md
index fc0b7099d..1b8043210 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-ret-cont-recursive-nix.md
@@ -136,14 +136,16 @@ Finally, any uses of the original derivation can be substituted to instead use t
 
 To faux-formalize everything in the vein of a small-step semantics:
 ```
-immediatelyDependsOn(drv0, drv1)
-immediatelyDependsOn(drv1, drv2)
--------------------------------------------------- deps-trans
-transitivelyDependsOn(drv0, drv2)
+∀<d : immediatelyDependsOn(drv, -)>
+  isValue(d)
+drv ? __recursive == false
+drv ? outputHash == false
+-------------------------------------------------- normalized
+isValue(drv)
 ```
 ```
-∀<d : transitivelyDependsOn(drv, -)>
-  d ? __recursive == false
+∀<d : immediatelyDependsOn(drv, -)>
+  isValue(d)
 -------------------------------------------------- build-readiness
 isReadyToBuild(drv)
 ```
@@ -152,29 +154,34 @@ drv0 : Drv
 ∀<o : outputs(drv0)> build_o : StorePath
 ∀<o : outputs(drv0)> build(drv0) = { ${o} = build_o; }
 isReadyToBuild(drv0)
-drv0 ? __recursive == false
--------------------------------------------------- normal-build
+-------------------------------------------------- simple-build
 ∀<o : outputs(drv0)> assoc(drv0, o, build_o)
 ```
 ```
+drv : Drv
+drv.outputs = { "out" = ...; }
+drv ? outputHash == true
+-------------------------------------------------- immediate-reduction-fixed-output
+reducesTo(drv0, stubDrv(drv.outputHash))
+```
+```
 drv0, drv1 : Drv
 drv1path : RelativePath
+drv0.outputs = { "store" = ...; "drv" = ...; }
 ∀<o : outputs(drv0)> build0_o : StorePath
-isReadyToBuild(drv0)
+∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
 drv0 ? __recursive == true
-drv0.outputs = { "store" = ...; "drv" = ...; }
-build(drv0) = { succeeded = build0; }
 isTrustlessStore(build0_store)
 drv1 = read(build0_store + drv1path)
 readlink(build0_drv) = build0.store + drv1path
--------------------------------------------------- immediate-drv-deligation
+-------------------------------------------------- immediate-reduction-drv-deligation
 reducesTo(drv0, drv1)
 ```
 ```
 drv0, drv1, drv2 : Drv
 reducesTo(drv1, drv2)
 immediatelyDependsOn(drv0, drv1)
--------------------------------------------------- transitive-drv-deligation
+-------------------------------------------------- transitive-reduction
 reducesTo(drv0, drv0[drv2/drv1])
 ```
 ```
@@ -190,13 +197,18 @@ reducesTo(drv0, drv1)
 
 There's a few things we can call out from the faux-formalization.
 
+ - When we build a derivation, we don't want it's associated build.
+   This is because it may not exist if it is not in normal form, and may be an intermediate result if the derivation is recursive.
+   Instead, we we want the associated build of the *evaluation* of the given derivation.
+
  - `isTrustlessStore` is called that because the restricted on the contents—fixed output builds / plain data and drvs—is fully and cheaply verifiabled.
    This is in contrast to normal builds,
    where the relationship between the derivation and build can only be verified by redoing the build,
    and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.
 
- - The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.
-   We should muse on this point, and hopefully write a small-step semantics for both together that is more elegant than the above.
+ - The substitution of drvs in a downstream derivation is very realted to the substitution of drvs for content hashes with the intensional store.
+   I included the normalization of content adressible stores, today done with `hashDerivationModulo` to to help make that clear, and show that we already normalize derivations.
+   I hope to eventually share a formalism for this and RFC #62.
 
  - The *building* itself of derivations is unchanged.
    All the magic happens through the `reducesTo` relation.

From a38f2457691bd08d197eb04ae5c52ce00b7f7560 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Thu, 18 Feb 2021 23:09:53 -0500
Subject: [PATCH 22/58] drv-build-drv: Start drafting from old
 ret-cont-recursive-nix RFC

---
 rfcs/{0000-ret-cont-recursive-nix.md => 0000-drv-build-drv.md} | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 rename rfcs/{0000-ret-cont-recursive-nix.md => 0000-drv-build-drv.md} (99%)

diff --git a/rfcs/0000-ret-cont-recursive-nix.md b/rfcs/0000-drv-build-drv.md
similarity index 99%
rename from rfcs/0000-ret-cont-recursive-nix.md
rename to rfcs/0000-drv-build-drv.md
index 1b8043210..fa6b50266 100644
--- a/rfcs/0000-ret-cont-recursive-nix.md
+++ b/rfcs/0000-drv-build-drv.md
@@ -1,5 +1,5 @@
 ---
-feature: ret-cont-recursive-nix
+feature: drv-build-drv
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
 co-authors: (find a buddy later to help our with the RFC)

From 6c01e4d2415609d091e3fcc1859cd430a37bdcbf Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Sun, 4 Apr 2021 01:34:17 -0400
Subject: [PATCH 23/58] drv-buiild-drv: WIP rewrite

---
 rfcs/0000-drv-build-drv.md | 246 +++++++++++++++++++++----------------
 1 file changed, 141 insertions(+), 105 deletions(-)

diff --git a/rfcs/0000-drv-build-drv.md b/rfcs/0000-drv-build-drv.md
index fa6b50266..489125d10 100644
--- a/rfcs/0000-drv-build-drv.md
+++ b/rfcs/0000-drv-build-drv.md
@@ -2,23 +2,157 @@
 feature: drv-build-drv
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
-co-authors: (find a buddy later to help our with the RFC)
-shepherd-leader: Franz Pletz
-shepherd-team: Franz Pletz, Eelco Dolstra, Shea Levy, Daniel Peebles
+co-authors: (find a buddy later to help out with the RFC)
+shepherd-team: (names, to be nominated and accepted by RFC steering committee)
+shepherd-leader: (name to be appointed by RFC steering committee)
 related-issues: (will contain links to implementation PRs)
 ---
 
 # Summary
 [summary]: #summary
 
-"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds.
-This avoids some platform-specific contortions relating to nested sandboxing.
-More importantly, it prevents imperative and overly linear direct-style build scripts;
-easy to write but throwing away the benefits of Nix.
+Allow derivations to build derivations, and extend our notion of derivation dependencies to include built and not just pre-existing derivations.
+This provides an alternative to "import from derivation" and recursive Nix that avoids many of their drawbacks.
 
 # Motivation
 [motivation]: #motivation
 
+Nix's design encourages a separation of build *planning* from build *execution*:
+evaluation of the Nix language produces derivations, and then then those derivations are built.
+This usually a great thing.
+It's enforced the separation of the more complex Nix expression language from the simpler derivation language.
+It's also encouraged Nixpkgs to take the "birds eye" view and successful grapple a ton of complexity that would have overwhelmed a more traditional package repository.
+
+However, there are sometimes situations where planning and execution has to be intermixed.
+Often this is in the converting of other build systems to Nix derivations, with `cabal2nix`, `crate2nix`, etc.
+Other times this is with extracting dependency information from the source itself, like C++20 modules or, hopefully someday, Haskell modules.
+While it's possible to do all this in one big expensive planning phase, it's much better to start with a courser dependency graph and then refine it.
+
+The two tools we have today for this, "import from derivation" and (still unstable) recursive have serious drawbacks.
+Import from derivation builds imported derivations one at a time due to the design of evaluator.
+Even if that were fixed (and it should be!), it more fundamentally flawed because it requires the evaluation heap/interpreter process to still persist until all the imported derivations are built.
+That makes it is a bad architecture for fine-grained, resumable work.
+
+The other, recursive Nix, has similar issues.
+With `nix-build` in `nix-build`, we again, we have an outer step that blocks awkwardly while the inner one continues.
+And whereas at least evaluation state is a known-entity that could be persisted or garbage collected, the output process tree and other state is a truly inscrutable black box.
+At the very least, all this breaks `--dry-run`.
+
+There's also a social argument.
+"`nix-build` in `nix-build`"is an extremely crude cudgel.
+As I wrote above, it's very good that we push people towards separating planning and building.
+"`nix-build` in `nix-build`" is just too mindlessly imperative, and I fear as Nix grows more popular it could be the vuvuzela of an External September.
+It's very important that people only use dynamism as a last resort, and it's properly functional, and as powerful and flexible as possible.
+
+Note that none of this applies to "`nix-instantiate` in `nix-build`".
+The limited form of recursive Nix that just allows adding data to the store is fine with me, and I will use it in conjunction with this
+Morally, it's just an implementation of derivation producing store path graphs that have dynamic shape, instead of multiple outputs static shape.
+That's a great feature.
+
+Derivation that build derivations is just that.
+It's quote simple to imagine each step: once a derivation is built it's treated like any other.
+And now that we have content-addressed derivation outputs (since the derivations themselves must be content address, built or otherwise) we've done all the hard parts!
+Yet, this one small feature makes the derivation language fully higher order and capable of embedding e.g. the lambda calculus.
+That's awesome, in both the positive and terrifying senses.
+
+# Detailed design
+[design]: #detailed-design
+
+We can break this down nicely into steps
+
+#. Derivation outputs can be valid derivations:
+
+   #. Allow derivation outputs to be content addressed with the "text hashing" scheme.
+
+   #. Lift the restriction barring derivations output paths from ending in `.drv` if they are so content-addressed
+
+#. Extend the CLI to take advantage of such derivations:
+
+   We hopefully will soon allow CLI "installable" args in the form
+   ```
+   single-installable ::= <path> ! <output-name>
+   ```
+   where the first path is a derivation, and the second is the output we want to build.
+
+   We should generalize the grammar like so:
+   ```
+   single-installable ::= <single-installable> ! <output-name>
+                       |  <path>
+
+   multi-installable  ::= <single-installable>
+                       |  <single-installable> ! *
+   ```
+
+   Now that we have `path` vs `path!*`, we also don't need `---derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.
+   (`toBuildables` in the the nix commands should always be pure functions and not consult the store.)
+
+#. Extend the scheduler and derivation dependencies similarly:
+
+   #. Extend data structures.
+
+     The `DerivationGoal` keys are currently a derivation path and a set up output names, with an empty set denoting "all outputs".
+     We should generalize the derivation path to be as powerful as our new `<single-installable>`.
+     Thus, we'll have a vector of output names "between" the root drv path, and the final set of output names.
+
+     The type of `inputDrvs` is conceptually `Set<SingleInstallable>`, but actually a map from drv paths to outputs.
+     It should stay conceptually `Set<SingleInstallable>`, but perhaps to keep the map structure we will do some fancy recursive thing for its concrete implementation.
+
+   #. Derivation goals working from a built derivation during the derivation loading step will instead spawn a subgoal for the derivation goal bilding the derivation.
+      Inductively we eventually reach the base case of static derivation, and those are built just like today.
+
+# Examples and Interactions
+[examples-and-interactions]: #examples-and-interactions
+
+I'll make Haskell and cabal2nix my running example.
+
+`cabal2nix` today is used something like:
+```nix
+haskellPackages.callPackage "something" (runCommand {} "cabal2nix ${mySrc} > $out") {}
+```
+This is pretty good, except for the issues given in the motivation section.
+
+With "`nix-build` in `nix-build`" it's would be something like:
+```nix
+runCommand "something" {} ''
+  cabal2nix ${mySrc} > default.nix
+  nix-build -E '(import ${pkgs.path} {}).haskellPackages.callPackage ./. {};'
+''
+```
+This is terrible.
+In addition to the problems given inn the motivation section, we redo the `cabal2nix` whenever *anything at all* changes in the Nixpkgs source.
+That's a lot of unnecessary rebuilds.
+```nix
+runCommand "something-continued" {} ''
+  nix-build -E '(import ${pkgs.path} {}).haskellPackages.callPackage \
+    ${runCommand "something-continued" {} "cabal2nix ${mySrc} > $out"} \
+    {};'
+''
+```
+This solves that additional problem, but is quite a mouthful.
+Do we really think people are going to write this without lineters nagging?
+The ease of use is trap that will lead people to think this about Nix "simple things work bad, and things that work well are ugly".
+And even after all that, the motivation section problems still remain.
+
+The new way is
+```nix
+runCommand "something.drv" {} "cabal2drv ${mySrc} > $out"
+```
+if we want to bypass the Nix language entirely.
+
+If we don't, there is:
+```nix
+runCommand "something-evaluated" {} ''
+  nix-instantiate -E '(import ${pkgs.path} {}).haskellPackages.callPackage \
+    ${runCommand "something" {} "cabal2nix ${mySrc} > $out"} \
+    {};'
+''
+```
+Yes, this is the same ugliness as before.
+However, there's at least no temptation to write any more commands after the `nix-instantiate`, since we must finish the derivation to build what we've instantiated.
+That will compel people to have fine grained derivations with a single `nix-instantiate` each.
+Also, I hope we could in fact auto-generate this from import-from-derivations, which remains the nicest interface.
+
+
 The benefits of recursive Nix have been described in many places.
 One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
 For this case, people usually imagine derivations like
@@ -122,104 +256,6 @@ stdenv.mkDerivation {
 This further simplifies the implementation.
 Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.
 
-# Detailed design
-[design]: #detailed-design
-
-Derivations today build outputs, and are associated to those outputs.
-We extend the derivation language by allowing a derivation to indicate their output is more derivations, and ultimately be associated with one of *those* derivations's associated outputs.
-Derivations that do so indicate this with some special attribute, say `__recursive`.
-Such derivations must have two outputs, `store` and `drv`.
-`store` would be a local Nix store limited to just drvs and fixed output builds.
-`drv` would contain a symlink to one of the derivations in the store, the root.
-After the build completes, Nix verifies all the drv files and fixed outputs are valid (contents match hashes, etc.) and merges the built store into the ambient store.
-Finally, any uses of the original derivation can be substituted to instead use the symlinked derivation.
-
-To faux-formalize everything in the vein of a small-step semantics:
-```
-∀<d : immediatelyDependsOn(drv, -)>
-  isValue(d)
-drv ? __recursive == false
-drv ? outputHash == false
--------------------------------------------------- normalized
-isValue(drv)
-```
-```
-∀<d : immediatelyDependsOn(drv, -)>
-  isValue(d)
--------------------------------------------------- build-readiness
-isReadyToBuild(drv)
-```
-```
-drv0 : Drv
-∀<o : outputs(drv0)> build_o : StorePath
-∀<o : outputs(drv0)> build(drv0) = { ${o} = build_o; }
-isReadyToBuild(drv0)
--------------------------------------------------- simple-build
-∀<o : outputs(drv0)> assoc(drv0, o, build_o)
-```
-```
-drv : Drv
-drv.outputs = { "out" = ...; }
-drv ? outputHash == true
--------------------------------------------------- immediate-reduction-fixed-output
-reducesTo(drv0, stubDrv(drv.outputHash))
-```
-```
-drv0, drv1 : Drv
-drv1path : RelativePath
-drv0.outputs = { "store" = ...; "drv" = ...; }
-∀<o : outputs(drv0)> build0_o : StorePath
-∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
-drv0 ? __recursive == true
-isTrustlessStore(build0_store)
-drv1 = read(build0_store + drv1path)
-readlink(build0_drv) = build0.store + drv1path
--------------------------------------------------- immediate-reduction-drv-deligation
-reducesTo(drv0, drv1)
-```
-```
-drv0, drv1, drv2 : Drv
-reducesTo(drv1, drv2)
-immediatelyDependsOn(drv0, drv1)
--------------------------------------------------- transitive-reduction
-reducesTo(drv0, drv0[drv2/drv1])
-```
-```
-drv0, drv1 : Drv
-reducesTo(drv0, drv1)
-∀<o : outputs(drv0)> build0_o : StorePath
-∀<o : outputs(drv0)> assoc(drv0, o, build0_o)
--------------------------------------------------- delegative-build
-∀<o : outputs(drv0)> assoc(drv1, o, build1_o)
-```
-
-## Design Notes
-
-There's a few things we can call out from the faux-formalization.
-
- - When we build a derivation, we don't want it's associated build.
-   This is because it may not exist if it is not in normal form, and may be an intermediate result if the derivation is recursive.
-   Instead, we we want the associated build of the *evaluation* of the given derivation.
-
- - `isTrustlessStore` is called that because the restricted on the contents—fixed output builds / plain data and drvs—is fully and cheaply verifiabled.
-   This is in contrast to normal builds,
-   where the relationship between the derivation and build can only be verified by redoing the build,
-   and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.
-
- - The substitution of drvs in a downstream derivation is very realted to the substitution of drvs for content hashes with the intensional store.
-   I included the normalization of content adressible stores, today done with `hashDerivationModulo` to to help make that clear, and show that we already normalize derivations.
-   I hope to eventually share a formalism for this and RFC #62.
-
- - The *building* itself of derivations is unchanged.
-   All the magic happens through the `reducesTo` relation.
-
- - Because drvs can produce plans of drvs producing more drvs ad-infinitum, it's possible to never terminate (no `reducesTo` from a derivation to an `isReadyToBuild` derivation) but that's the user's fault.
-   We can detect simple cycles analogous to black holes in thunks: if a derivation produces a redirected derivation depending on the original, a cycle is effectively recreated even though we don't have a hash fixed point.
-   Nix should raise an error rather than looping, but either behavior is permissible.
-
-# Examples
-[examples]: #examples
-
 As a running example, I'll use @matthewbauer's [Reproducible résumé].
 (Do steal the method; don't steal Matt; he works with me!)
 Here's the original `default.nix`, which uses IFD:

From 6d97de1c2aae7634464741844dfa5a4fd5623a12 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Mon, 26 Apr 2021 01:20:13 -0400
Subject: [PATCH 24/58] plan-dynamism: Rewrite RFC yet again

---
 rfcs/0000-drv-build-drv.md | 514 +++++++++++--------------------------
 1 file changed, 146 insertions(+), 368 deletions(-)

diff --git a/rfcs/0000-drv-build-drv.md b/rfcs/0000-drv-build-drv.md
index 489125d10..22b3944e5 100644
--- a/rfcs/0000-drv-build-drv.md
+++ b/rfcs/0000-drv-build-drv.md
@@ -1,5 +1,5 @@
 ---
-feature: drv-build-drv
+feature: plan-dynamism
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
 co-authors: (find a buddy later to help out with the RFC)
@@ -11,8 +11,9 @@ related-issues: (will contain links to implementation PRs)
 # Summary
 [summary]: #summary
 
-Allow derivations to build derivations, and extend our notion of derivation dependencies to include built and not just pre-existing derivations.
-This provides an alternative to "import from derivation" and recursive Nix that avoids many of their drawbacks.
+We need build plan dynamism -- interleved building and planning -- to cope with the every growing world of language-specific package managers.
+Propose to allow derivations to build derivations, and depend on those built derivations, as the core primitive for this.
+Additionally, introduce a new primop to leverage this in making IFD, still the gold standard for ease of use, more efficient and compatible with `hydra.nixos.org`'s queue runner.
 
 # Motivation
 [motivation]: #motivation
@@ -23,50 +24,49 @@ This usually a great thing.
 It's enforced the separation of the more complex Nix expression language from the simpler derivation language.
 It's also encouraged Nixpkgs to take the "birds eye" view and successful grapple a ton of complexity that would have overwhelmed a more traditional package repository.
 
-However, there are sometimes situations where planning and execution has to be intermixed.
-Often this is in the converting of other build systems to Nix derivations, with `cabal2nix`, `crate2nix`, etc.
-Other times this is with extracting dependency information from the source itself, like C++20 modules or, hopefully someday, Haskell modules.
-While it's possible to do all this in one big expensive planning phase, it's much better to start with a courser dependency graph and then refine it.
-
-The two tools we have today for this, "import from derivation" and (still unstable) recursive have serious drawbacks.
-Import from derivation builds imported derivations one at a time due to the design of evaluator.
-Even if that were fixed (and it should be!), it more fundamentally flawed because it requires the evaluation heap/interpreter process to still persist until all the imported derivations are built.
-That makes it is a bad architecture for fine-grained, resumable work.
-
-The other, recursive Nix, has similar issues.
-With `nix-build` in `nix-build`, we again, we have an outer step that blocks awkwardly while the inner one continues.
-And whereas at least evaluation state is a known-entity that could be persisted or garbage collected, the output process tree and other state is a truly inscrutable black box.
-At the very least, all this breaks `--dry-run`.
-
-There's also a social argument.
-"`nix-build` in `nix-build`"is an extremely crude cudgel.
-As I wrote above, it's very good that we push people towards separating planning and building.
-"`nix-build` in `nix-build`" is just too mindlessly imperative, and I fear as Nix grows more popular it could be the vuvuzela of an External September.
-It's very important that people only use dynamism as a last resort, and it's properly functional, and as powerful and flexible as possible.
-
-Note that none of this applies to "`nix-instantiate` in `nix-build`".
-The limited form of recursive Nix that just allows adding data to the store is fine with me, and I will use it in conjunction with this
-Morally, it's just an implementation of derivation producing store path graphs that have dynamic shape, instead of multiple outputs static shape.
-That's a great feature.
-
-Derivation that build derivations is just that.
-It's quote simple to imagine each step: once a derivation is built it's treated like any other.
-And now that we have content-addressed derivation outputs (since the derivations themselves must be content address, built or otherwise) we've done all the hard parts!
-Yet, this one small feature makes the derivation language fully higher order and capable of embedding e.g. the lambda calculus.
-That's awesome, in both the positive and terrifying senses.
+Nixpkgs, along with every other distro, also faces a looming crisis: new open source software is increasingly not intended to be packaged by distros at all.
+Many languages now support very large library ecosystems, with dependencies expressed in a language-specific package manager.
+To this new generation of developers, the distro (or homebrew) is a crufty relic from an earlier age to bootstrap modernity, and then be forgotten about.
+
+Right now, to deal with these packages, we either convert by hand, or commit lots generated code into Nixpkgs.
+But it's don't think it's either of those is healthy or sustainable.
+The problem with the first is sheer effort; we'll never be able to keep up.
+The problem with the second is bloating Nixpkgs but more importantly reproducability: If someone wants to update that generated code it is unclear how.
+All these mean that potential users coming from this new model of development find Nix / Nixpkgs cumbersome and unsuited to their needs.
+
+The core feature here, derivations that build derivations, is the best fundamental primitive for this problem.
+It's very performant, being well-adapted for Nix's current scheduler.
+Unlike recursive Nix, there's is no potential for half-built dependencies to sit around waiting for other builds, wasting resources.
+Each build step (derivation) always runs start to finish blocking on nothing.
+It's very efficient, because it doesn't obligate the use of the Nix expression language.
+Finally, it's quite compatible with `--dry-run`.
+
+However, "import from derivation" is still far and away the easiest method to use, and the one that existing tools to convert to Nix use.
+We should continue to support it, finding a way for `hydra.nixos.org` to allow it, so those tools with IFD can be used in Nixpkgs and become first-class members of the Nix ecosystem.
+We have a fairly straightforward mechanism, only slightly more cumbersome than IFD today, to allow evaluation to be deferred after imported things are built.
+This frees up the Hydra evaluator to finish before building, and also meshes well with any plan to build more than one eval-realized path at a time.
+This should allow us to drop the `hydra.nixos.org` restriction.
+
+With these steps, I think we will be able to successfully convert to Nix a bunch of developers that mainly work in one language, and didn't even think they were in need of a better distro.
+In turn, I hope these upstream packages and ecosystems might even care about packaging and integration of the sort that we do.
+This would create a virtuous cycle where Nix is easier to use by more people, and Nixpkgs is easier to maintain as upstream packages better match our values.
 
 # Detailed design
 [design]: #detailed-design
 
-We can break this down nicely into steps
+We can break this down nicely into steps.
 
-#. Derivation outputs can be valid derivations:
+## Dynamic derivations
 
-   #. Allow derivation outputs to be content addressed with the "text hashing" scheme.
+*This is implemented in https://github.com/NixOS/nix/pull/4628.*
 
-   #. Lift the restriction barring derivations output paths from ending in `.drv` if they are so content-addressed
+1. Derivation outputs can be valid derivations:
 
-#. Extend the CLI to take advantage of such derivations:
+   1. Allow derivation outputs to be content addressed with the "text hashing" scheme.
+
+   2. Lift the restriction barring derivations output paths from ending in `.drv` if they are so content-addressed
+
+2. Extend the CLI to take advantage of such derivations:
 
    We hopefully will soon allow CLI "installable" args in the form
    ```
@@ -86,300 +86,134 @@ We can break this down nicely into steps
    Now that we have `path` vs `path!*`, we also don't need `---derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.
    (`toBuildables` in the the nix commands should always be pure functions and not consult the store.)
 
-#. Extend the scheduler and derivation dependencies similarly:
-
-   #. Extend data structures.
+3. Extend the scheduler and derivation dependencies similarly:
 
-     The `DerivationGoal` keys are currently a derivation path and a set up output names, with an empty set denoting "all outputs".
-     We should generalize the derivation path to be as powerful as our new `<single-installable>`.
-     Thus, we'll have a vector of output names "between" the root drv path, and the final set of output names.
+  - Derivations can depended on the outputs of derivations that are themselves derivation outputs.
+    The schedule will substitute derivations to simplify dependencies as computed derivations are built, just like how floating content addressed derivations are realized.
 
-     The type of `inputDrvs` is conceptually `Set<SingleInstallable>`, but actually a map from drv paths to outputs.
-     It should stay conceptually `Set<SingleInstallable>`, but perhaps to keep the map structure we will do some fancy recursive thing for its concrete implementation.
-
-   #. Derivation goals working from a built derivation during the derivation loading step will instead spawn a subgoal for the derivation goal bilding the derivation.
-      Inductively we eventually reach the base case of static derivation, and those are built just like today.
-
-# Examples and Interactions
-[examples-and-interactions]: #examples-and-interactions
-
-I'll make Haskell and cabal2nix my running example.
-
-`cabal2nix` today is used something like:
-```nix
-haskellPackages.callPackage "something" (runCommand {} "cabal2nix ${mySrc} > $out") {}
-```
-This is pretty good, except for the issues given in the motivation section.
+  - Missing derivations get their own full fledged goals so they can be built, not just fetched from substituters.
 
-With "`nix-build` in `nix-build`" it's would be something like:
-```nix
-runCommand "something" {} ''
-  cabal2nix ${mySrc} > default.nix
-  nix-build -E '(import ${pkgs.path} {}).haskellPackages.callPackage ./. {};'
-''
-```
-This is terrible.
-In addition to the problems given inn the motivation section, we redo the `cabal2nix` whenever *anything at all* changes in the Nixpkgs source.
-That's a lot of unnecessary rebuilds.
-```nix
-runCommand "something-continued" {} ''
-  nix-build -E '(import ${pkgs.path} {}).haskellPackages.callPackage \
-    ${runCommand "something-continued" {} "cabal2nix ${mySrc} > $out"} \
-    {};'
-''
-```
-This solves that additional problem, but is quite a mouthful.
-Do we really think people are going to write this without lineters nagging?
-The ease of use is trap that will lead people to think this about Nix "simple things work bad, and things that work well are ugly".
-And even after all that, the motivation section problems still remain.
-
-The new way is
-```nix
-runCommand "something.drv" {} "cabal2drv ${mySrc} > $out"
-```
-if we want to bypass the Nix language entirely.
-
-If we don't, there is:
-```nix
-runCommand "something-evaluated" {} ''
-  nix-instantiate -E '(import ${pkgs.path} {}).haskellPackages.callPackage \
-    ${runCommand "something" {} "cabal2nix ${mySrc} > $out"} \
-    {};'
-''
-```
-Yes, this is the same ugliness as before.
-However, there's at least no temptation to write any more commands after the `nix-instantiate`, since we must finish the derivation to build what we've instantiated.
-That will compel people to have fine grained derivations with a single `nix-instantiate` each.
-Also, I hope we could in fact auto-generate this from import-from-derivations, which remains the nicest interface.
+4. Add a new `outputOf` primop:
 
+   `builtins.outputOf drv outputName` produces a placeholder string with the appropriate string context to access the output of that name produced by that derivation.
+   The placeholder string is quite analogous to that used for floating content addressed derivation outputs.
+   This isn't needed today when we statically know the derivation and thus it's outputs to make attributes for them, but it is needed for built derivations because we don't know either the final derivation or its output names in advanced.
 
-The benefits of recursive Nix have been described in many places.
-One main reason is if we want Nix to function as a build system and package manager, we need upstream packages to use Nix too without duplicating their build systems in Nixpkgs.
-For this case, people usually imagine derivations like
-```nix
-{ stdenv, pkgs, nix }:
+## Deferred import from derivation.
 
-stdenv.mkDerivation {
-  name = "foo";
-  version = "1.2.3";
+Create a new primop `assumeDerivation` that takes a single expression.
+It's semantics are as follows:
 
-  src = ...;
+ - If the underlying expression evaluates (shallowly) without needing to build derivations (for `import`, `readFile`, etc.), and is a derivation, simply return that.
+   ```
+   e ⇓ e'
+   e' is derivation
+   ------
+   builtins.assumeDerivation e ⇓ e'
+   ```
 
-  nativeBuildInputs = [ nix ];
-  NIX_PATH = "nixpkgs=${pkgs.path}";
+- If the underlying expression cannot evaluate (shallowly) without building on or more paths, defer evaluation into a derivation-producing-derivation, and take it's output with `builtins.outputOf`:
+  ```
+  e gets stuck on builds
+  defer = derivation {
+    inputDrvs = builds;
+    buildCommand = "nix-instantiate ${reified e} > $out"
+  }
+  ------
+  builtins.assumeDerivation e ⇓ builtins.outputOf e "out"
+  ```
 
-  outputs = [ "out" "dev" ];
+This allows downstream non-dynamic derivations to be evaluated without getting stuck on their dynamic upstream ones.
+They just depend on the derivation computed by the deferred eval derivations, and those substitutions are performed by the scheduler.
 
-  doConfigure = false;
-  doBuild = false;
+# Examples and Interactions
+[examples-and-interactions]: #examples-and-interactions
 
-  installPhase = ''
-    for o in $outputs; do
-      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
-      cp -r $pkg ${!o}
-    done
-  '';
-}
-```
-The other main reason is other build systems should be translated to Nix without vendoring tons of autogenerated code in Nixpkgs.
-For this, case, the one difference is we need to generate some Nix first.
-```nix
-stdenv.mkDerivation {
-  # ...
-  installPhase = ''
-    bazel2nix # new bit
-    for o in $outputs; do
-      pkg=$(nix-build -E '((import <nixpkgs> {}).callPackage ./. {}).'"$o")
-      cp -r $pkg ${!o}
-    done
-  '';
-}
-```
+Here is everything put together with a Haskell and `cabal2nix` example.
 
-"Ret-cont" recursive Nix, short for "return-continuation" recursive Nix, is a different take on recursive Nix.
-The normal variant in the examples above might be termed "direct-style" recursive Nix.
-Consider what happens with the recursive "nix-build" in those examples:
-the outer build blocks while the inner one builds, and then the other one continues.
-Just as we can CPS-transform programs, reifying the context of a function call as another function (which is passed as an argument), so we can imagine splitting the derivation in two at this blocking point.
-This gives the "continuation" part of the name.
-But whereas the CPS transformation makes the continuation an argument, the Nix *derivation* language is first order.
-Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
-Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
-This gives the "return" part of the name.
-Both differences together, the first example becomes something like:
+Given the following code, and assuming `bar` will depend on `foo`
 ```nix
-{ stdenv, pkgs, nix }:
-
-stdenv.mkDerivation {
-  name = "foo";
-  version = "1.2.3";
-
-  src = ...;
-
-  nativeBuildInputs = [ nix ];
-  NIX_PATH = "nixpkgs=${pkgs.path}";
-
-  __recursive = true;
-
-  outputs = [ "drv" ];
-
-  doConfigure = false;
-  doBuild = false;
-
-  installPhase = ''
-    mv $(nix-instantiate -E '((import <nixpkgs> {}).callPackage ./. {})') $drv
-  '';
-}
+mkScope (self: {
+  foo = builtins.assumeDerivation (self.callCabal2nix "foo" ./foo);
+  bar = builtins.assumeDerivation (self.callCabal2nix "bar" ./bar);
+})
 ```
-Note how in this case we don't need to do any "post-processing" of the produced derivation.
-When the outer derivation can just "become" the inner derivation, explicitly copying the derivation outputs like before becomes unnecessary.
-
-So why prefer this variation of the standard design?
-I've always been concerned with the ease of which someone can just "nix-build ...; nix-build ...; nix-build ..." within a derivation with recursive Nix.
-This creates a linear chain of dependencies, which isn't terribly performant: shorter critical paths are crucial for parallelism and incrementality and this fails with both.
-Building derivations is less convenient, but makes linear chains and the proper dependency graph *equally* less convenient, removing the perverse incentive.
-And in general, dynamism in the dependency graph, which is the essence of what recursive Nix provides, is only a feature of last resort, so making it more difficult across the board isn't concerning.
-
-Additionally, see https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717 for Eelco's draft implementation of recursive Nix, and the Darwin sandboxing restrictions that make it a Linux-only feature.
-Sandboxing and Darwin are crucial to Nix today, and we shouldn't sacrifice either of them.
-With "ret-cont" recursive Nix, actual builds are never nested, so we don't need any fancy constraints on the derivation "runtime" (i.e. the code that actually performs and isolates builds).
-Furthermore, we can skip needing to talk to the daemon by just producing a local store:
-```nix
-stdenv.mkDerivation {
-  # ...
-  outputs = [ "drv" "store" ];
-  installPhase = ''
-    mv $(nix-instantiate --store $store -E '((import <nixpkgs> {}).callPackage ./. {})') $drv
-  '';
-}
+We get something like the following derivations with dependencies
 ```
-This further simplifies the implementation.
-Derivations remain built exactly as today, with only logic *between* building steps that is entirely platform-agnostic changing.
-
-As a running example, I'll use @matthewbauer's [Reproducible résumé].
-(Do steal the method; don't steal Matt; he works with me!)
-Here's the original `default.nix`, which uses IFD:
-
-```nix
-{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
-let
-
-README = stdenv.mkDerivation {
-  name = "README";
-  unpackPhase = "true";
-  buildInputs = [ emacs ];
-  installPhase = ''
-    mkdir -p $out
-    cd $out
-    cp -r ${./fonts} fonts
-    cp ${./README.org} README.org
-    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
-    cp resume.nix default.nix
-  '';
-};
-
-in import README {inherit nixpkgs;}
+(cabal2nix foo)!out ----> deFoo = deferred eval (fooNix: self.callPackage fooNix)
+(cabal2nix bar)!out ----> deBar = deferred eval (barNix: self.callPackage barNix)
 ```
-
-The shortest way to make it instead use "Ret-cont" recursive Nix is this:
-
+and evaluated nix
 ```nix
-{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
-
-stdenv.mkDerivation {
-  name = "README";
-  unpackPhase = "true";
-  outputs = [ "drv" "store" ];
-  __recursive = true;
-  buildInputs = [ emacs nix ];
-  installPhase = ''
-    mkdir -p $out
-    cd $out
-    cp -r ${./fonts} fonts
-    cp ${./README.org} README.org
-    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
-    mv $(nix-instantiate --store $store resume.nix --arg nixpkgs 'import ${nixpkgs.path}') > $drv
-  '';
-}
+mkScope (self: {
+  foo = builtins.outputOf /nix/store/deFoo.drv "out";
+  bar = builtins.outputOf /nix/store/deBar.drv "out";
+})
 ```
 
-But note how this means we re-run emacs every time anything in Nixpkgs changes, no good!
-Here's a better version which is more incremental:
+If we then build `bar` we will get something steps like:
 
-```nix
-{nixpkgs ? <nixpkgs>}: with import nixpkgs {};
-let
-
-# Just like original
-README = stdenv.mkDerivation {
-  name = "README";
-  unpackPhase = "true";
-  buildInputs = [ emacs ];
-  installPhase = ''
-    mkdir -p $out
-    cd $out
-    cp -r ${./fonts} fonts
-    cp ${./README.org} README.org
-    emacs --batch -l ob-tangle --eval "(org-babel-tangle-file \"README.org\")"
-    cp resume.nix default.nix
-  '';
-};
-
-in stdenv.mkDerivation {
-  name = "readme-outer";
-  unpackPhase = "true";
-  buildInputs = [ nix ];
-  outputs = [ "drv" "store" ];
-  __recursive = true;
-  installPhase = ''
-    mv $(nix-instantiate --store $store ${README} --arg nixpkgs 'import ${nixpkgs.path}') > $drv
-  '';
-}
-```
+1. ```
+   deBar!out
+   ```
+2. Expand `deBar` in to see dep
+   ```
+   ((cabal2nix bar)!out ----> (deferred eval (fooNix: self.callPackage barNix {}))!out
+   ```
+3. Build cabal2nix and inline path for simplicity (or if cabal2nix job is floating CA derivation)
+   ```
+   (deferred eval self.callPackage built-barNix)!out
+   ```
+4. Build deferred eval job and substitute
+   ```
+   deFoo!out ----> bar.drv!out
+   ```
+5. Expand `deFoo` in to see dep
+   ```
+   ((cabal2nix foo)!out ----> (deferred eval (fooNix: self.callPackage fooNix {}))!out ----> bar.drv!out
+   ```
+6. Build cabal2nix and inline path for simplicity (or if cabal2nix job is floating CA derivation)
+   ```
+   (deferred eval self.callPackage built-fooNix)!out ----> bar.drv!out
+   ```
+7. Build deferred eval job and substitute
+   ```
+   foo.drv!out ----> bar.drv!out
+   ```
+8. Build foo and realize bar
+   ```
+   bar.drv[foo-path/foo!out]!out
+   ```
+9. Build bar
+   ```
+   bar-path
+   ```
 
-Now only `readme-outer` is rebuilt when Nixpkgs and Nix changes.
-This may still seem wasteful, but remember we still need to reevaluate whenever those changes.
-Now some of that evaluation work is pushed into the derivations themselves.
+The above is no doubt hard to read -- I am sorry about that --- but here are a few things to note:
 
-This is actually a crucial observation:
-A limit to scaling Nix today is that while our builds are very incremental, evaluation isn't.
-But if we can "push" some of the work of evaluation "deeper" into the derivaiton graph itself, we can be incremental with respect to both.
-This means we are incremental at all levels.
+ - The scheduler "substitutes on demand" giving us a lazy evaluation of sorts.
+   This means that in the extreme case where we to make to, e.g., make a derivation for every C compiler invocation, we can avoid storing a very large completely static graph all at once.
+   
+ - At the same time, the derivations can be built in many different orders, so one can intentionally build all the `cabal2nix` derivations first and try to accumulate up the biggest static graph with `--dry-run`.
+   This approximates what would happen in the "infinitely parallel" case when the scheduler will try to dole out work to do as fast as it can.
 
 # Drawbacks
 [drawbacks]: #drawbacks
 
- - The opinionated nature may put off those who think Nix is too hard to learn already, and think simple recursive "nix-build" is good for newcomers.
-
- - If we ever want full recursive Nix, this doesn't really build in that direction.
-   It sidesteps the bulk of the difficulty which is in making the nested sandboxing and daemon communication secure.
-   To me though, this is a feature not a bug; I don't want to go in that direction just yet.
+The main drawback is that the *only* the stub expressions are "pure" derivation.
+For other sort of values, we have no choice but wait.
+That means we cannot defer the `pname` `meta` etc. fields: either make do with the bare string `builtins.outputOf` provides, or *statically* add a fake `name` and `meta` etc. that must be manually synced with the deferred eval derivation if it is to match.
 
 # Alternatives
 [alternatives]: #alternatives
 
- - Don't allow fixed-output builds.
-   All data can be stuck inside the drv file, so this can be cut without limiting expressive power.
-   But this is much less efficient, and more cumbersome for whatever produces the data.
-
- - Use a socket to talk to the host daemon.
-   https://github.com/edolstra/nix/commit/1a27aa7d64ffe6fc36cfca4d82bdf51c4d8cf717, a draft implementation of full recursive Nix, has done this and we can take the details from that.
-   This might sightly more efficient by reducing moving files, but is conceptual overkill given this design.
-   No direct access to the host daemon rules out a bunch of security concerns, and simplifies the interface for non-Nix tools producing derivations.
-   The latter I very much hope will happen, just as Ninja is currently used with CMake, Meson, etc., today.
-
- - Full recursive Nix (builds within builds)
-
- - Import from derivation.
-   This has been traditionally considered an alternative to this, but I will soon propose an implementation of that relying on this; I no longer consider the two in conflict.
-
- - `builtins.exec` runs an arbitrary command at eval time as the user triggering evaluation.
-   This is highly impure; nothing at all tries to make the environment deterministic.
-   It is useful for writing fetchers that need the impurities to access the internet and secrets, while also managing their own caching.
-   But for everything else, I view it strictly worse than IFD.
-
- - Keeping the status quo and use vendoring.
-   But then Nix will never scale to bridging the package manager and build system divide.
+ - Do nothing, and continue to have no good answer for language-specific package mangers.
+ 
+ - Instead of deferring only certain portions of the evaluation with `builtins.asssumeDerivation`, simply restart the entire eval.
+   Quite simple, and no existing IFD code today needs to change.
+   
+ - Abandon IFD and just let users use the underlying dynamic derivation feature.
+   This was my first idea, but I became worried that this was too hard to use for simple experiments.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions
@@ -389,69 +223,13 @@ The exact way the outputs refer to the replacement derivations / their outputs i
 # Future work
 [future]: #future-work
 
-1. As the example shows, we can push the work of evaluation into builds.
-   This unlocks lots of future work in Nixpkgs:
-
-   - Leveraging language-specific tools to generate plans Nix builds, rather than reimplementing much of those tools.
-     We do this with IFD today, but both due to Hydra's architecture, and concerns about eval times regardless of Hydra, we don't allow this in Nixpkgs.
-     This is a huge loss as we either do things entirely manually (python) Or vender tons of code (Haskell).
-     We will save valuable human time, and start to bridge the distribution / ops vs developer cultural divide by making it easier to work on your own packages.
-
-   - Simply fetching and importing packages which use Nix for their build system, like Nix itself and hydra, rather than vendoring that build system in.
-     This is an easier special case of the above, where the upstream developer knows and loves Nix, and their package has a Nix-based build system.
-     Flakes are supposed to help with this, as is `builtins.fetchTarball` skirting the IFD prohibition.
-     But, it's better if the hydra evaluator can avoid blocking on the download and/or evaluating the Nix expressions therein.
-     "Ret-cont" recursive Nix would allow this by just putting the "outer" derivation in Nixpkgs.
-
-   As an example of the former, Ninja (is getting)[Ninja Dyndeps] a very similar notion they call `dyndep`s in their upcoming release.
-   This is needed for C++ and Fortran modules.
-   If CMake and other tools use it as expected, we would need "Ret-cont" to automatically translate their build plans for fine-grained builds of large projects like LLVM or Chromium.
-   Shake, soon Ninja, and eventually [LLBuild], are the only general purpose build systems I know that do or aim to do dynamic dependencies, but none of them sandbox.
-   If we become the only way to both correctly and incrementally build modern C++, that will be a huge opportunity for further growth.
-
-2. Better still, we can try to automatically transform evaluation without writing manually "outer" derivations.
-   With `--pure` mode, Eelco has also talked about opening the door to caching evaluation.
-   "Ret-cont" recursive Nix is wonderful foundation for that.
-
-   I hope to at least later proposal automatically converting IFD into "Ret-cont":
-
-   IFD is also slow because the evaluator isn't concurrent and so imported derivations get built one at a time.
-   We can fix this somewhat by modifying the evaluator so that evaluation continues where the *value* of the previously-imported derivation isn't needed.
-   But, we still will inevitably get stuck somewhere shallower in the expression when the value being built is needed.
-
-   With "Ret-cont", we can cleverly avoid needing that value in certain common situations.
-   Quite often, IFD is creating a derivation, so we will have something like:
-
-   ```nix
-   "blah blah blah ${/*some expression ... is stuck because deep inside: */ (import /* ... */) /* ... */} blah blah blah"
-   ```
-
-   Without knowing what the value of that expression is, we may reasonably assume it's coercible to a string.
-   If it isn't, well, evaluation will fail anyways.
-   We can then make a "scratch" derivation, however, that reifies the evaluation of the stuck term, we can splice the scratch derivation's hash instead:
-
-   ```nix
-   "bash balh blash /nix/store/123asdf4sdf1g2dfgh34fg8h7fg69876-i-like-to-procrastinate-and-hope-for-the-best blah blah blah"
-   ```
-
-   Now, evaluation truly isn't stuck at all, and we are as free to continue as if there was no IFD at all!
-   The only failure mode would be if the import was a string but *wasn't* a derivation.
-   But, I imagine we can annotate things such that Nix knows when to speculate like this.
-   (c.f. Compiler hot and cold pragmas.)
-   The author of the Code almost always knows what *type* of thing they are splicing, so I would think we could so annotate quite faithfully.
-   I emphasize "type" because if we ever get a type system, this becomes much easier:
-   Specifying types for imported expressions (along with other explicit signature to guide inference) is wholly sufficient to derive the type of all such splices.
-
-  Another future project would be some speculative strictness to allow one round of evaluation to return *both* a partial build plan and stuck imported derivations.
-  Currently the plan must still be evaluated entirely before any building of "actually needed" derivations, i.e. those which *aren't* imported, begins.
-
-3. More broadly, we can get rid of a special notion of "evaluation" entirely.
-   We can think of evaluation today as basically a special case single layer of dynamism, where the outer work (evaluation) is impure.
-   If all `--pure` and IFD evaluation is done inside Nix builds, then the Nix daemon need not even know about the nix language at all.
-   We can have completely separate tools running inside sandboxes that deal with the Nix expression langauge.
-
-[Reproducible résumé]: https://github.com/matthewbauer/resume
-
-[Ninja Dyndeps]: https://github.com/ninja-build/ninja/pull/1521
+1. Actually use this stuff in Nixpkgs with modification to the existing "lang2nix" tools.
+   This is the lowest hanging fruit and most import thing.
 
-[LLBuild]: https://github.com/apple/swift-llbuild
+2. Try to breach the build system package manager divide.
+   Just as there are foreign packages graphs to convert to Nix, there are Ninja and Make graphs we can also convert to Nix.
+   This might really help with big builds like Chromium and LLVM.
+   
+3. Try to convince upstream tools to use Nix like CMake, Meson, etc. use Ninja.
+   Rather than converting Ninja plans, we might convince those tools to have purpose-built Nix backends.
+   Language-specific package mangers that don't use Ninja today might also be modified to "let Nix do that actual building".

From 2079528a157e175079532509e09d2e000543d918 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Mon, 26 Apr 2021 01:20:55 -0400
Subject: [PATCH 25/58] plan-dynamism: Rename file accordingly

---
 rfcs/{0000-drv-build-drv.md => 0000-plan-dynanism.md} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename rfcs/{0000-drv-build-drv.md => 0000-plan-dynanism.md} (100%)

diff --git a/rfcs/0000-drv-build-drv.md b/rfcs/0000-plan-dynanism.md
similarity index 100%
rename from rfcs/0000-drv-build-drv.md
rename to rfcs/0000-plan-dynanism.md

From fd7bb41943d1d9513e3cd9cc656c99f98633b389 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Mon, 26 Apr 2021 10:26:12 -0400
Subject: [PATCH 26/58] plan-dynanism: Fix typo

Thanks @mweinelt.
---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 22b3944e5..20ee0113e 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -11,7 +11,7 @@ related-issues: (will contain links to implementation PRs)
 # Summary
 [summary]: #summary
 
-We need build plan dynamism -- interleved building and planning -- to cope with the every growing world of language-specific package managers.
+We need build plan dynamism -- interleaved building and planning -- to cope with the ever-growing world of language-specific package managers.
 Propose to allow derivations to build derivations, and depend on those built derivations, as the core primitive for this.
 Additionally, introduce a new primop to leverage this in making IFD, still the gold standard for ease of use, more efficient and compatible with `hydra.nixos.org`'s queue runner.
 

From ec622cdbc51fdfe32916278e036c85761179d281 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Mon, 26 Apr 2021 15:18:42 -0400
Subject: [PATCH 27/58] plan-dynanism: Fix formalism slightly

---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 20ee0113e..4754f9411 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -120,7 +120,7 @@ It's semantics are as follows:
     buildCommand = "nix-instantiate ${reified e} > $out"
   }
   ------
-  builtins.assumeDerivation e ⇓ builtins.outputOf e "out"
+  builtins.assumeDerivation e ⇓ builtins.outputOf defer "out"
   ```
 
 This allows downstream non-dynamic derivations to be evaluated without getting stuck on their dynamic upstream ones.

From ad74b6882c4a9d0d70ce3ae6907b96367bd56e06 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Mon, 26 Apr 2021 15:22:15 -0400
Subject: [PATCH 28/58] Apply suggestions from code review

Thanks!

Co-authored-by: Rosario Pulella <Rosariopulella@gmail.com>
Co-authored-by: Ninjatrappeur <NinjaTrappeur@users.noreply.github.com>
---
 rfcs/0000-plan-dynanism.md | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 4754f9411..78e8e4159 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -28,8 +28,8 @@ Nixpkgs, along with every other distro, also faces a looming crisis: new open so
 Many languages now support very large library ecosystems, with dependencies expressed in a language-specific package manager.
 To this new generation of developers, the distro (or homebrew) is a crufty relic from an earlier age to bootstrap modernity, and then be forgotten about.
 
-Right now, to deal with these packages, we either convert by hand, or commit lots generated code into Nixpkgs.
-But it's don't think it's either of those is healthy or sustainable.
+Right now, to deal with these packages, we either convert by hand, or commit lots of generated code into Nixpkgs.
+But I don't think either of those options are healthy or sustainable.
 The problem with the first is sheer effort; we'll never be able to keep up.
 The problem with the second is bloating Nixpkgs but more importantly reproducability: If someone wants to update that generated code it is unclear how.
 All these mean that potential users coming from this new model of development find Nix / Nixpkgs cumbersome and unsuited to their needs.
@@ -112,7 +112,7 @@ It's semantics are as follows:
    builtins.assumeDerivation e ⇓ e'
    ```
 
-- If the underlying expression cannot evaluate (shallowly) without building on or more paths, defer evaluation into a derivation-producing-derivation, and take it's output with `builtins.outputOf`:
+- If the underlying expression cannot evaluate (shallowly) without building one or more paths, defer evaluation into a derivation-producing-derivation, and take it's output with `builtins.outputOf`:
   ```
   e gets stuck on builds
   defer = derivation {
@@ -200,7 +200,7 @@ The above is no doubt hard to read -- I am sorry about that --- but here are a f
 # Drawbacks
 [drawbacks]: #drawbacks
 
-The main drawback is that the *only* the stub expressions are "pure" derivation.
+The main drawback is that the stub expressions are *only* "pure" derivations.
 For other sort of values, we have no choice but wait.
 That means we cannot defer the `pname` `meta` etc. fields: either make do with the bare string `builtins.outputOf` provides, or *statically* add a fake `name` and `meta` etc. that must be manually synced with the deferred eval derivation if it is to match.
 

From 0eaa6bbb793a03f53fc04bd3114a1de7036e541b Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 27 Apr 2021 00:31:42 -0400
Subject: [PATCH 29/58] plan-dynamism: `Buildables` -> `DerivedPathsWithHints`

Thanks @sternenseemann for catching.
---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 78e8e4159..2fbb748c1 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -84,7 +84,7 @@ We can break this down nicely into steps.
    ```
 
    Now that we have `path` vs `path!*`, we also don't need `---derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.
-   (`toBuildables` in the the nix commands should always be pure functions and not consult the store.)
+   (`toDerivedPathsWithHints` in the the nix commands should always be pure functions and not consult the store.)
 
 3. Extend the scheduler and derivation dependencies similarly:
 

From 49070db293fe11bce95c06949fd51f32a003f0ae Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 27 Apr 2021 00:50:32 -0400
Subject: [PATCH 30/58] plan-dynamism: Add semantics and examples for `!`
 syntax

---
 rfcs/0000-plan-dynanism.md | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 2fbb748c1..42a55b1f9 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -83,6 +83,22 @@ We can break this down nicely into steps.
                        |  <single-installable> ! *
    ```
 
+   Plain paths just mean that path itself is the goal, while `!` indexing indicates one more outputs of the derivation to the left of the `!` is the goal.
+
+   > For example,
+   > ```
+   > nix build /nix/store/…foo.drv
+   > ```
+   > would just obtain `/nix/store/…foo.drv` and not build it, while
+   > ```
+   > nix build /nix/store/…foo.drv!*
+   > ```
+   > would obtain (by building or substituting) all its outputs.
+   > ```
+   > nix build /nix/store/…foo.drv!out!out
+   > ```
+   > would obtain the `out` output of whatever derivation `/nix/store/…foo.drv!out` produces.
+
    Now that we have `path` vs `path!*`, we also don't need `---derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.
    (`toDerivedPathsWithHints` in the the nix commands should always be pure functions and not consult the store.)
 

From c50ee4395b5df7b20a1b31fdceb25895e109d9d1 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 27 Apr 2021 00:52:39 -0400
Subject: [PATCH 31/58] plan-dynamism: Too many dashes in `--derivation`

---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 42a55b1f9..99251194b 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -99,7 +99,7 @@ We can break this down nicely into steps.
    > ```
    > would obtain the `out` output of whatever derivation `/nix/store/…foo.drv!out` produces.
 
-   Now that we have `path` vs `path!*`, we also don't need `---derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.
+   Now that we have `path` vs `path!*`, we also don't need `--derivation` as a disambiguator, and so that should be removed along with all the complexity that goes with it.
    (`toDerivedPathsWithHints` in the the nix commands should always be pure functions and not consult the store.)
 
 3. Extend the scheduler and derivation dependencies similarly:

From 98ea32a14cbd973340a6a06b5d6c2e119a28b78e Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 6 Jun 2021 15:06:57 -0400
Subject: [PATCH 32/58] plan-dynanism: Put pupose of text hashing before name

---
 rfcs/0000-plan-dynanism.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 99251194b..09e1ff39a 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -62,7 +62,8 @@ We can break this down nicely into steps.
 
 1. Derivation outputs can be valid derivations:
 
-   1. Allow derivation outputs to be content addressed with the "text hashing" scheme.
+   1. Allow derivation outputs to be content addressed in the same manner as drv files.
+      (The little-exposed name for this is "text" content addressing).
 
    2. Lift the restriction barring derivations output paths from ending in `.drv` if they are so content-addressed
 

From c01f07cc25b68e0734f8ab97c6060b5f0611ad8b Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 6 Jun 2021 15:51:17 -0400
Subject: [PATCH 33/58] Apply suggestions from code review

Co-authored-by: Ollie Charles <ollie@ocharles.org.uk>
---
 rfcs/0000-plan-dynanism.md | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 09e1ff39a..2b590778a 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -13,7 +13,7 @@ related-issues: (will contain links to implementation PRs)
 
 We need build plan dynamism -- interleaved building and planning -- to cope with the ever-growing world of language-specific package managers.
 Propose to allow derivations to build derivations, and depend on those built derivations, as the core primitive for this.
-Additionally, introduce a new primop to leverage this in making IFD, still the gold standard for ease of use, more efficient and compatible with `hydra.nixos.org`'s queue runner.
+Additionally, introduce a new primop to leverage this in making "import from derivation" (IFD), still the gold standard for ease of use, more efficient and compatible with `hydra.nixos.org`'s queue runner.
 
 # Motivation
 [motivation]: #motivation
@@ -42,6 +42,9 @@ It's very efficient, because it doesn't obligate the use of the Nix expression l
 Finally, it's quite compatible with `--dry-run`.
 
 However, "import from derivation" is still far and away the easiest method to use, and the one that existing tools to convert to Nix use.
+\[Actually it's not just `import` which is notable, one can `builtins.readFile` a not-yet-buit path and it will also block today, and probably other such primops.
+The exact primop doesn't matter --- all are noticeably more ergonomic than alternatives, and our proposal here is agnostic to the prim-op which would trigger the blocking.
+I will continue to use "IFD" as an umbrella acronym despite its deficiencies because it is best known.\]
 We should continue to support it, finding a way for `hydra.nixos.org` to allow it, so those tools with IFD can be used in Nixpkgs and become first-class members of the Nix ecosystem.
 We have a fairly straightforward mechanism, only slightly more cumbersome than IFD today, to allow evaluation to be deferred after imported things are built.
 This frees up the Hydra evaluator to finish before building, and also meshes well with any plan to build more than one eval-realized path at a time.
@@ -60,12 +63,15 @@ We can break this down nicely into steps.
 
 *This is implemented in https://github.com/NixOS/nix/pull/4628.*
 
-1. Derivation outputs can be valid derivations:
+1. Derivation outputs can be valid derivations.
+   \[If one tries to output a drv file today, they will find Nix doesn't accept the output as such because these small paper cuts.
+   This list item and its children should be thought of as "lifting artificial restrictions".\]
 
    1. Allow derivation outputs to be content addressed in the same manner as drv files.
       (The little-exposed name for this is "text" content addressing).
 
-   2. Lift the restriction barring derivations output paths from ending in `.drv` if they are so content-addressed
+   2. Lift the (perhaps not yet documented) restriction barring derivations output paths from ending in `.drv`, but only for derivation outputs that are so content-addressed.
+      \[There are probably other ways to make store paths that end in `.drv` that aren't valid derivations, so we could make the simpler change of lifting this restriction entirely without breaking invariants. But I'm fine keeping it for the wrong sorts of derivations as a useful guard rail.\]
 
 2. Extend the CLI to take advantage of such derivations:
 
@@ -105,8 +111,8 @@ We can break this down nicely into steps.
 
 3. Extend the scheduler and derivation dependencies similarly:
 
-  - Derivations can depended on the outputs of derivations that are themselves derivation outputs.
-    The schedule will substitute derivations to simplify dependencies as computed derivations are built, just like how floating content addressed derivations are realized.
+  - Derivations can depend on the outputs of derivations that are themselves derivation outputs.
+    The scheduler will substitute derivations to simplify dependencies as computed derivations are built, just like how floating content addressed derivations are realized.
 
   - Missing derivations get their own full fledged goals so they can be built, not just fetched from substituters.
 
@@ -114,7 +120,9 @@ We can break this down nicely into steps.
 
    `builtins.outputOf drv outputName` produces a placeholder string with the appropriate string context to access the output of that name produced by that derivation.
    The placeholder string is quite analogous to that used for floating content addressed derivation outputs.
-   This isn't needed today when we statically know the derivation and thus it's outputs to make attributes for them, but it is needed for built derivations because we don't know either the final derivation or its output names in advanced.
+   \[With just floating content-addressed derivations but no computed derivations, derivations are always known statically but their outputs aren't.
+   With this RFC, since derivations themselves can floating CA derivation outputs, we also might not know them statically, so we need "deep" placeholders to account for arbitrary layers of dynamism.
+   This also corresponds to the use of arbitrary many `!` in the CLI.\]
 
 ## Deferred import from derivation.
 

From 3f187a3c58c9a9217a3532cc05f5e734856a4497 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 6 Jun 2021 16:04:06 -0400
Subject: [PATCH 34/58] Apply suggestions from code review

Co-authored-by: Ollie Charles <ollie@ocharles.org.uk>
---
 rfcs/0000-plan-dynanism.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 2b590778a..b6aa0f41b 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -29,7 +29,7 @@ Many languages now support very large library ecosystems, with dependencies expr
 To this new generation of developers, the distro (or homebrew) is a crufty relic from an earlier age to bootstrap modernity, and then be forgotten about.
 
 Right now, to deal with these packages, we either convert by hand, or commit lots of generated code into Nixpkgs.
-But I don't think either of those options are healthy or sustainable.
+But I don't think either of those options is healthy or sustainable.
 The problem with the first is sheer effort; we'll never be able to keep up.
 The problem with the second is bloating Nixpkgs but more importantly reproducability: If someone wants to update that generated code it is unclear how.
 All these mean that potential users coming from this new model of development find Nix / Nixpkgs cumbersome and unsuited to their needs.
@@ -225,14 +225,15 @@ The above is no doubt hard to read -- I am sorry about that --- but here are a f
 # Drawbacks
 [drawbacks]: #drawbacks
 
-The main drawback is that the stub expressions are *only* "pure" derivations.
+The main drawback is that these stub expressions are *only* "pure" derivations --- placeholder strings (with the proper string context) and not attrsets with all the niceties we are used to getting from `mkDerivation`.
+This is true even when the deferred evaluation in fact *does* use `mkDerivation` and would provide those niceties.
 For other sort of values, we have no choice but wait.
 That means we cannot defer the `pname` `meta` etc. fields: either make do with the bare string `builtins.outputOf` provides, or *statically* add a fake `name` and `meta` etc. that must be manually synced with the deferred eval derivation if it is to match.
 
 # Alternatives
 [alternatives]: #alternatives
 
- - Do nothing, and continue to have no good answer for language-specific package mangers.
+ - Do nothing, and continue to have no good answer for language-specific package managers.
  
  - Instead of deferring only certain portions of the evaluation with `builtins.asssumeDerivation`, simply restart the entire eval.
    Quite simple, and no existing IFD code today needs to change.

From 7d187800915f1f0f17f186e036a39a15b925c087 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 6 Jun 2021 16:04:32 -0400
Subject: [PATCH 35/58] Apply suggestions from code review

---
 rfcs/0000-plan-dynanism.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index b6aa0f41b..6c8713475 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -227,8 +227,8 @@ The above is no doubt hard to read -- I am sorry about that --- but here are a f
 
 The main drawback is that these stub expressions are *only* "pure" derivations --- placeholder strings (with the proper string context) and not attrsets with all the niceties we are used to getting from `mkDerivation`.
 This is true even when the deferred evaluation in fact *does* use `mkDerivation` and would provide those niceties.
-For other sort of values, we have no choice but wait.
-That means we cannot defer the `pname` `meta` etc. fields: either make do with the bare string `builtins.outputOf` provides, or *statically* add a fake `name` and `meta` etc. that must be manually synced with the deferred eval derivation if it is to match.
+For other sort of values, we have no choice but wait; that would require a fully incremental / deferral evaluation which is a completely separate feature not an extension of this.
+Concretely, our design means we cannot defer the `pname` `meta` etc. fields: either make do with the bare string `builtins.outputOf` provides, or *statically* add a fake `name` and `meta` etc. that must be manually synced with the deferred eval derivation if it is to match.
 
 # Alternatives
 [alternatives]: #alternatives

From 5e92cc93ce437e84e55963fca59982d780e7cfc3 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Sun, 6 Jun 2021 16:12:12 -0400
Subject: [PATCH 36/58] Update rfcs/0000-plan-dynanism.md

---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 6c8713475..77206d0c2 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -163,7 +163,7 @@ mkScope (self: {
   bar = builtins.assumeDerivation (self.callCabal2nix "bar" ./bar);
 })
 ```
-We get something like the following derivations with dependencies
+After some evaluation, we get something like the following derivations with dependencies:
 ```
 (cabal2nix foo)!out ----> deFoo = deferred eval (fooNix: self.callPackage fooNix)
 (cabal2nix bar)!out ----> deBar = deferred eval (barNix: self.callPackage barNix)

From 1e2012cc5df1d37ea2315c14d73061a5f833b1bd Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Wed, 21 Jul 2021 16:46:46 -0600
Subject: [PATCH 37/58] plan-dynanism: Fix bad sentence

Thanks @roberth!
---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 77206d0c2..a5ef0702b 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -121,7 +121,7 @@ We can break this down nicely into steps.
    `builtins.outputOf drv outputName` produces a placeholder string with the appropriate string context to access the output of that name produced by that derivation.
    The placeholder string is quite analogous to that used for floating content addressed derivation outputs.
    \[With just floating content-addressed derivations but no computed derivations, derivations are always known statically but their outputs aren't.
-   With this RFC, since derivations themselves can floating CA derivation outputs, we also might not know them statically, so we need "deep" placeholders to account for arbitrary layers of dynamism.
+   With this RFC, since drv files themselves can be floating CA derivation outputs, we also might not know the derivations statically, so we need "deep" placeholders to account for arbitrary layers of dynamism.
    This also corresponds to the use of arbitrary many `!` in the CLI.\]
 
 ## Deferred import from derivation.

From 5836c0273d937f4cc17b73ea18cb1b79c7af6691 Mon Sep 17 00:00:00 2001
From: John Ericson <git@JohnEricson.me>
Date: Fri, 3 Sep 2021 12:23:12 -0400
Subject: [PATCH 38/58] plan-dynamism: Number the two parts

---
 rfcs/0000-plan-dynanism.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index a5ef0702b..b8f24ac49 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -59,7 +59,7 @@ This would create a virtuous cycle where Nix is easier to use by more people, an
 
 We can break this down nicely into steps.
 
-## Dynamic derivations
+## Part 1: Dynamic derivations
 
 *This is implemented in https://github.com/NixOS/nix/pull/4628.*
 
@@ -124,7 +124,7 @@ We can break this down nicely into steps.
    With this RFC, since drv files themselves can be floating CA derivation outputs, we also might not know the derivations statically, so we need "deep" placeholders to account for arbitrary layers of dynamism.
    This also corresponds to the use of arbitrary many `!` in the CLI.\]
 
-## Deferred import from derivation.
+## Part 2: Deferred import from derivation.
 
 Create a new primop `assumeDerivation` that takes a single expression.
 It's semantics are as follows:

From 970ff43c8e94f0cdc48989bb7e283a77b6ab025e Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 12 Oct 2021 17:05:57 -0400
Subject: [PATCH 39/58] plan-dynamism: Rip out part 2

There is more to do I'm sure, but I wanted to get the ball rolling.
---
 rfcs/0000-plan-dynanism.md | 37 ++-----------------------------------
 1 file changed, 2 insertions(+), 35 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index b8f24ac49..492bd86ba 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -59,8 +59,6 @@ This would create a virtuous cycle where Nix is easier to use by more people, an
 
 We can break this down nicely into steps.
 
-## Part 1: Dynamic derivations
-
 *This is implemented in https://github.com/NixOS/nix/pull/4628.*
 
 1. Derivation outputs can be valid derivations.
@@ -124,33 +122,6 @@ We can break this down nicely into steps.
    With this RFC, since drv files themselves can be floating CA derivation outputs, we also might not know the derivations statically, so we need "deep" placeholders to account for arbitrary layers of dynamism.
    This also corresponds to the use of arbitrary many `!` in the CLI.\]
 
-## Part 2: Deferred import from derivation.
-
-Create a new primop `assumeDerivation` that takes a single expression.
-It's semantics are as follows:
-
- - If the underlying expression evaluates (shallowly) without needing to build derivations (for `import`, `readFile`, etc.), and is a derivation, simply return that.
-   ```
-   e ⇓ e'
-   e' is derivation
-   ------
-   builtins.assumeDerivation e ⇓ e'
-   ```
-
-- If the underlying expression cannot evaluate (shallowly) without building one or more paths, defer evaluation into a derivation-producing-derivation, and take it's output with `builtins.outputOf`:
-  ```
-  e gets stuck on builds
-  defer = derivation {
-    inputDrvs = builds;
-    buildCommand = "nix-instantiate ${reified e} > $out"
-  }
-  ------
-  builtins.assumeDerivation e ⇓ builtins.outputOf defer "out"
-  ```
-
-This allows downstream non-dynamic derivations to be evaluated without getting stuck on their dynamic upstream ones.
-They just depend on the derivation computed by the deferred eval derivations, and those substitutions are performed by the scheduler.
-
 # Examples and Interactions
 [examples-and-interactions]: #examples-and-interactions
 
@@ -233,13 +204,9 @@ Concretely, our design means we cannot defer the `pname` `meta` etc. fields: eit
 # Alternatives
 [alternatives]: #alternatives
 
- - Do nothing, and continue to have no good answer for language-specific package managers.
+ - Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
  
- - Instead of deferring only certain portions of the evaluation with `builtins.asssumeDerivation`, simply restart the entire eval.
-   Quite simple, and no existing IFD code today needs to change.
-   
- - Abandon IFD and just let users use the underlying dynamic derivation feature.
-   This was my first idea, but I became worried that this was too hard to use for simple experiments.
+ - Embrace recursive Nix in its current form.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions

From d6d6b17789f31080a4593e9fcb1163ffe2fe751f Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 12 Oct 2021 17:59:23 -0400
Subject: [PATCH 40/58] plan-dynamism: New motivation

---
 rfcs/0000-plan-dynanism.md | 64 +++++++++++++++++++++++---------------
 1 file changed, 39 insertions(+), 25 deletions(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 492bd86ba..6c92df353 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -18,41 +18,55 @@ Additionally, introduce a new primop to leverage this in making "import from der
 # Motivation
 [motivation]: #motivation
 
+> Instead of recursive Nix builds, the alternative is to have one gigantic build graph.
+> For instance, if we are building a component that needs a C compiler, the Nix expression for that component simply imports the Nix expression that builds the compiler.
+> The problem with this approach is scalability: the resulting build graphs would become huge.
+> The graph for a simple component such as GNU Hello would include the build graphs for dozens of large components, such as Glibc, GCC, etc.
+> The resulting graph could easily have hundreds of thousands of nodes, far exceeding the graphs typically occurring in deployment (e.g., the one in Figure 1.5).
+> However, apart from its efficiency, this is possibly the most desirable solution because of its conceptual simplicity.
+> Thus it is interesting to develop efficient ways of dealing with very large build graphs
+
+-- Eelco Dolstra's this, page 240.
+
 Nix's design encourages a separation of build *planning* from build *execution*:
 evaluation of the Nix language produces derivations, and then then those derivations are built.
 This usually a great thing.
 It's enforced the separation of the more complex Nix expression language from the simpler derivation language.
 It's also encouraged Nixpkgs to take the "birds eye" view and successful grapple a ton of complexity that would have overwhelmed a more traditional package repository.
 
-Nixpkgs, along with every other distro, also faces a looming crisis: new open source software is increasingly not intended to be packaged by distros at all.
-Many languages now support very large library ecosystems, with dependencies expressed in a language-specific package manager.
-To this new generation of developers, the distro (or homebrew) is a crufty relic from an earlier age to bootstrap modernity, and then be forgotten about.
-
-Right now, to deal with these packages, we either convert by hand, or commit lots of generated code into Nixpkgs.
-But I don't think either of those options is healthy or sustainable.
-The problem with the first is sheer effort; we'll never be able to keep up.
-The problem with the second is bloating Nixpkgs but more importantly reproducability: If someone wants to update that generated code it is unclear how.
-All these mean that potential users coming from this new model of development find Nix / Nixpkgs cumbersome and unsuited to their needs.
+The core feature here, derivations that build derivations, is a nice sneaky fundamental primitive for the problem Eelco point's out.
 
-The core feature here, derivations that build derivations, is the best fundamental primitive for this problem.
 It's very performant, being well-adapted for Nix's current scheduler.
 Unlike recursive Nix, there's is no potential for half-built dependencies to sit around waiting for other builds, wasting resources.
 Each build step (derivation) always runs start to finish blocking on nothing.
 It's very efficient, because it doesn't obligate the use of the Nix expression language.
-Finally, it's quite compatible with `--dry-run`.
 
-However, "import from derivation" is still far and away the easiest method to use, and the one that existing tools to convert to Nix use.
-\[Actually it's not just `import` which is notable, one can `builtins.readFile` a not-yet-buit path and it will also block today, and probably other such primops.
-The exact primop doesn't matter --- all are noticeably more ergonomic than alternatives, and our proposal here is agnostic to the prim-op which would trigger the blocking.
-I will continue to use "IFD" as an umbrella acronym despite its deficiencies because it is best known.\]
-We should continue to support it, finding a way for `hydra.nixos.org` to allow it, so those tools with IFD can be used in Nixpkgs and become first-class members of the Nix ecosystem.
-We have a fairly straightforward mechanism, only slightly more cumbersome than IFD today, to allow evaluation to be deferred after imported things are built.
-This frees up the Hydra evaluator to finish before building, and also meshes well with any plan to build more than one eval-realized path at a time.
-This should allow us to drop the `hydra.nixos.org` restriction.
+It's also quite compatible with `--dry-run`.
+Because derivations don't get new dependencies *mid build*, we have no need to mess with individual steps to explore the plan.
+There still becomes multiple sorts of `--dry-run` policies, but all of them just have to do with building or not buidling derivations which *themselves* are unchanged.
+
+To make that more, clear, if you *do* want one big ("hundreds of thousands of nodes"-big), static graph, you can still have it!
+Build all the derivations that compute derivations, but not nothing else.
+Then the results of those can be substituted (think partial eval, also remember we already do this sort of thing for CA derivations), and one has just that.
+
+If one *doesn't* want that however, do a normal build, and graph in "goals" form in Nixpkgs can stay small.
+Graphs evaluate into large graphs, but goals are GC'd as they are built.
+This keeps the "working set" small, at least in the archetypal use-case where the computed subgraphs are disjoint, coming from the `Makefile`s of individual packages.
+
+Finally there is a sense in which this extension is very natural.
+The opening sentence of every revised scheme report is:
 
-With these steps, I think we will be able to successfully convert to Nix a bunch of developers that mainly work in one language, and didn't even think they were in need of a better distro.
-In turn, I hope these upstream packages and ecosystems might even care about packaging and integration of the sort that we do.
-This would create a virtuous cycle where Nix is easier to use by more people, and Nixpkgs is easier to maintain as upstream packages better match our values.
+> Programming languages should be designed not by piling feature on top of feature,
+> but by removing the weaknesses and restrictions that make additional features appear necessary.
+
+We already have a dynamic scheduler that doesn't need to know all the goals up front.
+We also already rewrite derivations based on previous builds for CA-derivations.
+All the underlying mechanisms are thus there, and the patch implementing this in a sense wrote itself.
+
+Now, there is a good argument that maybe the Nix derivation language today has other implementation strategies where this *wouldn't* be so natural and easy.
+This is like saying "we can add this axiom for free in our current model, but not in all possible models of our current axioms".
+Well, if such a concrete other strategy ever arises, it is very easy to statically prohibit the new features this RFC proposes.
+Until then, down with the artificial restrictions!
 
 # Detailed design
 [design]: #detailed-design
@@ -189,7 +203,7 @@ The above is no doubt hard to read -- I am sorry about that --- but here are a f
 
  - The scheduler "substitutes on demand" giving us a lazy evaluation of sorts.
    This means that in the extreme case where we to make to, e.g., make a derivation for every C compiler invocation, we can avoid storing a very large completely static graph all at once.
-   
+
  - At the same time, the derivations can be built in many different orders, so one can intentionally build all the `cabal2nix` derivations first and try to accumulate up the biggest static graph with `--dry-run`.
    This approximates what would happen in the "infinitely parallel" case when the scheduler will try to dole out work to do as fast as it can.
 
@@ -205,7 +219,7 @@ Concretely, our design means we cannot defer the `pname` `meta` etc. fields: eit
 [alternatives]: #alternatives
 
  - Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
- 
+
  - Embrace recursive Nix in its current form.
 
 # Unresolved questions
@@ -222,7 +236,7 @@ The exact way the outputs refer to the replacement derivations / their outputs i
 2. Try to breach the build system package manager divide.
    Just as there are foreign packages graphs to convert to Nix, there are Ninja and Make graphs we can also convert to Nix.
    This might really help with big builds like Chromium and LLVM.
-   
+
 3. Try to convince upstream tools to use Nix like CMake, Meson, etc. use Ninja.
    Rather than converting Ninja plans, we might convince those tools to have purpose-built Nix backends.
    Language-specific package mangers that don't use Ninja today might also be modified to "let Nix do that actual building".

From 6ab3338bf7f3e74811f8c24017e22c9127965173 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Tue, 12 Oct 2021 18:11:09 -0400
Subject: [PATCH 41/58] plan-dynamism: Fix typo

Thanks @L-as
---
 rfcs/0000-plan-dynanism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0000-plan-dynanism.md
index 6c92df353..7d52ab31e 100644
--- a/rfcs/0000-plan-dynanism.md
+++ b/rfcs/0000-plan-dynanism.md
@@ -26,7 +26,7 @@ Additionally, introduce a new primop to leverage this in making "import from der
 > However, apart from its efficiency, this is possibly the most desirable solution because of its conceptual simplicity.
 > Thus it is interesting to develop efficient ways of dealing with very large build graphs
 
--- Eelco Dolstra's this, page 240.
+-- [*The Purely Functional Software Deployment Model*](https://edolstra.github.io/pubs/phd-thesis.pdf), Eelco Dolstra's dissertation, page 240.
 
 Nix's design encourages a separation of build *planning* from build *execution*:
 evaluation of the Nix language produces derivations, and then then those derivations are built.

From 5f7d4da69e0d2c3bd3354351818e0c215940fb94 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Fri, 17 Sep 2021 19:21:47 -0400
Subject: [PATCH 42/58] TEMP PLES AMEND

---
 rfcs/0000-plan-dynanism-example.nix | 125 ++++++++++++++++++++++++++++
 1 file changed, 125 insertions(+)
 create mode 100644 rfcs/0000-plan-dynanism-example.nix

diff --git a/rfcs/0000-plan-dynanism-example.nix b/rfcs/0000-plan-dynanism-example.nix
new file mode 100644
index 000000000..e49d94b1f
--- /dev/null
+++ b/rfcs/0000-plan-dynanism-example.nix
@@ -0,0 +1,125 @@
+# with import ../config.nix;
+with import <nixpkgs> {};
+with pkgs;
+with stdenv;
+
+# A simple content-addressed derivation.
+# The derivation can be arbitrarily modified by passing a different `seed`,
+# but the output will always be the same
+rec {
+  root = mkDerivation {
+    # Name must be identical to the name of "dependent" because the name is
+    # part of the hash scheme
+    name = "text-hashed-root";
+    buildCommand = ''
+      set -x
+      echo "Building a CA derivation"
+      mkdir -p $out
+      echo "Hello World" > $out/hello
+    '';
+    __contentAddressed = true;
+    outputHashMode = "recursive";
+    outputHashAlgo = "sha256";
+  };
+  recursive-test = mkDerivation {
+    name = "dir-of-text-hashed-root";
+    buildCommand = ''
+      echo "Copying the derivation"
+      export NIX_PATH=nixpkgs=${pkgs.path}
+
+      # replace this with assumeDerivation
+      # nix-instantiate is okay, not nix-build
+      # does assumeDerivation enforces the "tail-call recursion"
+
+      # HASH=$(readLockFile {./Go.mod} {./go.sum})
+
+      # builtins.fetchThisForMe
+      # builtins.instantiate
+
+      # Pros
+      # cleaner
+      # no nested builds, only instantiation
+      # easier to stabilize
+
+      # Cons
+      # not same power?
+
+      # 1) recursive-nix
+      # 2) RFC92 can output .drv
+      # 3) "full recursive-nix": complexity
+
+      # ninja2nix, splice Nix into derivations
+
+      THING=$(${pkgs.nix}/bin/nix-instantiate -E '
+         derivation {
+          name = "text-hashed-root";
+          system = "x86_64-linux";
+          builder = "/bin/sh";
+          args = ["-c" "echo hi there ''${(import <nixpkgs>{}).blender} > $out"];
+
+          ## Assert that .drv IS a derivation and _contentAddressed, correct everything
+
+          __contentAddressed = true;
+          outputHashMode = "recursive"; # flat (hash a file), text, recursive (hash a dir)
+          outputHashAlgo = "sha256";
+         }')
+
+      mkdir $out
+      echo $THING
+      cp $THING $out/out1
+      cp $THING $out/out2
+    '';
+    outputs = ["out" ];
+    __contentAddressed = true; outputHashMode = "recursive"; outputHashAlgo = "sha256";
+  };
+  nixSourceGenerator ::: drv -> Nix Source Code
+  nixSourceGenerator = drv: runCommand some_name {} ''
+    export NIX_PATH=nixpkgs=${pkgs.path}
+    cat > $out <<EOF
+      derivation {
+       name = "text-hashed-root";
+       system = "x86_64-linux";
+       builder = "/bin/sh";
+       args = ["-c" "echo hi there ''${(import <nixpkgs>{}).${drv.buildInputs.pname} > $out"];
+       __contentAddressed = true;
+       outputHashMode = "recursive";
+       outputHashAlgo = "sha256";
+      }')
+    EOF
+    '';
+
+
+  dependent2 = mkDerivation { # {{{
+    name = "text-hashed-root.drv";
+    buildCommand = ''
+      echo "Link the derivation"
+      ln -s ${root.drvPath} $out
+    '';
+    # Can we link it instead? It will resolve symlink?
+    __contentAddressed = true;
+    outputHashMode = "text";
+    outputHashAlgo = "sha256";
+  }; # }}}
+
+  # Part two allows this?
+  #splitter-new = builtins.assumeDerivation dependent;
+
+  # this is awkward, needs the right name
+  splitter = mkDerivation {
+    name = "text-hashed-root.drv";
+    buildCommand = ''
+      cp ${recursive-test}/out2 $out
+    '';
+    __contentAddressed = true; outputHashMode = "text"; outputHashAlgo = "sha256";
+    # BUG outputHashMode is not used?
+    # contentAddressed T/F is not checked?
+  };
+  wrapper = mkDerivation {
+    name = "put-it-all-together";
+    buildCommand = ''
+      echo "Copying the output of the dynamic derivation"
+      cp -r ${builtins.outputOf splitter.out "out"} $out
+    '';
+    __contentAddressed = true; outputHashMode = "recursive"; outputHashAlgo = "sha256";
+  };
+}

From ea388e707bd2143f047ec3cfef1787c3643791ae Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 19:36:55 +0000
Subject: [PATCH 43/58] [RFC 0092] Rename file

---
 rfcs/{0000-plan-dynanism.md => 0092-plan-dynamism.md} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename rfcs/{0000-plan-dynanism.md => 0092-plan-dynamism.md} (100%)

diff --git a/rfcs/0000-plan-dynanism.md b/rfcs/0092-plan-dynamism.md
similarity index 100%
rename from rfcs/0000-plan-dynanism.md
rename to rfcs/0092-plan-dynamism.md

From f6741c499ae83ec770c92b3ce0a513b985a459ea Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 19:45:51 +0000
Subject: [PATCH 44/58] [RFC 0092] Fix YAML header

---
 rfcs/0092-plan-dynamism.md | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 7d52ab31e..921edbee9 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -2,10 +2,10 @@
 feature: plan-dynamism
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
-co-authors: (find a buddy later to help out with the RFC)
-shepherd-team: (names, to be nominated and accepted by RFC steering committee)
-shepherd-leader: (name to be appointed by RFC steering committee)
-related-issues: (will contain links to implementation PRs)
+co-authors: Las Safin (@L-as)
+shepherd-team: @tomberek, @ldesgoui, @gytis-ivaskevicius, @L-as
+shepherd-leader: @tomberek
+related-issues: https://github.com/NixOS/nix/pull/4628 https://github.com/NixOS/nix/pull/5364 https://github.com/NixOS/nix/pull/4543 https://github.com/NixOS/nix/pull/3959
 ---
 
 # Summary
@@ -18,7 +18,7 @@ Additionally, introduce a new primop to leverage this in making "import from der
 # Motivation
 [motivation]: #motivation
 
-> Instead of recursive Nix builds, the alternative is to have one gigantic build graph.
+> Instead of Recursive Nix builds, the alternative is to have one gigantic build graph.
 > For instance, if we are building a component that needs a C compiler, the Nix expression for that component simply imports the Nix expression that builds the compiler.
 > The problem with this approach is scalability: the resulting build graphs would become huge.
 > The graph for a simple component such as GNU Hello would include the build graphs for dozens of large components, such as Glibc, GCC, etc.
@@ -37,7 +37,7 @@ It's also encouraged Nixpkgs to take the "birds eye" view and successful grapple
 The core feature here, derivations that build derivations, is a nice sneaky fundamental primitive for the problem Eelco point's out.
 
 It's very performant, being well-adapted for Nix's current scheduler.
-Unlike recursive Nix, there's is no potential for half-built dependencies to sit around waiting for other builds, wasting resources.
+Unlike Recursive Nix, there's is no potential for half-built dependencies to sit around waiting for other builds, wasting resources.
 Each build step (derivation) always runs start to finish blocking on nothing.
 It's very efficient, because it doesn't obligate the use of the Nix expression language.
 
@@ -79,7 +79,7 @@ We can break this down nicely into steps.
    \[If one tries to output a drv file today, they will find Nix doesn't accept the output as such because these small paper cuts.
    This list item and its children should be thought of as "lifting artificial restrictions".\]
 
-   1. Allow derivation outputs to be content addressed in the same manner as drv files.
+   1. Allow derivation outputs to be content-addressed in the same manner as drv files.
       (The little-exposed name for this is "text" content addressing).
 
    2. Lift the (perhaps not yet documented) restriction barring derivations output paths from ending in `.drv`, but only for derivation outputs that are so content-addressed.
@@ -124,14 +124,14 @@ We can break this down nicely into steps.
 3. Extend the scheduler and derivation dependencies similarly:
 
   - Derivations can depend on the outputs of derivations that are themselves derivation outputs.
-    The scheduler will substitute derivations to simplify dependencies as computed derivations are built, just like how floating content addressed derivations are realized.
+    The scheduler will substitute derivations to simplify dependencies as computed derivations are built, just like how floating content-addressed derivations are realized.
 
   - Missing derivations get their own full fledged goals so they can be built, not just fetched from substituters.
 
 4. Add a new `outputOf` primop:
 
    `builtins.outputOf drv outputName` produces a placeholder string with the appropriate string context to access the output of that name produced by that derivation.
-   The placeholder string is quite analogous to that used for floating content addressed derivation outputs.
+   The placeholder string is quite analogous to that used for floating content-addressed derivation outputs.
    \[With just floating content-addressed derivations but no computed derivations, derivations are always known statically but their outputs aren't.
    With this RFC, since drv files themselves can be floating CA derivation outputs, we also might not know the derivations statically, so we need "deep" placeholders to account for arbitrary layers of dynamism.
    This also corresponds to the use of arbitrary many `!` in the CLI.\]
@@ -220,7 +220,7 @@ Concretely, our design means we cannot defer the `pname` `meta` etc. fields: eit
 
  - Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
 
- - Embrace recursive Nix in its current form.
+ - Embrace Recursive Nix in its current form.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions

From b5aa21df694dd4c4a220cf9b16b1a1d9e03064d2 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:06:14 +0000
Subject: [PATCH 45/58] [RFC 0092] Rewrite summary

---
 rfcs/0092-plan-dynamism.md | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 921edbee9..8dbaa5126 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -11,9 +11,28 @@ related-issues: https://github.com/NixOS/nix/pull/4628 https://github.com/NixOS/
 # Summary
 [summary]: #summary
 
-We need build plan dynamism -- interleaved building and planning -- to cope with the ever-growing world of language-specific package managers.
-Propose to allow derivations to build derivations, and depend on those built derivations, as the core primitive for this.
-Additionally, introduce a new primop to leverage this in making "import from derivation" (IFD), still the gold standard for ease of use, more efficient and compatible with `hydra.nixos.org`'s queue runner.
+We introduce three fundamental new features:
+- The ability to have derivations which output store path end in `.drv`
+  (e.g. `$out` is /nix/store/something.drv).
+- The ability for a derivation to depend on the output of a derivation,
+  that isn't yet built but has to be built by another derivation.
+- A primitive `builtins.outputOf` to make use of this feature from within
+  the Nix language.
+
+These features work best in combination with Recursive Nix, such that you
+can add to the host store from within the build.
+It can replace doing `nix build` within a build with a mechanism
+that works better with the design constraints of Nix.
+
+Notable improvements it allows:
+- We can split up big builds like the Linux kernel into
+  smaller derivations without introducing automatically generated
+  code into Nixpkgs.
+- We can do the above automatically for many *2nix tools,
+  allowing us to have source-file-level derivations for most
+  languages (forget crate-level!).
+- We can fetch Merkle trees by just knowing the hash of the root,
+  with Θ(n) derivations for n nodes in the tree.
 
 # Motivation
 [motivation]: #motivation

From 866dc5ec37de00278590234ccde01807e4d3701d Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:10:30 +0000
Subject: [PATCH 46/58] [RFC 0092] Add link to documentation

---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 8dbaa5126..68031cbaa 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -99,7 +99,7 @@ We can break this down nicely into steps.
    This list item and its children should be thought of as "lifting artificial restrictions".\]
 
    1. Allow derivation outputs to be content-addressed in the same manner as drv files.
-      (The little-exposed name for this is "text" content addressing).
+      (`outputHashMode = "text";`, see [Advanced Attributes](https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html)).
 
    2. Lift the (perhaps not yet documented) restriction barring derivations output paths from ending in `.drv`, but only for derivation outputs that are so content-addressed.
       \[There are probably other ways to make store paths that end in `.drv` that aren't valid derivations, so we could make the simpler change of lifting this restriction entirely without breaking invariants. But I'm fine keeping it for the wrong sorts of derivations as a useful guard rail.\]

From 098fe688a44c7fbb15087cbf812da3e81a5dc79a Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:30:23 +0000
Subject: [PATCH 47/58] [RFC 0092] Rewrite example section

---
 rfcs/0092-plan-dynamism.md | 90 +++++++++++++-------------------------
 1 file changed, 31 insertions(+), 59 deletions(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 68031cbaa..24607be25 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -158,73 +158,45 @@ We can break this down nicely into steps.
 # Examples and Interactions
 [examples-and-interactions]: #examples-and-interactions
 
-Here is everything put together with a Haskell and `cabal2nix` example.
+A good example is available at https://github.com/L-as/nix-build.nix.
 
-Given the following code, and assuming `bar` will depend on `foo`
+Specifically, we can do the following:
 ```nix
-mkScope (self: {
-  foo = builtins.assumeDerivation (self.callCabal2nix "foo" ./foo);
-  bar = builtins.assumeDerivation (self.callCabal2nix "bar" ./bar);
-})
-```
-After some evaluation, we get something like the following derivations with dependencies:
-```
-(cabal2nix foo)!out ----> deFoo = deferred eval (fooNix: self.callPackage fooNix)
-(cabal2nix bar)!out ----> deBar = deferred eval (barNix: self.callPackage barNix)
-```
-and evaluated nix
-```nix
-mkScope (self: {
-  foo = builtins.outputOf /nix/store/deFoo.drv "out";
-  bar = builtins.outputOf /nix/store/deBar.drv "out";
-})
+{ pkgs, nixBuild }:
+
+let
+  drv = pkgs.runCommand "hello-drv.nix" {} ''
+    echo "with import ${pkgs.path} {}; hello" > $out
+  '';
+in
+nixBuild pkgs.system "hello" drv
 ```
 
-If we then build `bar` we will get something steps like:
+`nixBuild` essentially runs the following builder internally:
+```bash
+cp $(nix-instantiate $input) $out
+```
 
-1. ```
-   deBar!out
-   ```
-2. Expand `deBar` in to see dep
-   ```
-   ((cabal2nix bar)!out ----> (deferred eval (fooNix: self.callPackage barNix {}))!out
-   ```
-3. Build cabal2nix and inline path for simplicity (or if cabal2nix job is floating CA derivation)
-   ```
-   (deferred eval self.callPackage built-barNix)!out
-   ```
-4. Build deferred eval job and substitute
-   ```
-   deFoo!out ----> bar.drv!out
-   ```
-5. Expand `deFoo` in to see dep
-   ```
-   ((cabal2nix foo)!out ----> (deferred eval (fooNix: self.callPackage fooNix {}))!out ----> bar.drv!out
-   ```
-6. Build cabal2nix and inline path for simplicity (or if cabal2nix job is floating CA derivation)
-   ```
-   (deferred eval self.callPackage built-fooNix)!out ----> bar.drv!out
-   ```
-7. Build deferred eval job and substitute
-   ```
-   foo.drv!out ----> bar.drv!out
-   ```
-8. Build foo and realize bar
-   ```
-   bar.drv[foo-path/foo!out]!out
-   ```
-9. Build bar
-   ```
-   bar-path
-   ```
+However, you don't have to use the Nix language, nor do you have to use `nix-instantiate`.
+The following also works:
+```bash
+cat > $out <<END
+Derive([("out","/nix/store/15c875mwri8xx3s0gqsdkdw7sqqyv55c-hello-2.10","","")],[("/nix/store/3x7l9pm7hqbhz2s59hsrg2y1dxr8glw8-hello-2.10.tar.gz.drv",["out"]),("/nix/store/bqfy8ydlxs0xhzqmy7rc3zjw60vwab6j-stdenv-linux.drv",["out"]),("/nix/store/k6c94x4g937sh8wh5sx90czd9wn9apks-bash-5.1-p8.drv",["out"])],["/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],"x86_64-linux","/nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8/bin/bash",["-e","/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],[("buildInputs",""),("builder","/nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8/bin/bash"),("configureFlags",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck","1"),("doInstallCheck",""),("name","hello-2.10"),("nativeBuildInputs",""),("out","/nix/store/15c875mwri8xx3s0gqsdkdw7sqqyv55c-hello-2.10"),("outputs","out"),("patches",""),("pname","hello"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("src","/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz"),("stdenv","/nix/store/qcq1y0nfxv8za7w6c682s93gk87r2xy1-stdenv-linux"),("strictDeps",""),("system","x86_64-linux"),("version","2.10")])
+END
+```
 
-The above is no doubt hard to read -- I am sorry about that --- but here are a few things to note:
+The way you would use a derivation that outputs a derivation to `out` is then
+as such:
+```nix
+{ pkgs, drv }
 
- - The scheduler "substitutes on demand" giving us a lazy evaluation of sorts.
-   This means that in the extreme case where we to make to, e.g., make a derivation for every C compiler invocation, we can avoid storing a very large completely static graph all at once.
+pkgs.runCommand "example" {} ''
+  ls ${builtins.outputOf drv.out "out"} > $out
+''
+```
 
- - At the same time, the derivations can be built in many different orders, so one can intentionally build all the `cabal2nix` derivations first and try to accumulate up the biggest static graph with `--dry-run`.
-   This approximates what would happen in the "infinitely parallel" case when the scheduler will try to dole out work to do as fast as it can.
+Given a path to a derivation that might not yet be built, `builtins.outputOf`
+gives us the path to an output of it.
 
 # Drawbacks
 [drawbacks]: #drawbacks

From fed89917e9cf7c61f7ed0a5c7504cc63e7ef5191 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:31:47 +0000
Subject: [PATCH 48/58] [RFC 0092] Small fix

---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 24607be25..1bf6779a9 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -21,7 +21,7 @@ We introduce three fundamental new features:
 
 These features work best in combination with Recursive Nix, such that you
 can add to the host store from within the build.
-It can replace doing `nix build` within a build with a mechanism
+It can replace invoking `nix build` within a build with a mechanism
 that works better with the design constraints of Nix.
 
 Notable improvements it allows:

From 854fd9b5d1c247e0a36825092e6cb8f262d12725 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:43:18 +0000
Subject: [PATCH 49/58] [RFC 0092] Rewrite drawbacks and alternatives

---
 rfcs/0092-plan-dynamism.md | 34 +++++++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 9 deletions(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 1bf6779a9..411bd76cf 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -201,22 +201,38 @@ gives us the path to an output of it.
 # Drawbacks
 [drawbacks]: #drawbacks
 
-The main drawback is that these stub expressions are *only* "pure" derivations --- placeholder strings (with the proper string context) and not attrsets with all the niceties we are used to getting from `mkDerivation`.
-This is true even when the deferred evaluation in fact *does* use `mkDerivation` and would provide those niceties.
-For other sort of values, we have no choice but wait; that would require a fully incremental / deferral evaluation which is a completely separate feature not an extension of this.
-Concretely, our design means we cannot defer the `pname` `meta` etc. fields: either make do with the bare string `builtins.outputOf` provides, or *statically* add a fake `name` and `meta` etc. that must be manually synced with the deferred eval derivation if it is to match.
+- We add a bit of complexity to Nix.
+- There is currently no way of getting a "derivation object"
+  as you do from `builtins.derivation` with `builtins.outputOf`.
+  There are reasons for why this can't be done, mainly that you
+  don't actually know the attributes the built derivation will have,
+  but it might be an ergonomic issue.
+- This is for many things not an alternative to IFD, since we
+  still can not build derivations and then use them at evaluation
+  time, meaning that you can't have an attribute set whose contents
+  are determined by some build, and then access that attribute set
+  outside of build that dependens on that derivation.
+- We unfortunately expose the `text` `outputHashMode` to users.
+  Preferably this should be removed entirely, in addition to `flat`,
+  and everything should just use `recursive`.
 
 # Alternatives
 [alternatives]: #alternatives
 
- - Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
-
- - Embrace Recursive Nix in its current form.
+- Do what we would do with this RFC with only Recursive Nix.
+  The issue with doing something like `nixBuild` with Recursive Nix,
+  is that the derivations you're building inside the build will not be registered
+  as dependencies. This makes logging essentially useless if an inner build fails.
+  NB: This is **not** a replacement for Recursive Nix. We still need the ability to
+  access the store inside the build for many usages of this RFC's features.
+- Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
 
 # Unresolved questions
 [unresolved]: #unresolved-questions
 
-The exact way the outputs refer to the replacement derivations / their outputs is subject to bikeshedding.
+- The exact way the outputs refer to the replacement derivations / their outputs is subject to bikeshedding.
+- Do we need the new CLI?
+- Can we make `builtins.outputOf` more ergonomic?
 
 # Future work
 [future]: #future-work
@@ -230,4 +246,4 @@ The exact way the outputs refer to the replacement derivations / their outputs i
 
 3. Try to convince upstream tools to use Nix like CMake, Meson, etc. use Ninja.
    Rather than converting Ninja plans, we might convince those tools to have purpose-built Nix backends.
-   Language-specific package mangers that don't use Ninja today might also be modified to "let Nix do that actual building".
+   Language-specific package managers that don't use Ninja today might also be modified to "let Nix do that actual building".

From f853a6f6b5c80c2d3d802ae9f8eb4020b7ab711f Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:56:16 +0000
Subject: [PATCH 50/58] [RFC 0092] Improve alternatives section

---
 rfcs/0092-plan-dynamism.md | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 411bd76cf..658578144 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -34,6 +34,9 @@ Notable improvements it allows:
 - We can fetch Merkle trees by just knowing the hash of the root,
   with Θ(n) derivations for n nodes in the tree.
 
+NB: This is **not** a replacement for Recursive Nix. We still need the ability to
+access the store inside the build for many usages of this RFC's features.
+
 # Motivation
 [motivation]: #motivation
 
@@ -219,12 +222,14 @@ gives us the path to an output of it.
 # Alternatives
 [alternatives]: #alternatives
 
-- Do what we would do with this RFC with only Recursive Nix.
-  The issue with doing something like `nixBuild` with Recursive Nix,
-  is that the derivations you're building inside the build will not be registered
-  as dependencies. This makes logging essentially useless if an inner build fails.
-  NB: This is **not** a replacement for Recursive Nix. We still need the ability to
-  access the store inside the build for many usages of this RFC's features.
+- Restrict ourselves to a subset of what we can do with this RFC,
+  and implement that using only Recursive Nix without making use
+  of this RFC.
+  Notably, we can still run `nix build` at the end of builds. This isn't as great,
+  since 1) the daemon will consider the build doing `nix build`
+  as an active build, 2) it messes with logging, often the log
+  of a failing inner build will not be easily accessible, and
+  3) we can't actually have derivations that build derivations.
 - Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
 
 # Unresolved questions

From 1abaf30a914cf35227336c679472697fa67db0b9 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 20:58:22 +0000
Subject: [PATCH 51/58] [RFC 0092] Fix syntax error

---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 658578144..385d077b1 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -191,7 +191,7 @@ END
 The way you would use a derivation that outputs a derivation to `out` is then
 as such:
 ```nix
-{ pkgs, drv }
+{ pkgs, drv }:
 
 pkgs.runCommand "example" {} ''
   ls ${builtins.outputOf drv.out "out"} > $out

From ef1d9aadf6208279591604e780080787f2ff1396 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Tue, 19 Oct 2021 22:23:33 +0000
Subject: [PATCH 52/58] [RFC 0092] Small change

---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 385d077b1..a71ee7151 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -229,7 +229,7 @@ gives us the path to an output of it.
   since 1) the daemon will consider the build doing `nix build`
   as an active build, 2) it messes with logging, often the log
   of a failing inner build will not be easily accessible, and
-  3) we can't actually have derivations that build derivations.
+  3) we can't actually have derivations that output derivations.
 - Do nothing, and continue to have no good answer for large builds like Linux and Chromium.
 
 # Unresolved questions

From 5115ebb5ee040b9cef18022c741de7e3c972f6b4 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Wed, 20 Oct 2021 07:10:01 +0000
Subject: [PATCH 53/58] [RFC 0092] Remove unnecessary file

---
 rfcs/0000-plan-dynanism-example.nix | 125 ----------------------------
 1 file changed, 125 deletions(-)
 delete mode 100644 rfcs/0000-plan-dynanism-example.nix

diff --git a/rfcs/0000-plan-dynanism-example.nix b/rfcs/0000-plan-dynanism-example.nix
deleted file mode 100644
index e49d94b1f..000000000
--- a/rfcs/0000-plan-dynanism-example.nix
+++ /dev/null
@@ -1,125 +0,0 @@
-# with import ../config.nix;
-with import <nixpkgs> {};
-with pkgs;
-with stdenv;
-
-# A simple content-addressed derivation.
-# The derivation can be arbitrarily modified by passing a different `seed`,
-# but the output will always be the same
-rec {
-  root = mkDerivation {
-    # Name must be identical to the name of "dependent" because the name is
-    # part of the hash scheme
-    name = "text-hashed-root";
-    buildCommand = ''
-      set -x
-      echo "Building a CA derivation"
-      mkdir -p $out
-      echo "Hello World" > $out/hello
-    '';
-    __contentAddressed = true;
-    outputHashMode = "recursive";
-    outputHashAlgo = "sha256";
-  };
-  recursive-test = mkDerivation {
-    name = "dir-of-text-hashed-root";
-    buildCommand = ''
-      echo "Copying the derivation"
-      export NIX_PATH=nixpkgs=${pkgs.path}
-
-      # replace this with assumeDerivation
-      # nix-instantiate is okay, not nix-build
-      # does assumeDerivation enforces the "tail-call recursion"
-
-      # HASH=$(readLockFile {./Go.mod} {./go.sum})
-
-      # builtins.fetchThisForMe
-      # builtins.instantiate
-
-      # Pros
-      # cleaner
-      # no nested builds, only instantiation
-      # easier to stabilize
-
-      # Cons
-      # not same power?
-
-      # 1) recursive-nix
-      # 2) RFC92 can output .drv
-      # 3) "full recursive-nix": complexity
-
-      # ninja2nix, splice Nix into derivations
-
-      THING=$(${pkgs.nix}/bin/nix-instantiate -E '
-         derivation {
-          name = "text-hashed-root";
-          system = "x86_64-linux";
-          builder = "/bin/sh";
-          args = ["-c" "echo hi there ''${(import <nixpkgs>{}).blender} > $out"];
-
-          ## Assert that .drv IS a derivation and _contentAddressed, correct everything
-
-          __contentAddressed = true;
-          outputHashMode = "recursive"; # flat (hash a file), text, recursive (hash a dir)
-          outputHashAlgo = "sha256";
-         }')
-
-      mkdir $out
-      echo $THING
-      cp $THING $out/out1
-      cp $THING $out/out2
-    '';
-    outputs = ["out" ];
-    __contentAddressed = true; outputHashMode = "recursive"; outputHashAlgo = "sha256";
-  };
-  nixSourceGenerator ::: drv -> Nix Source Code
-  nixSourceGenerator = drv: runCommand some_name {} ''
-    export NIX_PATH=nixpkgs=${pkgs.path}
-    cat > $out <<EOF
-      derivation {
-       name = "text-hashed-root";
-       system = "x86_64-linux";
-       builder = "/bin/sh";
-       args = ["-c" "echo hi there ''${(import <nixpkgs>{}).${drv.buildInputs.pname} > $out"];
-       __contentAddressed = true;
-       outputHashMode = "recursive";
-       outputHashAlgo = "sha256";
-      }')
-    EOF
-    '';
-
-
-  dependent2 = mkDerivation { # {{{
-    name = "text-hashed-root.drv";
-    buildCommand = ''
-      echo "Link the derivation"
-      ln -s ${root.drvPath} $out
-    '';
-    # Can we link it instead? It will resolve symlink?
-    __contentAddressed = true;
-    outputHashMode = "text";
-    outputHashAlgo = "sha256";
-  }; # }}}
-
-  # Part two allows this?
-  #splitter-new = builtins.assumeDerivation dependent;
-
-  # this is awkward, needs the right name
-  splitter = mkDerivation {
-    name = "text-hashed-root.drv";
-    buildCommand = ''
-      cp ${recursive-test}/out2 $out
-    '';
-    __contentAddressed = true; outputHashMode = "text"; outputHashAlgo = "sha256";
-    # BUG outputHashMode is not used?
-    # contentAddressed T/F is not checked?
-  };
-  wrapper = mkDerivation {
-    name = "put-it-all-together";
-    buildCommand = ''
-      echo "Copying the output of the dynamic derivation"
-      cp -r ${builtins.outputOf splitter.out "out"} $out
-    '';
-    __contentAddressed = true; outputHashMode = "recursive"; outputHashAlgo = "sha256";
-  };
-}

From fdbf7784ac5ded47d76425d39dd9a3b2bf6d2d7d Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Wed, 20 Oct 2021 11:09:36 +0000
Subject: [PATCH 54/58] [RFC 0092] Add comment about IFD

---
 rfcs/0092-plan-dynamism.md | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index a71ee7151..8125fe4fc 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -33,6 +33,10 @@ Notable improvements it allows:
   languages (forget crate-level!).
 - We can fetch Merkle trees by just knowing the hash of the root,
   with Θ(n) derivations for n nodes in the tree.
+- It is a better way of evaluating Nix code inside a build compared
+  to standard Recursive Nix, and can serve as an alternative to **i**mport-**f**rom-**d**erivation
+  in many cases. (IFD is where you import the output of a derivation into
+  the "evaluation stage", e.g. `import drv` or `builtins.readFile drv`).
 
 NB: This is **not** a replacement for Recursive Nix. We still need the ability to
 access the store inside the build for many usages of this RFC's features.

From 2388cbe1de2ffcc1d4eb0f5df3806550c7b5bdc2 Mon Sep 17 00:00:00 2001
From: Las Safin <me@las.rs>
Date: Wed, 20 Oct 2021 14:14:37 +0000
Subject: [PATCH 55/58] [RFC 0092] Fix typo

---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 8125fe4fc..20e970c30 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -12,7 +12,7 @@ related-issues: https://github.com/NixOS/nix/pull/4628 https://github.com/NixOS/
 [summary]: #summary
 
 We introduce three fundamental new features:
-- The ability to have derivations which output store path end in `.drv`
+- The ability to have derivations which output store paths end in `.drv`
   (e.g. `$out` is /nix/store/something.drv).
 - The ability for a derivation to depend on the output of a derivation,
   that isn't yet built but has to be built by another derivation.

From 404ad6d2da4461375132f957925295c9b6b5769e Mon Sep 17 00:00:00 2001
From: Eelco Dolstra <edolstra@gmail.com>
Date: Wed, 3 Nov 2021 15:28:29 +0100
Subject: [PATCH 56/58] Update rfcs/0092-plan-dynamism.md
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 20e970c30..918b2dc7d 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -3,7 +3,7 @@ feature: plan-dynamism
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
 co-authors: Las Safin (@L-as)
-shepherd-team: @tomberek, @ldesgoui, @gytis-ivaskevicius, @L-as
+shepherd-team: @tomberek, @ldesgoui, @gytis-ivaskevicius, @edolstra 
 shepherd-leader: @tomberek
 related-issues: https://github.com/NixOS/nix/pull/4628 https://github.com/NixOS/nix/pull/5364 https://github.com/NixOS/nix/pull/4543 https://github.com/NixOS/nix/pull/3959
 ---

From a906a7c8fc75bda840a0948ab5e5cbcbb67f0da1 Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Wed, 8 Dec 2021 11:44:58 -0500
Subject: [PATCH 57/58] plan-dynamism-experiment: Make clear is experimental

---
 rfcs/0092-plan-dynamism.md | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 918b2dc7d..57ed33085 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -1,5 +1,5 @@
 ---
-feature: plan-dynamism
+feature: plan-dynamism-experiment
 start-date: 2019-02-01
 author: John Ericson (@Ericson2314)
 co-authors: Las Safin (@L-as)
@@ -11,7 +11,7 @@ related-issues: https://github.com/NixOS/nix/pull/4628 https://github.com/NixOS/
 # Summary
 [summary]: #summary
 
-We introduce three fundamental new features:
+Guarded under an experimental feature, introduce three fundamental new features:
 - The ability to have derivations which output store paths end in `.drv`
   (e.g. `$out` is /nix/store/something.drv).
 - The ability for a derivation to depend on the output of a derivation,
@@ -97,7 +97,11 @@ Until then, down with the artificial restrictions!
 # Detailed design
 [design]: #detailed-design
 
-We can break this down nicely into steps.
+Really, this RFC is just proposing that we create the expirmental feature.
+All details are subject to change.
+But so we aren't just proposing an arbitrary experiment, with nothing concrete to judge, we include here the initial design.
+
+We can break the initial experimental feature down nicely into steps.
 
 *This is implemented in https://github.com/NixOS/nix/pull/4628.*
 
@@ -256,3 +260,7 @@ gives us the path to an output of it.
 3. Try to convince upstream tools to use Nix like CMake, Meson, etc. use Ninja.
    Rather than converting Ninja plans, we might convince those tools to have purpose-built Nix backends.
    Language-specific package managers that don't use Ninja today might also be modified to "let Nix do that actual building".
+
+4. Another RFC when we finalize the feature and propose its stabilization.
+   This is a bit speculative, as we haven't pinned down an official experimental feature process/lifecycle.
+   But we include it here to reiterate this RFC is *not* mandating the final design.

From 4d579ed779b1845b2d84ba210d7ec5165ae1d9cf Mon Sep 17 00:00:00 2001
From: John Ericson <John.Ericson@Obsidian.Systems>
Date: Wed, 8 Dec 2021 11:46:54 -0500
Subject: [PATCH 58/58] plan-dynamism-experiment: Fix typo

Thanks @L-as
---
 rfcs/0092-plan-dynamism.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rfcs/0092-plan-dynamism.md b/rfcs/0092-plan-dynamism.md
index 57ed33085..5df788eed 100644
--- a/rfcs/0092-plan-dynamism.md
+++ b/rfcs/0092-plan-dynamism.md
@@ -97,7 +97,7 @@ Until then, down with the artificial restrictions!
 # Detailed design
 [design]: #detailed-design
 
-Really, this RFC is just proposing that we create the expirmental feature.
+Really, this RFC is just proposing that we create the experimental feature.
 All details are subject to change.
 But so we aren't just proposing an arbitrary experiment, with nothing concrete to judge, we include here the initial design.