Deferred derivations

[copumpkin]

This is a generic feature request but I'll use an example to clarify what I want and why:

I'm packaging buck, a fancy build tool from facebook. Among its various features is support for all sorts of build ecosystems, including Go (and Java, etc.)

I have two goals:

1. Give buck a rigid build of Go (and other tools) without having it reach out to the environment at runtime to figure out what Go to use
2. Don't give buck an unnecessarily large runtime closure

I'd basically like buck to be able to "lazily" refer to our Go package, without including it in its closure unless we actually end up using it.

I'm mostly looking for ideas for how to achieve this goal. I realize that we'd need to patch buck to call back into Nix to build/download the appropriate Go package before trying to use it. GC issues would also need to be worked out, of course.

One approach might be to drop a .drv file for Go somewhere and leave it around and have the hook build it, while at the same time giving buck the uninstantiated Nix store path for Go. The callback would build the derivation and make the store path exist, so the existing package would not need to change after build, and would automatically retain Go afterwards. I'm not sure if the "bad reference" before would run afoul of store integrity checks though. I also think this might shift from retaining the runtime closure of Go to retaining the compile-time closure of Go, which doesn't feel great. #620 might be an alternative to storing the .drv file, but might also be a rat's nest.

Any better ideas? Is the whole concept too weird?

[Ericson2314]

I think of it is one build having a weak reference to another. Need whole closure at compile time, but the Go stuff can be chucked after build / not initially downloaded from binary cache until store path is accessed. FUSE filesystem for store could accomplish help implement that.

(As with most proposed features) the intensional store makes this safer, in this case by crying foul if the rebuilt Go package is different than what was used to build buck.

[copumpkin]

I wouldn't go so far as doing FUSE, although that would be neat (call it NixFS). I'm quite fine with adding minor patches to the package in question, for now.

Also, more than a weak reference, I see it as a lazy reference at runtime. Weak would probably be to allow the GC to get rid of it after I've forced it.

[Ericson2314]

That sounds fair, furthermore as a practical measure probably best to ensure binary cache has build of it, rather than relying on rebuild.

I am hoping that someday this and conventional binary substitution would just fall out of a separate storage layer (e.g. IPFS), it's just a matter of subscribing to a .drv->build map.

Also, getting rid of builds once even they are used are good for a cluster with central storage. The local store just holds things needed for boot, and acts as a cache.

[Mathnerd314]

Buck reads /etc/buckconfig, so on NixOS you could have a Buck service as follows:

config.settings = { go.path = "${pkgs.go}/bin/go"; };
...
mkIf config.enable {
  systemPackages = [ pkgs.buck ];
  environment.etc."buckfile" = generate-ini-file config.settings;
}

OSX could have a user-level program similar to NixOS; there is some work on nixuser which could be used.

giving buck the uninstantiated Nix store path for Go

I think this is a horrible idea; buck (the executable program) does not need Go (it's written in Java), and there is no reason to regenerate the .class files on every Go update.

[copumpkin]

@Mathnerd314 this isn't about Buck or a NixOS service; it's about Buck having "go integration". I know Buck is written in Java, but it "knows how to build go programs", and as such knows how to invoke a go compiler. The question then becomes how to find that go compiler. If we wrap Buck to point at a Nix-packaged go, its closure grows. If we don't, it pulls an unpredictable version of go at runtime, or complains that go isn't installed. This is proposing a third option that doesn't significantly grow the closure, but also leads to deterministic runtime behavior even if you tell Buck to build a go project.

[Mathnerd314]

Sure. I'm proposing a fourth option: look for the go compiler in /etc/buckconfig. Which, unlike your option, does not require any changes to Nix. The dependency structure is like so:

It's completely deterministic w.r.t. the NixOS config file, and go can be excluded by modifying the config.

For building Nixpkgs packages that use Buck, a global setting like /etc/buckconfig is not right, and instead we would set BUCK_CONFIG=/nix/store/${buck-config} (after patching Buck to use such an environment variable). But the dependency graph's structure is unchanged.

[copumpkin]

Isn't that just creating an implicit dependency on a piece of global state though, which can vary with time? Nix doesn't know about any of your system on any level. We just happen to have a convention that a NixOS module puts a file in a place, and a package happens to look in that place. Doesn't seem very appealing to me.

[Mathnerd314]

Yes, it's a dependency on some state. It's not implicit, unless you forget to read the Buck manual's page on configuration.

But your solution does not escape this; you just want the state to be stored in Nix, instead of the filesystem. "Dropping a .drv file for Go somewhere" is exactly as stateful as storing a reference to Go in /etc/buckconfig.

Environment variables have the advantage of scoping, but for some reasons everyone has decided to use files instead (http://peterlyons.com/problog/2010/02/environment-variables-considered-harmful).

[copumpkin]

Yes, except it's now tracked in the hash, and I can use a variety of nix tools for telling me all the dependencies that went into the program's execution.

Put this differently: I want NixOS to (e.g.,) automatically write AppArmor profiles for me. In theory, everything but runtime state (/var, databases, etc.) should be statically knowable today. Mutable state is a pain, and I think we have ideas but no implementations. I would not be shoving things that could be resolved at compile time to being resolved at runtime, as that just complicates reasoning and we're aiming to minimize mutable state, not increase it.

[Mathnerd314]

So you want Go to be tracked in the buck package's hash, i.e. recompile buck on every Go update? That is what "resolving at compile time" implies.

Adding a config file does not add any mutable state, it's just a level of indirection. Unless you mean that writing let x=3 in Haskell adds mutable state?

I think this should move to #nixos, having long discussions on GitHub is painful.

[copumpkin]

I can't get on IRC now, but happy to defer discussion to later. In short, my answer to your "recompile on every Go update", would be yes though.

[Mathnerd314]

Alright; I'm almost always on, so just ping me.

[copumpkin]

Connecting this to #709 which is actually pretty closely related. What I basically want to do is serialize a Nix closure (what #709 does in some sense) and have someone evaluate it later, on demand.

[stale]

I marked this as stale due to inactivity. → More info

[stale]

I closed this issue due to inactivity. → More info