[RFC/WIP] multiple-outputs: Change the order of outputs to have binaries in the primary output

[dezgeg]

This series reorders the outputs of each multiple-output splitted package in nixpkgs to have the output that is installed into the user environment be the first one (though I made a single exception: glibc has its libraries first to keep usages like ${stdenv.cc.libc}/lib/ld-linux-x86-64.so.2 working). Previously the dev output was typically the primary one, although there were some exceptions like the kernel package. To accommodate this, mkDerivation was changed to use the dev output by default in packages passed to buildInputs and friends.

The rationale for doing this is to avoid breakage to users' scripts and systemd services that reference binaries like ${pkgs.perl}/bin/perl (e.g. https://github.com/NixOS/nixos-org-configurations/blob/master/delft/common.nix#L62). Currently, those usages will just silently produce a nonsensical path, which IMHO is a significant regression to the pre-multiple-outputs state.

Also this should fix some of our in-tree systemd units which suffer from the same issue and avoid the need for changes like 3e38a71

As a part of doing this, I manually eyeballed all usages of ${pkgs.multiout-package} . I collected this list of packages where the implicit dev is currently incorrectly used in place of the correct output: https://github.com/dezgeg/nixpkgs/commits/bad-multiout-refs. So this PR fixes some of them as a side-effect (though not all, there are other things wrong as well).

Note: I'm still compiling most of this (at least stdenv builds...). It's certainly possible that chooseDevOutputs needs to be used in more places...

cc @edolstra @vcunat @ttuegel @joachifm @lethalman

[vcunat]

Hmm, yes, probably. At least in principle, though I haven't thought through many potential consequences yet. I'm not certain how fast I get to this properly. I suppose I'd first solve issues like #14682, as that seems some easily fixable sloppy programming (of mine).

[cstrahan]

The general idea here looks good to me. The changes to multiple-outputs.sh look good (particularly with respect to the defaulting of $propagaterOutput), and the exception for output order for glibc seems reasonable given how pervasive the current typical use case is. Hopefully this will make things a little easier on newcomers; it could be my own bias from my familiarity with the old status quo, but I think "${foo}/bin/foo" and "${foo.dev}/include" is more intuitive than "${foo.bin}/bin/foo" and "${foo}/include".

👍

[cstrahan]

I forgot to mention, I also approve of having buildInputs in the stdenv default to using the dev output, expecting it to follow the convention of propagating the other requisite outputs.

I'd like to see this get pushed along, assuming no one comes along with a reasonable objection.

[abbradar]

I don't have a strong opinion either way, but to new users this would probably be more consistent with Debian and Red Hat-based distributions' -devel packages.

[joachifm]

Agreed, this sounds like a good idea, so better to just deal with any bugs.

[dezgeg]

I built closures.smallContainer.x86_64-linux and the minimal installation CD successfully (way better than I expected...). There are some minor evaluation errors exposed by this series (e.g. GHC using a version of gmp that is not split) that I need to deal with.

[edolstra]

Doesn't this break

buildInputs = [ foo ];

since you would now have to write

buildInputs = [ foo.dev ];

? That was the whole reason for putting dev first.

[dezgeg]

@edolstra: No, that is handled by 090c967

[edolstra]

Okay, I see there is some special-casing for this in mkDerivation. But that seems confusing to me, i.e. foo will mean foo.dev in some contexts but not others.

[vcunat]

Well, the whole point is to try guessing what output was meant when none was specified... obviously the best guess depends on context.

[dezgeg]

Yes, that is pretty much inevitable. Even currently (as in before this PR) environment.systemPackages = [ pkgs.curl ]; doesn't refer to curl.dev.

Also cases like services.httpd.package = pkgs.apacheHttp_2_4; - that probably shouldn't refer to the dev output either.

[vcunat]

The last one is correct, I believe. The users of services.httpd.package should choose the particular output, as they might need different on different places.

[dezgeg]

#14902 seems another victim of silent breakage by accidentally using the dev output.

[vcunat]

I don't think there can be any argument against the commits that make output choice explicit (they make majority of the diff). These shouldn't even change hashes and could go to master directly (before conflicts can arise).

[trishume]

I ran into some issues with this except with libraries defaulting to dev instead of out, which perhaps makes more sense than binaries. Example issue caused by this problem: #15124

[vcunat]

I suppose we can fixup various details after staging this.

I wonder whether to re-consider the default setting of meta.outputsToInstall, but probably not.

[dezgeg]

Right, I would just like a confirmation from @edolstra if he agrees with this or not. Also @domenkozar might have something to say about one more mass-rebuild thing to 16.09 ;)

[domenkozar]

No objections for mass-rebuild if it's done before Thursday and it works (doesn't break too many packages and all tests pass).

[edolstra]

Looks good to me.

[lucabrunox]

Good. Can you please update release notes about this? Also the dev / devEnv change of ruby.

[lucabrunox]

There are some more places where .dev needs to be used I think, like for nss nspr gcc glibc glib readline ncurses gio and all these libraries that sometimes are used in build flags of packages. Searching for "-I" might help.

[dezgeg]

I had already done most of those changes in master (e.g. 4db1657) via some regexps just to be able to spot breakages (e.g. d801e8a) but now that you mention, I indeed managed to miss some, thanks.

Should now be in a condition where I can stage this in the morning after some overnight build testing.

[dezgeg]

Oh, and cc @zimbatm for the Ruby change (ba6d94e). Does this rename sound good, or would some other name be better?

Reason for changing is that the name dev is used in the multiple-outputs stuff and causes problem there.

Also, any documentation references to update? Didn't notice any but the name dev is not very greppable.

[zimbatm]

Looks good. I don't think ruby.dev isn't documented right now and I'm considering dropping it in the future.

[lucabrunox]

Wow, this is on staging? Awesome work!

[vcunat]

It's in master, actually, since 8c4aeb1.

[grahamc]

(and in 16.09!)