clangMultiStdenv is unable to link i686 code

[OPNA2608]

Describe the bug
Currently on a local checkout of c83e133, clangMultiStdenv is incapable of linking i686 code. If attempting to do so, ld finds only the x86_64 variants of some of the crucial compiler dependencies. I think this Hydra test is a relevant failure, started failing 2020-07-03.

To Reproduce
Steps to reproduce the behavior:

1. Try to build a 32-bit binary in a derivation via clangMultiStdenv (specifically this stdenv, not just clang_multi)

Linker will be unable to find any of the gcc dependencies (-lgcc, libgcc_s.so.1, crtbeginS.o, crtendS.o, …)

Expected behavior
The code compiles fine without any problems. (https://github.com/darlinghq/darling/tree/master/src/libelfloader/native)

Screenshots
(Code is a derivation I've been working on, shouldn't matter for this bug.)

Additional context
Fetching a shell with clang_multi and running the same command works fine.

Notify maintainers
Don't know who or what broke it, only that it did break at some point.

Metadata

• system: "x86_64-linux"
• host os: Linux 4.14.176, NixOS, 19.09.2518.8260cd5bc65 (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.3
• channels(root): "nixos-19.09.2518.8260cd5bc65"
• channels(bt1cn): "unstable-20.09pre235279.5717d9d2f7c, stable-19.09.1686.69ed29f5f41"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

(I know I should update my system)

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
- clangMultiStdenv
# a list of nixos modules affected by the problem
module:

[kira-bruneau]

I've also encountered this problem, but I switched to gccMultiStdenv and forgot about it.

[OPNA2608]

It's unfortunately not that easy for my use case, I am trying to package something that specifically requires Clang capable of linking both 64-bit and 32-bit code.

I rebased my derivation on a checkout of 1f03d49 (last known-good commit for this test) and the build indeed progresses past the first 32-bit linking. I'll try to figure out the exact commit that broke this when I have abit more time at hand.

[kira-bruneau]

@OPNA2608 Oh sorry, I meant that I forgot to report an issue because I found a workaround. I didn't mean to suggest it as a solution. I would still prefer to use clangMultiStenv so I will spend some time looking into this too.

[OPNA2608]

Been running a git bisect since yesterday, still need to do the final build but it seems like it's homing in on 704daf7 being the breaking commit.

[OPNA2608]

Bisect indeed blames 704daf7. CC @Ericson2314 because you committed it, can you take a closer look at this?

[Ericson2314]

Yes this is my fault and I need to fix it.

[davidtwco]

Has there been any progress on fixing this? It would be great to have clangMultiStdenv working again before the 20.09 release.

[SCOTT-HAMILTON]

I found 4a78b2b to be the first commit introducing the break.

[stale]

I marked this as stale due to inactivity. → More info

[OPNA2608]

Still broken, see https://hydra.nixos.org/build/143310420.

checking whether compiler builds valid C binaries... ok
checking whether compiler builds valid 32bit C binaries... /nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libc.so when searching for -lc
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: skipping incompatible /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: i386:x86-64 architecture of input file `/nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crt1.o' is incompatible with i386 output
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: i386:x86-64 architecture of input file `/nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crti.o' is incompatible with i386 output
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: i386:x86-64 architecture of input file `/nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crtbegin.o' is incompatible with i386 output
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: i386:x86-64 architecture of input file `/nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crtend.o' is incompatible with i386 output
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: i386:x86-64 architecture of input file `/nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crtn.o' is incompatible with i386 output
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: /nix/store/dky9y2lh8q8wk8v55mbh5vjcbvj9klb8-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crt1.o: file class ELFCLASS64 incompatible with ELFCLASS32
/nix/store/77i6h1kjpdww9zzpvkmgyym2mz65yff1-binutils-2.35.1/bin/ld: final link failed: file in wrong format
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
builder for '/nix/store/fzqxsfi21kjqkw0j6s37dl597zln3cfv-cc-multilib-test.drv' failed with exit code 1

[peperunas]

Are there any updates on the issue or what we could do to help speed up the development of a fix?

For reference, I have tried to use clang_multi from the current master branch:

$ clang -m32 /tmp/testcase001.c 
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libc.so when searching for -lc
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: skipping incompatible /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgcc.a when searching for -lgcc
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: i386:x86-64 architecture of input file `/nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crt1.o' is incompatible with i386 output
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: i386:x86-64 architecture of input file `/nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crti.o' is incompatible with i386 output
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: i386:x86-64 architecture of input file `/nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crtbegin.o' is incompatible with i386 output
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: i386:x86-64 architecture of input file `/nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crtend.o' is incompatible with i386 output
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: i386:x86-64 architecture of input file `/nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crtn.o' is incompatible with i386 output
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: /nix/store/vg6pnga4110qjbm8s3sz4i2pwnv968qr-gcc-multi-sysroot/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/crt1.o: file class ELFCLASS64 incompatible with ELFCLASS32
/nix/store/wa59khxfvx6nf1wgdr5bjyyf9rdk23jc-binutils-2.35.2/bin/ld: final link failed: file in wrong format
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)

EDIT: Referencing in @Ericson2314

[peperunas]

The following patch fixes the issue but we are currently working on a refactoring of the multilib wrappers in #166947.

diff --git a/pkgs/development/compilers/llvm/multi.nix b/pkgs/development/compilers/llvm/multi.nix
index 60db622a73a..ecea5d44037 100644
--- a/pkgs/development/compilers/llvm/multi.nix
+++ b/pkgs/development/compilers/llvm/multi.nix
@@ -19,9 +19,9 @@ let
       lib = gcc_multi_sysroot;
     };
   } ''
-    mkdir -p $out/lib/gcc
+    mkdir -p $out/lib{,64}/gcc
 
-    ln -s ${combine gcc64}/lib/gcc/* $out/lib/gcc/
+    ln -s ${combine gcc64}/lib/gcc/* $out/lib64/gcc/
     ln -s ${combine gcc32}/lib/gcc/* $out/lib/gcc/
     # XXX: This shouldn't be needed, clang just doesn't look for "i686-unknown"
     ln -s $out/lib/gcc/i686-unknown-linux-gnu $out/lib/gcc/i686-pc-linux-gnu

[peperunas]

@davidtwco @OPNA2608 @kira-bruneau, can you please test the PR and confirm if it fixes the issue on your end?

Thank you!