Building rust lib with wasm-pack is not reproducible

[iFreilicht]

Describe the bug
Trying to build my rust lib with wasm-pack inside a pure nix shell, I get the following error, which seems to be caused by wasm-pack trying to use my systems glibc (which is severly outdated) instead of the one installed inside the nix shell.

To Reproduce
Steps to reproduce the behavior:

1. Clone my repo
2. Enter the pure nix shell as defined by shell.nix
3. Try to compile using wasm-pack

In more detail:

$ git clone [email protected]:iFreilicht/iris-rs.git # or https://github.com/iFreilicht/iris-rs.git
$ cd iris-rs
$ nix-shell --pure
$ cd iris-hub
$ ldd --version
ldd (GNU libc) 2.32
# [...]
$ wasm-pack build
[INFO]: Checking for the Wasm target...
[INFO]: Compiling to Wasm...
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/[USERNAME]/repos/iris-rs/iris-hub/Cargo.toml
workspace: /home/[USERNAME]/repos/iris-rs/Cargo.toml
   Compiling proc-macro-error v1.0.4
   Compiling serde v1.0.121
   Compiling wasm-bindgen v0.2.69
   Compiling palette v0.5.0
error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/[USERNAME]/repos/iris-rs/target/release/deps/libwasm_bindgen_macro-3a22ad8fd558cc68.so)
  --> /home/felix/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/wasm-bindgen-0.2.69/src/lib.rs:53:13
   |
53 |     pub use wasm_bindgen_macro::__wasm_bindgen_class_marker;
   |             ^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

# [more errors like this, all with the same error message]

error: aborting due to previous error

error: build failed
Error: Compiling your crate to WebAssembly failed
Caused by: failed to execute `cargo build`: exited with exit code: 101
  full command: "cargo" "build" "--lib" "--release" "--target" "wasm32-unknown-unknown"

Expected behavior
I would expect wasm-pack to successfully build the library and not be dependent on the glibc version installed on my OS.

Additional context
This build also fails when I run it outside the machine, with the exact same errors. So

Notify maintainers

@dhl

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-123-generic, Ubuntu, 18.04.5 LTS (Bionic Beaver)`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.10`
 - channels(ga82vac): `"nixpkgs-21.03pre268853.d9c6f13e13f"`
 - nixpkgs: `/home/felix/.nix-defexpr/channels/nixpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[iFreilicht]

I've looked into this issue further and it seems that neither wasm-pack itself isn't responsible for this. LDD shows that all dynamic libraries are linked to the ones installed inside /nix/store:

$ ldd /nix/store/az7jkjqz9mp7pcxpax7x7q00364x62qh-wasm-pack-0.9.1/bin/wasm-pack
	linux-vdso.so.1 (0x00007fff33fa5000)
	libssl.so.47 => /nix/store/m0b1iw8dpqsf6h6x7jbhrjrpgi9idx11-libressl-3.0.2/lib/libssl.so.47 (0x00007f81f78b9000)
	libcrypto.so.45 => /nix/store/m0b1iw8dpqsf6h6x7jbhrjrpgi9idx11-libressl-3.0.2/lib/libcrypto.so.45 (0x00007f81f76b7000)
	libc.so.6 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/libc.so.6 (0x00007f81f74f6000)
	/nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/ld-linux-x86-64.so.2 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib64/ld-linux-x86-64.so.2 (0x00007f81f8033000)
	libgcc_s.so.1 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/libgcc_s.so.1 (0x00007f81f74dc000)
	libpthread.so.0 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/libpthread.so.0 (0x00007f81f74bb000)
	libm.so.6 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/libm.so.6 (0x00007f81f7378000)
	libdl.so.2 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/libdl.so.2 (0x00007f81f7371000)
	librt.so.1 => /nix/store/gafigwfaimlziam6qhw1m8dz4h952g1n-glibc-2.32-35/lib/librt.so.1 (0x00007f81f7366000)

I tried to do the same for cargo and rustc, which are installed as part of the rustup package, but they are installed and weird way where all rust tools are just symlinks of rustup, which seems to be a monolithic binary similarly to how BusyBox works. This binary is not dynamically linked, though:

$ ls -l /nix/store/3zbjid3nhzai4y7phag1r0wacim4zvq7-rustup-1.23.1/bin
total 48
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 cargo -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 cargo-clippy -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 cargo-fmt -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 cargo-miri -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 clippy-driver -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 rls -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 rust-gdb -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 rust-lldb -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 rustc -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 rustdoc -> rustup
lrwxrwxrwx 1 felix felix   6 Jan  1  1970 rustfmt -> rustup
-r-xr-xr-x 1 felix felix 295 Jan  1  1970 rustup
$ ldd /nix/store/3zbjid3nhzai4y7phag1r0wacim4zvq7-rustup-1.23.1/bin/rustup
	not a dynamic executable

So there's nothing I can really investigate there, but I have a feeling this is where the problem is. I also found this SE thread about using patchelf which seems to touch on a similar problem.

However, I don't know which binary actually uses the wrong libc, so I can patchelf it to see if that makes a difference.

[iFreilicht]

It also seems like rustc will override the linker for wasm, see #89426.

[iFreilicht]

Ok, I found a workaround for this, but maybe there's a better solution. From what I understand, Rust builds with nix aren't reproducible at all, unless you're using an overlay that specifically re-packages Rust and Cargo and handles the toolchain management. There's two options here: the mozilla overlay and the oxalica overlay. The latter is specifically geared to be 100% reproducible, while the former actually pulls package hashes on-demand as I understand it, meaning you need a network connection to build.

So for my setup, I went with the latter. This is my minimal shell.nix that works perfectly for building a crate with wasm-pack:

# Use nixpkgs with oxalica rust-bin overlay
let
  rust_overlay = import (builtins.fetchTarball
    "https://github.com/oxalica/rust-overlay/archive/master.tar.gz");
  nixpkgs = import <nixpkgs> { overlays = [ rust_overlay ]; };
  rust_channel = nixpkgs.rust-bin.fromRustupToolchainFile ./rust-toolchain;
in
# Avoid typing `nixpkgs.` before each package name
with nixpkgs;

# Define the shell
pkgs.mkShell {
  nativeBuildInputs = [
    rust_channel # Full rust from overlay, includes cargo
    nodePackages.npm # For all node packages
    wasm-pack # Compiling to WASM and packing with web-stuff
  ];
}

So this is great, but I wonder if maybe something like this overlay should be the default way to install rust tooling.

[stale]

I marked this as stale due to inactivity. → More info

[sporksmith]

Came here to report basically the same issue.

Using strace I was able to figure out that while the rustup package installs a rustc in the nix store, that's just a wrapper that invokes a rustc outside the store - in my case inside ~/.rustup. That toolchain is patched to link against nix libraries, but since it is not itself versioned in the nix store, they're not necessarily the right versions. In my case the version in ~/.rustup was hard-linked against glibc 2.33, which was causing problems from a nix shell that was using glibc 2.35.

I was able to work around the issue by setting the RUSTUP_HOME directory to a directory inside my project (i.e. next to my shell.nix), so that it wouldn't interfere or be interfered with by toolchains on the system or other projects. It's not a great workaround, though.

[sporksmith]

@iFreilicht maybe it would help to change the name of this issue? I happened to also run into it while trying to do something with wasm-pack, but I agree that it looks like a general issue with the rustup package.

[sporksmith]

Demo:

Initial setup of what's expected to be a pure shell environment:

$ cd `mktemp -d`
$ cat <<EOF > shell.nix
with (import (fetchTarball https://github.com/NixOS/nixpkgs/archive/8a7f50dee4ca6bb8a788c477b6253b762d9b7831.tar.gz) {});
mkShell {
  buildInputs = [
    nix
    rustup
    strace
  ];
}
EOF
$ echo 1.65.0 > rust-toolchain
$ nix-shell --pure

We use strace to examine what actually happens when we run rustc:

$ strace -e execve -f rustc --version > /dev/null
execve("/nix/store/0nn2s2i80jxlvfv2hbknpk9hkll98vlh-rustup-1.25.1/bin/rustc", ["rustc", "--version"], 0x7fffb7400f50 /* 77 vars */) = 0
execve("/nix/store/0nn2s2i80jxlvfv2hbknpk9hkll98vlh-rustup-1.25.1/bin/.rustup-wrapped", ["rustc", "--version"], 0x21e92a0 /* 78 vars */) = 0
execve("/home/jnewsome/.rustup/toolchains/1.65.0-x86_64-unknown-linux-gnu/bin/rustc", ["/home/jnewsome/.rustup/toolchain"..., "--version"], 0x55ef4eb3d710 /* 82 vars */) = 0
+++ exited with 0 +++

In the last execve we see that it ultimately executes a rustc in my home directory with the other per-user rustup toolchains.

In my case, this happens to be one that was installed from an older nix environment; it is hard-linked against a glibc 2.33 in the nix store:

$ ldd /home/jnewsome/.rustup/toolchains/1.65.0-x86_64-unknown-linux-gnu/bin/rustc | grep -E 'glibc.*=>'
	/nix/store/ff88p8pnhdmf8bflzbxldys21djw9dp0-glibc-2.33-56/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007f62bc544000)

This is surprising and problematic since the glibc in the current --pure nix shell is 2.35:

$ nix search nixpkgs#glibc
* legacyPackages.x86_64-linux.glibc (2.35-163)

If I override RUSTUP_HOME to force a fresh toolchain install, I get one linked against glibc 2.35 as expected.

$ strace -e execve -f -E RUSTUP_HOME=`pwd`/.rustup rustc --version > /dev/null
...
[pid 3786554] execve("/tmp/tmp.UHB2CtQKuD/.rustup/toolchains/1.65.0-x86_64-unknown-linux-gnu/bin/rustc", ["/tmp/tmp.UHB2CtQKuD/.rustup/tool"..., "--version"], 0x56404543f6c0 /* 83 vars */ <unfinished ...>
...
$ ldd /tmp/tmp.UHB2CtQKuD/.rustup/toolchains/1.65.0-x86_64-unknown-linux-gnu/bin/rustc | grep -E 'glibc.*=>'
	/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007fdf59d71000)

The last bit seems to fix the wasm-pack link errors for me, though I'm a bit unclear why; maybe rustc is hard-coded to link target binaries against the same glibc that it itself was linked against? In any case though, the root of the problem seems to be that the rustup package ends up installing wrappers around the rust toolchain that invoke non-deterministic binaries outside of the nix store.