wasm-ld should ignore .rlib files in archives

[hoodmane]

As per the rust issue rust-lang/rust#80775, if you try to build an empty rust file:

rustc \ 
    -C target-feature=+mutable-globals \
    -C relocation-model=pic \
    -C link-args='-s SIDE_MODULE' \
    --target wasm32-unknown-emscripten \
    --crate-type cdylib \
    test.rs

the result is the following error:

note: wasm-ld: error: unknown file type: lib.rmeta

Rust libraries contain an index file called lib.rmeta. The linker is expected to ignore them. I can work around this with the following patch to the linker:

https://github.com/pyodide/pyodide/pull/2378/files#diff-c21c39739d2d98d34e3063aceea3d659b11bc10471c403a0199a5bc01587e58b

Emscripten linker patch

if any(arg.endswith(".rlib") for arg in cmd):
  from tempfile import mkstemp
  from pathlib import Path
  tempfiles = []
  new_cmd = []
  try:
    for arg in cmd:
      if arg == "-lc":
        continue
      if not arg.endswith(".rlib"):
        new_cmd.append(arg)
        continue
      fd, temp_path = mkstemp()
      shutil.copy2(arg, temp_path)
      tempfiles.append(Path(temp_path))
      subprocess.call(["ar", "-d", temp_path, "lib.rmeta"])
      subprocess.call(["emranlib", temp_path])
      new_cmd.append(temp_path)
    cmd = new_cmd
    cmd = get_command_with_possible_response_file(cmd)
    check_call(cmd)
  finally:
    for file in tempfiles:
      file.unlink()
else:
  cmd = get_command_with_possible_response_file(cmd)
  check_call(cmd)

xref Pyodide PR pyodide/pyodide#2378

@sbc100

[llvmbot]

@llvm/issue-subscribers-lld-wasm

[sbc100]

I believe the behaviour of wasm-ld here matches other linkers.

Specifically unknown archive members are ignored unless -Wl,--whole-archive is passed. Can you confirm if you are using -Wl,--whole-archive?

Here is an example of the native GNU linker giving the same error:

$ clang -Wl,--whole-archive libtest.a 
/usr/bin/ld: libtest.a: member libtest.a(lib.rmeta) in archive is not an object
clang: error: linker command failed with exit code 1 (use -v to see invocation

[hoodmane]

Both --whole-archive and then later --no-whole-archive are present:

Response file

root:WARNING: Creating response file /tmp/emscripten_w2dsyqtv.rsp with following contents: -o
/src/packages/cryptography/build/cryptography-36.0.2/src/rust/target/wasm32-unknown-emscripten/release/deps/cryptography_rust.wasm
--whole-archive
... tons of files
-L/src/packages/cryptography/build/cryptography-36.0.2/src/rust/target/wasm32-unknown-emscripten/release/deps
-L/src/packages/cryptography/build/cryptography-36.0.2/src/rust/target/release/deps
-L/src/.docker_home/.rustup/toolchains/nightly-2022-05-28-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-emscripten/lib
/src/packages/cryptography/build/cryptography-36.0.2/src/rust/target/wasm32-unknown-emscripten/release/deps/libcompiler_builtins-07685c9166f0770f.rlib
-lc
-L/src/.docker_home/.rustup/toolchains/nightly-2022-05-28-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-emscripten/lib
-L/src/.docker_home/.rustup/toolchains/nightly-2022-05-28-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-emscripten/lib/self-contained
--fatal-warnings
-L/src/emsdk/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic
--no-whole-archive
-mllvm
-combiner-global-alias-analysis=false
-mllvm
-enable-emscripten-cxx-exceptions
-mllvm
-enable-emscripten-sjlj
-mllvm
-disable-lsr
--import-undefined
--import-memory
--strip-debug
--export-all
--no-gc-sections
--experimental-pic
-shared

[hoodmane]

Stripping out the first --whole-archive indeed fixes the problem. So is it a bug that if we pass --whole-archive --no-whole-archive, the --no-whole-archive doesn't win?

[sbc100]

I think that is working as intended then.. you would see the same error from GNU ld or ELF lld.

[hoodmane]

Presumably the point is that --whole-archive applies to all the files and then --no-whole-archive switches it back off but too late?

[sbc100]

--whole-archive operated on any library on the command until the the --no-whole-archive..

[hoodmane]

So this is either a bug in rust, PyO3, or Emscripten, not in wasm-ld. That's good news.

[sbc100]

If the problem is that --whole-archive/--no-whole-archive is being injected by emscripten.. then this is arguably an emscripten bug.

[hoodmane]

I will open a new issue in Emscripten.

[hoodmane]

I feel like github is putting words into my mouth with "closed this as completed"...

[sbc100]

Yes, I think they recently changes the "closed" messages .. but I've need seen how to control them