error_index_generator stage 0 tests don't work with download-ci-llvm: "error while loading shared libraries: libLLVM-11-rust-1.49.0-nightly.so"

[jyn514]

command did not execute successfully: "/home/joshua/rustc/build/x86_64-unknown-linux-gnu/stage0-tools-bin/error_index_generator" "markdown" "/home/joshua/rustc/build/x86_64-unknown-linux-gnu/test/error-index.md"
expected success, got: exit code: 127

stdout ----

stderr ----
/home/joshua/rustc/build/x86_64-unknown-linux-gnu/stage0-tools-bin/error_index_generator: error while loading shared libraries: libLLVM-11-rust-1.49.0-nightly.so: cannot open shared object file: No such file or directory

failed to run: /home/joshua/rustc/build/bootstrap/debug/bootstrap test --stage 0 src/tools/error_index_generator/

[jyn514]

I have download-ci-llvm = "if-available" in case that makes a difference.

[RalfJung]

Miri also runs into this when doing x.py test --stage 0 src/tools/miri (which used to work)

[jyn514]

From zulip:

I think the fix is indeed that these tools should not be run without being in the sysroot.

[Michael-F-Bryan]

@jyn514 I believe this may have been resolved some time between the issue being created and now. When i tried to reproduce it locally (a4cbb44) the error index tests were able to start and run just fine.

The tests failed with "error[E0658]: const generics are unstable" because some snippets use #![feature(const_generics)] instead of the recently stabilised min_const_generics, but that doesn't have anything to do with being unable to load shared libraries. Similarly I was able to run x.py test --stage 0 src/tools/miri without an issue.

@jyn514 and @RalfJung are you still able to reproduce this?

[RalfJung]

Similarly I was able to run x.py test --stage 0 src/tools/miri without an issue.

I just tried this and it still fails:

     Running `build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/cargo-miri miri setup`
thread 'main' panicked at 'failed to determine underlying rustc version of Miri: UnexpectedVersionFormat', src/tools/miri/cargo-miri/bin.rs:132:38
stack backtrace:
   0: rust_begin_unwind
             at ./library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at ./library/core/src/panicking.rs:92:14
   2: core::option::expect_none_failed
             at ./library/core/src/option.rs:1266:5
   3: core::result::Result<T,E>::expect
             at ./library/core/src/result.rs:929:23
   4: cargo_miri::version_info
             at ./src/tools/miri/cargo-miri/bin.rs:132:5
   5: cargo_miri::setup
             at ./src/tools/miri/cargo-miri/bin.rs:325:16
   6: cargo_miri::phase_cargo_miri
             at ./src/tools/miri/cargo-miri/bin.rs:418:5
   7: cargo_miri::main
             at ./src/tools/miri/cargo-miri/bin.rs:757:25
   8: core::ops::function::FnOnce::call_once
             at ./library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I haven't checked, but I assume the "incorrect version format" is precisely because there's still some error such as "cannot open shared object file" being reported. (That's what used to happen.)

Note that I have download-ci-llvm = true; maybe that's important to reproduce the problem.

[Michael-F-Bryan]

Note that I have download-ci-llvm = true; maybe that's important to reproduce the problem.

It looks like download-ci-llvm has something to do with this because when I switched to download-ci-llvm = true I also got the same "incorrect version format" error.

I haven't checked, but I assume the "incorrect version format" is precisely because there's still some error such as "cannot open shared object file" being reported. (That's what used to happen.)

I printed the output of rustc -vV (here) and apparently rustc doesn't generate any output. I'm not sure if that is relevant.

[/home/michael/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/rustc_version-0.2.3/src/lib.rs:112] str::from_utf8(&out.stdout) = Ok(
    "",
)

[RalfJung]

I printed the output of rustc -vV (here) and apparently rustc doesn't generate any output. I'm not sure if that is relevant.

That's stdout; what does stderr say?

[Michael-F-Bryan]

Ah yep, it's the "unable to load shared libraries" error message:

[/home/michael/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/rustc_version-0.2.3/src/lib.rs:112] str::from_utf8(&out.stderr) = Ok(
    "/home/michael/Documents/forks/rust/build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri: error while loading shared libraries: libLLVM-11-rust-1.51.0-nightly.so: cannot open shared object file: No such file or directory\n",
)

[Michael-F-Bryan]

Inspecting the failing miri binary shows it isn't able to find rustc-driver or std:

$ ldd build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri
	linux-vdso.so.1 (0x00007ffeac5f7000)
	librustc_driver-110e2444bfbc1fba.so => not found
	libstd-6593c2e4e017f5fb.so => not found
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f0f3d890000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f0f3d86e000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f0f3d728000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f0f3d55f000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f0f3dced000)

Should bootstrap be fiddling with LD_LIBRARY_PATH or setting rpath when compiling/running? I was able to make things work by setting LD_LIBRARY_PATH to the stage1 directory.

$ build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri --version
build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri: error while loading shared libraries: librustc_driver-110e2444bfbc1fba.so: cannot open shared object file: No such file or directory

$ export LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage1/lib/
$ build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri --version
rustc 1.51.0-dev

[RalfJung]

The error is about libLLVM-11-rust-1.51.0-nightly.so though, not rustc_driver or std.

I think x.py already sets LD_LIBRARY_PATH.

[RalfJung]

I was able to make things work by setting LD_LIBRARY_PATH to the stage1 directory.

The error you got first is a different one though. Possibly it's not loading the codegen backend. Can you try miri -vV instead of miri --version?

[Michael-F-Bryan]

I think x.py already sets LD_LIBRARY_PATH.

Ah, good to know.

The error is about libLLVM-11-rust-1.51.0-nightly.so though, not rustc_driver or std.

That's odd... Given the dependency chain miri > rustc_driver > libLLVM, on my machine the first link is broken while it looks like the second link is broken for you.

This is what I see when looking at the dependencies for rustc_driver.

$ ldd build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-110e2444bfbc1fba.so
	linux-vdso.so.1 (0x00007fff511fb000)
	libstd-6593c2e4e017f5fb.so => /home/michael/Documents/forks/rust/build/x86_64-unknown-linux-gnu/stage1/lib/../lib/libstd-6593c2e4e017f5fb.so (0x00007fd8e30e2000)
	libLLVM-11-rust-1.51.0-nightly.so => /home/michael/Documents/forks/rust/build/x86_64-unknown-linux-gnu/stage1/lib/../lib/libLLVM-11-rust-1.51.0-nightly.so (0x00007fd8de6a1000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fd8de494000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fd8de47a000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fd8de458000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007fd8de312000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fd8de30a000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007fd8de141000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007fd8e776f000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007fd8de127000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007fd8de11c000)

Can you try miri -vV instead of miri --version?

Looks like it's picking up LLVM just fine 🤷‍♂️

$ export LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage1/lib/  
$ build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri -vV
rustc 1.51.0-dev
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.51.0-dev
LLVM version: 11.0.1

[RalfJung]

That's odd... Given the dependency chain miri > rustc_driver > libLLVM, on my machine the first link is broken while it looks like the second link is broken for you.

Even on your machine it complained about LLVM, not rustc_driver. So far it doesn't look like there is a difference between our machines.

FWIW, this also works here:

$ LD_LIBRARY_PATH=build/x86_64-unknown-linux-gnu/stage1/lib/ build/x86_64-unknown-linux-gnu/stage0-tools-bin/miri -vV
rustc 1.51.0-dev
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.51.0-dev
LLVM version: 11.0.1

Probably x.py sets some other LD_LIBRARY_PATH...

[jyn514]

It sounds like the only change that needs to happen is setting LD_LIBRARY_PATH correctly?

[RalfJung]

Possibly? I don't know anything about our LD_LIBRARY_PATH handling.^^

[Michael-F-Bryan]

Based on these comments on Zulip from @Mark-Simulacrum...

download-LLVM-from-CI could be indirectly responsible by making your local build a shared link to LLVM (not the default on most platforms)

I think the fix is indeed that these tools should not be run without being in the sysroot (or I guess we can try to compile them with a really weird rpath of like ../../stageN/lib but that seems bad

... we should set tool_cmd.current_dir(sysroot) whenever running a tool, probably somewhere around here.

Is that the correct interpretation?

Otherwise we could explicitly add the sysroot to the lib_paths, which is effectively what you meant when you said

It sounds like the only change that needs to happen is setting LD_LIBRARY_PATH correctly?

[Michael-F-Bryan]

So it looks like the issue is that we need some way to tell tools where to find LLVM and the other dynamic libraries it needs.

What I've done is found where we create the std::process::Command which runs the tools, then inside an if build.config.llvm_from_ci { ... } I have tried:

• Setting the command's LD_LIBRARY_PATH (or rather, the result of utils::dylib_path_var()) environment variable to include the sysroot, or
• Just set the command's current working directory to the sysroot

The places I've tried making this modification are:

• Builder::tool_cmd() in bootstrap::tool because it creates the std::process::Command used to run a tool, and
• bootstrap::tool::prepare_tool_cargo() because the the ./x.py test --stage 0 src/tools/miri command seems to do a cargo run --manifest-path src/tools/miri/Cargo.toml and that is where we start constructing its std::process::Command

I believe the fix for this issue should only take a couple lines, but I don't know where to put them because of my unfamiliarity with Rustbuild. I wouldn't be surprised if I was actually doing the right thing, but trying to link to the wrong stage or something. @jyn514 do you have any pointers?

[RalfJung]

Those sound like questions for @Mark-Simulacrum

[Mark-Simulacrum]

tool_cmd sounds like the right place to me at a high level, yes - it'll be hard for me to be sure without rebuilding all of the context, but if you propose a PR that'll make it easier for me to evaluate.

I don't think there's necessarily a "right place", if you find something that works for the stage 0 tools pretty much across the board that would be good enough most likely.

[RalfJung]

... we should set tool_cmd.current_dir(sysroot) whenever running a tool, probably somewhere around here.

I don't think it is a good idea to change the working directory; the tool might be invoked with relative paths that would then not work any more. But I also have no idea how the working directory is even determined currently.

However, I also don't see why that would even help. The RPATH is set to $ORIGIN/../lib, so the working directory dos not matter, does it?

[RalfJung]

@Mark-Simulacrum Moreover, tool_cmd does not even seem involved here? I added some debugging output but it doesn't show. The code that invokes the tool from bootstrap looks like

            let mut cargo = tool::prepare_tool_cargo(
                builder,
                compiler,
                Mode::ToolRustc,
                host,
                "run",
                "src/tools/miri/cargo-miri",
                SourceType::Submodule,
                &[],
            );

[Mark-Simulacrum]

Oh, that seems wrong. IIRC we generally don't build+run in one step, rather we run cargo build and then tool_cmd on the binary? Setting up the environment in prepare_tool_cargo seems feasible, too, though.

[Mark-Simulacrum]

We do already modify this variable in some places, but I don't think in a very principled way (so I'm not terribly surprised it's not working). I'm not sure I have enough time to investigate though, unfortunately.

[RalfJung]

Oh, that seems wrong. IIRC we generally don't build+run in one step, rather we run cargo build and then tool_cmd on the binary? Setting up the environment in prepare_tool_cargo seems feasible, too, though.

We later want to do cargo test in the Miri crate and that has all the same problems. I doubt we want to re-do all the logic of cargo test. So given that cargo test has to work, why is using cargo run wrong?

Also, this does not build+run in one step. The build already happened earlier. But we use cargo run for running since it does a lot of path setup for us.

We do already modify this variable in some places, but I don't think in a very principled way (so I'm not terribly surprised it's not working). I'm not sure I have enough time to investigate though, unfortunately.

Which variable?

---

I did some more experiments and I can confirm that adding build/x86_64-unknown-linux-gnu/ci-llvm/lib to the LD_LIBRARY_PATH helps. Now the cargo run part works. However, cargo test still fails, because something is overwriting our LD_LIBRARY_PATH with something else and thereby removes the LLVM dir from it...

[RalfJung]

Okay, I found who's overwriting LD_LIBRARY_PATH, it was add_rustc_lib_path. Here's my hackfix that makes ./x.py test --stage 0 src/tools/miri work with the downloaded LLVM:

diff --git a/src/bootstrap/test.rs b/src/bootstrap/test.rs
index 86d940cd733..96a94d43428 100644
--- a/src/bootstrap/test.rs
+++ b/src/bootstrap/test.rs
@@ -393,6 +392,12 @@ fn run(self, builder: &Builder<'_>) {
             );
             cargo.arg("--").arg("miri").arg("setup");
 
+            // Make sure we can access all the dynamic libraries.
+            let mut dylib_path = dylib_path();
+            let ci_llvm_lib = builder.out.join(&*host.triple).join("ci-llvm/lib");
+            dylib_path.insert(0, ci_llvm_lib);
+            cargo.env(dylib_path_var(), env::join_paths(&dylib_path).unwrap());
+            eprintln!("{} = {:?}", dylib_path_var(), env::join_paths(&dylib_path).unwrap());
             // Tell `cargo miri setup` where to find the sources.
             cargo.env("XARGO_RUST_SRC", builder.src.join("library"));
             // Tell it where to find Miri.
@@ -442,6 +447,7 @@ fn run(self, builder: &Builder<'_>) {
                 SourceType::Submodule,
                 &[],
             );
+            cargo.env(dylib_path_var(), env::join_paths(&dylib_path).unwrap());
 
             // miri tests need to know about the stage sysroot
             cargo.env("MIRI_SYSROOT", miri_sysroot);
@@ -450,7 +456,7 @@ fn run(self, builder: &Builder<'_>) {
 
             cargo.arg("--").args(builder.config.cmd.test_args());
 
-            cargo.add_rustc_lib_path(builder, compiler);
+            //cargo.add_rustc_lib_path(builder, compiler);
 
             if !try_run(builder, &mut cargo.into()) {
                 return;

I hope at some point someone can take this and turn it into something proper. ;)

[RalfJung]

It is also worth noting that this is a regression -- testing Miri on stage 0 used to work just fine some time in summer/fall last year. I have no idea which rustbuild change broke things, though.

EDIT: ah I remember, it's probably the pre-built LLVM download.

[RalfJung]

So how can we make progress on this? I produced a working diff above, but it's a crude hack and only fixes the problem for Miri. I don't know nearly enough about rustbuild to figure out the proper fix for this though -- there are lots of moving pieces here, so this is unlikely to be fixed until someone from the rustbuild team can take a look. That would be much appreciated. :)

[jyn514]

@RalfJung there is not really a rustbuild team - there's me and Mark-Simulacrum, and ehuss has a fair bit of knowledge I think. I'm on break for a while and Simulacrum is always busy.

[RalfJung]

I mean, I guess that makes you two (or three) the rustbuild team. At least this is exactly how I became part of the Miri and const-eval teams. ;)

[RalfJung]

Regarding the problem at hand, I could probably try patching add_rustc_lib_path so that it sets up dylib_path_var to include ci-llvm/lib (hopefully I can check to only do this when we are actually using CI LLVM). But would that make any sense?

[jyn514]

@RalfJung sure, that seems reasonable.

[RalfJung]

Reopening as the issue persists for error_index_generator specifically, which has some custom logic for dylib paths that still needs adjusting:

rust/src/bootstrap/tool.rs

Lines 396 to 400 in 903e369

	add_dylib_path(
	vec![
	PathBuf::from(&builder.sysroot_libdir(compiler, compiler.host)),
	PathBuf::from(builder.rustc_libdir(compiler)),
	],

[jyn514]

@RalfJung the error index is already tracked by #80096