error while loading shared libraries: librustc_driver-fb0866b1cd913c20.so: cannot open shared object file: No such file or directory

[DeepeshvRajan]

In Yocto environment , we could see a test failure after upgrading to rust_1.82 . (May be because of the feature (#122362) )

After updating to rust 1.82 , we ran "python3 src/bootstrap/bootstrap.py test" . Then we came across the below failure . Can anyone point me out do we need to make any changes in poky sources to adapt this feature .

Compiling rustc_driver_impl v0.0.0 (rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/compiler/rustc_driver_impl)
  Compiling rustc_driver v0.0.0 (rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/compiler/rustc_driver)
   Finished `release` profile [optimized] target(s) in 5m 29s
    Running unittests src/main.rs (build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_main-92223b15c9f2d827)
uploaded "rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_main-92223b15c9f2d827", waiting for result
/tmp/work/test4056/rustc_main-92223b15c9f2d827: error while loading shared libraries: librustc_driver-fb0866b1cd913c20.so: cannot open shared object file: No such file or directory
error: test failed, to rerun pass `-p rustc-main --bin rustc-main`

Caused by:
 process didn't exit successfully: `rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client run 0 rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_main-92223b15c9f2d827 -Z unstable-options --format json` (exit status: 127)
note: test exited abnormally; to see the full output pass --nocapture to the harness.
    Running unittests src/lib.rs (build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_abi-ec634b1002c52b95)
uploaded "rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_abi-ec634b1002c52b95", waiting for result

running 1 tests
test tests::align_constants ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.99ms

According to the above given error log, it seems like it cannot able to find the librustc_driver.so but I can able to find librustc_driver.so file in the following build path where the rustc_main-92223b15c9f2d827 is also present :

rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps

I found the following files related to librustc_driver in that directory:

librustc_driver-fb0866b1cd913c20.so
librustc_driver_impl-41ab001b5a5d6c41.rlib
librustc_driver_impl-41ab001b5a5d6c41.rmeta

However, I am unsure about the below path:

/tmp/work/test4056/rustc_main-92223b15c9f2d827

I'm not certain where this path is coming from, and whether it indicates that the system is trying to find the librustc_driver*.so file in the wrong location.

CC: @bjorn3 @Zoxc

[Noratrieb]

are you removing the rpath? the rustc binary is supposed to have an rpath of $ORIGIN/../lib so it can find the library correctly. there have been other reports from distros that removed the rpath and got this issue

[jyn514]

that "uploading" comment is suspicious. do you have some external test tool running this in a docker container or something like that? if so, the files there won't match the files in your build directory unless you upload all of them. please show a listing of that /tmp/work/test4056 directory and verify that it contains librustc_driver.so.

[DeepeshvRajan]

that "uploading" comment is suspicious. do you have some external test tool running this in a docker container or something like that?

Yeah, in our Yocto environment, we're performing cross-compilation and testing by transferring the image and other artifacts to QEMU to run the tests. During this process, the rustc-main binaries are successfully uploaded to QEMU, but we are uncertain whether the librustc_driver.so file is also being uploaded. When attempting to run rustc-main inside QEMU, it is unable to locate the librustc_driver.so file in the expected directory where the rustc-main binaries were uploaded.

if so, the files there won't match the files in your build directory unless you upload all of them. please show a listing of that /tmp/work/test4056 directory and verify that it contains librustc_driver.so.

Since the files are uploaded for testing via the remote-test-client to run on the QEMU target using the remote-test-server, we are unable to list the files in the /tmp/work/test4056 directory present inside QEMU to verify if the librustc_driver.so file is actually present. However, it can be found in the build directory, located at where exactly rustc-main binaries also present as follows:

rust_1.82/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps.

PS: While adding the comment I mistakenly closed the issue . Can someone please reopen this issue?

[jyn514]

i am not sure what to tell you. it doesn't matter if that file is present on the host if it's not present in the QEMU image.

note that target/release/deps is not the final path to the binary, bootstrap does various file copying to set up the sysroot. like @Noratrieb said, librustc_driver.so needs to be in the right relative path so it can be loaded with -rpath.

[DeepeshvRajan]

Please find the detailed analysis of the reported issue.

1. Testing above reported issue on the native rust build.

We noticed that the rustc_main-92223b15c9f2d827 execution works fine in the Rust native build and test environment. Because, as indicated in the logs, it sets the LD_LIBRARY_PATH to include the location of the librustc_driver-fb0866b1cd913c20.so file.

Running CARGO=rust_up/build1/build/x86_64-unknown-linux-gnu/stage0/bin/cargo CARGO_MANIFEST_DIR=/native_rust-lang/compiler/rustc CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=rustc-main CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.0.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=0 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/psm-fb7f76d9ecc79812/out:/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/rustc_llvm-02021b9c3da99d7e/out:/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps:/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release:/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib:/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib' OUT_DIR=/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/rustc-main-3e9e1c14c3f0b5a0/out /native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-78364219a439ba0d --quiet -Z unstable-options --format json

As shown in the log above, the librustc_driver-fb0866b1cd913c20.so file is located at the following path: "/native_rust-lang/build1/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps".

2. Details of above reported issue on the target (using qemu image)

When running the test for yocto target "x86_64-poky-linux-gnu" using the remote-test-server and remote-test-client method, the tests are uploaded to the QEMU image and executed on the image. The cause for failure is, the rustc_main-92223b15c9f2d827 binary attempts to locate the librustc_driver-fb0866b1cd913c20.so file in the "/tmp/work" directory of the QEMU image (target image). Since the librustc_driver-fb0866b1cd913c20.so file is not found in the /tmp/work of qemu.

Command throwing error:
Running CARGO=/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/recipe-sysroot-native/usr/bin/cargo CARGO_MANIFEST_DIR=/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/compiler/rustc CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=rustc-main CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.0.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=0 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/build/psm-3d0e360728375387/out:/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/build/rustc_llvm-17b300843e67d221/out:/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps:/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release:/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-poky-linux-gnu/lib:/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-poky-linux-gnu/lib' OUT_DIR=/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/build/rustc-main-5f53942dc1bae8eb/out /rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client run 0 /rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_main-92223b15c9f2d827 -Z unstable-options --format json

uploaded "/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_main-92223b15c9f2d827", waiting for result
/tmp/work/test4056/rustc_main-92223b15c9f2d827: error while loading shared libraries: librustc_driver-fb0866b1cd913c20.so: cannot open shared object file: No such file or directory
error: test failed, to rerun pass `-p rustc-main --bin rustc-main`

We observed that librustc_driver-fb0866b1cd913c20.so from the path rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/ is linked to rustc_main-92223b15c9f2d827, but it is not copied to the image before executing the test. This library is only present in the mentioned path and is not found in the standard dylib path. We also attempted to copy librustc_driver-fb0866b1cd913c20.so to the standard dylib path, but when the test is triggered, the libraries in the standard dylib path are recompiled, causing the librustc_driver-fb0866b1cd913c20.so to be removed.

3. Verified test executes successfully by copying librustc_driver-fb0866b1cd913c20.so to qemu image manually.

We pushed the librustc_driver-fb0866b1cd913c20.so to qemu using below command,

"/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "push" "/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/librustc_driver-fb0866b1cd913c20.so"

The "librustc_driver-fb0866b1cd913c20.so" got copied to /tmp/work using above command.

Executed the test command on qemu image from host machine by setting other necessary environment variables.

/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client run 0 /rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-poky-linux-gnu/release/deps/rustc_main-92223b15c9f2d827 -Z unstable-options --format json

The execution got passed without any errors.

4. Behaviour in rust-1.81.0 sources.

In rust-1.81.0 sources testing the rustc_main-92223b15c9f2d827 binary depended on libstd-cf5d1b1b7d767bcd.so, which got pushed into the /tmp/work/ directory of the qemu by the time client server communication started. As shown below by the remote-test-client. Because of that ./rustc_main-92223b15c9f2d827 ran successfully.

"/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "push" "/rust82-poky/testnew-st/tmp/work/core2-64-poky-linux/rust/1.82.0/rustc-1.82.0-src/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-poky-linux-gnu/lib/libstd-dc8753f602037ec0.so"

[thesamesam]

See also #133629.