Compiling cargo binaries from build script deadlocks

[farazshaikh]

Problem
deadlock with .cargo-lock file lock when using build.rs (build script) with cargo crate/

usecase
A workspace with 2 binary packages Pa, Pb.

Pb wants to compile Pa as a binary dependency. Both Pa and Pb are rust packages.

Currently, IMO this can only be done using a build script for Pb which compiles Pa as a binary. I am using the cargo crate to programmatically invoke the compilation of Pa, This compilation blocks on locking TARGET_DIR/.cargo.lock file. The file is locked by the parent cargo process.

(attaching link to sample workspace for repro)

Steps

Workspace layout
https://drive.google.com/file/d/19Ab4TQcy0pl90o8keN42h8Y-e05dCiSP/view?usp=sharing
▶ tree
.
├── Cargo.lock
├── Cargo.toml
├── sub1
│   ├── Cargo.toml
│   └── src
│   └── main.rs
└── sub2
├── build.rs << Compiles sub1 using the cargo crate.
├── Cargo.toml
├── op.txt
└── src
└── main.rs

Possible Solution(s)
I cannot think of a reason why .cargo.lock has to lock during the invocation of the build script.
Maybe this can be relaxed.

Notes

Output of cargo version:
cargo --version
cargo 1.47.0 (f3c7e06 2020-08-28)

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal

Workaround

1. build a release profile of binary dependency, when compiling dev and vice-versa.
2. move the functionality out of build.rs and have the compilation done in the test itself.

Code snippet

.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/cargo-0.49.0/src/cargo/core/compiler/layout.rs:162
//
// For now we don't do any more finer-grained locking on the artifact
// directory, so just lock the entire thing for the duration of this
// compile.
// let lock = dest.open_rw(".cargo-lock", ws.config(),
// "build directory")?;`

/// build.rs from the workspace. (gdrivelink )
use cargo::core::compiler::CompileMode;
use cargo::ops::{compile, CompileOptions};
use cargo::util::Config;

fn main() {
    let mut cargo_manifest = std::env::var("CARGO_MANIFEST_DIR").expect(
        "Can be only
        invoked from cargo",
    );

    cargo_manifest.push_str("/Cargo.toml");

    let config = cargo::util::config::Config::default().unwrap();

    let ws = cargo::core::Workspace::new(std::path::Path::new(&cargo_manifest), &config)
        .expect("Cannot create workspace");

    let mut compile_options = CompileOptions::new(&config, CompileMode::Build).unwrap();

    compile_options.spec = cargo::ops::Packages::Packages(vec!["sub1".to_string()]);
    compile_options.build_config.requested_profile =
        cargo::util::interning::InternedString::new("dev");
    println!(
        "Requested Profile {:?}",
        compile_options.build_config.requested_profile
    );

    // hangs here
    let result = compile(&ws, &compile_options).unwrap();

    for x in result.binaries {
        println!("{:?}", x.1);
    }

}

[tv42]

I think this is the same issue as #6412