Way to enable caching across cargo invocations in different directories

[alex]

Describe the problem you are trying to solve

The genesis of this bug report is investigating CI times for the pyca/cryptography project. We're a Python project that uses Rust for some of our extensions (via pyo3). We set up caching of src/rust/target in CI, to no avail.

The problem is that when pip (Python package manager) builds and installs our Python code in CI, it does the build process in an isolated environment (i.e. tempdir). This means that the target/ directory with cacheable artifacts lives somewhere like /tmp/pip-req-build-r9mof20a/src/rust/target/. And naturally this temp dir changes from run to run. It's not clear how to enable CI caching of these artifacts under this constraint.

Our Rust build process contributes substantially to our CI cycle times (something on the order of 1.5 minutes out of a 6.5 minute CI time), so we're motivated to find a solution.

Describe the solution you'd like
One possible solution (though certainly not the only one) would be a way to tell cargo to place all cache artifacts in a specific directory (e.g. env CARGO_CACHE_DIR=$XDG_CACHE_DIR/.cargo/cache), this would let you tell the CI to cache that dir and not worry about where the build dir is.

One might reasonable ask why CARGO_TARGET_DIR is not sufficient for this. The problem is that this also moves the final build artifacts, which would mean the external build system wouldn't know where to find them! Moving only deps/ artifacts would address this.

[alex]

After prodding further, @tomprince suggested to me that maybe setuptools-rust should just be taught about CARGO_TARGET_DIR. At first I thought teaching every build system about CARGO_TARGET_DIR was kind of a pain.

But it turns out setuptools-rust works with CARGO_TARGET_DIR already! It uses cargo metadata to get a bunch of stuff, and that includes target directory, and it passes the env to that subprocess. So in conclusion, getting what I wanted seems to already be possible.

I'll leave this open in case the cargo team finds it interesting/useful though.

[ehuss]

Thanks for the clearly written issue! I think this is a duplicate of #6227, so closing in favor of that.