--artifact-dir (nee --out-dir) Tracking Issue

[ehuss]

Note: this was renamed from --out-dir to --artifact-dir in #13809

Original issue: #4875
Implementation PR: #5203
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#artifact-dir
Issues: Z-out-dir Nightly: --out-dir

Summary
Adds --artifact-dir=PATH flag to cargo build to specify a directory to place final artifacts.

Tasks:

• cargo build --out-dir #5203
• Rename --out-dir to --artifact-dir #13809
• Remove --out-dir

Unresolved questions

• Multi-target builds (--target <one> --target <two>)
• MSVC examples cannot be exported with out-dir #7493
• Is --out-dir justified or can we find another way of accomplishing it? #6100 — concern about similarity with --target-dir, proliferation of flags
• Duplicate artifact tracking issue. #6313 — concern about name collisions in workspaces or with examples
• What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?
• Confusion with rustc's --out-dir (Rename --out-dir to --artifact-dir #13809)
• Discrepancy or confusion with build.rss OUT_DIR (depends on if build.rs is generating intermediate or final artifacts) (Rename --out-dir to --artifact-dir #13809)

[ultrasaurus]

I find this feature very helpful, since I get output in a directory named like this ./target/debug/build/mylib-79d5eadf36eadf53 and with a build.rs file there are two similarly named directories.

However, it doesn't seem that OUT_DIR environment variable is set to the value of --out-dir when build.rs is executed. Is that intentional? is there some other way to discover the value of --out-dir?

[haze]

+1 on this issue. I have a program that uses a template directory for rust compilation and would like to now have to recompile everything. Yes I could symlink all the folders within target, but it would be nicer to be able to specify the binary output path. I'm not sure why this is such an overlooked issue.

[robinbudd]

When will this feature be available in the stable release? It would greatly simplify my build/CI pipeline.

[Chris--B]

+1 on this. This greatly improves the workflow on our CI, and we'd like to avoid using a nightly compiler just for this.

What needs to happen for this to move forward?

[lf-]

This issue makes nmattia/naersk unusable on stable rust. (another build system issue)

[golddranks]

This would streamline some of my Dockerfiles and build scripts quite nicely.

[Diggsey]

Is there a reason this doesn't work with cargo rustc?

[benesch]

I just published a crate called cargo-out-dir to crates.io, which can ease some of the frustration while we wait for this to stabilize. All it does is print out the current' crate's out directory, as determined by running cargo check --metadata-format=json:

$ cargo install cargo-out-dir
$ cargo out-dir
/home/benesch/path/to/crate/target/debug/build/foo-9d9f15d3d084bc3a/out

[simonbuchan]

Another potential use case for out-dir is making cargo run work nicely with building MacOS apps, which expect a structure like:

My App.app/Contents/
  Info.plist
  MacOS/actual-binary
  Resources/foo.png

in order for native APIs like (the moral equivalent of) [NSImage imageNamed: @"foo"] to work.

But in this case it would be more helpful to be able to put out-dir = "dist/My App.app/Contents/MacOS" in Cargo.toml somewhere, presumably in the [bin] section. Not sure if that makes anyone more or less happy?

[workingjubilee]

--out-dir should not have this name because --out-dir on rustc performs almost the exact function of --target-dir, and to have this be a copy directory command on cargo instead would be confusing. A name like --copy-dir would suffice, however. I believe one of the proposed solutions for #6100, such as having the ability to independently specify "scratch" and "final" build artifact directories is worth investigating.

[piegamesde]

What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

I run into this regularly (multiple times a year), and I am not the only one. At the moment, people hard-code ./target/release/binary or some variant of it like $CARGO_TARGET_DIR/$TARGET/$APPNAME for copying their binaries in installation scripts. This quickly breaks apart in the following cases:

• The cargo target dir has been moved to a custom value
• Cross compilation.

Cargo's current file handling is unfortunate, and stabilizing this value would help a lot. Specifically, all temporary build files should be freely movable to an external directory, while the build outputs (i.e. the binaries or shared objects) should land in a fixed path for easy packaging.

(In an ideal world, ./target would only contain what --out-dir currently provides by default, and all other build artifacts would land somewhere user-configurable with a default to $XDG_CACHE_HOME/cargo-target/appname or something)

Related: #6100 (comment)

[bugproof]

@piegamesde exactly, I'm used to having bin and obj nice and clean. There should be an environment variable or make it configurable from file as well (--out-dir )

[vabka]

Why is it still unstable?
I need to use some dark magic to build in debug/release profiles in one dockefile

...
# If release is set to any value then PROFILE env var becomes 'release' and then left unchanged
# Otherwise PROFILE env var becomes empy and then changes to debug 
ARG release
ENV PROFILE=${release:+release}
ENV PROFILE=${PROFILE:-debug}

RUN cargo build --frozen --offline --target x86_64-unknown-linux-gnu ${release:+--release}

FROM docker.io/ubuntu:22.04 as prod
COPY --from=build /target/${PROFILE}/executable /
...

[dpaoliello]

What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

The key scenario for this is integrating with other build systems, especially where you are cross-compiling: since Cargo will place the build artifacts into a target-triple directory, you need to teach the outer build system about Rust's target-triples so that you know where to look to find the build artifacts.

I'd mostly be happy if there was an option to suppress the target-triple directory (and the release or debug directory).

[workingjubilee]

The key scenario for this is integrating with other build systems, especially where you are cross-compiling: since Cargo will place the build artifacts into a target-triple directory, you need to teach the outer build system about Rust's target-triples so that you know where to look to find the build artifacts.

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

[detly]

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

Just from experience with various (non-embedded) CI and embedded build systems, it's often easier to determine a path for the binary artifacts early and up-front and then use that everywhere, than try to reverse engineer your build tool's scheme for placement. Especially if there's no agreement that the build tool's placement will remain stable.

Consider also that with CI/CD you're configuring things in YAML, so no functions to call like cargo_path_for_binary(app_name, target_triple, profile) - it's copy and paste everywhere. GNU Make at least has functions, but they're not fun to figure out.

There may be more concrete reasons too, that's just what came to mind.

[workingjubilee]

CI configuration languages generally accept variables, indeed this is virtually required for making a CI matrix work. And in a project I have recently been working on which requires constructing a custom build system, I made it so cargo is always invoked with a --target, even if it is just echoing the host tuple, precisely because I don't like the fact that binary artifacts appear in /target/release or /target/release/blah-blah-blah-blah/ based on whether you invoke cargo with --target or not. But neither does it work well to potentially admix the artifacts of different targets.

[piegamesde]

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

One problematic use case I ran into was with some application that was built with Meson. The problem is that it just copied ./target/release/appname into the installation folder. Now I wanted to add cross compilation support for it, and had to hack my way around so that it would take ./target/target-triple/release/appname instead. I think the situation would be half as bad if we didn't have this weird two-level stuff going on based on whether the project is cross-compiled or not.

[dpaoliello]

...? But in that case you need to have invoked them using --target in the first place, which means you already know what the triple is.

Just from experience with various (non-embedded) CI and embedded build systems, it's often easier to determine a path for the binary artifacts early and up-front and then use that everywhere, than try to reverse engineer your build tool's scheme for placement. Especially if there's no agreement that the build tool's placement will remain stable.

Consider also that with CI/CD you're configuring things in YAML, so no functions to call like cargo_path_for_binary(app_name, target_triple, profile) - it's copy and paste everywhere. GNU Make at least has functions, but they're not fun to figure out.

You might know the target triple within the Rust project, but if you are wrapping calls to Cargo from another build system then there are likely other non-Rust projects in that build system that are unaware of Rust's target triples. It's much easier in a build system to assume that the built artifact is $OUTPUT/$PROJECT.so than to have to thread the target triple (and knowledge that a subset of projects are Rust) throughout the rest of the project so that you can generate $OUTPUT/target/$TARGET_TRIPLE_IF_RUST/$FLAVOR_IF_RUST/$PROJECT.so.

Things only get worse if a project can customize its target triple: it might be trivial to create some Make/CMake function to generate the target triple based on the target architecture of the build, but if an individual project can customize that triple (e.g., MUSL vs GNU, MSVC vs GNU, or using a none target for no_std) then the external projects need to rely on implementation details of how that project is configured to know where its output is.

[hashworks]

As a practical example, this is a problem when one builds and packages mdcat - besides the binary target/release/mdcat additional files are produced, which one needs to locate in a (IMO) unstable way:

$ git clone https://codeberg.org/flausch/mdcat
$ cargo fetch --locked
$ cargo build --frozen --release --target-dir=target
$ ls target/release/mdcat
target/release/mdcat
$ ls -d target/release/build/mdcat-*
target/release/build/mdcat-2a83e790d2b3f7a6  target/release/build/mdcat-7decb70a642f31fb
$ find target/release/build -type d -path "target/release/build/mdcat-*/out"
target/release/build/mdcat-7decb70a642f31fb
$ ls target/release/build/mdcat-7decb70a642f31fb/out
completions  _mdcat  mdcat.bash  mdcat.fish  _mdcat.ps1

[mcandre]

I need this --out-dir feature in order to automate archiving binary artifacts (e.g. tar -czvf ..., zip -r ...), when I release my Rust apps.

Not a fan of how the flag is teased in non-nightly cargo's help menu, only to bail when invoked. Don't bother me. Either let me use the flag, or at least drop it from the CLI documentation.

Alternatively, I could use nightly, and require my users to use nightly. For DevOps users and other fans of pinned, immutable versions, nightly is not a valid option.

Alternatively, I would have less need for this feature if cargo simply nested binary artifacts in a more archive amenable tree structure, like target/bin, by default. Currently, cargo flattens out binary artifacts--the most important part of the build--polluting recursive archives with sibling junk files.

Alternatively, one could write a custom shell script to accomplish this. But that wouldn't work very well across all Rust projects, and it would needlessly restrict Windows devs, and proper robust shell programming is rather a dark art, and shell scripts substantially depart from the Rust ecosystem.

Alternatively, one could configure archive commands to strip out the junk sibling files, but that's a maintenance nightmare, as cargo will surely (as any package manager reasonably would) introduce additional file patterns in the build tree internal directory over time.

Alternatively, one could configure archive commands to look exclusively for file patterns matching target/(debug|release)/<all the binaries mentioned in Cargo.toml>(\.(exe|js|wasm))*. Another maintenance nightmare.

I don't want to get too involved with low level details of cargo internal build files. I just want more reasonable defaults. Or failing that, landing --out-dir in the next Rust release.

[nils-werner]

I'm also looking forward to seeing this option merged quickly. In that spirit: Is it really necessary to rename the option?

I understand that the worry is that there could be confusion between --out-dir for cargo and the environment variable OUT_DIR for build.rs, but doesn't it mean exactly the same in both cases?

In both cases you're asking the build tool to place it's output artifact in a certain directory, it just happens that in one case you're asking cargo, and in the other instance someone (cargo) is asking build.rs. But the operation and meaning is exactly the same, no?

[valadaptive]

In both cases you're asking the build tool to place it's output artifact in a certain directory, it just happens that in one case you're asking cargo, and in the other instance someone (cargo) is asking build.rs. But the operation and meaning is exactly the same, no?

No. As I understand it, the OUT_DIR environment variable tells build scripts where to place intermediate artifacts (it's the build directory within the target folder) whereas --out-dir tells cargo where to place final artifacts (binaries and libraries).

[workingjubilee]

Two people have asked "why rename the option?" with an air of "...when we could ship what's currently there, and get it faster?" I wonder if people imagine that, since we have been waiting so long, if someone renames the flag we will have to wait another long time for it to stabilize?

That doesn't happen, in my experience. As the minimum wait from FCP to stable release is 6~12 weeks¹, and the only prior users were necessarily using the nightly toolchain, I have seen teams rename something and then immediately start the stabilization clock.

But I have also seen a team member that does not think something is "finished" decide to raise a blocking concern. Including on naming. And that by definition can delay the stabilization process indefinitely. I speak in generalities of course.

"Should the name be --out-dir? Probably not?" is not one but three of the Unresolved questions on this very tracking issue. Having observed a lot of FCPs, asking for an FCP to start without a good answer for all unresolved questions is a good way to materialize a blocking concern for each unanswered question. That is why features like this one are made closer to their finished, and thus stabilized, forms, by removing a highly contentious and confusable trait.

Footnotes

1. https://doc.rust-lang.org/nightly/cargo/reference/unstable.html ↩

[workingjubilee]

@valadaptive In #6790 (comment) you asked this:

I'm not familiar with the Cargo governance structure--what's needed to push this feature over the finish line?

Note: I am not a T-cargo member. But to answer to the best of my ability:

T-cargo uses rfcbot to do FCPs like most of the Rust teams. That means any T-cargo member can call for a T-cargo FCP, and that is what you're looking to convince someone to start. There is T-cargo's unstable feature process that explains everything but... probably skip towards the end, as this has been baking quite a while. More of a background note, but might make things make more sense: like most teams this process is partially templated on the rustc stabilization guide.

There's also interesting tidbits¹ of related documentation in the Cargo contributor guide. Can't see anything about deprecation tags though!

Footnotes

1. https://doc.crates.io/contrib/team.html#decision-process ↩

[valadaptive]

#7493 also seems to be a blocker here.

[valadaptive]

Here's a recap of the unresolved questions:

• Is --out-dir justified or can we find another way of accomplishing it? #6100 — concern about similarity with --target-dir, proliferation of flags

Over the years, many people in that issue thread have proposed their dissatisfaction with --out-dir and suggested some sort of alternative. As far as I can tell, no design work has actually been put into any of these alternatives. Also, they all seem to require large refactorings of Cargo, and may be breaking changes considering that many build systems already assume the layout of the target directory. As such, I consider them untenable.

• Duplicate artifact tracking issue. #6313 — concern about name collisions in workspaces or with examples

Still relevant and will need to be solved.

• What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

See above for all the developers who want this to be added.

An important use case is being able to directly output artifacts where you want them, without having to know where the target directory is or copy out of it as an extra step. Figuring out the former is complicated by the --target-dir flag. This is very helpful when integrating with other build systems, CI pipelines, Docker, etc.

A couple pain points with the current situation seem to be knowing whether to reach into the target/release or target/debug folder, and accessing the folder for the correct target triple when cross-compiling. Passing such information around the build system you're trying to integrate with can be tedious and error-prone.

• Confusion with rustc's --out-dir

• Discrepancy or confusion with build.rss OUT_DIR (depends on if build.rs is generating intermediate or final artifacts)

Solved by renaming to --artifact-dir (#13809).

As I noted above, #7493 is also a blocker that will need to be resolved before we can stabilize this flag.

[epage]

Over the years, many people in that issue thread have proposed their dissatisfaction with --out-dir and suggested some sort of alternative. As far as I can tell, no design work has actually been put into any of these alternatives. Also, they all seem to require large refactorings of Cargo, and may be breaking changes considering that many build systems already assume the layout of the target directory. As such, I consider them untenable.

That is not sufficient reason to ignore those concerns.

However, since then (besides team members rotating)

• We've started talking more concretely on TARGET_DIR layout changes in Per-user compiled artifact cache #5931
• RFC: Templating CARGO_TARGET_DIR to make it the parent of all target directories rfcs#3371 was created and is moving forward. A potential shiny future for that is that everyone has their unstable-layout TARGET_DIR hidden away with everyone accessing the stable-layout ARTIFACT_DIR

Looking at those, I agree with #6100 (comment) that having --artifact-dir is likely the way forward for allowing more evolution of intermediate artifacts.

See above for all the developers who want this to be added.

What would be important for organizing all of the use cases is to understand what is in or out of scope (or if we need more, lower level knobs as mentioned).

e.g. #6100 (comment)

Is it possible to make --out-dir option for cargo test also? The use case that I need is to run the binary test file to run under a debugger

A feature may sound like it works for one person or it may indeed work for one person but we need to consider the overall ecosystem and "where will this lead" both in terms of second order effects (not concerned about that here) and what will people find lacking and ask to have changed or extended. We don't have to solve everyone's problem but we need to have a rough idea of what the future is to make sure we are solving the right problems. For me, I'm less concerned with #6100 for this flag but whether we'll need several additional flags after this because we didn't quite solve the right problem.

[detly]

What would be important for organizing all of the use cases is to understand what is in or out of scope (or if we need more, lower level knobs as mentioned).

[...]

A feature may sound like it works for one person or it may indeed work for one person but we need to consider the overall ecosystem and "where will this lead" both in terms of second order effects (not concerned about that here) and what will people find lacking and ask to have changed or extended.

I think that to a large extent, the need being addressed can be split into two parts:

1. The need to predict where certain build artifacts will be placed.
2. The need to control where certain build artifacts will be placed.

The "certain build artifacts" can also be split into two parts:

1. Build artifacts that are needed for actual deployment, perhaps called "final" build artifacts.
2. Build artifacts that are needed to analyse coverage, cache build stages etc. perhaps called "intermediate" build artifacts.

By way of example, I have recently been trying to implement fine-grained code coverage reporting in CI on a Rust project. It is necessary to tell the coverage tools where both the coverage instrumented binary artifacts are, and where the coverage data is. The latter can be controlled through environment variables, so perhaps we can ignore that. But the former is extremely hard right now.

• For a native dev build, they're in target/debug.
• For a native test build, they're in target/debug.
• For a native release build or any other profile, they're in target/<profile>
• For a cross build, they're in target/$RUSTC_TARGET_ARCH/<profile> where Cargo has been invoked with --target=$RUSTC_TARGET_ARCH

So to simply generate the command lines for a coverage tool in CI, I need to implement this logic in CI or a build script. I can't just have CARGO_ARTIFACT_DIR="target/$CARGO_PROFILE", because of the maybe-there maybe-not target component, although @workingjubilee's trick works for that. I cannot have CARGO_ARTIFACT_DIR=target/$RUSTC_TARGET_ARCH/$CARGO_PROFILE, because dev and test profiles map to a debug directory name.

I also can't just use the JSON output of Cargo because there is no part of that that gives me the base directory, only the full path to each individual artifact. The base directory is required eg. for most instrumentation analysis tools, for caching and storing artifacts consistently, and because the full list might actually be too long for many shells' limit on environment variables.

So I need to implement a more complex mapping that duplicates Cargo's internal artifact directory construction logic. In shell script. Or the CI config syntax. Or another build script language. This is undesirable because:

• it's non-trivial and error prone
• (AFAIK) it's not an API detail of Cargo, and could change, so it's fragile
• it's making everyone who needs this do the work over and over instead of doing it in one place ie. Cargo

This is not just an issue for code coverage. This applies to:

• integrating Rust projects into more complex build configurations eg. OpenWRT, Meson projects, installer build systems, almost any mixed-language project really
• trying to manage caching in CI systems to reduce test turnaround time
• trying to pass artifacts between CI jobs

Now consider that if I were just able to predict (or extract from Cargo's output) where artifacts will end up, I can effectively meet "the need to control where certain build artifacts will be placed" by writing cp "$PREDICTED_ARTIFACT_DIR" "$DESIRED_ARTIFACT_DIR". Hell, someone could write a tool cargo-with-outdir that does nothing but force certain flags/parse certain output and do the copy for you. There may be other cases where it is simply necessary to dictate this directory and not predict it, I don't know.

I'll also preempt the argument that there are tools that abstract this away for some of the use cases I mention eg. cargo-llvm-cov for coverage, cargo-chef for CI caching, cargo-nextest for test artifact passing, OpenWRT's own Rust build integration. That's a bit of an own-goal, because it's actually just listing places where the internal logic of Cargo's directory naming has to be duplicated, or the logic of capturing Cargo's output just to collect artifacts in a common directory has to be duplicated. Those tools do not cover every use case eg. to use exclusion filters for Rust code coverage, I need to ditch cargo-llvm-cov and use eg. grcov or gcovr, which... need telling where the artifacts are. When cargo-nextest's intermediate archive needs winnowing out to be copied to various CI jobs with different responsibilities... I need to know where the artifacts are. Not every niche is going to be covered by a 3rd party tool, and not every 3rd party tool is going to have a common subset of Cargo-output-path-predicting code that can be extracted into a 4th party tool.

In the last four years of using Rust in earnest and trying to do novel things related to code quality, instrumentation and testing, I just keep running up against this. I don't mean this as a kind of pressure to just "get it done", because I respect (a) Cargo's commitment to quality of API and (b) the fact that (veeery much like the --deps-only issue) a lot of different goals have kind of been simplified into "give us --some-flag!", and it might not actually work in a way that achieves those goals. But it is to say that there is probably a very common pain point here between users and use cases, and hopefully the above articulates that a bit more, and it really does seem like --artifact-dir or something like it must be pretty close to addressing those.

[soloturn]

rust and cargo has so many flags and environment variables, which i find quite confusing. there are legacy tools which have a logic variable naming, like cmake: https://cmake.org/cmake/help/v3.0/manual/cmake-variables.7.html, it has a couple of directories, but for profiles it operates with postfixes. for building, the default cmake is very simple. either change directory and build, or call cmake -Bbuilddir -Ssource. there is no cache, no artifact, no out, no target, no lib etc etc. if the file is there it uses it, if not it produces it.

if you want to have a more modern example, bazel has the following: https://bazel.build/remote/output-directories. it is designed for buils on a cluster, and extreme reuse of build artifacts.

therefor i woudl support a design ... if the designer would be prepared to remove 90% of the existing varialbes would be already a huge advancement :) my personal opinion of course - just coming from my current frustration of hours of build time of COSMIC desktop because it recompiles the same libraries again and again.

[epage]

So I need to catch up on some things earlier in the thread but needed to record some thoughts real quick.

Someone brought up multi-target builds (cargo build --target <one> --target <two>). If could error if we build for multiple targets and the artifact-dir is the same value. This could be better controlled with templating in the artifact-dir.

While I don't dig into the blockers for this, #14125 (comment) explores a potential cohesive picture of what we should be doing

[krh]

Can this also copy the corresponding dependency file (artifact.d)? In a build system where we're using meson to drive cargo to build the same bin with different features, we need the correct per-build dependency file to pass to meson.

What exactly is the use case for this? In what scenario would it be used where you don't already have an outer build system or script that can copy the files into the correct location?

I hope this answers that question. We're building the same binary, same profile and target, but different flags, resulting in different artifacts but with the same file name in the same output directory. I suppose there's related, nagging concern there, that if this is just a copy step from the canonical artifact dir under --target-dir then the files still conflict if they're built with different features.