Is --out-dir justified or can we find another way of accomplishing it?

[withoutboats]

I noticed that cargo build has two subcommands, both added this year. I believe this accurately describes their behavior:

• --target-dir: Replaces $PWD/target with this directory (e.g. a debug build will be built into target-dir/debug). Stable flag, added in April 2018. Add target directory parameter --target-dir #5393 In addition to CARGO_TARGET_DIR env var, support --target-dir command line option #5308. Mimicks the behavior of CARGO_TARGET_DIR
• --out-dir: Things will be built into $PWD/target as normal, but copies some of the artifacts into the directory specified by out-dir (not a profile specific subdirectory). Unstable flag, added in March 2018. cargo build --out-dir #5203 Ability to specify output artifact name #4875. Mimicks the behavior of OUT_DIR.

(I'm not 100% certain I've accurately described their behavior in relationship to the profile specific subdirectories.)

--out-dir is still unstable, but --target-dir was instantly stabilized (by mistake, I believe).

I can't help but notice that these two flags do very similar things! They also seem to have been designed for very similar purposes (build integration, the meson project is mentioned specifically in both of them).

I'm not certain I agree that either of them should have been added as flags (an argument is made that flags are better than env vars, and while I agree in principle for user oriented APIs, I don't think that build system integration is a user oriented API in the relevant sense), but I'm definitely suspicious of having both of these very similar flags provided. Since target-dir is stable, I'm not sure if there's an available avenue to unify these flags other than just only providing target-dir.

cc @rust-lang/cargo

[joshtriplett]

I do think it makes sense to keep these separate. I like the idea of having --out-dir to put the final build artifacts and not all the intermediate bits.

[matklad]

I think it is important that we have both of these features:

• ability to override "scratch space" dir for intermediate build artifacts
• ability to override the place where the final build results go

UI-wise, using flags for these seems better: the "build system" code, like ci-scripts, tends to be poked by many people, so it's important to keep it easy to understand without prior knowledge, and flags are better than env vars in this respect.

I do agree that for most of the users, interacting with Cargo's CLI, these flags are irrelevant. However, this could be said about some other flags as well, like --manifest-path, --message-format, --build-plan, --frozen, --locked. Perhaps we should hide such "useful in very specific circumstances" flags from cargo cmd --help by default?

[withoutboats]

I do agree that for most of the users, interacting with Cargo's CLI, these flags are irrelevant. However, this could be said about some other flags as well, like --manifest-path, --message-format, --build-plan, --frozen, --locked. Perhaps we should hide such "useful in very specific circumstances" flags from cargo cmd --help by default?

I am unhappy with the UX of having all of these flags also. For a long time we've been creeping in the direction of having flags to customize every user visible aspect of the behavior, because this has been how CLI tools traditionally handle users wanting to customize their behavior. However, I think this has a bad UX overall, because it makes the tool overwhelming and ultimately makes all of the flags undiscoverable. I think hiding these tools from --help unless you pass another flag is a half measure, which improves the situation but is probably not the best solution.

I think a preferable approach to what we've done would be to provide some lower level command which takes as many flags as you want and can be used to build up the user-oriented commands like cargo build, cargo run, and cargo test, but with all of the flags to customize the precise nature of its operations. The higher level commands could be defined in terms of that lower level command, and people writing very bespoke integrations can also build their own build command out of it.

So I noticed a few days ago that when I ran cargo build --help it went beyond the length of my terminal. This surprised and concerned me, so I took statistics of how cargo build --help's output has grown over time:

It was in the process of investigating how the flags to cargo build had proliferated that I noticed these two flags. I think there's a larger problem here, but these two flags in particular have such similar behavior that keeping them both has unique negative UX consequences aside from the general concern. From the --help output I could not understand how they were different, and I don't think their names indicate that the most material difference between them is that one excludes the intermediate artifacts. Its the kind of customization creep that makes me groan and turn to stack overflow instead whenever I try to read the man pages of any of the standard UNIX utilities: their man pages are an information overload that make it too difficult to learn things.

An obvious way to help resolve their close similarity is to clarify the differences in the --help output. But as we've already touched on, these are not useful for most users, and so in the common case these two commands are both cruft I am trying to sort through to figure out the flag I actually want - increasing the amount of time we dwell on them makes the situation worse. So I don't think that would be a solution either.

[alexcrichton]

I'm personally fine adding these flags basically wherever makes sense, but to echo @matklad I don't think we'll want to remove the functionality. In that sense I would want to make sure that these sorts of configurations (which you're 100% right @withoutboats, will continue to get added to seemingly no end) have some home somewhere. They're definitely not important to surface in high-profile locations, just need to be located somewhere!

[withoutboats]

We discussed this & also the overarching issue a bit in the cargo meeting yesterday. The sort of overarching viewpoint was that it would be ideal to expose these kinds of knobs through a set of lower level primitive commands on top of which sit the more user oriented commands (git has this distinction for example, calling them "porcelain" and "plumbing").

However, the problem with doing that is that it requires major refactors which seem difficult or infeasible. On the other hand, by adding all of these knobs incrementally we end up with the same functionality, but with worse UX (from having the functionality exposed through a bunch of flags on the main commands), and worse maintainability (because those flags are being threaded through the entire codebase). But back to the problem of pulling off a refactor, and so we ended at a standstill.

[alexcrichton]

Was there a good idea of what the porcelain/plumbing mode would actually look like in Cargo? In abstract it makes sense but if there's a concrete idea where to go it may not be so bad to draw up an issue and start soliciting help for making progress

[mark-i-m]

Is there any chance of --out-dir being stabilized any time soon? It is really useful when integrating cargo with another build system, e.g. Makefile.

[kornelski]

It seems to be related, but a bit backwards, to temp vs non-temp separation of directories proposed in #6227

[elichai]

I would also like to see --target-dir for cargo install

[igrep]

I found the --out-dir option is very useful for me to copy .so files built for Android to app/src/main/jniLibs/<arch_dir>.
--target-dir emits too many unnecessary things, and I have to resolve the dylib's name (in debug? in release? etc.) by myself in addition...

[yaahc]

I recently had a couple of people express an interest in --out-dir being stabilized and from my initial digging it seems like what they may actually want is to switch to OUT_DIR, which is already stable. If we don't intend to stabilize --out-dir since flags are the "user oriented API" then I think we might want to more clearly document how to get the functionality provided by --out-dir on stable to make the environment variable more discoverable.

[sophie-h]

@yaahc

from my initial digging it seems like what they may actually want is to switch to OUT_DIR, which is already stable

OUT_DIR is read only if I understand the docs correctly. (I also tried to set it and it didn't worked.)

[matklad]

Oh wow, I think there's a very confusing point here, which we somehow didn't noticed before. OUT_DIR and --out-dir are different and unrelated. They just happen to share the same name.

OUT_DIR is used for build scripts. It is set by Cargo, and tells the build scripts where to put the generated code.

--out-dir is used to instruct Cargo itself where Cargo should put its output. It is set by the user.

[matklad]

Coming back to this after a couple of years, I feel that maybe adding --out-dir is indeed premature, and a brainstorm of "layout of ./target dir" is required before that. I feel there's a bunch of problems with target, and maybe some of them are worth solving in batch.

Of the top of my head:

• bad mental model: ./target is both where the final output goes, and cargo's private "scartch space". For a new user, it's unclear which parts of the ./target they can use, and which they can't Configurable directory for temp files that isn't the target dir #6227
• ad-hoc rules for where final output goes: it's ./target for binaries, ./target/examples for examples, and ./target/deps for tests
• Running tests "by hand" is hard. The test binaries are buried in the ./target directory among a huge number of intermediate artifacts
• difficult to cache on How to effectively clean target folder for CI caching #5885, because it's impossible to separate "workspace crates which need to be rebuild every time" from "dependencis which stay mostly the same" How to effectively clean target folder for CI caching #5885
• hard to do system-wide cache. setting the same TARGET_DIR for all projects kinda works (as long there are no name collisions), but is accidental, rather than intentional, and breaks UX of running stuff by hand.
• similarly, workspace-wide TARGET_DIR is prone to name collisions

[Zingam]

Can I configure build.target-dir on a workspace? I don't see a definite answer that it is not possible and I don't get an error but it also doesn't work with rust-analyzer configuring the project. I get the default behavior.

[dpaoliello]

Coming back to this after a couple of years, I feel that maybe adding --out-dir is indeed premature, and a brainstorm of "layout of ./target dir" is required before that.

Unfortunately changing the layout of the target dir may require the --out-dir feature: currently it is the only officially supported location where the Rust build artifacts can be found, so changing the layout would be a breaking change.

Instead, if we could have an output directory that is guaranteed to only contain the final artifacts, that would enable us to declare the layout of the target directory to be unstable and push folks away from relying on its contents.

An alternative is to move towards deprecating the target directory altogether, and instead introduce the concept of an output directory (or output directory per crate-type) and a temp directory.

[kornelski]

@dpaoliello I've imagined this done in reverse: keep the final build artifacts that people care about in the target dir where they are, but start moving everything else (temp files, compiler-private junk) to some other directory (such as platform's preferred cache and temp dirs).

#14125

[stevenhansel]

Is it possible to make --out-dir option for cargo test also? The use case that I need is to run the binary test file to run under a debugger

[bazhenov]

+1 for @stevenhansel suggestion. I have different environments for building tests and running them. It's quote cumbersome to find test in target/debug/deps. At the moment I find them using

$ cargo test -q --no-run --message-format=json \
    | jq 'select(.reason == "compiler-artifact" and .target.test and (.target.kind | index("bin"))) | .executable'

[jakobhellermann]

Another reason for providing this is that you can't know where the target folder is without not just reading the CARGO_TARGET_DIR, but also parsing the possible .cargo/config.toml that may set a global shared target folder.

So if you want to write e.g. a vscode task that compiles the code and moves the binary somewhere, you can either

1. not have it work for people with non-default configuration
2. use --message-format=json and assume people have installed jq

both of which aren't good options.

[bazhenov]

Just leave it here: https://github.com/bazhenov/cargo-export

[JosiahParry]

Adding a vote for making --out-dir a safe operation. It is nice to be able to specify where the final artifact will be placed. Currently opting for a subsequent cp statement.

[andrewbaxter]

I don't think the use case was explicitly mentioned (although related CI use cases were): I use a global CARGO_TARGET_DIR to help manage disk space, but it means I need to hunt for compiled binaries in that dir. I'd like a way to keep the binaries in the current directory, ideally without needing a per-project --out-dir.

[epage]

Access to artifacts is also coming up in rust-lang/rfcs#3371