Add cargo tree command.

[ehuss]

This migrates cargo-tree into Cargo as a built-in command. This is based on a recent master (https://github.com/sfackler/cargo-tree/tree/4108d216ec072d2e7c9befb4de32175f97a9dbc4), and should be mostly similar in functionality. There are a variety changes:

• --all-targets renamed to --no-filter-targets to avoid confusion with the --all-targets flag used in other Cargo commands with a different meaning.
• --all/-a renamed to --no-dedupe to avoid confusion with the -all flag which means "all workspace crates" in other Cargo commands.
• --duplicate renamed to --duplicates (with alias), just a personal preference.
• Added support for multiple roots (workspace support).
• Added the --graph-features flag for including features in the graph (to "explain" why a feature is enabled).
• Added {f} to format string to show features.
• Handles new feature resolver.
• Handles cyclical dev dependencies.
• Added a test suite.
• Dropped the dependency on petgraph, in favor of a simpler custom graph.

Closes #7286.

[rust-highfive]

r? @Eh2406

(rust_highfive has picked a reviewer for you, use r? to override)

[ehuss]

cc @sfackler

This is a proposal to merge cargo-tree as a built-in command.

Motivation
Currently, Cargo does not have a built-in method to display or visualize the dependency graph. The information is available in JSON (or Cargo.lock), but not in a human-readable form. The cargo-tree command provides a way to see the dependencies. It also has a variety of options for performing useful actions, such as displaying duplicate dependencies.

cargo-tree has been available as a Cargo plugin for over 4 years. Making it a built-in command reduces the user's burden for installing and updating a separate plugin. It also helps ensure that the command stays compatible with all changes in Cargo.

Drawbacks
This adds to the maintenance burden on the Cargo team.

Alternatives
We could leave cargo-tree as an external plugin. There are a variety of different plugins available for visualizing and querying dependencies, and we could leave it up to the user to pick the one they want.

There are a variety of other ways to visualize a dependency graph. Using graphviz is a common alternate solution. It is my opinion that the low-tech solution of formatting the tree in the terminal offers a simpler solution that is more broadly accessible. Additionally, large graphs can rapidly become unreadable when naively rendered in graphviz, requiring additional effort from the user to prune or organize it. Alternate output formats can be added at a later date.

Prior Art
Several package managers offer similar functionality.

• (Javascript) npm ls — Nearly equivalent to cargo-tree.
• (Javascript) yarn list — Nearly equivalent to cargo-tree.
• (Go) dep status -dot option will emit graphviz.
• (Go) go list can emit dependencies/imports, not in tree form.
• (.NET) — Visual Studio has built-in methods to graphically browse the dependency tree. nuget and dotnet have ways to list dependencies, though not in a hierarchical fashion that I can see.
• (Haskell) stack dot will emit a graphviz graph of dependencies.
• (Julia) The pkg REPL can interactively inspect the graph, but doesn't have a tree view I can see.
• (Ruby) bundle viz will emit a graphviz PNG.
• (Dart) pub deps emits an ASCII tree.

Most dependency managers have additional third-party tools for displaying and graphing dependencies.

Future possibilities

• Any of the enhancements listed at https://github.com/sfackler/cargo-tree/issues/
• Make package downloading lazy, so that extra packages are only downloaded if needed. Currently it downloads all packages (even if not displayed or needed), just for simplicity. (This is no different from the cargo-tree plugin, which uses cargo metadata which forces downloading everything.)

Questions

• Any hard objection to the backwards-incompatible changes to the flags mentioned above (--all and --all-targets)? I think maintaining consistency with the rest of Cargo justifies the temporary disruption when users upgrade.
• Instead of adding --no-filter-targets, maybe another option would be to make a special target like --target=all to mean all targets?
• Are there any other changes we should make before merging?
  • A more general way to select dependency kinds? Currently it has --no-dev-dependencies, but there are requests like Add an option to ignore build dependencies sfackler/cargo-tree#51 to also skip build dependencies. I was thinking of a flag like --dep-kinds=… where you can list the dep kinds you want (maybe with negative options like --dep-kinds=-dev or --dep-kinds=no-dev). This could be useful for other commands (like cargo doc?).
  • A more general way to select the prefix format? Currently it has --prefix-depth and --no-indent. Maybe a single flag like --prefix=depth and --prefix=no-indent?
    • (BTW, I don't really know the use case for changing the prefix).
  • Any of the other issues listed at https://github.com/sfackler/cargo-tree/issues?
  • The format string is a little strange. It doesn't handle optional values very well. It also has some parsing bugs. I'm not sure if we should make changes or not?
• Bikeshed any flag names. --graph-features?

[sfackler]

As one other bit of prior art, I based the original implementation off of Gradle's dependencies task: https://docs.gradle.org/current/userguide/viewing_debugging_dependencies.html

It seems like a fine time to just make breaking changes to the CLI interface - people will have to opt-in anyway by deleting the old cargo-tree.

WRT to dependency kinds, it's kind of an interesting question. I don't think you really ever want to say "show me all dev dependencies recursively" - you'd want to see your dev dependencies, their regular (and maybe build dependencies) etc. It might make more sense to structure it around "what gets built if I run this kind of operation" than specific dependency kinds.

[metajack]

I often want to query the first party deps or the third party ones only. --kind workspace and --kind thirdparty in cargo guppy. I've also added --kind directthirdparty now to get only the direct dependencies (a depth isn't sufficient as there are many edges from first party into first party deps). I've also found it useful to omit parts of the tree (for instance you might want to ignore edges into rustc-workspace-hack).

[alexcrichton]

Thanks for spearheading this @ehuss, this is all looking great! I mostly have some various nits here and there, some of them are probably just holdovers from age-old code at this point, so no worries!

I'm pretty stoked for this, it seems like an awesome feature to add to Cargo

people will have to opt-in anyway by deleting the old cargo-tree.

Oh I don't think this is actually true because cargo $builtin_cmd takes precedence over anything in $PATH. We had to handle this when we moved in cargo vendor where some flags were added but they specifically say "if you want to use this use cargo-vendor vendor --flag", so we may want to do something similar for cargo-tree if there's frequently used flags that changed?

[alexcrichton]

Over time I've tended to prefer "positive" CLI flags to negative ones. The main reason is more rustc-specific when these flags take values, but I find double-negatives really hard to read (-C no-foo=no means "foo=yes" actually). Do we want to consider renaming these flags so it's something like --deduplicate=no or --dev-dependencies=no perhaps?

[ehuss]

I agree that avoiding double negatives is good, but in this case I don't think it is important. There isn't a corresponding flag for dedupe since that is the default. I don't feel like this needs to be super flexible, so I don't think it should take a value (and swapping the "no" from the front to the back doesn't substantially change things).

I do agree the flag itself is potentially confusing. Other alternatives I thought of was --show-dupes or --show-duplicates, but it's starting to get long and wordy. I was also concerned about confusion with the --duplicates flag, which means something completely different.

One (tangentially related) idea I wanted to toss out is changing (*) to deduped. This is what NPM uses, so there might be some familiarity there. With the explicit wording, it might make more sense? I mention this because I remember the first time I used cargo tree I was confused what the (*) signified.

Any other ideas?

[alexcrichton]

Oh right I forgot about --duplicates. Given the other consolidations that have been happening, I agree let's leave this as --no-dedupe

I like the idea though of changing (*) to something else. Do you think it'd be useful to take that a bit further and do (deduped from line 13) so if you pipe the output to a text editor you can jump around? Or something that points to where the deduplication happened at least.

[ehuss]

Sounds interesting. I'd like to put this on the future enhancements pile, though.

[alexcrichton]

So a the recommendation of @ehuss I decided to give this a spin locally to see if I could debug an actual issue I had in builds. One thing I've noticed is that in the wasmtime repo these two commands cause quite a few rebuilds:

$ cargo build --manifest-path crates/api/Cargo.toml
$ cargo test --all --exclude lightbeam --exclude wasmtime-c-api --no-run

Using another tool of mine I got output that looks like this:

duplicate artifacts found when compiling, this typically means that something was recompiled because a transitive dependency has different features activated than in a previous build:

the following dependencies are duplicated although they have the same features enabled:
the following dependencies have different features:
  regex-syntax 0.6.17 (registry+https://github.com/rust-lang/crates.io-index)
    build1 additionally enabled features {"unicode", "default"} at "/home/alex/code/wasmtime/target/debug/deps/libregex_syntax-37d0dc3aa6c97f66.rlib"
    build0 additionally enabled features {} at "/home/alex/code/wasmtime/target/debug/deps/libregex_syntax-025cc46211ab3255.rlib"
  syn 1.0.17 (registry+https://github.com/rust-lang/crates.io-index)
    build1 additionally enabled features {"extra-traits"} at "/home/alex/code/wasmtime/target/debug/deps/libsyn-5c3d8b7c7d3bf908.rlib"
    build0 additionally enabled features {} at "/home/alex/code/wasmtime/target/debug/deps/libsyn-edabbd70c7a2db6e.rlib"
  object 0.18.0 (registry+https://github.com/rust-lang/crates.io-index)
    build1 additionally enabled features {"indexmap", "write_core", "crc32fast", "write"} at "/home/alex/code/wasmtime/target/debug/deps/libobject-70dfec46d12ad840.rlib"
    build0 additionally enabled features {} at "/home/alex/code/wasmtime/target/debug/deps/libobject-af359f2409925054.rlib"
  cranelift-codegen 0.61.0 (path+file:///home/alex/code/wasmtime/cranelift/codegen)
    build1 additionally enabled features {"all-arch", "riscv", "x86", "arm32", "arm64", "testing_hooks"} at "/home/alex/code/wasmtime/target/debug/deps/libcranelift_codegen-1a33ce6f0cb8fa3b.rlib"
    build0 additionally enabled features {} at "/home/alex/code/wasmtime/target/debug/deps/libcranelift_codegen-82466da147164ef8.rlib"

builds traced are:

	build0 - ["build", "--manifest-path", "crates/api/Cargo.toml"]
	build1 - ["test", "--all", "--exclude", "lightbeam", "--exclude", "wasmtime-c-api", "--no-run"]

So here's some things I did given this output:

regex-syntax

No idea why these features are changing.

First I ran cargo tree --graph-features -p regex-syntax -i --manifest-path crates/api/Cargo.toml giving this. Sure enough the regex-syntax crate doesn't have the "default" or "unicode" features activated.

Next I wasn't actually sure what next to do since there weren't --all analogues for cargo tree. Maybe we should have similar package selection flags for cargo tree? Additionally I wasn't actually sure how I'd handle cargo test or not. I suspect that it's from [dev-dependencies] though?

From prior knowledge I know the crates/wiggle crate depends on proptest which enables these features, and sure enough going into that directory and executing cargo tree --graph-features -p regex-syntax -i I immediately found proptest through [dev-dependencies] as the culprit.

So I think my conclusion from this would be that the package selection arguments --all and such may want to be added? (basically expanding the set of roots)

syn

Next I wanted to check syn and where it got extra-traits from. Using cargo tree in crates/api it sure enough shows no extra-traits anywhere.

Like before I didn't know what to do next, but I had prior knowledge that this is due to the derive_arbitrary crate's dependency on synstructure, so going into crates/fuzzing I could immediately see this.

object

Out of idle curiosity I looked into object. Again I had prior knowledge of where to look but it was immediately obvious who enabled the extra features in the test builds once I ran cargo tree there.

---

Basically I found this really useful for my use case of figuring out "why was this enabled?" It'd be helped to support the same --all family of package selection arguments but I suspect that isn't too hard. It was a lot of output and I found myself leaning on --no-dedupes to make sense of anything. My personal use case was "given this node in the graph print me a singular path to the root" so it was easiest to --no-dedupes, pipe to less, search, and follow visually to the end.

Overall I'm still quite excited for this :)

[sfackler]

With respect to "why am I rebuilding things" questions, I think they're common enough to potentially have a more tailored tool specifically for that question. Trying to track the answer down by manually diffing two feature activation trees seems awkward when ideally you'd be able to directly ask cargo "what's changing in my transitive dependency feature selection between operation X and operation Y?".

[alexcrichton]

I agree yeah my use case here specifically wasn't really targeted for cargo tree, but I can at least say that despite having so much information in cargo tree --graph-features I was still able to find what I needed pretty quickly.

[ehuss]

there weren't --all analogues for cargo tree. Maybe we should have similar package selection flags for cargo tree?

It does have package selection flags (-p, --workspace, --exclude). I excluded --all since in the original cargo tree it meant something different (--no-dedupe) and Cargo's --all has been deprecated in favor of --workspace to avoid confusion. I added an error message if you pass --all that explains the ambiguity.

But perhaps I'm not understanding what you mean here?

Additionally I wasn't actually sure how I'd handle cargo test or not. I suspect that it's from [dev-dependencies] though?

I think the answer here is yes, though I'm not sure what you mean. Features only differ if you have -Zfeatures=dev_dep, but I don't think that's what you mean.

[ehuss]

There are plenty of good ideas here (like limiting to "workspace only", or being able to prune parts of the tree). I'd like to land this first before doing too many more major additions. I'll keep track of them and file issues later.

[alexcrichton]

@ehuss oh my age is showing again, sorry --workspace is perfect and I should have been using that originally.

Although now testing it out, I'm not sure it's working? Or at least I'm getting somewhat confusing output after executing:

$ cargo tree -p regex-syntax --graph-features -i --workspace

In that output the regex-syntax crate doesn't actually show up?

[ehuss]

$ cargo tree -p regex-syntax --graph-features -i --workspace

-p is ignored with --workspace. We should probably have a warning about that.

I think in this case, you just want to run that same command without --workspace, though that will only give you the answer for one workspace member at a time (in wasmtime, it looks like regex-syntax is only used by the root package, so it doesn't matter there).

-p for a non-member is a bit of an oddball. It can only select a non-member package inside the tree of the "current" member (the one in the current directory). This is true of all other commands (cargo build, etc.). That's likely not easy to change (I can explain more if you want).

[alexcrichton]

Ah right yeah, that makes sense. Do you know why, though, cargo tree -i --workspace only shows path dependencies? That seems like it may be a bit of a bug?

For my initial question cargo tree --graph-features --workspace plus "search the output" is what I wanted.

[ehuss]

Do you know why, though, cargo tree -i --workspace only shows path dependencies?

It is because registry packages cannot depend on workspace members (ignoring [patch]). For example, run cargo tree --workspace, and you can see the only things that depend directly on witx are wig, wiggle, wiggle-generate, and wiggle-macro. So when you run cargo tree --workspace -i, only those 4 are listed as inverse dependencies of witx.

[alexcrichton]

Hm ok so digging in a bit, I feel like the behavior with roots and --invert is a bit confusing. I don't think the --workspace flag is needed here, but cargo tree and cargo tree -i, for example, don't produce the same graph. With cargo tree -i you only get one package whereas with cargo tree you get the whole tree.

This makes sense with respect to the implementation, but it feels surprising to me and a bit counter-intuitive. Does it makes sense to invert the roots as well when we invert the graph? (so not just the edges). It does mean that cargo tree -i -p some-cratesio-crate doesn't make much sense.

Maybe what I'm looking for is root selection orthogonal to -p/--workspace/etc package selection. What I really want is cargo tree --workspace -i --root regex-syntax (terrible flag name). Or maybe -i doesn't make sense unless you explicitly specify some root you're interested in?

Hm, do you agree that this is counter-intuitive in some situations? I don't really think this is a showstopper at all, just some thoughts I have from using it.

[ehuss]

I agree that it is confusing at first. The first time I tried cargo tree -i and saw only one item, that was surprising. But I don't know what else it could show that would make sense. The whole point of -i is to show the edges flowing into a given node, and by default it chooses the current member which normally doesn't have any edges flowing into it.

I tried to emphasize in the documentation that -i usually needs -p to be useful.

Contemplating it more… We could change -i to take a package name. Since it is only really useful with a named package, maybe it should emphasize that? So maybe something like this:

• cargo tree -i syn: Show what depends on syn for the current member.
• cargo tree -i syn --workspace: Show what depends on syn across the entire workspace.
• cargo tree -i syn -p cranelift: Show what depends on syn inside the cranelift member.

And then bare cargo tree -i could just print a long help message explaining how to use it (since it is generally useless right now without arguments). I could also probably make cargo tree -p syn -i work for backwards-compatibility.

I could also maybe add a hint to the error if you do cargo tree -i foo when foo isn't in the current member's tree, but exists elsewhere.

WDYT?

[joshtriplett]

This is a minor nit, but I've never seen "dedupe" (singular) as an abbreviation for "deduplicate". I've always seen that written as dedup (which has the advantage of being a prefix of "deduplicate"). Would it be possible to rename the option before this is stabilized? (We could also have an alias.)

[ehuss]

I think an alias would be fine, though I think "dupe" is a well-established shorthand for duplicate. I also see more hits on google for "dedupe" than "dedup". Either is ok with me, though.

[rivy]

This hitting nightly just broke the uutils/coreutils CI build (see https://github.com/rivy/rs.coreutils/runs/587889104?check_suite_focus=true) because of the lack of support for the prior flags (specifically, hard faults for using --all or -V). I'm in the process of moving the CI to use beta for the time being (rust v1.43 is required). But, this took a while to track down and it's a hard, unexpected break for users of cargo-tree.

Maybe it would be better to just deprecate the old options with warnings until this hits at least beta or stable? Or, at least with all the warnings/errors, add some pointer to this PR so that users can more easily figure out the problem and possible solutions, specifically using cargo-tree tree ... as a workaround. That's an odd construction not easily otherwise discoverable without some pointer.

[ehuss]

@rivy Sure, I have opened #8115 to support those two flags.

[rivy]

@ehuss , thank you for the response and the rather quick PR solution!