Implement --extern-location
#72603
Merged
Implement --extern-location
#72603
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
(rust_highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
When I saw |
|
@tshepang Yes, I have a tendency to over-abbr. |
jsgf
force-pushed
the
extern-loc
branch
2 times, most recently
from
9c0c9fe
to
bd9b98c
Compare
This comment has been minimized.
This comment has been minimized.
jsgf
force-pushed
the
extern-loc
branch
2 times, most recently
from
2951ba5
to
bf1c504
Compare
This comment has been minimized.
This comment has been minimized.
|
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
petrochenkov
added
T-cargo
|
This is great, thank you. I'll leave a few notes and questions in comments below. |
Note 1 - MotivationThis is useful not only for the All "cannot find or load crate" errors, name ambiguity errors during resolution, lint for private dependencies (#44663), perhaps something else. |
Note 2 - Dependency trackingWhen file is loaded into source map during conversion into a span that file also becomes a dependency for the build. Looks like PR puts the files specified in Lazy loading may be inconvenient in some cases when we want to work with spans uniformly, though. |
Note 3 - Relative pathIf the path in Will cargo and other tools use absolute or relative paths in practice? |
Note 4 - Command line length compressionSomewhat related to note 3. If tools start using absolute paths for In practice all the paths from cargo will refer to the same |
Note 5 - Over-engineeringAre 4 types of extern locations really necessary? |
petrochenkov
added
S-waiting-on-team
|
@petrochenkov Thank you very much for taking the time to look at this. 1: Yes, I hadn't really considered other uses, but I'm glad there are some. 2: I was concerned about the Spans becoming part of the state of the build, both as dependencies, and for incremental builds. I wasn't sure if it was desirable or not, or what the effects would be. It sounds like it's not an issue, and even desirable. 3: I think the path should be relative to 4: I was a little concerned about command-line expansion, but ultimately decided that it isn't an issue we need to solve now - it could be added later if experience shows that it's necessary. It's proportional to the number of direct dependencies a crate has, which doesn't seem to be large - I haven't done a systematic survey, but my gut says that it's going to be almost always <50. It would be a different matter if it also affected indirect dependencies. If it is needed, then I'm thinking of some scheme of adding a new option to give short ids to paths, and a new location type to reference them, eg, 5: I have a local change to remove One thing I'd like to highlight is that the |
|
Thinking about raw vs json a bit more - the |
|
✌️ @jsgf can now approve this pull request |
|
☔ The latest upstream changes (presumably #79078) made this pull request unmergeable. Please resolve the merge conflicts. |
This allows a build system to indicate a location in its own dependency
specification files (eg Cargo's `Cargo.toml`) which can be reported
along side any unused crate dependency.
This supports several types of location:
- 'json' - provide some json-structured data, which is included in the json diagnostics
in a `tool_metadata` field
- 'raw' - emit the provided string into the output. This also appears as a json string in
`tool_metadata`.
If no `--extern-location` is explicitly provided then a default json entry of the form
`"tool_metadata":{"name":<cratename>,"path":<cratepath>}` is emitted.
...so we can skip serializing `tool_metadata` if it hasn't been set. This makes the output a bit cleaner, and avoiding having to update a bunch of unrelated tests.
This will make sure the encoder will get updated if any new fields are added to Diagnostic.
|
@bors r+ |
|
📌 Commit 91d8c3b has been approved by |
bors
added
S-waiting-on-bors
|
@bors r=nikomatsakis |
|
💡 This pull request was already approved, no need to approve it again.
|
|
📌 Commit 91d8c3b has been approved by |
|
☀️ Test successful - checks-actions |
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Apr 4, 2021
…crum Add an unstable --json=unused-externs flag to print unused externs This adds an unstable flag to print a list of the extern names not used by cargo. This PR will enable cargo to collect unused dependencies from all units and provide warnings. The companion PR to cargo is: rust-lang/cargo#8437 The goal is eventual stabilization of this flag in rustc as well as in cargo. Discussion of this feature is mostly contained inside these threads: rust-lang#57274 rust-lang#72342 rust-lang#72603 The feature builds upon the internal datastructures added by rust-lang#72342 Externs are uniquely identified by name and the information is sufficient for cargo. If the mode is enabled, rustc will print json messages like: ``` {"unused_extern_names":["byteorder","openssl","webpki"]} ``` For a crate that got passed byteorder, openssl and webpki dependencies but needed none of them. ### Q: Why not pass -Wunused-crate-dependencies? A: See [ehuss's comment here](rust-lang#57274 (comment)) TLDR: it's cleaner. Rust's warning system wasn't built to be filtered or edited by cargo. Even a basic implementation of the feature would have to change the "n warnings emitted" line that rustc prints at the end. Cargo ideally wants to synthesize its own warnings anyways. For example, it would be hard for rustc to emit warnings like "dependency foo is only used by dev targets", suggesting to make it a dev-dependency instead. ### Q: Make rustc emit used or unused externs? A: Emitting used externs has the advantage that it simplifies cargo's collection job. However, emitting unused externs creates less data to be communicated between rustc and cargo. Often you want to paste a cargo command obtained from `cargo build -vv` for doing something completely unrelated. The message is emitted always, even if no warning or error is emitted. At that point, even this tiny difference in "noise" matters. That's why I went with emitting unused externs. ### Q: One json msg per extern or a collective json msg? A: Same as above, the data format should be concise. Having 30 lines for the 30 crates a crate uses would be disturbing to readers. Also it helps the cargo implementation to know that there aren't more unused deps coming. ### Q: Why use names of externs instead of e.g. paths? A: Names are both sufficient as well as neccessary to uniquely identify a passed `--extern` arg. Names are sufficient because you *must* pass a name when passing an `--extern` arg. Passing a path is optional on the other hand so rustc might also figure out a crate's location from the file system. You can also put multiple paths for the same extern name, via e.g. `--extern hello=/usr/lib/hello.rmeta --extern hello=/usr/local/lib/hello.rmeta`, but rustc will only ever use one of those paths. Also, paths don't identify a dependency uniquely as it is possible to have multiple different extern names point to the same path. So paths are ill-suited for identification. ### Q: What about 2015 edition crates? A: They are fully supported. Even on the 2015 edition, an explicit `--extern` flag is is required to enable `extern crate foo;` to work (outside of sysroot crates, which this flag doesn't warn about anyways). So the lint would still fire on 2015 edition crates if you haven't included a dependency specified in Cargo.toml using `extern crate foo;` or similar. The lint won't fire if your sole use in the crate is through a `extern crate foo;` statement, but that's not its job. For detecting unused `extern crate foo` statements, there is the `unused_extern_crates` lint which can be enabled by `#![warn(unused_extern_crates)]` or similar. cc `@jsgf` `@ehuss` `@petrochenkov` `@estebank`
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Apr 4, 2021
…crum Add an unstable --json=unused-externs flag to print unused externs This adds an unstable flag to print a list of the extern names not used by cargo. This PR will enable cargo to collect unused dependencies from all units and provide warnings. The companion PR to cargo is: rust-lang/cargo#8437 The goal is eventual stabilization of this flag in rustc as well as in cargo. Discussion of this feature is mostly contained inside these threads: rust-lang#57274 rust-lang#72342 rust-lang#72603 The feature builds upon the internal datastructures added by rust-lang#72342 Externs are uniquely identified by name and the information is sufficient for cargo. If the mode is enabled, rustc will print json messages like: ``` {"unused_extern_names":["byteorder","openssl","webpki"]} ``` For a crate that got passed byteorder, openssl and webpki dependencies but needed none of them. ### Q: Why not pass -Wunused-crate-dependencies? A: See [ehuss's comment here](rust-lang#57274 (comment)) TLDR: it's cleaner. Rust's warning system wasn't built to be filtered or edited by cargo. Even a basic implementation of the feature would have to change the "n warnings emitted" line that rustc prints at the end. Cargo ideally wants to synthesize its own warnings anyways. For example, it would be hard for rustc to emit warnings like "dependency foo is only used by dev targets", suggesting to make it a dev-dependency instead. ### Q: Make rustc emit used or unused externs? A: Emitting used externs has the advantage that it simplifies cargo's collection job. However, emitting unused externs creates less data to be communicated between rustc and cargo. Often you want to paste a cargo command obtained from `cargo build -vv` for doing something completely unrelated. The message is emitted always, even if no warning or error is emitted. At that point, even this tiny difference in "noise" matters. That's why I went with emitting unused externs. ### Q: One json msg per extern or a collective json msg? A: Same as above, the data format should be concise. Having 30 lines for the 30 crates a crate uses would be disturbing to readers. Also it helps the cargo implementation to know that there aren't more unused deps coming. ### Q: Why use names of externs instead of e.g. paths? A: Names are both sufficient as well as neccessary to uniquely identify a passed `--extern` arg. Names are sufficient because you *must* pass a name when passing an `--extern` arg. Passing a path is optional on the other hand so rustc might also figure out a crate's location from the file system. You can also put multiple paths for the same extern name, via e.g. `--extern hello=/usr/lib/hello.rmeta --extern hello=/usr/local/lib/hello.rmeta`, but rustc will only ever use one of those paths. Also, paths don't identify a dependency uniquely as it is possible to have multiple different extern names point to the same path. So paths are ill-suited for identification. ### Q: What about 2015 edition crates? A: They are fully supported. Even on the 2015 edition, an explicit `--extern` flag is is required to enable `extern crate foo;` to work (outside of sysroot crates, which this flag doesn't warn about anyways). So the lint would still fire on 2015 edition crates if you haven't included a dependency specified in Cargo.toml using `extern crate foo;` or similar. The lint won't fire if your sole use in the crate is through a `extern crate foo;` statement, but that's not its job. For detecting unused `extern crate foo` statements, there is the `unused_extern_crates` lint which can be enabled by `#![warn(unused_extern_crates)]` or similar. cc ``@jsgf`` ``@ehuss`` ``@petrochenkov`` ``@estebank``
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Apr 4, 2021
…crum Add an unstable --json=unused-externs flag to print unused externs This adds an unstable flag to print a list of the extern names not used by cargo. This PR will enable cargo to collect unused dependencies from all units and provide warnings. The companion PR to cargo is: rust-lang/cargo#8437 The goal is eventual stabilization of this flag in rustc as well as in cargo. Discussion of this feature is mostly contained inside these threads: rust-lang#57274 rust-lang#72342 rust-lang#72603 The feature builds upon the internal datastructures added by rust-lang#72342 Externs are uniquely identified by name and the information is sufficient for cargo. If the mode is enabled, rustc will print json messages like: ``` {"unused_extern_names":["byteorder","openssl","webpki"]} ``` For a crate that got passed byteorder, openssl and webpki dependencies but needed none of them. ### Q: Why not pass -Wunused-crate-dependencies? A: See [ehuss's comment here](rust-lang#57274 (comment)) TLDR: it's cleaner. Rust's warning system wasn't built to be filtered or edited by cargo. Even a basic implementation of the feature would have to change the "n warnings emitted" line that rustc prints at the end. Cargo ideally wants to synthesize its own warnings anyways. For example, it would be hard for rustc to emit warnings like "dependency foo is only used by dev targets", suggesting to make it a dev-dependency instead. ### Q: Make rustc emit used or unused externs? A: Emitting used externs has the advantage that it simplifies cargo's collection job. However, emitting unused externs creates less data to be communicated between rustc and cargo. Often you want to paste a cargo command obtained from `cargo build -vv` for doing something completely unrelated. The message is emitted always, even if no warning or error is emitted. At that point, even this tiny difference in "noise" matters. That's why I went with emitting unused externs. ### Q: One json msg per extern or a collective json msg? A: Same as above, the data format should be concise. Having 30 lines for the 30 crates a crate uses would be disturbing to readers. Also it helps the cargo implementation to know that there aren't more unused deps coming. ### Q: Why use names of externs instead of e.g. paths? A: Names are both sufficient as well as neccessary to uniquely identify a passed `--extern` arg. Names are sufficient because you *must* pass a name when passing an `--extern` arg. Passing a path is optional on the other hand so rustc might also figure out a crate's location from the file system. You can also put multiple paths for the same extern name, via e.g. `--extern hello=/usr/lib/hello.rmeta --extern hello=/usr/local/lib/hello.rmeta`, but rustc will only ever use one of those paths. Also, paths don't identify a dependency uniquely as it is possible to have multiple different extern names point to the same path. So paths are ill-suited for identification. ### Q: What about 2015 edition crates? A: They are fully supported. Even on the 2015 edition, an explicit `--extern` flag is is required to enable `extern crate foo;` to work (outside of sysroot crates, which this flag doesn't warn about anyways). So the lint would still fire on 2015 edition crates if you haven't included a dependency specified in Cargo.toml using `extern crate foo;` or similar. The lint won't fire if your sole use in the crate is through a `extern crate foo;` statement, but that's not its job. For detecting unused `extern crate foo` statements, there is the `unused_extern_crates` lint which can be enabled by `#![warn(unused_extern_crates)]` or similar. cc ```@jsgf``` ```@ehuss``` ```@petrochenkov``` ```@estebank```
jsgf
added a commit
to jsgf/rust
that referenced
this pull request
Apr 15, 2022
`--extern-location` was an experiment to investigate the best way to generate useful diagnostics for unused dependency warnings by enabling a build system to identify the corresponding build config. While I did successfully use this, I've since been convinced the alternative `--json unused-externs` mechanism is the way to go, and there's no point in having two mechanisms with basically the same functionality. This effectively reverts rust-lang#72603
Dylan-DPC
added a commit
to Dylan-DPC/rust
that referenced
this pull request
Apr 19, 2022
…dtwco Remove `--extern-location` and all associated code `--extern-location` was an experiment to investigate the best way to generate useful diagnostics for unused dependency warnings by enabling a build system to identify the corresponding build config. While I did successfully use this, I've since been convinced the alternative `--json unused-externs` mechanism is the way to go, and there's no point in having two mechanisms with basically the same functionality. This effectively reverts rust-lang#72603
Dylan-DPC
added a commit
to Dylan-DPC/rust
that referenced
this pull request
Apr 19, 2022
…dtwco Remove `--extern-location` and all associated code `--extern-location` was an experiment to investigate the best way to generate useful diagnostics for unused dependency warnings by enabling a build system to identify the corresponding build config. While I did successfully use this, I've since been convinced the alternative `--json unused-externs` mechanism is the way to go, and there's no point in having two mechanisms with basically the same functionality. This effectively reverts rust-lang#72603
Dylan-DPC
added a commit
to Dylan-DPC/rust
that referenced
this pull request
Apr 19, 2022
…dtwco Remove `--extern-location` and all associated code `--extern-location` was an experiment to investigate the best way to generate useful diagnostics for unused dependency warnings by enabling a build system to identify the corresponding build config. While I did successfully use this, I've since been convinced the alternative `--json unused-externs` mechanism is the way to go, and there's no point in having two mechanisms with basically the same functionality. This effectively reverts rust-lang#72603
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements
--extern-locationas a followup to #72342 as part of the implementation of #57274. The goal of this PR is to allow rustc, in coordination with the build system, to present a useful diagnostic about how to remove an unnecessary dependency from a dependency specification file (eg Cargo.toml).EDIT: Updated to current PR state.
The location is specified for each named crate - that is, for a given
--extern foo[=path]there can also be--extern-location foo=<location>. It supportsthreetwo styles of location:~~1.
--extern-location foo=file:<path>:<line>- a file path and line specification--extern-location foo=span:<path>:<start>:<end>- a span specified as a file and start and end byte offsets~~--extern-location foo=raw:<anything>- a raw string which is included in the output--extern-location foo=json:<anything>- an arbitrary Json structure which is emitted via Json diagnostics in atool_metadatafield.1 & 2 are turned into an internalSpan, so long as the path exists and is readable, and the location is meaningful (within the file, etc). This is used as theSpanfor a fix suggestion which is reported like other fix suggestions.rawandjsonare for the case where the location isn't best expressed as a file and location within that file. For example, it could be a rule name and the name of a dependency within that rule.rustcmakes no attempt to parse the raw string, and simply includes it in the output diagnostic text.jsonis only included in json diagnostics.rawis emitted as text and also as a json string intool_metadata.If no
--extern-locationoption is specified then it will emit a default json structure consisting of{"name": name, "path": path}corresponding to the name and path in--extern name=path.This is a prototype/RFC to make some of the earlier conversations more concrete. It doesn't stand on its own - it's only useful if implemented by Cargo and other build systems. There's also a ton of implementation details which I'd appreciate a second eye on as well.
NOTE The first commit in this PR is #72342 and should be ignored for the purposes of review. The first commit is a very simplistic implementation which is basically raw-only, presented as a MVP. The second implements the full thing, and subsequent commits are incremental fixes.cc @ehuss @est31 @petrochenkov @estebank