Tracking issue for -Zrustdoc-map

[ehuss]

Original issue: #6279
Implementation: #8287
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#rustdoc-map
Issues: Z-rustdoc-map Nightly: rustdoc-map

Summary
This feature adds the ability for Cargo to pass external documentation mappings to rustdoc, so that links to dependencies will use those external sites (like docs.rs). See the documentation for details on how to use it.

Unresolved issues

• Does not properly handle renamed dependencies. The links rustdoc generates are to the package name, not the renamed name. Cargo is written to pass in package names, but if there are multiple dependencies to the same package, it won't work properly.
• If there are multiple versions of the same package within the dep graph, rustdoc will only link to one of them. To fix this, Cargo would need to pass metadata info into rustdoc (such as the package version).
• If a dependency is built with different features than what is on docs.rs, some links may break. This is probably unsolvable.
• This increases the command-line length significantly. Before stabilizing, we may want to consider addressing that. I'm not sure if it would make sense to change rustdoc's interface, or to use response files?
• Decide if Cargo should pass mappings for transitive dependencies (currently it does not). This normally isn't an issue, but can arise for re-exports (see the alt_registry test for an example). I'm not sure if this is a bug in rustdoc or not (there is a large number of issues regarding reexports and rustdoc). Cargo could include these, but this would make the command-line length even longer. Not sure what to do here.
• The config value does not support being set via environment variables (like CARGO_CONFIG_DOC_EXTERN_MAP_REGISTRIES_CRATES_IO="…"). This would be very difficult to support, because Cargo doesn't retain the registry name in SourceId. I looked into fixing that, but it is very difficult, and hard to make it reliable. I would lean towards not bothering with this (it has similar issues as the source table).
• Add a config option to control the default deps behavior of cargo doc. I can think of 3 options: all-deps, no-deps, direct-deps only. I think this would be useful, otherwise to use the rustdoc-map, one would always need to pass --no-deps. This would probably also need a new command-line flag to override the config option (maybe a generalization of --no-deps).
• Consider changing the defaults. This is an important part of the UX story.
  • Default to crates-io = "https://docs.rs/". I think this should be relatively safe, since it only enhances the current behavior (people who use --no-deps will now have links instead of no links in their docs). DONE: Set docs.rs as the default extern-map for crates.io #8877
  • Change default of the deps behavior of cargo doc. I would prefer to get more community feedback to see what makes sense. Two options:
    • Direct dependencies only. I think this option is more realistic. It retains local browsing and indexing, and removes transitive dependencies which are usually not interesting. This would not engage rustdoc-map, so this is somewhat orthogonal.
    • Default to --no-deps. This would engage rustdoc-map, but is a much more drastic change. The user would not have local indexing available.
• Should there be some way for the crate author to control the link destination? The documentation URL set in Cargo.toml is never consulted. I think this is unlikely to work with rustdoc, because it expects a certain structure to the API docs, and documentation URLs often point to user guides, or other things that aren't API docs. The #![doc(html_root_url = "...")] attribute is overridden by the --extern-html-root-url flag. This may be a good or bad thing depending on your perspective (should the user have control, or should the crate author?).

Future extensions
These things can be done after stabilization, but should be considered to ensure this feature doesn't make them more difficult.

• Single tab browsing. This would be a mode where the std docs are merged with the local crate's docs so that the std docs are shown in the same place (and included in the index). This could be expressed with something like doc.extern-map.std = "include" or something like that. (Or maybe just use build-std?) Support rustdoc root-url feature #6279 (comment)
• Direct-dependencies only. Often transitive dependencies aren't that interesting, and take up a lot of space in the output, and clog the search index. Some users want the ability to (locally) document their package + direct dependencies only. I think this could be implemented with some kind of command-line flag, perhaps with a config setting in the [doc] table. --extern-html-root-url flag will automatically handle second-level dependencies. cargo doc only direct dependencies #2025 feature request: build docs for dependencies only #2801 Is there a way to run cargo doc and only get the documentation for direct dependencies of my project?  #6421 https://crates.io/crates/cargo-makedocs
• Manual-exclusions. Sometimes there are specific dependencies that are very expensive to document locally, but you still want everything else. I think this could be implemented with a command-line flag (--exclude winapi?), and the rustdoc-map feature would automatically link those excluded crates' items to docs.rs. This could also be added to the [doc] table. Allow skipping of specific dependencies in 'cargo doc' #4049

[jyn514]

Decide if Cargo should pass mappings for transitive dependencies (currently it does not). This normally isn't an issue, but can arise for re-exports (see the alt_registry test for an example). I'm not sure if this is a bug in rustdoc or not (there is a large number of issues regarding reexports and rustdoc). Cargo could include these, but this would make the command-line length even longer. Not sure what to do here.

There's another example at https://docs.rs/bevy_tiled_prototype/0.1.0/bevy_tiled_prototype/struct.TileMapChunk.html, which has broken links to Byteable and RenderResource, which come from bevy_render, re-exported through bevy: https://docs.rs/bevy/0.3.0/bevy/render/pass/trait.RenderPass.html.

I think this isn't a rustdoc bug because it has neither a documentation cache in target/doc, nor an --extern-html-root-url. So it doesn't know where to generate the links. In an ideal world, cargo would pass --extern-html-root-url only for transitive dependencies that have types visible in the direct dependencies, but I don't think cargo has a way to find out that info. Maybe rustc could expose it?

[jyn514]

This increases the command-line length significantly. Before stabilizing, we may want to consider addressing that. I'm not sure if it would make sense to change rustdoc's interface, or to use response files?

Rustdoc has response files since the latest nightly: rust-lang/rust#82261

[jyn514]

If there are multiple versions of the same package within the dep graph, rustdoc will only link to one of them. To fix this, Cargo would need to pass metadata info into rustdoc (such as the package version).

See also rust-lang/rust#78909, which is waiting on review.

[jyn514]

The #![doc(html_root_url = "...")] attribute is overridden by the --extern-html-root-url flag. This may be a good or bad thing depending on your perspective (should the user have control, or should the crate author?).

I opened rust-lang/rust#82776 changing this; in practice, it should only hurt docs.rs (and I've asked the team to sign off before merging) and it gives crate authors more flexibility to point the links at their own site if they want to.

[jyn514]

This increases the command-line length significantly. Before stabilizing, we may want to consider addressing that. I'm not sure if it would make sense to change rustdoc's interface, or to use response files?

FWIW, rustdoc supports response files since 1.52: rust-lang/rust#82261

[jyn514]

Decide if Cargo should pass mappings for transitive dependencies (currently it does not). This normally isn't an issue, but can arise for re-exports (see the alt_registry test for an example). I'm not sure if this is a bug in rustdoc or not (there is a large number of issues regarding reexports and rustdoc). Cargo could include these, but this would make the command-line length even longer. Not sure what to do here.

I don't think this is a rustdoc bug, see #8296 (comment).

The way cargo manages this for normal dependencies is by passing -L dependency=target/debug/deps, which only needs a single flag for all dependencies. I wonder if we can do something like that here? Add some --extern-html-root-url-format=https://docs.rs/{package_name}/{version}/x86_64-unknown-linux-gnu flag to rustdoc? it would need to account for host vs target deps (probably just proc macros) somehow, but that seems doable, and it would avoid passing hundreds more flags than cargo currently passes.

[Nemo157]

s/crate_name/package_name, other than that that sounds like a good approach if the package name is recorded in the .rmeta for rustdoc to use (or if it's plausible to add it).

[jyn514]

Currently it's not recorded, but I think it's plausible to add :) cargo already sets it through CARGO_PKG_NAME so we just need to change rustc to add it to metadata.

[xeniarose]

I'm interested in being able to build a fully offline documentation tree for rust development. Does this feature currently work? doc.extern-map.std doesn't seem to do what I expect according to the docs

example:

mkdir demo
cd demo
cat >rust-toolchain.toml <<EOF
[toolchain]
channel = "nightly-2024-11-25"
components = [ "rust-std", "rustc", "rust-docs"  ]
targets = [ "x86_64-unknown-linux-gnu" ]
profile = "minimal"
EOF

cargo init --lib .

mkdir .cargo
cat >.cargo/config.toml <<EOF
[doc.extern-map]
std = "local"
EOF

cargo doc --no-deps -Zrustdoc-map -Zunstable-options

this creates documentation for the default template library project, which as an example contains an add function. if you open the docs for add and click on eg u64 it goes to doc.rust-lang.org instead of the local copy of std docs

running strace as:

strace -ff -v -s 1024 cargo doc --no-deps -Zrustdoc-map -Zunstable-options 2>&1 | grep execve > doc.log

confirms that --extern-html-root-url with the correct file paths is being passed to the rustdoc process, however the option seems to be getting ignored

has the implementation of this feature changed from what the docs currently state? am i just missing something?

[Nemo157]

AFAIK this feature hasn't changed, but rustdoc now prefers the #![doc(html_root_url)] in the crate over what is specified on the command line, you need to use --extern-html-root-takes-precedence to revert this to the previous behavior.

cargo rustdoc --config='doc.extern-map.std="local"' -Zrustdoc-map -Zunstable-options -- --extern-html-root-takes-precedence -Zunstable-options