Host compiler documentation #49193

davidtwco · 2018-03-20T02:09:17Z

Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40

davidtwco · 2018-03-20T02:11:00Z

I'm expecting that this will have some issues (particularly in the naming of some of the steps) and I didn't have an opportunity to throughly test the dist-x86_64-linux builder changes but I wanted to get a PR open for feedback.

alexcrichton

Thanks! Have you tried running this locally as well to ensure the tarball is generated correctly?

alexcrichton · 2018-03-20T14:37:04Z

src/ci/docker/dist-x86_64-linux/Dockerfile

@@ -84,7 +84,8 @@ ENV HOSTS=x86_64-unknown-linux-gnu
 ENV RUST_CONFIGURE_ARGS \
      --enable-full-tools \
      --enable-sanitizers \
-      --enable-profiler
+      --enable-profiler \
+      --enable-compiler-args


s/args/docs/

I think?

Oops, I think you're right, will change this.

alexcrichton · 2018-03-20T14:37:56Z

src/bootstrap/doc.rs

@@ -620,19 +625,86 @@ impl Step for Rustc {
        let mut cargo = builder.cargo(compiler, Mode::Librustc, target, "doc");
        compile::rustc_cargo(build, &mut cargo);

-        if build.config.compiler_docs {


Could you expand a bit on how come this step was separated? I'd naively expect it to be ok if it continued to just be one step that internally dispatched to all crates or just proc_macro

When the standard documentation is being generated - this is done by merging the standard library, test and procedural macro documentation (normally, the compiler_docs flag would add everything else here too). This was achieved through symlinking docs to a shared folder for those three steps.

In order to produce the compiler documentation such that we would still have standard documentation separately which contains the same crates as it did pre-PR, I had to ensure that the symlink directory was different when generating all the compiler documentation. This is because the distribution step would a copy from that generated documentation and that would contain the compiler documentation too if it wasn't generated into somewhere else (this is because all documentation steps happen before dist steps).

(It occurs to me now when writing this that since each doc step copies from the shared symlink directory to its output directory at the end of each step that we could still use the same shared folder which might speed things up. This is because the dist step for standard documentation will use the folder that was copied before compiler documentation was added - will try this). Done this now.

I assumed that we'd want the standard documentation to still include the proc_macro crate and so I kept one step that would generate that into the shared location and created another that would generate the compiler documentation into the separate location. These two locations then produce the two tarballs in the dist step. These could be one step that just called to cargo doc twice.

Updated that commit to contain the improvement I noticed in the above comment.

kennytm · 2018-03-20T15:49:38Z

src/bootstrap/doc.rs

+        // Like with libstd above if compiler docs aren't enabled then we're not
+        // documenting internal dependencies, so we have a whitelist.
+        cargo.arg("--no-deps");
+        for krate in &["proc_macro"] {


... why? 😕

Currently, the documentation on doc.rust-lang.org contains the standard library, test and proc_macro. This step generates the proc_macro documentation into the same folder that already contains the standard library and test documentation. At the end of this step, this documentation is copied into the doc folder and joined by the various books/error index/etc. - this is what produces the tarball and the contents match the contents before this PR.

After the above, we add the compiler crates in another step to the original folder (not the doc folder it was copied into) and then make a new compiler-doc folder from that. This produces the compiler docs tarball.

In order to actually perform the above, I modified the existing step that produced compiler documentation so that it would only include the docs (ie. only proc_macro for the original tarball; and added a new step that produced the rest.

Does this clarify why?

@davidtwco Sorry, let me clarify.

Why iterate on an array with a single element when you could just write cargo.arg("-p").arg("proc_macro") directly? The step name doesn't suggest you'll need to extend this array.

Before this PR, when compiler docs were disabled, a whitelist of crates from the compiler were documented for the standard documentation. This was clear from the context and that it was an array. However, as you point out, that context has been lost somewhat.

I had mentioned in my initial comment that I wasn't sure about the name of this step. I named it ProcMacro just so that it had a name for me to work with, in practice it should be PublicRustc or WhitelistRustc or something like that since it could contain more crates that we choose to include in the standard available documentation.

I'm not sure what name is best for this step but I'll change it and clarify the point of it in a comment.

Modified that commit to clarify this.

alexcrichton · 2018-03-21T14:18:14Z

Ok thanks for the explanations @davidtwco! This looks good to me with all that in mind, so let's merge!

Before that though let's give this a spin to make sure it'll work

@bors: try

bors · 2018-03-21T14:18:25Z

⌛ Trying commit 79d0572 with merge b2c6936...

@alexcrichton

Host compiler documentation Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

bors · 2018-03-21T17:18:35Z

💔 Test failed - status-travis

davidtwco · 2018-03-21T17:25:02Z

@alexcrichton I'm not sure that the error on Travis is related to the contents of this PR?

alexcrichton · 2018-03-21T17:28:26Z

@davidtwco

The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over).

It looks like we're uploading a bit too much? The raw HTML files look to be getting uploaded instead of just the tarballs

davidtwco · 2018-03-21T18:00:56Z

@alexcrichton Ah, I saw the following and assumed that was the issue:

[01:37:57] Unable to build RLS, skipping dist

Locally, I was seeing the folder structure with various tar archives for the documentation and compiler, etc. But I was also seeing a folder with the raw documentation for both doc and rustc-doc - that was being produced by this code:

rust/src/bootstrap/dist.rs

Lines 123 to 129 in 1e73c1d

    
           // As part of this step, *also* copy the docs directory to a directory which 
        
           // buildbot typically uploads. 
        
           if host == build.build { 
        
               let dst = distdir(build).join("doc").join(build.rust_package_vers()); 
        
               t!(fs::create_dir_all(&dst)); 
        
               cp_r(&src, &dst); 
        
           }

The above is from the existing Docs step from dist.rs on master, it is the same logic (it changes the directory name where appropriate) in both steps in this PR.

So I assume that it is intended that we produce regular directories alongside the tarballs? Perhaps that will need to change if the rustc-doc folder is too large?

Also, I've noticed another issue locally - it seems that while the source directories for the archives have the files I expected in them, the actual archives somehow combine the files. I hadn't checked the archives before (my bad), just looked at the source directories all the way through the dist process and assumed the taring would be fine.

alexcrichton · 2018-03-21T18:50:50Z

Hm yeah I'm not really sure what's up? In any case that's a very outdated comment this point, we only produce/upload tarballs from the builders nowadays

…tion

davidtwco · 2018-03-21T19:03:48Z

I've pushed modified commits that remove that section since it doesn't seem like we need it. I think I've also fixed the other issue I'd noticed.

alexcrichton · 2018-03-21T22:10:06Z

@bors: try

bors · 2018-03-21T22:10:16Z

⌛ Trying commit 1392179 with merge 00ba863...

@alexcrichton

Host compiler documentation Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

bors · 2018-03-21T22:47:15Z

💔 Test failed - status-travis

kennytm · 2018-03-22T10:20:28Z

@bors retry

bors · 2018-03-22T10:20:38Z

⌛ Trying commit 1392179 with merge 8ad3c68...

@alexcrichton

Host compiler documentation Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

bors · 2018-03-22T13:01:20Z

☀️ Test successful - status-travis
State: approved= try=True

davidtwco · 2018-03-22T20:50:01Z

After chatting with @alexcrichton on Gitter, we've decided to remove the std and test documentation from the compiler tarball. Pushed a commit that does this.

alexcrichton · 2018-03-22T21:38:55Z

@bors: try

bors · 2018-03-22T21:39:05Z

⌛ Trying commit 73fa6d5 with merge fd9b5ca0fa489432dfb0fa200555316a5ef04a03...

bors · 2018-03-22T23:23:21Z

☀️ Test successful - status-travis
State: approved= try=True

alexcrichton · 2018-03-23T01:31:55Z

@bors: try

alexcrichton · 2018-03-23T01:32:10Z

er, wrong pr...

alexcrichton · 2018-03-23T01:33:18Z

@bors: r+

Ok looks great!

bors · 2018-03-23T01:33:19Z

📌 Commit 73fa6d5 has been approved by alexcrichton

@alexcrichton

Host compiler documentation Fixes rust-lang#29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

@alexcrichton

Host compiler documentation Fixes rust-lang#29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

@alexcrichton

Host compiler documentation Fixes rust-lang#29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

@alexcrichton

Host compiler documentation Fixes rust-lang#29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

Rollup of 21 pull requests - Successful merges: #49046, #49076, #49089, #49120, #49121, #49122, #49162, #49193, #49194, #49203, #49229, #49235, #49254, #49268, #49273, #49274, #49290, #49312, #49314, #49318, #49299

rust-highfive assigned alexcrichton Mar 20, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 20, 2018

alexcrichton reviewed Mar 20, 2018

View reviewed changes

davidtwco force-pushed the issue-29893 branch from eb29fae to feb116f Compare March 20, 2018 15:46

kennytm reviewed Mar 20, 2018

View reviewed changes

davidtwco mentioned this pull request Mar 20, 2018

Host compiler documentation rust-lang/rust-central-station#40

Merged

davidtwco force-pushed the issue-29893 branch from feb116f to 79d0572 Compare March 20, 2018 17:40

bors added a commit that referenced this pull request Mar 21, 2018

Auto merge of #49193 - davidtwco:issue-29893, r=<try>

b2c6936

Host compiler documentation Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 21, 2018

davidtwco added 2 commits March 21, 2018 19:02

Add support to rustbuild for a 'rustc docs' component tarball

178652a

Configure the dist-x86_64-linux builder to produce compiler documenta…

1392179

…tion

davidtwco force-pushed the issue-29893 branch from 79d0572 to 1392179 Compare March 21, 2018 19:03

bors added a commit that referenced this pull request Mar 21, 2018

Auto merge of #49193 - davidtwco:issue-29893, r=<try>

00ba863

Host compiler documentation Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

bors added a commit that referenced this pull request Mar 22, 2018

Auto merge of #49193 - davidtwco:issue-29893, r=<try>

8ad3c68

Host compiler documentation Fixes #29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 23, 2018

frewsxcv mentioned this pull request Mar 23, 2018

Rollup of 12 pull requests #49302

Closed

alexcrichton mentioned this pull request Mar 23, 2018

Rollup of 20 pull requests #49317

Closed

kennytm mentioned this pull request Mar 24, 2018

Rollup of 21 pull requests #49325

Closed

kennytm added a commit to kennytm/rust that referenced this pull request Mar 24, 2018

Rollup merge of rust-lang#49193 - davidtwco:issue-29893, r=alexcrichton

22447cf

Host compiler documentation Fixes rust-lang#29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

kennytm added a commit to kennytm/rust that referenced this pull request Mar 24, 2018

Rollup merge of rust-lang#49193 - davidtwco:issue-29893, r=alexcrichton

311814a

Host compiler documentation Fixes rust-lang#29893. Rust Central Station PR: rust-lang/rust-central-station#40 r? @alexcrichton

kennytm mentioned this pull request Mar 24, 2018

Rollup of 21 pull requests #49337

Merged

bors merged commit 73fa6d5 into rust-lang:master Mar 24, 2018

davidtwco deleted the issue-29893 branch March 25, 2018 11:17

ollie27 mentioned this pull request Mar 27, 2018

Term and getopt crates show in the standard library documentation #49388

Closed

QuietMisdreavus mentioned this pull request Apr 16, 2018

include librustdoc in the distributed compiler docs #50004

Closed

kennytm mentioned this pull request Aug 24, 2018

Add docs for codegen_llvm to d.r-l.o/nightly/nightly-rustc #51737

Closed

jyn514 mentioned this pull request Feb 22, 2023

rust-docs and rustc-docs components conflict at share/doc/rust/html/rustc #108535

Closed

Host compiler documentation #49193

Host compiler documentation #49193

Conversation

davidtwco commented Mar 20, 2018 • edited Loading

davidtwco commented Mar 20, 2018

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidtwco Mar 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Mar 21, 2018

bors commented Mar 21, 2018

bors commented Mar 21, 2018

davidtwco commented Mar 21, 2018

alexcrichton commented Mar 21, 2018

davidtwco commented Mar 21, 2018 • edited Loading

alexcrichton commented Mar 21, 2018

davidtwco commented Mar 21, 2018

alexcrichton commented Mar 21, 2018

bors commented Mar 21, 2018

bors commented Mar 21, 2018

kennytm commented Mar 22, 2018

bors commented Mar 22, 2018

bors commented Mar 22, 2018

davidtwco commented Mar 22, 2018

alexcrichton commented Mar 22, 2018

bors commented Mar 22, 2018

bors commented Mar 22, 2018

alexcrichton commented Mar 23, 2018

alexcrichton commented Mar 23, 2018

alexcrichton commented Mar 23, 2018

bors commented Mar 23, 2018

davidtwco commented Mar 20, 2018 •

edited

Loading

davidtwco Mar 20, 2018 •

edited

Loading

davidtwco commented Mar 21, 2018 •

edited

Loading