[WIP] Add rustdoc GUI tests

[GuillaumeGomez]

[TODO]

[Mark-Simulacrum]

Sorry for the delay here, have been quite busy.

[GuillaumeGomez]

Updated: I moved what was needed into bootstrap (so the cloning, the setup and the run). Only kept the required dependencies into the docker file.

[GuillaumeGomez]

Applied the comments, sorry for the long delay.

[Mark-Simulacrum]

Can we just use curl? Looks like the apparent use is for the wget below but that should be fine to just use curl for I imagine?

[GuillaumeGomez]

I guess, I tend to prefer wget because it's simpler but we can use curl.

[GuillaumeGomez]

However, I just copied the code from the chromium repository so I'd prefer not to change it until everything is working so we can use the current code as witness.

[GuillaumeGomez]

Updated.

[Mark-Simulacrum]

Okay, so this is looking relatively good to me -- I guess maybe we should try to test it? We'll want to add a dedicated Makefile target I think, like check-aux but with GUI tests enabled too (perhaps check-aux-and-gui or something). That can be done roughly here:

rust/src/bootstrap/mk/Makefile.in

Lines 43 to 47 in a8cf399

	check-aux:
	$(Q)$(BOOTSTRAP) test \
	src/tools/cargo \
	src/tools/cargotest \
	$(BOOTSTRAP_ARGS)

By adding something like:

check-aux-and-gui: check-aux
	$(Q)$(BOOTSTRAP) test \
        src/test/rustdoc-gui \
		$(BOOTSTRAP_ARGS)

We'll also want to enable this builder on try branch, by applying this diff:

diff --git a/src/ci/azure-pipelines/try.yml b/src/ci/azure-pipelines/try.yml
index 818306a0092..eea08d1f26a 100644
--- a/src/ci/azure-pipelines/try.yml
+++ b/src/ci/azure-pipelines/try.yml
@@ -26,6 +26,7 @@ jobs:
   strategy:
     matrix:
       dist-x86_64-linux: {}
+      x86_64-gnu-aux: {}
 
 # The macOS and Windows builds here are currently disabled due to them not being
 # overly necessary on `try` builds. We also don't actually have anything that

Once that's done we can run bors try here on this PR and see what the results are.

[rust-log-analyzer]

The job x86_64-gnu-aux of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

 finished in 416.838
Build completed successfully in 0:37:19
    Finished dev [unoptimized + debuginfo] target(s) in 0.52s
Cloning into '/checkout/obj/build/browser-UI-test'...
Note: checking out 'e47b1a8697f628429cb684e0b3716737a3a0fa78'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.


If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at e47b1a8... Merge pull request #152 from GuillaumeGomez/up-docker
Cloning into '/checkout/obj/build/test-rust-docs-ui'...
Note: checking out 'd74abb60d1bd1e3fbf647d2d2eb521f2aa357f27'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.


If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d74abb6... Merge pull request #84 from GuillaumeGomez/wait
    Finished release [optimized] target(s) in 0.21s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Could not determine the LLVM submodule commit hash. Assuming that an LLVM rebuild is not necessary.
To force LLVM to rebuild, remove the file `/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/llvm-finished-building`
---
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
HEAD is now at e47b1a8... Merge pull request #152 from GuillaumeGomez/up-docker
HEAD is now at d74abb6... Merge pull request #84 from GuillaumeGomez/wait

> [email protected] install /checkout/obj/build/browser-UI-test/node_modules/puppeteer
> node install.js


Chromium downloaded to /checkout/obj/build/browser-UI-test/node_modules/puppeteer/.local-chromium/linux-722234
added 164 packages from 138 contributors and audited 164 packages in 11.278s
found 14 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details
    Finished release [optimized] target(s) in 0.21s
=> Starting doc-ui tests...

basic-code... ok
basic-code... ok
basic... ok
code-sidebar-toggle-mobile... ok
code-sidebar-toggle... ok
list_code_block... ok
module-mobile... ok
theme-change... FAILED (images "failures/theme-change-test.png" and "theme-change" are different)
toggle-docs... FAILED (images "failures/toggle-docs-test.png" and "toggle-docs" are different)

<= doc-ui tests done: 6 succeeded, 2 failed


command did not execute successfully: "/usr/bin/node" "../browser-UI-test/src/index.js" "--no-sandbox" "--test-folder" "ui-tests" "--failure-folder" "failures" "--variable" "DOC_PATH" "/checkout/obj/build/test-rust-docs-ui/test-docs/doc/test_docs" "--show-text" "--generate-images"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --stage 2 src/test/rustdoc-gui
Build completed unsuccessfully in 0:00:38
Build completed unsuccessfully in 0:00:38
make: *** [check-aux-and-gui] Error 1
Makefile:49: recipe for target 'check-aux-and-gui' failed
  local time: Wed Aug 26 16:55:48 UTC 2020
  network time: Wed, 26 Aug 2020 16:55:48 GMT
== end clock drift check ==
== end clock drift check ==
##[error]Process completed with exit code 2.
Terminate orphan process: pid (3953) (python)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

[bors]

💔 Test failed - checks-actions

[GuillaumeGomez]

So at this point, only 2 tests are failing, however I need to be able to get the images to understand why. More globally, we need to provide an easy way for contributors to get the images to see what went wrong. cc @rust-lang/infra

[Mark-Simulacrum]

I still am uncertain what exactly you mean by images. Is that the docker images? A model like the one I proposed here might be necessary: rust-lang/triagebot#760

[jyn514]

I still am uncertain what exactly you mean by images. Is that the docker images? A model like the one I proposed here might be necessary: rust-lang/triagebot#760

Images like PNGs. https://github.com/GuillaumeGomez/test-rust-docs-ui/tree/master/ui-tests

[GuillaumeGomez]

The images generated in case of failures (the PNG screenshots).

[Mark-Simulacrum]

Yeah, then I think the triagebot issue I linked to is the best bet for the most up to date mentor-ish issue on integrating retrieval of artifacts into CI. I would comment there to sync up with @lcnr though who expressed interest in working on this.

[GuillaumeGomez]

Fine by me. In the meantime, I'd be really interested into being able to know where the images are (no need for automation on this part I hope) so I can at least move forward on this PR. :)

[Mark-Simulacrum]

I am confused - you believe you have uploaded them? We don't upload by default from non dist builders at all...

[GuillaumeGomez]

Oh snap... Any idea on how we could get access to the images generated then?

[Mark-Simulacrum]

...I have already pointed you at an issue typed up with docs on how I would suggest going about doing this? If something in that description is unclear, please comment on that issue with questions. It doesn't require triagebot integration, just work in CI here, for basically all of the steps.

[GuillaumeGomez]

I misundestood the issue then... Going to re-read it.

[bors]

☔ The latest upstream changes (presumably #77294) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

[Mark-Simulacrum]

I'm going to close this for now to keep the list of PRs assigned to me cleaner, but feel free to reopen when you have more time to work on it!