document testing for broader set of targets

[pnkfelix]

We need to document the recommended way to go about testing for broader set of targets (e.g. asmjs) other than just throwing it at bors and hoping it works.

Spawned off of T-compiler meeting discussion of rust-lang/rust#54223

Some choice quotes:

"I was recommended changing the .travis.yml to enable other platforms temporarily for the PR testing instead of queuing for bors each time for the other platforms." -- davidtwco

• "you can manually fiddle with .travis.yml if you want to enable more, just don't enable all at once (go 1-2 at a time)" -- simulacrum

Examples of PR's that have been doing the "throw it at bors and hope" dance: PR rust-lang/rust#54004, PR rust-lang/rust#54223

[tromey]

In the case of #54004 it was not a a documentation problem but a problem reproducing the failure locally - the docker setup did not work here, and only after many tries (and I still don't really know what the key change was) did I manage to get a local failure. Possibly it would have helped if the various ci scripts didn't hard-code paths, but even then I am not so sure it would have helped.

[pnkfelix]

So what I just tried doing was:

1. opening the .travis.yml
2. Near the top of the file, there's a directive like:

    # Images used in testing PR and try-build should be run first.
    - env: IMAGE=x86_64-gnu-llvm-5.0 RUST_BACKTRACE=1
      if: type = pull_request OR branch = auto

3. Looking for the relevant target by skimming the env: IMAGE=foo-bar-baz lines which look like:

    - env: IMAGE=asmjs
      if: branch = auto

4. In my own experiment, I swapped the image directives, so that the one at the top said IMAGE=asmjs (above the if: type = pull_request ...) and the x86_64-gnu-llvm-5.0 was now where asmjs previous was. I did it this way as an experiment just because I saw that as the thing that was least likely to cause any problems for travis.

I suspect that the "right" thing to do, in terms of recommended best practices here, is not to worry about swapping, but instead to just change if: branch = auto to if: type = pull_request for the one or two additional targets that you want to pull into the mix. I haven't tested whether that works yet, but the next time I need to do this, I will try it.

---

Also, one might want to disable fail_fast on the testing of such runs, since its (probably?) more important to get an account of as many failures as possible, rather than trying to get a result back quickly... Unless turning off fail_fast would end up being a big drain on resource...?

[pnkfelix]

@tromey the claim I am making is that if we provided docs for how to leverage the CI to assist with more targeted testing of specific targets, then you wouldn't need to reproduce the setup locally...

[tromey]

Aha, ok, thanks. Someone did give me the advice about how to tweak .travis.yml to run just the configuration I wanted on a push -- it was as you say above, adding type = pull_request. I also removed all the other configs to avoid any excess server time. The main thing that helped in the end was figuring out how to hack llvm to build with -g1, to get a stack trace with line numbers. Full -g was not doable due to lack of disk space.

[pnkfelix]

I also removed all the other configs to avoid any excess server time.

ah, yes, so best practices might to cut out all but one of the IMAGE directives in the .travis.yml? I'm pretty ignorant of Travis so I'm largely making this up as we go almong.

[tromey]

ah, yes, so best practices might to cut out all but one of the IMAGE directives in the .travis.yml? I'm pretty ignorant of Travis so I'm largely making this up as we go almong.

I also know little about Travis. Cutting it down to just the one I was interested in just seemed like a good idea, to avoid spending resources when not needed.

[pnkfelix]

But now that I think about it further: It shouldn't cost anything to have those other directives, as long as you don't r+ or bors try the PR with them engaged... because the only thing that should run is the single IMAGE that has if: type = pull_request, right?

[Mark-Simulacrum]

The easiest thing (and this is also fine for Travis) is to simply comment out the if auto line, making no other changes. That will run said builder, and doesn't impact other builders.

[pnkfelix]

Thanks @Mark-Simulacrum !

[ehuss]

FWIW, there are now two chapters related to this:

• Testing with Docker shows how to locally build and run the same as CI. Of course this doesn't cover macOS or Windows testing, which can be hard to replicate.
• Testing with CI covers how to use CI to run tests.

I realize I probably should have added some information for simulating CI locally without Docker (#447).

[jieyouxu]

Triage:

• Jakub implemented arbitrary try jobs and documented in https://rustc-dev-guide.rust-lang.org/tests/ci.html#try-builds. This allows running some cross-compiled tests as try builds.
• Nothing beats a rollup=iffy (if it might fail) or rollup=never (if it will fail) full build because that will include all currently exercised CI jobs, some you might not have ever considered.