Avoid calling rustup in tests

[oxalica]

I'm on NixOS and use nixpkgs-mozilla to setup cargo, rustc and other components, without rustup.

cargo test in rust-analyzer results in 3 failure for me:

• cli::check_code_formatting
• cli::generated_assists_are_fresh
• cli::generated_grammar_is_fresh

They are all trying to invoke rustup run stable -- ... and then rustup toolchain install stable. Obviously, they failed because there is no rustup.
But I already have cargo and rustfmt installed!

Why not invoking cargo ... or rustfmt ... directly? Since we are expecting stable, I think it should not lead to problem even if nightly components are invoked.

[lnicola]

See also #3243 for a fun issue (the tests run in parallel, but rustup is racy). It's supposed to make things easier for contributors, but I suspect that a "please install the stable toolchain" panic would be nicer.

Why not invoking cargo ... or rustfmt ... directly? Since we are expecting stable, I think it should not lead to problem even if nightly components are invoked.

It can. I use nightly and there were formatting differences between nightly and stable rustfmt (because of bugs, nightly was clearly better). So I ran cargo fmt before commit, but CI failed with a nonsensical formatting diff.

EDIT: but yes, running cargo +stable instead of rustup run stable -- cargo might help with your issue.

[bjorn3]

cargo +stable is also a rustup feature.

[oxalica]

@lnicola

See also #3243 for a fun issue.

I also think installing toolchains/components during tests is a bad idea due to the side effects.
Panicking should be better.

We can simply check to refuse nightly rustfmt and directly invoke rustfmt/cargo in tests.

> nix run nixpkgs.latest.rustChannels.nightly.rustfmt-preview -c rustfmt --version
rustfmt 1.4.14-nightly (a5cb5d2 2020-04-14)
> nix run nixpkgs.latest.rustChannels.stable.rustfmt-preview -c rustfmt --version
rustfmt 1.4.12-stable (a828ffe 2020-03-11)

If this is acceptable, I'm willing to implement it.

[flodiebold]

Just running rustfmt --version would break the tests for users who have stable installed, but nightly as default. That could be solved by a rust-toolchain file though, I guess 🤔

[oxalica]

@flodiebold
How about setting RUSTUP_TOOLCHAIN=stable to run cargo fmt? This works in both rustup and bare installations (the latter one ignores it, so we still need version check).

[flodiebold]

Sounds plausible...

[flodiebold]

Regarding the installation, my guess is it's a holdover from when we were using a specific beta version for formatting instead of just stable. For stable, it seems fine to me to bail if it's not installed...

[polyzen]

Still getting the test failures with rust-analyzer 2020-05-04 and rustfmt provided by the Arch Linux rust package:

> rustfmt --version
rustfmt 1.4.12-
thread 'cli::check_code_formatting' panicked at 'Failed to run rustfmt from toolchain 'stable'. ...

[polyzen]

Bump @oxalica @matklad

[matklad]

@polyzen rust-analyzer test suite is guaranteed to work only in rust-analyzer repo, we ship the binary, we don't ship test suite. PRs to improve testing in downstreams packagers are welcome though.