Run tests on all of our crate dependencies

[repi]

It would be useful and good to be able to ourselves run unit tests for all the crates that we are using (currently 478 crates!) and verify they pass on all platforms we are targeting.

This becomes extra important as we bring up new platforms (related: #18) and environments where 3rd party crates have never run and most likely don't have access to run on (consoles).

This is likely a fairly large undertaking to make robust, fast, and fully useful in practice across a wider set of platforms. So is unlikely a task for 2019 but something that we would like to eventually have in some capacity.

Requirements

One would want this to work transitively so the crate dependencies of our crate dependencies get tested as well. And that the exact versions we are using are tested (including if using multiple versions of the same crate). Easiest here is likely to simple test all crates that are in the Cargo.lock file for our repos, the metadata section does include a flat list of exact versions:

[metadata]
"checksum MacTypes-sys 2.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "eaf9f0d0b1cc33a4d2aee14fb4b2eac03462ef4db29c8ac4057327d8a71ad86f"
"checksum RustyXML 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "9222d58bccd9e6e3b82098a2ec142ad34e5d433de986d46cec03ad3a2b5fd529"
"checksum abomonation 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f6b72851223d8747938515812ef24e3f678452a4e5201d61e1954ab378ad8601"
"checksum abomonation_derive 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "0847ef820f2699bfaa7ceef120588946823806112974fb7a46eca946180049f7"
"checksum actix 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)" = "6c616db5fa4b0c40702fb75201c2af7f8aa8f3a2e2c1dda3b0655772aa949666"
...

It would be excellent to run crate dependency tests on our ordinary CI on commits where Cargo.lock has been changed. It would be ok for such commits to be verified a bit slower than ordinary commits, but can't block it for hours.

Challenges

Lots!

• Quite a few crates do not contain tests in the published version on crates.io.
  • An example of this is hyper who's tests/ folder is not included
  • This makes sense in general as there is no built in mechanism to run tests on downloaded crates so no expectation on it
  • An alternative would be to pull down the repo from its git repository tag and build that, but another challenge with that approach is to figure out which revision to pull and build.

Related

• cargo: Ability to run tests on all my dependencies (closed 😢)

[Relrin]

Oh, probably I have something. Give a try to this small tool that I've done for myself a long time ago. Perhaps it will cover somehow your needs (or at least part of it)

[repi]

@Relrin thanks! we should try it out.