Skip to content
This repository has been archived by the owner on Sep 28, 2022. It is now read-only.

Implement framework for comparing guppy's results to cargo #83

Closed
sunshowers opened this issue Mar 31, 2020 · 4 comments
Closed

Implement framework for comparing guppy's results to cargo #83

sunshowers opened this issue Mar 31, 2020 · 4 comments
Labels
better-testing Improvements to guppy's testing

Comments

@sunshowers
Copy link
Contributor

guppy is a reimplementation of parts of cargo. It would be cool to compare guppy's results with cargo's continuously against real-world repositories.

Here's one way to make that happen:

  • Add a new binary crate called (say) guppy-comparer which depends on both guppy and cargo.
  • In this crate, implement a runner which:
    • takes a Rust workspaces (using the one at cwd is ok to begin with, but may also take path/git/vcs sources or even be based on crater)
    • uses the cargo API to load up the repo
    • uses guppy to load up the cargo metadata
    • knows how to interchange guppy and cargo data structures
    • generates N (say 10k by default) random queries using proptest
    • runs those queries on guppy and cargo
    • compares results, and fails if any query has non-matching results.

Examples of things to test:

  • Once we have implementations of both the classic and new feature resolvers, it should be possible to compare guppy's and cargo's respective resolvers. The random queries would consist of which feature IDs to select and which resolver to use.
@metajack
Copy link
Contributor

Is there an API for queries in cargo? Or are you thinking of somehow adapting the graph to cargo's native structs?

@sunshowers
Copy link
Contributor Author

There are a few APIs for queries, e.g. https://docs.rs/cargo/0.43.1/cargo/ops/fn.resolve_ws_with_opts.html. It won't cover everything that guppy does but it'll still be useful probably!

@sunshowers
Copy link
Contributor Author

@bmwill asked me about why this would be useful when cargo metadata already has resolved data. The reason is that cargo metadata has resolve results for the full workspace; the resolution for a subset of workspace packages might be a bit more restricted. It'll be good to compare results for subgraph resolutions.

@sunshowers sunshowers added the better-testing Improvements to guppy's testing label Apr 7, 2020
@sunshowers sunshowers mentioned this issue Apr 7, 2020
Open
sunshowers added a commit to sunshowers/cargo-guppy that referenced this issue May 1, 2020
@sunshowers
[cargo-compare] add some comparison testing to CI
5b223ab 
This works by:
* setting up a real, fairly complex fixture
* generating random queries using proptest
* running them through cargo and guppy
* ensuring that they produce the same results

Not only does this give us high confidence that cargo
and guppy produce the same results, it also gets us
most of the way to facebookarchive#83.

Most of the work in this commit is simply setting up
all the necessary test infrastructure. We piggy-back on
existing proptest infrastructure as much as possible.
sunshowers added a commit to sunshowers/cargo-guppy that referenced this issue May 2, 2020
@sunshowers
[cargo-compare] add some comparison testing to CI
502cd49 
This works by:
* setting up a real, fairly complex fixture
* generating random queries using proptest
* running them through cargo and guppy
* ensuring that they produce the same results

Not only does this give us high confidence that cargo
and guppy produce the same results, it also gets us
most of the way to facebookarchive#83.

Most of the work in this commit is simply setting up
all the necessary test infrastructure. We piggy-back on
existing proptest infrastructure as much as possible.
sunshowers added a commit to sunshowers/cargo-guppy that referenced this issue May 3, 2020
@sunshowers
[cargo-compare] add some comparison testing to CI
abfa059 
This works by:
* setting up a real, fairly complex fixture
* generating random queries using proptest
* running them through cargo and guppy
* ensuring that they produce the same results

Not only does this give us high confidence that cargo
and guppy produce the same results, it also gets us
most of the way to facebookarchive#83.

Most of the work in this commit is simply setting up
all the necessary test infrastructure. We piggy-back on
existing proptest infrastructure as much as possible.
sunshowers added a commit to sunshowers/cargo-guppy that referenced this issue May 3, 2020
@sunshowers
[cargo-compare] add some comparison testing to CI
6171da7 
This works by:
* setting up a real, fairly complex fixture
* generating random queries using proptest
* running them through cargo and guppy
* ensuring that they produce the same results

Not only does this give us high confidence that cargo
and guppy produce the same results, it also gets us
most of the way to facebookarchive#83.

Most of the work in this commit is simply setting up
all the necessary test infrastructure. We piggy-back on
existing proptest infrastructure as much as possible.
sunshowers added a commit to sunshowers/cargo-guppy that referenced this issue May 5, 2020
@sunshowers
[cargo-compare] add some comparison testing to CI
d4085f4 
This works by:
* setting up a real, fairly complex fixture
* generating random queries using proptest
* running them through cargo and guppy
* ensuring that they produce the same results

Not only does this give us high confidence that cargo
and guppy produce the same results, it also gets us
most of the way to facebookarchive#83.

Most of the work in this commit is simply setting up
all the necessary test infrastructure. We piggy-back on
existing proptest infrastructure as much as possible.
sunshowers added a commit that referenced this issue May 6, 2020
@sunshowers
[cargo-compare] add some comparison testing to CI
a8cbe50 
This works by:
* setting up a real, fairly complex fixture
* generating random queries using proptest
* running them through cargo and guppy
* ensuring that they produce the same results

Not only does this give us high confidence that cargo
and guppy produce the same results, it also gets us
most of the way to #83.

Most of the work in this commit is simply setting up
all the necessary test infrastructure. We piggy-back on
existing proptest infrastructure as much as possible.
@sunshowers
Copy link
Contributor Author

Done with cargo-compare.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
better-testing Improvements to guppy's testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@metajack @sunshowers