Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a CI check that fails if we are missing a test for a user error #765

Open
cburgdorf opened this issue Jul 8, 2022 · 2 comments
Open
Labels

Comments

@cburgdorf
Copy link
Collaborator

cburgdorf commented Jul 8, 2022

What is wrong?

Some of the errors that the compiler is supposed to present to the user aren't covered by tests.

How can it be fixed

Add a CI check that:

  1. extracts a unique part of each user facing error message that the compiler produces
  2. Check that the unique part is somewhere present in our snapshots

That should ensure we do not end up in a situation where we have uncovered compiler errors.

@cburgdorf
Copy link
Collaborator Author

Here's an example of code that isn't exercised at all that would have been caught if we had such a check

#762 (comment)

@cburgdorf
Copy link
Collaborator Author

I'm looking back at this because I think it would be really helpful to improve coverage of our tests. There are probably a bunch of user presented errors that we currently do not cover as part of our compile errors test.

I think the simplest thing we could do is to give each error an id and then simply check if all ids are covered in our compile errors tests.

Rust also has ids for the errors and begins each error message with error[<error-id>]. E.g.

error[E0433]: failed to resolve: use of undeclared type `TType`
   --> crates/analyzer/src/traversal/expressions.rs:642:16
    |
642 |             Ok(TType::Mut(field_type(context, inner, field_name, field_span)?).id(context.db()))
    |                ^^^^^
    |                |
    |                use of undeclared type `TType`
    |                help: an enum with a similar name exists: `Type`

For more information about this error, try `rustc --explain E0433`.

We could turn that in a CI check so that it is always guaranteed that each user facing error has a corresponding test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant