-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add contributing docs specific to rule-testing patterns #4690
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,6 +8,7 @@ Welcome! We're happy to have you here. Thank you in advance for your contributio | |
- [Project Structure](#project-structure) | ||
- [Example: Adding a new lint rule](#example-adding-a-new-lint-rule) | ||
- [Rule naming convention](#rule-naming-convention) | ||
- [Rule testing: fixtures and snapshots](#rule-testing-fixtures-and-snapshots) | ||
- [Example: Adding a new configuration option](#example-adding-a-new-configuration-option) | ||
- [MkDocs](#mkdocs) | ||
- [Release Process](#release-process) | ||
|
@@ -110,7 +111,7 @@ At a high level, the steps involved in adding a new lint rule are as follows: | |
checks), `crates/ruff/src/checkers/tokens.rs` (for token-based checks), `crates/ruff/src/checkers/lines.rs` | ||
(for text-based checks), or `crates/ruff/src/checkers/filesystem.rs` (for filesystem-based | ||
checks). | ||
1. Add a test fixture. | ||
1. Add proper [testing](#rule-testing-fixtures-and-snapshots) for your rule. | ||
1. Update the generated files (documentation and generated code). | ||
|
||
To define the violation, start by creating a dedicated file for your rule under the appropriate | ||
|
@@ -125,18 +126,8 @@ collecting diagnostics as it goes. | |
If you need to inspect the AST, you can run `cargo dev print-ast` with a Python file. Grep | ||
for the `Check::new` invocations to understand how other, similar rules are implemented. | ||
|
||
To add a test fixture, create a file under `crates/ruff/resources/test/fixtures/[linter]`, named to match | ||
the code you defined earlier (e.g., `crates/ruff/resources/test/fixtures/pycodestyle/E402.py`). This file should | ||
contain a variety of violations and non-violations designed to evaluate and demonstrate the behavior | ||
of your lint rule. | ||
|
||
Run `cargo dev generate-all` to generate the code for your new fixture. Then run Ruff | ||
locally with (e.g.) `cargo run -p ruff_cli -- check crates/ruff/resources/test/fixtures/pycodestyle/E402.py --no-cache --select E402`. | ||
|
||
Once you're satisfied with the output, codify the behavior as a snapshot test by adding a new | ||
`test_case` macro in the relevant `crates/ruff/src/rules/[linter]/mod.rs` file. Then, run `cargo test`. | ||
Your test will fail, but you'll be prompted to follow-up with `cargo insta review`. Accept the | ||
generated snapshot, then commit the snapshot file alongside the rest of your changes. | ||
Once you're satisfied with your code, add tests for your rule. See [rule testing](#rule-testing-fixtures-and-snapshots) | ||
for more details. | ||
|
||
Finally, regenerate the documentation and generated code with `cargo dev generate-all`. | ||
|
||
|
@@ -154,6 +145,24 @@ This implies that rule names: | |
When re-implementing rules from other linters, this convention is given more importance than | ||
preserving the original rule name. | ||
|
||
#### Rule testing: fixtures and snapshots | ||
|
||
To test rules, Ruff uses snapshots of Ruff's output for a given file (fixture). Generally, there will be one file per rule (e.g., `E402.py`), and each file will contain all necessary examples of both violations and non-violations. `cargo insta review` will generate a snapshot file containing Ruff's output for each fixture, which you can then commit alongside your changes. | ||
|
||
Once you've completed the code for the rule itself, you can define tests with the following steps: | ||
|
||
1. Add a Python file to `crates/ruff/resources/test/fixtures/[linter]` that contains the code you want to test. The file name should match the rule name (e.g., `E402.py`), and it should include examples of both violations and non-violations. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you use the script |
||
|
||
1. Run Ruff locally against your file and verify the output is as expected. For example, if you added a new rule named `E402`, you could run `cargo run -p ruff_cli -- check crates/ruff/resources/test/fixtures/pycodestyle/E402.py --no-cache --select E402`. Once you're satisfied with the output (you see the violations you expect, and no others), proceed to the next step. | ||
|
||
1. Run `cargo dev generate-all` to generate the code for your new fixture. | ||
|
||
1. Add the test to the relevant `crates/ruff/src/rules/[linter]/mod.rs` file. If you're contributing a rule to a pre-existing set, you should be able to find a similar example to pattern-match against. If you're adding a new linter, you'll need to create a new `mod.rs` file. | ||
|
||
1. Run `cargo test`. Your test will fail, but you'll be prompted to follow-up with `cargo insta review`. Run `cargo insta review` and accept the generated snapshot, then commit the snapshot file alongside the rest of your changes. | ||
|
||
1. Run `cargo test` again to ensure that your test now passes. | ||
|
||
### Example: Adding a new configuration option | ||
|
||
Ruff's user-facing settings live in a few different places. | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Everything above is just drive-by formatting changes added by me after the initial PR.)