Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add default regression terms based on --regress option. #339

Merged
merged 1 commit into from
Jun 17, 2024

Conversation

ehuss
Copy link
Collaborator

@ehuss ehuss commented May 30, 2024

This adds new default text for the regression status based on the --regress option. These terms will hopefully be clearer to understand as opposed to just saying "Yes" and "No". Although "Yes" and "No" are directly related to whether or not the regression was found, you have to have a mental map of Yes→Current Regression Mode→Compile Status. This hopes to shorten that to just "Compile Status".

To review the basics here:

  • RegressOn is the --regress option.
  • Satisfies is whether or not the "regression" was found.
  • The definition of "regression" depends on the --regress option. By default, it is --regress=error which means a regression occurs when it fails to compile. --regress=success means the opposite (a regression is when it successfully compiles).
  • term-old is the text displayed for compilers older than the date of the regression.
  • term-new is the text displayed for compilers on or after the date of the regression.

Thus, "old" is equivalent to "Satisfies::No" and "new" is equivalent to "Satisfies::Yes".

Here is an abbreviated example of what this would look like on the console for the default --regress=error:

checking the start range to find a passing nightly
installing nightly-2024-03-15
testing...
RESULT: nightly-2024-03-15, ===> Successfully compiled

checking the end range to verify it does not pass
installing nightly-2024-04-15
testing...
RESULT: nightly-2024-04-15, ===> Compile error

16 versions remaining to test after this (roughly 5 steps)
installing nightly-2024-03-30
testing...
RESULT: nightly-2024-03-30, ===> Successfully compiled

8 versions remaining to test after this (roughly 4 steps)
installing nightly-2024-04-07
testing...
RESULT: nightly-2024-04-07, ===> Compile error

4 versions remaining to test after this (roughly 3 steps)
installing nightly-2024-04-03
testing...
RESULT: nightly-2024-04-03, ===> Compile error

2 versions remaining to test after this (roughly 2 steps)
installing nightly-2024-04-01
testing...
RESULT: nightly-2024-04-01, ===> Successfully compiled

1 versions remaining to test after this (roughly 1 steps)
installing nightly-2024-04-02
testing...
RESULT: nightly-2024-04-02, ===> Successfully compiled

searched toolchains nightly-2024-03-15 through nightly-2024-04-15

Here you can see it starts checking the start range (the "old" compiler) and finds the baseline successfully compiles. Then it checks the end range (the "new" compiler) and finds the regression ("Compile error").

The terms are essentially reversed for --regress=success.

You can customize these terms with --term-old and --term-new. This is mostly useful when using scripts or custom commands, which might provide confusing outputs when using the default --regress=error. Scripts might be inverting the logic themselves, or doing actions other than compiling (like grepping for docs).

This also corrects what I think was a misunderstanding in #330 when during review there was a change from --term-good and --term-bad where it did make sense to "flip" the meaning based on the regression mode. However, my intent with --term-old and --term-new is to avoid that flipping (and is also the reasoning I believe git bisect introduced the old/new terminology). "Old" means an older compiler by date, and "new" means a newer one. There's no need to flip the meaning.

I also customized the text when using the --script parameter, since scripts can do things other than "compiling", and the new default terms could otherwise be confusing. It now explicitly indicates what the script returned.

@ehuss ehuss mentioned this pull request May 30, 2024
@ehuss ehuss merged commit ee3e071 into rust-lang:master Jun 17, 2024
4 checks passed
@apiraino apiraino mentioned this pull request Jun 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants