Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable queries on test run history #9218

Closed
amunger opened this issue Mar 3, 2022 · 2 comments · Fixed by #9775
Closed

Enable queries on test run history #9218

amunger opened this issue Mar 3, 2022 · 2 comments · Fixed by #9775
Assignees
Labels
debt Code quality issues triage-needed Issue needs to be triaged
Milestone

Comments

@amunger
Copy link
Contributor

amunger commented Mar 3, 2022

I'd like to be able to easily check the run history of a test or general trends in test runs/pass rate.

Some cases I can think of:

  • I can easily check if a "flaky test" issue is still relevant. There are plenty of old ones that could probably just be closed.
  • If a test fails in CI, I can go check if that test has already been failing before I made my change.
  • When troubleshooting a flaky test, I can get a better idea of when it started failing to track down relevant changes

Notes:

  • The CI system doesn't send any telemetry, so we currently can't just emit our normal telemetry events
  • The "publish test results" step in the github action sounds promising, but is currently failing for auth reasons
@amunger amunger added debt Code quality issues engineering labels Mar 3, 2022
@DonJayamanne
Copy link
Contributor

DonJayamanne commented Mar 3, 2022

Even if we had fixed the auth issue, you won't be able to query the failures of older tests.
Github doesn't store such information, I think we'd need to use Azure Pipelines or the like.
We could try pushing the results to azure pipelines in some weird n funky way.

@amunger
Copy link
Contributor Author

amunger commented Sep 20, 2022

vscode-jupyter\.vscode\notebooks\CITestResults.ipynb is published and ready for usage.
Also relies on the Github action Aggregate Test Results which consolidates results into daily chunks

@amunger amunger closed this as completed Sep 20, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
debt Code quality issues triage-needed Issue needs to be triaged
Projects
None yet
3 participants