Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Showing test (not subtest) summary numbers #2290

Open
foolip opened this issue Dec 3, 2020 · 2 comments
Open

Showing test (not subtest) summary numbers #2290

foolip opened this issue Dec 3, 2020 · 2 comments

Comments

@foolip
Copy link
Member

foolip commented Dec 3, 2020

Since the inception of wpt.fyi, the summary numbers have been the number of subtests (plus harness status, grrr). We also show a summary like "Showing 39166 tests (1739296 subtests)" at the top. (I often find myself digging into a directory just to see the first of those numbers change.)

In any kind of analysis or metrics we've used, we have always switched to something based on the number of tests, since subtest counts inflate certain directories and types of tests out of proportion.

A simple alternative which could be supported by the current legacy_status field from the /api/search results would be to count tests. Tests would be treated as passing if all subtests pass, and otherwise fail. As a metric this has the downside of treating a 1/1000 subtest failures the same as 999/1000 subtest failures, but in as a rough guide to where there are many problematic tests, it's still OK.

Straw proposal: make the computation of summary numbers configurable and controlled by a URL parameter.

@foolip
Copy link
Member Author

foolip commented Dec 3, 2020

Put together with #2289, this would have made a recent analysis much more straightforward, and something that I could have linked to for others to see. I don't think it should be a goal for wpt.fyi to support any arbitrary analysis and turn into a swiss army knife, but I'm actually optimistic about being able to simplify sorting+filtering into a pretty simple widget at the top of results, and note merely adding more and more toggles up there.

@foolip
Copy link
Member Author

foolip commented Oct 28, 2021

When showing test counts, it would probably be best to normalize the score of each test to a value between 0 and 1. This would then match how metrics for https://wpt.fyi/compat2021 are computed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant