You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since the inception of wpt.fyi, the summary numbers have been the number of subtests (plus harness status, grrr). We also show a summary like "Showing 39166 tests (1739296 subtests)" at the top. (I often find myself digging into a directory just to see the first of those numbers change.)
In any kind of analysis or metrics we've used, we have always switched to something based on the number of tests, since subtest counts inflate certain directories and types of tests out of proportion.
A simple alternative which could be supported by the current legacy_status field from the /api/search results would be to count tests. Tests would be treated as passing if all subtests pass, and otherwise fail. As a metric this has the downside of treating a 1/1000 subtest failures the same as 999/1000 subtest failures, but in as a rough guide to where there are many problematic tests, it's still OK.
Straw proposal: make the computation of summary numbers configurable and controlled by a URL parameter.
The text was updated successfully, but these errors were encountered:
Put together with #2289, this would have made a recent analysis much more straightforward, and something that I could have linked to for others to see. I don't think it should be a goal for wpt.fyi to support any arbitrary analysis and turn into a swiss army knife, but I'm actually optimistic about being able to simplify sorting+filtering into a pretty simple widget at the top of results, and note merely adding more and more toggles up there.
When showing test counts, it would probably be best to normalize the score of each test to a value between 0 and 1. This would then match how metrics for https://wpt.fyi/compat2021 are computed.
Since the inception of wpt.fyi, the summary numbers have been the number of subtests (plus harness status, grrr). We also show a summary like "Showing 39166 tests (1739296 subtests)" at the top. (I often find myself digging into a directory just to see the first of those numbers change.)
In any kind of analysis or metrics we've used, we have always switched to something based on the number of tests, since subtest counts inflate certain directories and types of tests out of proportion.
A simple alternative which could be supported by the current
legacy_status
field from the/api/search
results would be to count tests. Tests would be treated as passing if all subtests pass, and otherwise fail. As a metric this has the downside of treating a 1/1000 subtest failures the same as 999/1000 subtest failures, but in as a rough guide to where there are many problematic tests, it's still OK.Straw proposal: make the computation of summary numbers configurable and controlled by a URL parameter.
The text was updated successfully, but these errors were encountered: