Avoid unintended scope changes to active focus areas

[zcorpan]

Maybe we should have a CI check (in wpt) to avoid test changes for tests in interop-* (for active focus areas). Require a commit message to link to an issue in the interop repo?

c.f. #345 , #338

[foolip]

@zcorpan any thoughts on how to implement this? The labels are in https://github.com/web-platform-tests/wpt-metadata and can be accessed via a wpt.fyi API. Either would be an external dependency which would increase the risk of CI failures if those dependencies are not reliable.

[zcorpan]

I had in mind doing a git clone of the wpt-metadata repo. I'm not so familiar with the wpt.fyi API. No opinion on which approach to use. If there are network issues we could have a retry (or several retries with increased timeout).

[foolip]

Looks like the API is undocumented (web-platform-tests/wpt.fyi#3337) but one can get all of the labels in a single API call at https://wpt.fyi/api/metadata?product=chrome&includeTestLevel.

Whether using that API or a clone of the wpt-metadata repo, we'd get a list of test names. I'm a bit unsure how to implement the rest of the check, but it is possible of course.

[zcorpan]

I think there's code to determine "affected tests" for a PR for running stability checks, which maybe we could reuse? cc @jgraham

[foolip]

We discussed this in #660. Concrete next steps here could be a GitHub Actions workflow that does:

• Get list of labels for active focus areas from https://wpt.fyi/static/interop-data.json
• Get the list of labeled tests from https://wpt.fyi/api/metadata?includeTestLevel=true&product=chrome
• Get affected tests with ./wpt tests-affected (this also includes tests affected indirectly through included scripts)
• Check if there's an intersection between the labeled tests and the affected tests
• Add a "affects-interop-2024" label to the pull request