Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run web platform tests in browsers #96

Merged
merged 29 commits into from
Sep 29, 2021
Merged

Run web platform tests in browsers #96

merged 29 commits into from
Sep 29, 2021

Conversation

MattiasBuelens
Copy link
Owner

Previously, we only ran the web platform tests (WPT) against Node using wpt-runner. This works fine, but it's not perfect:

  • wpt-runner uses jsdom, and although jsdom does a really good job at implementing all the necessary web APIs, it's not a "real" browser.
  • If the polyfill accidentally starts depending on a Node-only API, we may fail to spot it and break the polyfill for browser users. That would be pretty bad, after all this is still a web streams polyfill. 😛
  • If we want to integrate with the built-in streams implementation of a web browser (Investigate using native streams #20), we really want to test that integration properly.

This PR adds the test infrastructure to also run WPT against actual browsers. I use Playwright to spin up a headless Chromium or Firefox browser, start a local web server, run all the tests and collect the results back inside Node. I tried to keep it as close as possible to how we run WPT on Node, to make it easier to understand and maintain.

Alternatives considered:

  • Use wpt run as suggested in Running WPT #78. This is a Python tool used by the WPT project itself to run the tests. Although I eventually managed to get it to run, I decided not to go this route:
    • It's a very complex tool because it needs to support all of the test kinds in WPT, including Fetch tests with a HTTP server or Canvas tests with screenshots. For streams, most of this is not needed.
    • The polyfill does not pass all of the streams tests, so I need a way to have some expected failures. The tool provides wpt update-expectations, but I found it to not be granular enough.

@MattiasBuelens MattiasBuelens added enhancement domain: compliance Issues related to compliance with the streams standard labels Sep 28, 2021
@MattiasBuelens MattiasBuelens added this to the v4.0.0 milestone Sep 28, 2021
@MattiasBuelens MattiasBuelens self-assigned this Sep 28, 2021
The "pull_request" workflow acts more as a "sanity check"
that the PR branch is sufficiently up-to-date with the base branch.

We leave the real heavy lifting over to the "push" workflow.
@MattiasBuelens MattiasBuelens merged commit 6c7eb27 into next Sep 29, 2021
@MattiasBuelens MattiasBuelens deleted the run-wpt-in-browser branch September 29, 2021 21:06
@MattiasBuelens MattiasBuelens mentioned this pull request Sep 29, 2021
@foolip
Copy link

foolip commented Sep 30, 2021

Thanks @MattiasBuelens for outlining what issues you had with getting ./wpt run to work. I am not surprised that it was difficult to get working, but this is still useful feedback for the project. My takeaways:

  • Getting ./wpt run to work at all the first time is too hard (I've heard this many times, and felt it myself)
  • How to run a subset of tests or test types with ./wpt run is not very discoverable
  • Using expectations with ./wpt run is not documented and is kind of broken. These code paths are used in Gecko's CI, but not using the ./wpt run frontend. I myself have never gotten it to work.

cc @gsnedders @jgraham @past FYI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: compliance Issues related to compliance with the streams standard enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants