Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add testdriver support for generate_test_report WebDriver command. #15305

Merged
merged 3 commits into from
Feb 20, 2019

Conversation

paulmeyer90
Copy link
Contributor

Also adds Reporting API tests which utilize this command.

@paulmeyer90 paulmeyer90 requested a review from kereliuk February 8, 2019 21:01
@wpt-pr-bot wpt-pr-bot added infra testdriver.js wptrunner The automated test runner, commonly called through ./wpt run labels Feb 8, 2019
@paulmeyer90 paulmeyer90 requested review from LukeZielinski and removed request for kereliuk February 8, 2019 21:02
Copy link
Contributor

@jgraham jgraham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestDriver additions now require an RFC in https://github.com/web-platform-tests/rfcs/ to allow fedback from across the project.

In this case it's unclear what specifies the underlying WebDriver API since it isn't part of the WebDriver spec. So your RFC should link to the relevant spec text.

@paulmeyer90
Copy link
Contributor Author

I have opened an RFC issue here: web-platform-tests/rfcs#7

@paulmeyer90
Copy link
Contributor Author

TestDriver additions now require an RFC in https://github.com/web-platform-tests/rfcs/ to allow fedback from across the project.

In this case it's unclear what specifies the underlying WebDriver API since it isn't part of the WebDriver spec. So your RFC should link to the relevant spec text.

RFC has passed

@paulmeyer90
Copy link
Contributor Author

@jgraham are you satisfied with this change now?

@jgraham
Copy link
Contributor

jgraham commented Feb 19, 2019

I think the testdriver changes are OK, but I shouldn't review the actual tests :)

Copy link
Contributor

@LukeZielinski LukeZielinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, thanks Paul.

reporting/bufferSize.html Show resolved Hide resolved
@foolip
Copy link
Member

foolip commented Feb 20, 2019

The failing Taskcluster run on this PR is because testdriver.js was modified and tests including it were run. If they weren't stable, well then the job fails. Here are the Chrome failures:

Test Subtest Results Messages
/event-timing/event-timing-timingconditions.html Event Timing only times certain types of trusted event. FAIL: 2/10, PASS: 8/10 assert_equals: Should only observe one entry: []. expected 1 but got 0
/pointerevents/pointerevent_pointerout_pen.html ERROR: 1/10, OK: 9/10
/pointerevents/pointerevent_pointerout_pen.html pointerout event received PASS: 9/10, MISSING: 1/10
/speech-api/SpeechSynthesis-pause-resume.tentative.html OK: 1/10, TIMEOUT: 9/10
/speech-api/SpeechSynthesis-pause-resume.tentative.html speechSynthesis.pause() and resume() state and events FAIL: 1/10, TIMEOUT: 9/10 assert_unreached: error event Reached unreachable code;Test timed out

And Firefox:

Test Subtest Results Messages
/pointerevents/pointerevent_setpointercapture_inactive_button_mouse.html OK: 8/10, TIMEOUT: 2/10
/pointerevents/pointerevent_setpointercapture_inactive_button_mouse.html setPointerCapture + inactive button state FAIL: 8/10, MISSING: 2/10 InvalidPointerId: Invalid pointer id.

@paulmeyer90 if none of these are due to changes that you made, which it doesn't seem like, then just let me know when this is ready to merge. I'll file bugs about the remaining failures and admin merge this. Sorry about the inconveniece, this is tracked by #14564 but I don't have a good fix for it.

@paulmeyer90
Copy link
Contributor Author

@foolip None of those appear to be related to my change, since all the tests added by my change pass, and none of those failing tests use the new API that I added (and those are the only changes in this PR).

@Hexcles Hexcles merged commit c0a12da into web-platform-tests:master Feb 20, 2019
@Hexcles
Copy link
Member

Hexcles commented Feb 20, 2019

Force-merged as per the comment above.

@foolip
Copy link
Member

foolip commented Feb 20, 2019

@Hexcles can you file the bugs as well?

@Hexcles
Copy link
Member

Hexcles commented Feb 20, 2019

Filed #15486 for pointer events, as well as #15485 for the more general question of skipping testdriver.js altogether (same as testharness.js).

I'm looking into the event-timing and speech-api flakes. Those don't seem to be flaky on master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra testdriver.js wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants