ExecutorWebDriver

[foolip]

We should stop relying on Selenium for Chrome/Edge/Safari support, and instead use WebDriver directly.

Relying on Selenium adds further levels of indirection (and the potential to abstract away bugs) and makes it harder to use custom WebDriver extensions for testing.

• Support WebDriver as an option (Implement WebDriver executor for wptrunner #12380)
• Enable WebDriver by default for Chrome (Make wptrunner chrome always use the WebDriver executor #13200, Re-enable ExecutorWebDriver in Chrome #13225)
• Enable WebDriver by default for Edge
• Enable WebDriver by default for Safari (Rename "safari_webdriver" to "safari", get rid of "safari" #14213)
• Enable WebDriver by default for Chrome on Android
• Enable WebDriver by default for Opera
• Enable WebDriver by default for WebKit (Switch WebKit browser product to WebDriver executors #13339)

[foolip]

Previous PR at #10197.

[foolip]

New PR at #12380.

[foolip]

Reopening to also track this being the default. @gsnedders what's the plan for that?

[gsnedders]

I've been trying to get Windows CI up and running again, and got somewhat side-tracked. :)

I think the plan is to just try landing it, at this point?

[gsnedders]

#13200

[gsnedders]

So, one the above to-do is done, this leaves just IE and Sauce; I think we should probably just drop support for everything except wdspec tests for IE (cc @jimevans), given almost all will just break horribly, and Sauce we can't do anything about until they support the W3C protocol.

[jimevans]

Let me echo my desire to retain wdspec tests for IE, since the IE driver is still under active development and support (even if the browser isn't), and the test suite is incredibly useful for keeping track of compliance. Anything other than wdspec, I don't much care.

[foolip]

@gsnedders, can I suggest comparing Taskcluster runs before/after the change, finding the commit where this landed in https://staging.wpt.fyi/test-runs?label=taskcluster. Example diff URL:
https://staging.wpt.fyi/results/?product=firefox@59bed3508c&product=firefox@ddb3e30ef4&diff=true

[foolip]

Also, a similar comparison for the Buildbot runs on prod would be good. Although that'll also include test changes as it's not run per push to master.

[foolip]

https://staging.wpt.fyi/results/?product=chrome@9bc71d422a&product=chrome@1e7408fcd3&diff=true

[foolip]

@gsnedders, Safari 12.0 was just released with macOS 10.14 Mojave, can you check if the accompanying SafariDriver is updated so that we can use ExecutorWebDriver?

[jimevans]

@foolip In my experience, the driver shipped with Safari 12 uses the W3C protocol dialect by default. @burg would be able to say for sure, but modulo bugs in the implementation, it should be usable with any spec-compliant local end implementation.

[foolip]

Excellent! @gsnedders, can you give it a try to enable, and coordinate with @jugglinmike who is trying to set up runs for Safari stable?

[burg]

@foolip In my experience, the driver shipped with Safari 12 uses the W3C protocol dialect by default. @burg would be able to say for sure, but modulo bugs in the implementation, it should be usable with any spec-compliant local end implementation.

This is correct. JSONWP has been dropped entirely in safaridriver for Safari Technology Preview.

[gsnedders]

@jugglinmike do you want to try flipping us over to safari_webdriver for a build or two on wpt.fyi? or should we try (pretty blindly!) to do that upstream?

[foolip]

I've tried enabling safari_webdriver in foolip#5:
https://travis-ci.org/foolip/wpt/jobs/439114299

So at least the infrastructure/ tests work. Also curious to hear how @jugglinmike would like to roll this out. I suspect it depends on the status of Safari 12 runs?

[jugglinmike]

Thanks for the offer, @gsnedders. I would like to take you up on that, but I'm still working through an intermittent error in testing Safari. Resolving that is my top priority, and I hope to have it licked before the end of the week. You can refer to that issue for progress updates.

[mdittmer]

Ping from your friendly neighbourhood ecosystem infra rotation

Any updates on this issue, @gsnedders?

[gsnedders]

• Edge is mostly waiting for Windows 10 RS5 to rollout to more users (given plenty of people still don't have it)
• Opera is blocked on OperaDriver working in W3C mode (webdriver/tests/new_session/ totally fails)
• Chrome on Android is blocked on someone actually having something set up to test it

[foolip]

@gsnedders ExecutorWebDriver has been a thing for a while now. Can you mark the important follow-up as roadmap priority and close this issue?