WPT test harness crashes if test receives "unknown error" from WebDriver

[cookiecrook]

Test harness crashes if test receives unknown error from WebDriver.

You can reproduce this in HEAD by running either of the /html-aam/fragile tests in WebKit.

https://wpt.fyi/results/html-aam/fragile?label=experimental&label=master&aligned

At first I thought it may have been a bug in WebKit, but we did some further investigation and it looks like this is just an implementation difference that Chromium isn't hitting for this test.

In this case, there aren't backing accessibility objects for these elements (in WebKit), so attempting to get the computedrole is triggering WebDriver to return "unknown error" based on the error code definitions defined in https://w3c.github.io/webdriver/#dfn-error-code ... It seems possible that Chromium or Gecko may hit the same problem in a different scenario.

Implementation differences aside, the test harness should be able to handle any defined WebDriver error type (including unknown error) without crashing and preventing other tests in the same file from being run.

[gsnedders]

To me the most concerning thing here is "how do we end up with CRASH if nothing crashes"

[rakuco]

To me the most concerning thing here is "how do we end up with CRASH if nothing crashes"

That's because executorwebdriver.py treats WebDriver "unknown errors" as crashes ever since it was added in #12380, maybe modeled after executorselenium.py.

[gsnedders]

To me the most concerning thing here is "how do we end up with CRASH if nothing crashes"

That's because executorwebdriver.py treats WebDriver "unknown errors" as crashes ever since it was added in #12380, maybe modeled after executorselenium.py.

Ah, thanks for looking into this!

@jgraham do you have any opinion here?

[jgraham]

@jgraham do you have any opinion here?

Not really. Gecko isn't using this, but doesn't treat unknown marionette errors as if the browser crashed. In general trying to figure out when a crash, particularly a content process crash, occurred depends a bit on browser-specific heuristics, so it may be necessary to change this code, or to make the exact set of exceptions product-specific (i.e. different for Safari, Chrome, etc.)

[cookiecrook]

Seems like the test harness is no longer crashing. Now the test vends the result:

FAIL message: promise_test: Unhandled rejection with value: "error: Action get_computed_role failed"

Which I think is probably okay? @gsnedders?

[gsnedders]

Seems like the test harness is no longer crashing. Now the test vends the result:

FAIL message: promise_test: Unhandled rejection with value: "error: Action get_computed_role failed"

Which I think is probably okay? @gsnedders?

It is, but we should still maybe reconsider the wptrunner behaviour here.