Re-enable ExecutorWebDriver in Chrome

[gsnedders]

#13200 enabled it, but this was reverted in #13220.

See wpt-chrome-dev-testharness-5 and wpt-chrome-dev-testharness-6 logs both of which contain a fair number of tracebacks.

https://staging.wpt.fyi/results/?product=chrome@1e7408fcd3&product=chrome@9bc71d422a&diff=true shows a lot of tests vanishing due to these tracebacks. This suggests there is at least one of the following bugs here:

• We aren't exiting with non-zero when we fail to run all the tests in the job (because, e.g., of Critical TestManager errors should cause non-zero exit #13073)
• We aren't setting results to ERROR/CRASH in some cases

@jgraham pointed out that in many implementations when the content process crashes the WebDriver "execute script" command just returns null, and this is indistinguishable from wd.execute_script("return null").

[gsnedders]

Looking at wpt-chrome-dev-testharness-6 things start to go wrong like this (rerun locally):

 0:15.52 TEST_START: /background-fetch/fetch.https.window.html
 0:15.52 DEBUG Got command: u'log'
 0:15.52 DEBUG Executor setup done
 0:15.52 DEBUG Got command: u'log'
 0:15.52 DEBUG Loading https://web-platform.test:8443/testharness_runner.html
 0:27.25 pid:15032 [1538335571.677][SEVERE]: Unable to receive message from renderer
 0:27.26 DEBUG Got command: u'log'
 0:27.26 DEBUG 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 238, in _run
    self.result = True, self.func(self.protocol, self.url, self.timeout)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 313, in do_testharness
    done, rv = handler(result)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/base.py", line 515, in __call__
    url, command, payload = result
TypeError: 'NoneType' object is not iterable

 0:27.26 DEBUG Got command: 'test_ended'
 0:27.26 DEBUG Unexpected count in this thread 1
 0:27.26 TEST_END: ERROR, expected OK - 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 238, in _run
    self.result = True, self.func(self.protocol, self.url, self.timeout)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 313, in do_testharness
    done, rv = handler(result)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/base.py", line 515, in __call__
    url, command, payload = result
TypeError: 'NoneType' object is not iterable

 0:27.26 DEBUG new state: running
 0:27.26 DEBUG Dispatch run_test
 0:27.26 TEST_START: /background-fetch/get-ids.https.window.html
 0:27.26 DEBUG Got command: u'log'
 0:27.26 DEBUG (Session infochrome=71.0.3559.6)
Traceback (most recent call last):
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 238, in _run
    self.result = True, self.func(self.protocol, self.url, self.timeout)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 304, in do_testharness
    parent_window = protocol.testharness.close_old_windows()
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 80, in close_old_windows
    handles = [item for item in self.webdriver.handles if item != exclude]
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 20, in inner
    return func(self, *args, **kwargs)
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 579, in handles
    return self.send_session_command("GET", "window/handles")
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 509, in send_session_command
    return self.send_command(method, url, body)
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 473, in send_command
    raise err
WebDriverException: chrome not reachable (500): (Session infochrome=71.0.3559.6)

Remote-end stacktrace:

#0 0x0000005f4f1c <unknown>
#1 0x000000506bc0 <unknown>
#2 0x0000004fd47b <unknown>
#3 0x0000004f4333 <unknown>
#4 0x0000004e04f9 <unknown>
#5 0x0000004dd8d8 <unknown>
#6 0x0000004bff98 <unknown>
#7 0x0000004c0f42 <unknown>
#8 0x00000059f4d9 <unknown>
#9 0x00000059e6ee <unknown>
#10 0x00000059ea82 <unknown>
#11 0x0000005a10aa <unknown>
#12 0x0000005b20f5 <unknown>
#13 0x0000005d1266 <unknown>
#14 0x0000005fcddd <unknown>
#15 0x7f91bc0f16ba start_thread
#16 0x7f91ba90241d clone


 0:27.27 DEBUG Got command: 'test_ended'
 0:27.27 DEBUG Unexpected count in this thread 2
 0:27.27 TEST_END: ERROR, expected OK - (Session infochrome=71.0.3559.6)
Traceback (most recent call last):
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 238, in _run
    self.result = True, self.func(self.protocol, self.url, self.timeout)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 304, in do_testharness
    parent_window = protocol.testharness.close_old_windows()
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 80, in close_old_windows
    handles = [item for item in self.webdriver.handles if item != exclude]
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 20, in inner
    return func(self, *args, **kwargs)
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 579, in handles
    return self.send_session_command("GET", "window/handles")
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 509, in send_session_command
    return self.send_command(method, url, body)
  File "/home/test/web-platform-tests/tools/webdriver/webdriver/client.py", line 473, in send_command
    raise err
WebDriverException: chrome not reachable (500): (Session infochrome=71.0.3559.6)

Remote-end stacktrace:

#0 0x0000005f4f1c <unknown>
#1 0x000000506bc0 <unknown>
#2 0x0000004fd47b <unknown>
#3 0x0000004f4333 <unknown>
#4 0x0000004e04f9 <unknown>
#5 0x0000004dd8d8 <unknown>
#6 0x0000004bff98 <unknown>
#7 0x0000004c0f42 <unknown>
#8 0x00000059f4d9 <unknown>
#9 0x00000059e6ee <unknown>
#10 0x00000059ea82 <unknown>
#11 0x0000005a10aa <unknown>
#12 0x0000005b20f5 <unknown>
#13 0x0000005d1266 <unknown>
#14 0x0000005fcddd <unknown>
#15 0x7f91bc0f16ba start_thread
#16 0x7f91ba90241d clone

A few things that stand out:

 0:27.25 pid:15032 [1538335571.677][SEVERE]: Unable to receive message from renderer

This is a message from ChromeDriver, just prior to execute script returning None, which is likely related to w3c/webdriver#1308.

The TypeError we throw is due to ChromeDriver (arguably) bogusly returning None to execute script (which shouldn't be possible!) having been unable to receive a message; while @jgraham's first attempt at working around this was to catch and error when we got None, I think this is really a distraction (especially given arguably we should then be doing that everywhere) given we should be reasonably handling any Python exception throw by the harness (because I don't believe we'll ever have no bugs).

The bigger problem is when we then run the next test (and, indeed, all following tests) we've not restarted the browser (#13013 is vaguely related here), so we just get WebDriverExceptions for the remainder of the tests.

[gsnedders]

So I think the underlying problem is that we were raising ERROR instead of INTERNAL-ERROR and therefore not restarting the browser after this happened?