Nightwatch doesn't support W3C WebDriver standard

[nkovacs]

When creating a new session, geckodriver returns a response like this:

{"value": {"sessionId":"...","capabilities":{...}}}

Nightwatch.js can't parse this correctly, so it doesn't work. It fails here: https://github.com/nightwatchjs/nightwatch/blob/v0.9.15/lib/index.js#L475

See mozilla/geckodriver#714 (comment) for more information.

[nkovacs]

And it's probably not the only thing that doesn't work. My test times out while waiting for an element to become visible when using geckodriver standalone.

[andreastt]

To be clear, this issue is actually about making Nightwatch compliant to the W3C WebDriver standard. geckodriver is currently the only driver that implements it.

When geckodriver/Firefox is being used through the Selenium remote server, there is a shim that implicitly converts Selenium requests to W3C WebDriver request. This explains why calls being made directly to geckodriver fail.

If Nightwatch wants to provide support for both Selenium’s protocol (which is being phased out) and the forthcoming WebDriver standard, it needs to go into legacy or conformance mode based on the handshake when creating a new session.

W3C WebDriver conforming remote ends will return a JSON Object with a semi-top-level capabilities field, indicating the response comes from a conforming remote end.

This is an example from geckodriver:

{"value": {"sessionId":"8138ef57-e339-4a2e-a837-de2d2d1814df","capabilities":{"acceptInsecureCerts":false,"browserName":"firefox","browserVersion":"55.0a1","moz:accessibilityChecks":false,"moz:processID":6722,"moz:profile":"/tmp/rust_mozprofile.r1QHgPjx8Lac","pageLoadStrategy":"normal","platformName":"linux","platformVersion":"4.9.0-2-amd64","rotatable":false,"specificationLevel":0,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000}}}}

[jamesaspence]

Can confirm I have this issue as well. I have tried running selenium standalone in order to have my requests to geckodriver routed through selenium instead, but no such luck yet. I will make sure my configuration is not incorrect, however.

[bitmvr]

Here to confirm this issue:

Geckodriver Logging

1498494142392	webdriver::httpapi	DEBUG	Creating routes
1498494142395	geckodriver	INFO	Listening on 127.0.0.1:4444
1498494152306	webdriver::server	DEBUG	-> POST /session {"desiredCapabilities":{"browserName":"firefox","javascriptEnabled":true,"acceptSslCerts":true,"platform":"ANY","name":"Demo"}}
1498494152336	geckodriver::marionette	INFO	Starting browser /Applications/Firefox.app/Contents/MacOS/firefox-bin with args ["-marionette"]
1498494153727	Marionette	DEBUG	Setting recommended pref datareporting.policy.dataSubmissionPolicyAccepted to false
1498494153727	Marionette	INFO	Listening on port 50828
1498494153783	geckodriver::marionette	DEBUG	Connected to Marionette on localhost:50828
1498494153785	Marionette	DEBUG	Accepted connection 0 from 127.0.0.1:50861
1498494153790	Marionette	CONFIG	Matched capabilities: {"browserName":"firefox","browserVersion":"54.0","platformName":"darwin","platformVersion":"16.6.0","pageLoadStrategy":"normal","acceptInsecureCerts":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"rotatable":false,"specificationLevel":0,"moz:processID":79804,"moz:profile":"/var/folders/_c/vs9fm2294gd8cffpx3d7tjm59lg55t/T/rust_mozprofile.W80EXKqwSFgi","moz:accessibilityChecks":false}
1498494153923	Marionette	DEBUG	loaded listener.js
1498494153932	webdriver::server	DEBUG	<- 200 OK {"value": {"sessionId":"c29a7b77-4d08-5d49-bd3c-699c1cba3355","capabilities":{"acceptInsecureCerts":false,"browserName":"firefox","browserVersion":"54.0","moz:accessibilityChecks":false,"moz:processID":79804,"moz:profile":"/var/folders/_c/vs9fm2294gd8cffpx3d7tjm59lg55t/T/rust_mozprofile.W80EXKqwSFgi","pageLoadStrategy":"normal","platformName":"darwin","platformVersion":"16.6.0","rotatable":false,"specificationLevel":0,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000}}}}
2017-06-26 12:22:33.980 plugin-container[79806:2921602] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x993b, name = 'com.apple.tsm.portname'
See /usr/include/servers/bootstrap_defs.h for the error codes.
2017-06-26 12:22:33.981 plugin-container[79806:2921602] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x9d03, name = 'com.apple.CFPasteboardClient'
See /usr/include/servers/bootstrap_defs.h for the error codes.

Nightwatch Output

Error retrieving a new session from the selenium server

Connection refused! Is selenium server started?
{ value: 
   { sessionId: 'c29a7b77-4d08-5d49-bd3c-699c1cba3355',
     capabilities: 
      { acceptInsecureCerts: false,
        browserName: 'firefox',
        browserVersion: '54.0',
        'moz:accessibilityChecks': false,
        'moz:processID': 79804,
        'moz:profile': '/var/folders/_c/vs9fm2294gd8cffpx3d7tjm59lg55t/T/rust_mozprofile.W80EXKqwSFgi',
        pageLoadStrategy: 'normal',
        platformName: 'darwin',
        platformVersion: '16.6.0',
        rotatable: false,
        specificationLevel: 0,
        timeouts: [Object] } } }

[aamorozov]

Also having issues with running Nightwatch tests in Firefox. For example, setValue command that works across browsers fails while running in Firefox with "uknown error".
Firefox v. 54, Selenium Standalone 3.4.0

[nkovacs]

@aamorozov I don't think that's related, you're using selenium, not geckodriver in standalone mode (without selenium).

[aamorozov]

@nkovacs Yeah makes sense, will try using without selenium and update the comment here if still experience the issue

[nkovacs]

That won't work at all with more recent versions of geckodriver (and older versions are very incomplete and buggy) because nightwatch doesn't support the standard that geckodriver uses. That's what this issue is about.

[blueimp]

I've brought up the same issue in this ticket:
#1285

[OttoG]

@beatfactor this issue has become more relevant since the previous comments were posted and the referenced issue #1285 was closed.

For example, with current versions of Firefox, geckodriver and Selenium Server, it is impossible to switch focus between windows. This is because Selenium does not translate the JsonWireProtocol "POST /session/:sessionId/window" command into a W3C WebDriver command before passing it to geckodriver. Thus, geckodriver gets a "name" parameter where it expects a "handle" parameter, which causes the request to fail with an error message.

See https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidwindow and https://w3c.github.io/webdriver/webdriver-spec.html#switch-to-window for the protocol difference.

Most likely, there are also other commands where a translation done by Selenium would be needed but is missing.

I have raised the issue SeleniumHQ/selenium#5831 about the missing translation but received the reply that the Selenium project is phasing out the protocol translation. Quote from a comment:

It is my understanding that some commands are still translated, but they should not be relied upon. As of 3.5 I believe, passthrough mode is on by default so commands will pass through server which is why you aren't getting the handle parameter.

In short, nightwatch needs to update to handle w3c commands and not rely on selenium to translate commands for it.

At the same time, it seems that both Crome and Edge have made good progress with implementing the W3C WebDriver API. See:

https://docs.microsoft.com/en-us/microsoft-edge/webdriver
https://chromium.googlesource.com/chromium/src/+/master/docs/chromedriver_status.md

For Edge, it seems that the missing items from the W3C support are all about things that were either not supported by Edge with JsonWire, not part of the protocol with JsonWire, or take the same parameters in the both protocols and therefore may be largely compatible. It is not as easy to interpret the ChromeDriver list (does empty mean the same thing as "complete", for example?), but a reasonable conclusion from scanning the list is that they either are very close to supporting the same things that they have supported through JsonWire, or already do it.

Egde and Chrome might not drop the JsonWireProtocol compatibility any time in the near future, but Firefox driving is already starting to break (or has been broken since the introduction of geckodriver), and moving towards W3C WebDriver communication is now unlikely to break anything with modern browsers and drivers.

Therefore, please take the W3C WebDriver issue as a serious feature request, or even better, a bug fix request for Firefox compatibility.

[beatfactor]

@OttoG I'm not sure if you're aware, but we have launched a new major version a few days ago which should mitigate some issues regarding compatibility and also make it easier to fix remaining ones. So, at this point, I can assure you that WebDriver compatibility is on the top of our list of priorities. However, we need help from the community, so I encourage you to try out the new version (1.0.3), if you haven't hadn't already - it's on the releases/1.0 branch and also in NPM.

I will merge the code to master in the next few days and then I'll also revisit the existing issues and features requests.

[OttoG]

@beatfactor Thank you for the fast reply. I was not aware of the new version, as it is marked as beta in NPM, but with “npm install [email protected]” I could install it.

Unfortunately, the switchWindow API call still fails, but now with an exception, which is an improvement over the previous silent failure:

   Response 500 POST /wd/hub/session/557ab6d2-3f59-2946-ae66-9dc770f9eb00/window (48ms)
   { state: 'invalid argument',
     sessionId: null,
     value: 
      { systemInformation: 'System info: host: \'local\', ip: \'\', os.name: \'Mac OS X\', os.arch: \'x86_64\', os.version: \'10.13.4\', java.version: \'1.8.0_25\'',
        message: 'Missing \'handle\' parameter',

The tcpdump trace of the exchange with Selenium Server still looks the same, with "name" instead of "handle":

POST /wd/hub/session/557ab6d2-3f59-2946-ae66-9dc770f9eb00/window HTTP/1.1
content-type: application/json; charset=utf-8
content-length: 21
Host: localhost:4444
Connection: close

{"name":"2147483669"}

HTTP/1.1 500 Server Error
Connection: close
Date: Thu, 26 Apr 2018 13:47:59 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json;charset=utf-8
Content-Length: 1266
Server: Jetty(9.4.7.v20170914)

{"state":"invalid argument","sessionId":null,"value":{"additionalInformation":"\nDriver info: driver.version: unknown","localizedMessage":"Missing 'handle' parameter\nBuild info: version: '3.11.0', revision: 'e59cfb3', time: '2018-03-11T20:33:15.31Z'…

Off topic, on the 1.0.3 version in general, I noticed a few changes and an error.

1. Callbacks are no longer invoked with the browser object as this. This affects code that is autogenerated for export by https://www.snaptest.io, where it is frequently assumed that this refers to browser. If this is intentional, it should be documented in migration notes, so that SnapTest and other users can adapt. With manual adaptations, the auto-generated code works fine.

2. The callback parameter values from API calls such as element are now different between Firefox and Chrome. For example browser.element('css selector', '#id', v => { console.log(v.value); }) will log {'element-6066-11e4-a52e-4f735466cecf': '…'} in Firefox but {'ELEMENT': '…'} in Chrome. In 0.9.x, it was 'ELEMENT' with both browsers.

3. The browser.resizeWindow(width, height) method does not work in either Chrome or Firefox. Invoking it gives the following error.

Error while running "resizeWindow" command: "TypeError: client.api[method] is not a function"

[beatfactor]

@OttoG Thanks for your detailed feedback. I will look into fixing 1 and 3 - 1 is unintentional and 3 is a bug. Regarding 2 - this happens because Chrome still uses JsonWire for the time being, and so you'll still get {'ELEMENT': '…'} until they switch to WebDriver.

For switchWindow issues, would you mind opening a separate issue? Thanks.

[beatfactor]

@OttoG the issues you have reported should be fixed in v1.0.10. If there are other issues left, could we please address them as separate?

Regarding general W3C WebDriver compatibility we are trying to close the gap before the v1.0 stable release and we'll handle remaining inconsistencies separately.