Saucelabs failure does not make chimp exit non-zero

[samhatoum]

Issue by joscha
Friday Apr 27, 2018 at 05:31 GMT
Originally opened as xolvio/chimp#687

---

We sometimes see errors like these:

--
  | [chimp] Detected an unhandledRejection:
  | [chimp][hooks] Error: ERROR The test with session id 3faf1045a7fc424a80def395fc3bb824 has already finished, and can't receive further commands.
  | You can learn more at https://saucelabs.com/jobs/3faf1045a7fc424a80def395fc3bb824
  | For help, please check https://wiki.saucelabs.com/display/DOCS/Common+Error+Messages
  | at new RuntimeError (node_modules/webdriverio/build/lib/utils/ErrorHandler.js:144:12)
  | at Request._callback (node_modules/webdriverio/build/lib/utils/RequestHandler.js:297:39)
  | at Request.self.callback (node_modules/request/request.js:186:22)
  | at Request.emit (events.js:180:13)
  | at Request.emit (domain.js:422:20)
  | at Request.<anonymous> (node_modules/request/request.js:1163:10)
  | at Request.emit (events.js:180:13)
  | at Request.emit (domain.js:422:20)
  | at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
  | at Object.onceWrapper (events.js:272:13)
  | at IncomingMessage.emit (events.js:185:15)
  | at IncomingMessage.emit (domain.js:422:20)
  | at endReadableNT (_stream_readable.js:1106:12)
  | at process._tickCallback (internal/process/next_tick.js:178:19)
  | Done in 438.48s.

I am not 100% sure where these come from, some suggest it might be the seleniumVersion used.

Expected behaviour

chimp exits non-zero (fails)

Actual behaviour

chimp exits zero (successful)

Exact steps to reproduce / repository that demonstrates the problem

Please put long code snips in a Gist and provide a link here.

Version & tools:

• Chimp command line used: The exact command you used
• Chimp config file: Please provide a Gist if you use a config file
• Chimp: 0.51.1
• Node.js: v9.4.0
• Java: openjdk version "1.8.0_112"
• Operation system: Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64

---

Join our Slack xolv.io/community #chimp channel, where you can find help and help others.

[samhatoum]

Comment by rsshilli
Monday Jun 04, 2018 at 21:24 GMT

---

We use CircleCI / AWS Fargate and see this too. I don't think this has anything to do with SauceLabs.

[samhatoum]

Comment by rsshilli
Tuesday Jun 05, 2018 at 17:18 GMT

---

This bug is super dangerous. I found out that one of our tests has been reporting as passing for months now but it's actually been broken the whole time. Yuck. The error I'm getting is:

  ✖ And I close the Risk Tables report tab
[chimp] Detected an unhandledRejection:
[chimp][hooks] Error: no such window: target window already closed
    from unknown error: web view not found
    at new RuntimeError (node_modules/webdriverio/build/lib/utils/ErrorHandler.js:144:12)
    at Request._callback (node_modules/webdriverio/build/lib/utils/RequestHandler.js:316:39)
    at Request.self.callback (node_modules/webdriverio/node_modules/request/request.js:186:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (node_modules/webdriverio/node_modules/request/request.js:1163:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (node_modules/webdriverio/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at node_modules/continuation-local-storage/node_modules/async-listener/glue.js:188:31
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback [as _tickCallback] (internal/process/next_tick.js:218:9)

@sanjo and @samhatoum I went to go submit a PR for this, and discovered that the correct code is in there:

https://github.com/xolvio/chimp/blame/6f3c6ea90405f9d3bcd2e4d8b6d321dbf0f09d10/src/lib/cucumberjs/hooks.js#L110-L111

Unfortunately it's commented out with a comment from 2 different commits from you from 2015 saying

Don't exit until the waitUntil uncaught promise bug is fixed in WebdriverIO

What bug is that? Is it this one?

webdriverio/webdriverio#1227

I can't understand why commenting out this exit call fixes that. Help?

[samhatoum]

Comment by rsshilli
Tuesday Jun 05, 2018 at 18:54 GMT

---

I opened a PR anyway to hopefully get this resolved. xolvio/chimp#691

[samhatoum]

Comment by Sanjo
Tuesday Jun 05, 2018 at 18:59 GMT

---

I don't remember what the bug is referring to. I don't maintain this package anymore. I approve this PR.