Cypress encounters unrelevant timeouts within the pipeline when upgrade to v3.8-v4

[san-slysz]

Current behavior:

We used to be in 3.3.2. We decided to move to v4 (to try firefox). We started having pipleine issues, with scenario failing during the beforeEach (visit or request) due to timeout.

After a diagnosis, we identify that we were fine in 3.4.0 but not in 3.8.3. We will thus stay in 3.4.0 for the moment. Below, you'll find more details.

Desired behavior:

Running tests -> No timeout using Cypress 3.8 or 4 (such as when we run using Cypress 3.4).

Test code to reproduce

Prerequisites

• We have simple front and api tests. They use beforeEach().
• BeforeEach either starts with a cy.visit(), cy.request()
• All configuration leads to no issue locally (cypress open -> run all spec -> all is fine)
• Our pipeline runs within Gitlab, using Electron

With cypress 3.4.0 all our tests runs in our pipeline just fine (if red -> application bug, else green)

With cypress 3.8.3 some of our tests timeouts in our pipeline. We know it's false positive as we run it on the same app that tests in 3.4. We also do not produce running the tests locally. It's not always the same tests that fails. On our 30-ish tests pipeline, we always have around 10-ish tests that fails, so every pipeline is affected.

Here the error messages we get (depending on if the scenario starts with, visit or request).

//=================================
2) Front scenario 
        "before each" hook for "Front scenario we run":
      CypressError: cy.request() timed out waiting 30000ms for a response from your server.
 The request we sent was:
 Method: POST
 URL: https://ourApiDns/path
 No response was received within the timeout.
 Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite: 'Front scenario...'
       at Object.cypressErr (https://ourDNS/__cypress/runner/cypress_runner.js:86207:11)
       at Object.throwErr (https://ourDNS/__cypress/runner/cypress_runner.js:86162:18)
       at Object.throwErrByPath (https://ourDNS/__cypress/runner/cypress_runner.js:86194:17)
       at https://ourDNS/__cypress/runner/cypress_runner.js:72513:23
       at tryCatcher (ourDNS/__cypress/runner/cypress_runner.js:120203:23)
       at ourDNS/__cypress/runner/cypress_runner.js:115344:41
       at tryCatcher (ourDNS/__cypress/runner/cypress_runner.js:120203:23)
       at Promise._settlePromiseFromHandler (https://ourDNS/__cypress/runner/cypress_runner.js:118139:31)
       at Promise._settlePromise (https://ourDNS/__cypress/runner/cypress_runner.js:118196:18)
       at Promise._settlePromise0 (https://ourDNS/__cypress/runner/cypress_runner.js:118241:10)
       at Promise._settlePromises (https://ourDNS/__cypress/runner/cypress_runner.js:118316:18)
       at Async../node_modules/bluebird/js/release/async.js.Async._drainQueue (https://ourDNS/__cypress/runner/cypress_runner.js:114928:16)
       at Async../node_modules/bluebird/js/release/async.js.Async._drainQueues (https://ourDNS/__cypress/runner/cypress_runner.js:114938:10)
       at Async.drainQueues (https://ourDNS/__cypress/runner/cypress_runner.js:114812:14)`
//=================================
`Running:  CORE/ScenarioName/scenario.js                                                   (10 of 29)
   ScenarioName
     1) "before each" hook for "Scenario Name"
   0 passing (33s)
   1 failing
   1) ScenarioName
        "before each" hook for "Scenario Name":
      CypressError: cy.visit() failed trying to load:
 https://ourDNS/pageId
 We attempted to make an http request to this URL but the request failed without a response.
 We received this error at the network level:
   > Error: ETIMEDOUT
 Common situations why this would fail:
   - you don't have internet access
   - you forgot to run / boot your web server
   - your web server isn't accessible
   - you have weird network configuration settings on your computer
 The stack trace for this error is:
 Error: ETIMEDOUT
     at Timeout.<anonymous> (/root/.cache/Cypress/3.8.3/Cypress/resources/app/packages/server/node_modules/request/request.js:848:19)
     at listOnTimeout (internal/timers.js:531:17)
     at processTimers (internal/timers.js:475:7)
 Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite: 'Scenario Name'
       at Object.cypressErr (http://localhost:37257/__cypress/runner/cypress_runner.js:86207:11)
       at Object.throwErr (http://localhost:37257/__cypress/runner/cypress_runner.js:86162:18)
       at Object.throwErrByPath (http://localhost:37257/__cypress/runner/cypress_runner.js:86194:17)
       at http://localhost:37257/__cypress/runner/cypress_runner.js:71500:31
       at visitFailedByErr (http://localhost:37257/__cypress/runner/cypress_runner.js:71006:12)
       at http://localhost:37257/__cypress/runner/cypress_runner.js:71499:22
       at tryCatcher (http://localhost:37257/__cypress/runner/cypress_runner.js:120203:23)
       at Promise._settlePromiseFromHandler (http://localhost:37257/__cypress/runner/cypress_runner.js:118139:31)
       at Promise._settlePromise (http://localhost:37257/__cypress/runner/cypress_runner.js:118196:18)
       at Promise._settlePromise0 (http://localhost:37257/__cypress/runner/cypress_runner.js:118241:10)
       at Promise._settlePromises (http://localhost:37257/__cypress/runner/cypress_runner.js:118316:18)
       at Async../node_modules/bluebird/js/release/async.js.Async._drainQueue (http://localhost:37257/__cypress/runner/cypress_runner.js:114928:16)
       at Async../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:37257/__cypress/runner/cypress_runner.js:114938:10)
       at Async.drainQueues (http://localhost:37257/__cypress/runner/cypress_runner.js:114812:14)`
I wondered if it could be linked to overall performance issues such as https://github.com/cypress-io/cypress/issues/5987. But as this is not proven, I open this dedicated issue.
//=================================

Versions

3.8.3 and 4.0.1 are affected. Could affect earlier versions.
3.4.0 is not affected.
We did not test versions in between 3.4.0 and 3.8.3.

I could give more information on our scenarios via MP.

[jennifer-shehane]

Narrowing down to the exact version that breaks would be helpful. As this issue stands there are multiple versions this could have been released in:

3.8.3
3.8.2
3.8.1
3.8.0
3.7.0
3.6.1
3.6.0
3.5.0
3.4.1

We are not currently aware of an issue similar with timeouts being released in any of the above versions.

We'll also need a reproducible example in order to look into this. This does not mean that your issue is not happening - it just means that we do not have a path to move forward.

Please comment in this issue with a reproducible example and a narrower focus of where the issue was introduced. Otherwise we will have to close this issue.

[san-slysz]

I saw your feedback. I'll try to narrow it down and come back with data this week. I appreciate the clear feedback an fully understand. I need to help you help me ;).

[jennifer-shehane]

Unfortunately we have to close this PR due to inactivity. Please open a new PR addressing the original issue and any requested changes.

[san-slysz]

We solved everything and more with v5 perf improvements.