Cypress 4.0.1 - Electron is 1.5x slower in a test spec

[sandrina-p]

Current behavior:

In our project, we are updating Cypress from 3.5 to 4.0.1. See PR okTurtles/group-income#829

The same test spec "Large Group" takes 2:29m to run instead of 1:36m. The other specs take the same time. See run 284 - @3.5 vs run 285 - @4.01.

Desired behavior:

All specs should take the same time.

Test code to reproduce

1. Clone our project and checkout the PR (or master to see v 3.5.0)
2. Install dependencies npm install
3. Run the project grunt dev
4. In another terminal run cypress npm run cy:open
5. Run the test step "Group Large" on Electron.

The test in the spec taking too long is located at test/cypress/integration/group-large.spec.js. See file here

  it(`A group with ${groupLength} members shows correctly the pledging month overview widget`, () => {
    for (let i = 2; i <= groupLength; i++) {
      // The member joins the group through an invitation...
      cy.giAcceptGroupInvite(invitationLinks.anyone, {
        username: `user${i}-${userId}`,
        groupName,
        bypassUI: true,
        actionBeforeLogout: () => {
          if (i > 3) {
            cy.getByDT('contributionsLink').click()
          }
          cy.giAddRandomIncome()
        }
      })
    }

    cy.giLogin(`user1-${userId}`)
    cy.giAddRandomIncome()
    cy.get('.graph-bar')
      .should('have.length', groupLength)
    cy.giLogout()
  })

If anything else is missing, please let me know.

Versions

• Cypress 3.5 and 4.0.1
• Browser Electron 72 and 78 (maybe that's the reason?)
• OS Ubuntu 14.04

[Olgagr]

We have the same issue. However, we noticed the speed tests degradation from version 3.8.2 to 3.8.3 (15 minutes slower). We thought that maybe the speed improves in version 4 but it is not.

Versions

• Cypress 3.8.2 to 4.0.1
• Electron 78 (headless)
• Doker image: cypress/base:10

[slava-44]

I have the same problem on Chrome: my tests takes 12 min on Cypress 3.8.3 and 19 min on Cypress 4.0.1

[oneumyvakin]

In compare with 3.6.1 we have the same issue with slowing down 15 tests from 1 min 50 sec to 2 min 50 sec - which is very significant for us.

Cypress: 4.0.1
Browser: Electron 78 (headless)

[samar-77]

Same here:

cypress run --headless --browser chrome (on windows 10)

Cypress 3.8.3 with Headless chrome 80 --> specs run in 2:30
Cypress 4.0.1 with Headless chrome 80 --> specs run in 7:35

[Thundix]

Hello,

For information, I have the same issues as Olgagr, performance drastically slowdown when upgrading above 3.8.2 (3.8.3 + 4.0.2 tested)

I see network calls for static resources are very slow (css, png,...)

I'm running on Windows

Now, on CI, which is running on Linux, no degradation with 4.0.2 nor 3.8.3

But I cannot migrate the test to fit 4.0.2 as it's really too slow in local :(

(a bit like before this update https://www.cypress.io/blog/2019/05/22/how-we-improved-network-speed-by-300-in-cypress-3-3-0/)

I'm for the moment downgrading to 3.8.2, would be glad to have an update on this

[Thundix]

For info, it's actually working on my personal PC

We are using px software as proxy in local, I'm suspecting an issue with this

Will keep posted here if I find a workaround

[kriegster108]

Tried upgrading to 4.1.0 today after coming from 3.8.0 and got HUGE performance hit. Went from 7min runtime to 21Min+. Had to downgrade back to 3.8.0
Versions:
Cypress 3.8.0 to 4.1.0

Runtime with 3.8.0: 06:58

Runtime with 4.1.0: 21:02

It seems that when our client makes http requests while testing the app, it literally "freezes" up the browser(i cant interact with the webpage at all in this state) and ends up taking forever to run. I was able to replicate on my local machine using chrome and electron this behavior, and our linux docker image that cypress is installed onto also has this similar behavior in our build pipeline (electron headless)

Other information. Our app runs on angular 8. The slowdowns seem to occur when having to use our search function, as in when the user types in letters it goes out and retrieves search results in real time. The Test runner timer at the top of the page will literally freeze. I verified in dev tools the API calls that are being made get registred latency of less than 100ms so its not my network being a issue. 3.8.0 does not have this issue.

[samar-77]

Just updated Cypress to 4.3.0 on my Windows 10 machine.

Running with Chrome 80 headless and Cypress 3.8.3 => 3.47 minutes
Running with Chrome 80 headless and Cypress 4.3.0 => 16.10 minutes

Is there any update on this, Cypress team?
Can't really benefit from any Cypress updates from now on...

B.t.w. Going to a Linux machine is not an option at the moment ;-)

[kriegster108]

tried with Cypress 4.5 today, and the performance is still extremely slow. each XHR request my client is trying to make takes FOREVER to resolve and the entire runner is just frozen up. Our organization is still on 3.8.0 because of this. I dont know if i can get a working example to use but i'll see.

[yizha]

3.1.3 --> 4.5.0
155 passing (7m) --> 155 passing (14m)

[MoRTDooM]

Is there any future plan to fix the performance issue?...
In our organisation we have not updated since 3.4.1 as now it would not make much sense to move to 3.7 which seems to have still a good timings, but cypress 4.7 has just been released and there's no sign of the performance issue being addressed.

[filipsnopek]

Same in our organisation. We're still running on 3.7.0 because of the performance drop.

We've ran same 5 jobs in TeamCity on the same agent for 3.7.0 and 4.7.0 (following Migration Guide) and had following results:

[simologos]

Same here. Since we are using Cypress to test an application which controls phone calls, we unfortunately rely on timing (calls only ring for 10 seconds for example). Upgrading to 4+ (even 4.7) brakes half of our existing test.

Our dev machine + CI is also windows based, and tests are running at least 3-5 times slower.
On our end it does not matter if we run the tests with Chrome, Firefox or Electron. Tests which took previously 8s to complete, now take ~30s.

Happy to share a debug log or other information.

[fsteinbauer]

Hi @jennifer-shehane,
do you know if there are any updates from the Cypress Developers?
We are currently executing more than 6000 tests nightly and this would quite significantly slow down our execution.

We would be happy to provide more info information or help fix this topic.

[samar-77]

Hi @jennifer-shehane or any other Cypress developer.
Is there any news about this?
Seems that more than a few organisations are impacted by this performance issue.

Is there something you can say about when you going to look into this?

[jennifer-shehane]

The issue

From the comments, there have been various mentions of performance hits:

• 3.5.0 to 4.0.1 (52 secs slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.8.2 to 3.8.3 (15 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.8.3 to 4.0.1 (7 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.6.1 to 4.0.1 (1 min slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.8.3 to 4.0.1 (5 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.8.2 to 3.8.3 Window only on px software as proxy in local (slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.8.0 to 4.1.0 (14 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment) - this sounds identical to this issue Tests slow down over time when performing several XHR requests calls in single test #6783
• 3.8.3 to 4.3.0 (12.5 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.8.0 to 4.5.0 (slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.1.3 to 4.5.0 (7 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.7.0 to ?.?.? (slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• 3.7.0 to 4.7.0 (4 mins slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)
• ?.?.? to 4+ (slower) Cypress 4.0.1 - Electron is 1.5x slower in a test spec #6388 (comment)

Some people mention 3.8.3 being the breaking version, while others mention a version greater than 3.8.3 being the introduction of the performance issue.

Additionally there is variation in mentioning which browsers and environments this happens on among the comments although the majority to mention Electron.

From this, there can't be any real determination on which version, environment, or test code to focus on to narrow down the issue.

Related issues

Your issue may have already been identified elsewhere, please refer to those issue if your issue looks similar.

• Tests slow down over time when performing several XHR requests calls in single test Tests slow down over time when performing several XHR requests calls in single test #6783
• Running large amount of DOM related Cypress commands will freeze/slow up the Test Runner, likely due to visibility checks Performance issues of the Runner when using Cypress commands on many DOM elements at once #6967
• Performance issues when iterating over large object. Spike in CPU usage when iterating over large 'body' object with long timeout. #6944
• 3.8.0 Regression: Tests slow down after failing tests when accessing request body Failing tests slow down all following tests, even in headless mode #6881
• 3.8.0 Regression due to Electron upgrade: Several commands in a single test slow down over time, likely due to visibility checks. Tests that iterate over many DOM elements with .each() run slower in Electron during cypress run in 3.8.0 #5987

We need a reproducible example

The original provided example is not runnable (the grunt dev task fails on the rollup:watch task).

I don't doubt that there is some performance regressions, but we can't identify it without a reproducible example.

Please comment in this issue with a reproducible example - one that shows marked performance issues in on specific version of Cypress versus an older version so that we can investigate.

[sandrina-p]

@jennifer-shehane , what error exactly do you have when running grunt dev?

[MoRTDooM]

I have now been testing v5.0.0. and I do not see any significant improvement in performance to what has been discussed in this thread...
Has anyone have any clue to how could it be possible to have the same performance in all these new versions matching the performance in the old versions that we have been talking about... I would very much like to be able to update to recent version but I cannot compromise the overall performance doing it, any tip will be very much appreciated, thanks!!

Here's an example of the very same test run on one of our features with Electron, one with v3.4.1 (111.32s) and the other with v5.0.0 (232.50s), notice the different is quite significant:

Cypress 3.4.1

Cypress 5.0.0

[denisLoncaric]

Hi, I tried it with cypress 5.0.0 on macOS Catalina 10.15.2.

Running with Electron 78 headless and Cypress 3.8.3 takes 05:23 minutes
Running with Electron 83 headless and Cypress 5.0.0 takes 12:04 minutes

We would like to update the package for quite some time now, any updates here dear Cypress team?

[andrewmtam]

Building off of this thread: #2970 (comment)

we tried setting overflow:hidden to the command log only when the test is run in CI (via checking environment variables), and this has yielded large performance benefits when running tests through CI.

const sidebar = window.parent.document.querySelector('.reporter-wrap .container');
if (sidebar) {
    sidebar.setAttribute('style', 'overflow: hidden;');
}

On [email protected], our test suite took about 1.75hrs.
On [email protected] without these changes, the test suite timed out at 3hrs+
On [email protected] with these changes, we were back down to 1.5hrs

Of course this isn't entirely robust as the UI can change in future versions, and it doesn't necessarily fix the underlying issue, but it enabled us to upgrade cypress from 3.4.1 to 5.1.0 without inheriting the large performance hits described in this thread. One caveat is that in some recordings, it looked like the runner did not continually keep scrolling the command log, but in most instances it did.

For local development, we did something similar, where we added a button to the UI that toggles the command log. Here's an incomplete snippet, but it should convey the general approach:

        const icon = document.createElement('i');
        icon.setAttribute('class', 'fas fa-list');

        const button = document.createElement('button');
        button.setAttribute('class', 'toggle-command-log');
        button.appendChild(icon);


        button.onclick = toggleCommandLog;
        button.setAttribute('aria-label', 'Toggle Command Log');

        const newCommandLogExpandButton = document.createElement('span');
        newCommandLogExpandButton.appendChild(button);

        commandLogButtons.appendChild(newCommandLogExpandButton);

We tossed both of these snippets into a before hook. Hopefully others find this work-around handy!

[emilio-notable]

we just tested @andrewmtam 's workaround and we got pretty good results as well.

We've been trying to upgrade from 4.3.0, and every version was much slower, except 5.1.0 (which includes this PR) which was not as slow as the other, but still slower than our baseline with 4.3.0.

Setting overflow: hidden the test duration is pretty much the same as 4.3.0, and sometimes even faster

[peruukki]

I quickly tested using a windowed list in the command log in #6783 (comment) and saw similarly good results. Probably some kind of large list efficient technique would be needed in the command log to solve the performance issues.

[jennifer-shehane]

We're investigating some performance issues related to the rendering of the Command Log such as #6783 where I think the solution to that may also solve this issue.

Since there was never a reproducible example provided in this thread - we will have no way of actually testing if any solution we provide solves these however. But we'll followup in here when a PR is open for that.

There are some cases where hiding the Command Log does improve performance, and this is what we're investigating - what in the rendering of the component is causing the slowdown and how we can improve that.

[jennifer-shehane]

Closing this issue as we have resolved SEVERAL performance issues over the year, especially in the last few months. Please upgrade to the latest version and open a new issue with a repro if you're experiencing performance issues.