Tests slows down to a crawl and eventually times out/crashes

[joacim-boive]

• Operating System: MacOS High Sierra 10.13.1
• Cypress Version: 1.1.2
• Browser Version: Chrome 62, Canary 64, Electron 53

Is this a Feature or Bug?

BUG

Current behavior:

After loading about 50-80 links the test times out or crashes, what-ever comes first.
But it's not our environment that's slow, it's the test itself that's so slow it times out.

I've rewritten the same test using Headless Chrome and it works perfectly so, to me, that indicates a problem with Cypress itself.

The behavior is consistent regardless of browser or computer restart(s).

Desired behavior:

Running a successful test

This test, using puppeteer and headless chrome works and they achieve the same thing:

/* eslint-disable no-await-in-loop */
const puppeteer = require('puppeteer');
const devices = require('puppeteer/DeviceDescriptors');

const iPhone6 = devices['iPhone 6'];
const iPadLandscape = devices['iPad landscape'];

const BASE_HREF = 'http://tcy-wb040.lenslogistics.int';

async function run() {
  const browser = await puppeteer.launch({
    headless: true,
  });

  const page = await browser.newPage();

  await page.goto(`${BASE_HREF}/kontaktlinser/linslista?sort=name&_page=999`, { waitUntil: 'networkidle2' });

  await page.waitForSelector('.product-item');

  const lensProducts = await page.evaluate(async () => {
    const anchors = Array.from(document.querySelectorAll('.product-item'));
    return anchors.map(anchor => anchor.href);
  });

  for (let i = 0; i < lensProducts.length; i = i + 1) {
    const fileName = lensProducts[i].split(BASE_HREF)[1];

    await page.emulate(iPadLandscape);
    await page.goto(`${lensProducts[i]}`, { waitUntil: 'networkidle2' });
    await page.screenshot({ path: `screenshots/${fileName}-iPadLandscape.png` });

    await page.emulate(iPhone6);
    await page.reload();

    await page.screenshot({ path: `screenshots/${fileName}-iPhonePortrait.png` });
  }

  await browser.close();
}

run();

How to reproduce:

Run the test.

Test code:

const BASE_HREF = 'http://tcy-wb040.lenslogistics.int';
before(() => {
  cy.visit(`${BASE_HREF}/kontaktlinser/linslista?sort=name&_page=999`).scrollTo('bottom');
});

describe('Do all transformations for LENSES', () => {
  it('Should load all product pages and their thumbnails', () => {
    cy
      .get('#productList')
      .find('.product-item')
      .each($link => {
        cy
          .visit(`${BASE_HREF}${$link.attr('href')}`)
          .get('#productImage-section')
          .then($productSection => {
            if ($productSection.find('.js-thumbnail-image').length) {
              cy.get('.js-thumbnail-image').each($thumbnail => {
                cy.wrap($thumbnail).click({ force: true });

                cy.get('.product-image__image');
              });
            }
          });
      });
  });
});

Additional Info (images, stack traces, etc)

I'm not getting any stacktraces. The browser window freezes - turns white. Or times out after 60 seconds.

[jennifer-shehane]

This is almost the exact same issue I outlined here: #431 (comment)

Thanks for the extra work on implementing this with Headless Chrome, this is definitely helpful. :)

[brian-mann]

TLDR; run in CLI mode, not GUI mode to prevent OOM crashes on excessively long tests

Thank you for putting together a comprehensive example.

This is a known issue-ish because it's kind of a bug, but yet also a feature.

The reason Cypress is crashing and chrome headless isn't is because of the additional features Cypress provides that no other testing tool does. In this case it's the time travel functionality.

Cypress has two basic modes of operation:

• GUI mode
• CLI mode

There are substantial differences between the two.

GUI mode is what you use to iterate and write and run a single test at a time, whilst you build a feature in your application. This is what mode you're working in by default when you open Cypress.

CLI mode is when you run test(s) to completion and receive an exit code and summary at the end.

GUI mode is optimized for debugging and CLI mode is optimized for running.

The reason Cypress is slowing down and crashing is because in GUI mode it takes snapshots for every single command. Under normal circumstances, this isn't a problem. Even tests that have 100 commands rarely ever see this problem.

However your test does something pretty unusual - it uses cy.visit to navigate between pages. This is what we call "page crawling". The vast majority of tests do not do this, which is why it is not a problem everyone is experiencing. Cypress is especially not optimized for this mode in a single test because the snapshots it takes will preserve every single page transition.

This means that window and document are being preserved in memory for every single page transition. This is where it chews up memory and you'll experience those slow downs and crashes.

This problem is well known and documented in several places. Here are several issues and proposals that suggest ways to work around this problem:

• Cypress slows down drastically over long runs - restart renderer between spec files? #431 (comment)
• Proposal: enable spec globbing, enforce isolated renderer processes for each spec #681
• --spec flag should accept array of files or glob pattern #416 (comment)

There are some things Cypress does to try to prevent this from happening. For instance it begins to purge snapshots by default after the 50th test. This number is wayyyy too high, and we're going to be lowering it to something more like 5 in the future. But even if you were to set numTestsKeptInMemory to 0 your test would still crash.

Likely what we need to do is add another parameter to the configuration that sets a threshold for any given test which exceeds a certain limit of commands. After that number Cypress should begin purging snapshots FIFO style.

We should also investigate adding a new before:snapshot and / or snapshot event and enable you to return false from those to prevent adding new ones. That would allow you to control this behavior inside of code. Of course, the problem with any of these approaches is that the user has to be aware these options exist and know how to properly use them.

As a final note - if you were to run in CLI mode you would not see crashes because Cypress does not snapshot at all in that mode.

EDIT: There is one more potential problem I see. Even if you were to run from the CLI and/or turn of snapshotting, because you're using a closure from the cy.each, that means that every one of those DOM elements will be closed over and retained in memory until all of the callback functions exit and those $link elements are no longer reachable. This could potentially cause OOM as well. It's worth trying to run your test in CLI mode and see if it slows down. If it does, this is also why. If that's the case, you'll need to map over the $link elements and extract their href property (since thats all your care about) - and then use another .each on a basic array of strings. That would avoid the closure and enable garbage collection to happen on the previous window and documents.

I'll keep this issue open since it well documents the "single test" problem vs the "long runs" other people have opened issues about.

[joacim-boive]

Thank you for that thorough explanation! :)

Good catch on the .each I totally forgot about the closure it creates and that was indeed the source of the leak!

I modified the test but I can't remove all the closures since I need .then to be able to branch the test.
It manages more iterations now until it crashes when memory reaches around 4.5GB.

I have set numTestsKeptInMemory": 0 in cypress.json.

I did rewrite the test not using .each but I still run into the memory issues using the below test:

const BASE_HREF = 'http://tcy-wb040.lenslogistics.int';

const getProductItem = () => {
  cy.get('.product-item').then(links => {
    /**
     * We're using a for-loop to NOT create a closure, otherwise this will bleed memory
     */
    for (let i = 0; i < links.length; i = i + 1) {
      cy
        .visit(`${links[i].href}`)
        .get('#productImage-section')
        .then($productSection => {
          if ($productSection.find('.js-thumbnail-image').length) {
            // eslint-disable-next-line no-loop-func
            /**
             * We do want a closure here, just for these particular thumbnails
             */
            cy.get('.js-thumbnail-image').then(thumbnails => {
              for (let k = 1; k < thumbnails.length; k = k + 1) {
                cy
                  .wrap(thumbnails[k])
                  .click()
                  .get('.product-page__product-header')
                  .scrollIntoView();

                cy.get('#product-image__wrapper').find('.lazyloaded', { timeout: 20000 });
              }
            });
          }
        });
    }
  });
};

describe.only('Do all transformations for LENSES', () => {
  before(() => {
    const PATH = 'kontaktlinser/linslista';

    // cy.visit(`${BASE_HREF}/${PATH}?sort=name&_page=999`).scrollTo('bottom');
    cy.visit(`https://www.lensway.se/kontaktlinser/linslista?p_brand=Eyemed%20Technologies`).scrollTo('bottom');
  });

  it('Should load all product pages and their thumbnails', () => {
    getProductItem();
  });
});

For some odd reason, the test behaves differently depending on if I run them headlessly or not.
The above test works when I'm not using headless and fails when I do - every time.
So, I haven't been able to determine if it helps to run them in headless mode.

Any ideas as to why that is?
(I've modified the test so you can run it against or production environment if you so wish)

[jennifer-shehane]

Closing since this was already remarked as wontfix / duplicate. Please comment if you are still experiencing this issue and we can reopen.