Retry tests that failed when Chromium Renderer process crashed

[mirobo]

Description

A huge advantage of cypress-cloud compared to Cypress' Module API is that cypress-cloud could retry tests that failed due to a Chromium Renderer process crash.

I'm not sure what other types of crashes are possible but I guess this one could be handled by cypress-cloud.

I'm not sure if/how Cypress has the possibility to retry such a test, but currently they don't do it and they don't even produce an junit/xml result file (tested 10.7.0 with Cypress Module API and sorry-cypress). The crash is just logged into the console. But since cypress-cloud starts Cypress it could be possible to retry such failures. In my opinion it would even be legit to retry such failures even if test retries is not set (it still is a failure but it's not necessarily a flaky test in itself).

Or I create an issue at Cypress to make them retry those failures (which would be the preferrable solution anyway)..

What do you think?

Full error message:

We detected that the Chromium Renderer process just crashed.

This is the equivalent to seeing the 'sad face' when Chrome dies.

This can happen for a number of different reasons:

- You wrote an endless loop and you must fix your own code
- You are running Docker (there is an easy fix for this: see link below)
- You are running lots of tests on a memory intense application.
    - Try enabling experimentalMemoryManagement in your config file.
    - Try lowering numTestsKeptInMemory in your config file.
- You are running in a memory starved VM environment.
    - Try enabling experimentalMemoryManagement in your config file.
    - Try lowering numTestsKeptInMemory in your config file.
- There are problems with your GPU / GPU drivers
- There are browser bugs in Chromium

You can learn more including how to fix Docker here:

https://on.cypress.io/renderer-process-crashed

Example usage

N/A

[agoldis]

Great idea! Spec level retries could be an easy win, and a more granular retries policy that is based on the error message pattern of the filter could be a more advanced feature.

Internal reference: https://www.notion.so/currents/Renderer-crash-retries-fab2e160f94c4c6f9b7d8a2c241ed502?pvs=4

[mirobo]

Great idea on top of a great idea :-)
I mentioned this also here cypress-io/cypress#349 (comment)
But I don't think the issue in Cypress has a high priority.

It would be a good feature, not only for cypress-cloud but also for sorry-[whatever] in general (but then it would be much more difficult to implement because sorry-cypress would have to handle retries on top of Cypress retries).

But if it's done in cypress-cloud it is manageable. I guess the spec result state is stored solely on sorry-cypress instance and the test retries are handled within Cypress itself.

So before uploading the results, cypress-cloud would check the error message and check that it's only ONE attempt without any retries (so we know that cypress-cloud is not re-running a test that already had retries).

Then it would re-run the whole spec once and just use this one result (no matter if it failed again).

The only question remains if you would somehow include the first attempt or just discard it completely. I think discarding is ok, but still it would be nice to know if a certain error still occurs. So the best solution would be to add an "attempt" to the spec results object.

If the error message pattern was stored in sorry-cypress, it could later be also used for this idea sorry-cypress/sorry-cypress#761 :)

[mwyrodek]

I like the idea - since cypress 1x we are seeing much more of those errors