[CSR-638] Cypress-grep compatibilty

[agoldis]

See https://sorry-cypress.slack.com/archives/C04F7LJ1E3H/p1671443824388969

The plugin is widely used, and we need to restore its functionality. I don't have many details except for the user's report.

[Rikkun-1]

I am using:
cypress: 12.16.0
@cypress/grep: 3.1.5
sorry-cypress: 2.5.1
cypress-cloud: 1.8.2

Existing in repo test case is very simple and does not describe real world usage:

• 5d7f36d#diff-6bd6fd4f71bae5e0b20dfcc2a86a659b57a657f546d2bdf033b7825d985f9376R1-R17
• 5d7f36d#diff-5ba646cdcf865c7d8146e78da174190a6cc056645ad7ee916c4df28844b4ed0aR1-R5.

There is only one spec file and it satisfies grepTags.
But if we have two spec files and one of them does not contain any case that satisfies grepTags then run will fail.

For example if we have following configuration where we have two files each with different tag:

// testTagA.spec.js
it('My test', { tags: ['@tagA'] } () => {
  // ...
});

// testTagB.spec.js
it('My test', { tags: ['@tagB'] } () => {
  // ...
});

then if we run it like this:
npx cypress-cloud --env grepTags=@tagB ...

Then result will be like this:

Running: cypress/e2e/testTagA.spec.js (1/2)
@cypress/grep: filtering using tag(s) "@tagB"
Can't run because no spec files were found.

We searched for specs matching this glob pattern:

> /home/.../projects/frontend/cypress/e2e/testTagA.spec.js
WARNING  Cypress runner failed with message: "Could not find Cypress test run results"
WARNING  The following spec files will be marked as failed: 
- /home/.../projects/frontend/cypress/e2e/testTagA.spec.js

Reporting results and artifacts in background...  

I think thats because cy2 implementation tried to run all specs in one cypress run.
So when one of specs is filtered out by @cypress/grep there is still another one that will be run.

But cypress-cloud works differently.
It runs every spec file in separate cypress run command.
Thus, there is only one spec file per run so when @cypress/grep filters out that single file, cypress fails with an error that there is no test cases that satisfy spec and tag filtering.
Cypress programmed to fail if it can't find any test case in entire run.
So if, instead of one big run with all specs, we create many runs with one spec, then all runs that don't match the grep pattern will fail.

I can't use @cypress/grep after I moved from cy2 to cypress-cloud.
This issue makes @cypress/grep incompatible with cypress-cloud at the moment which is very sad for me.
@cypress/grep is crucial for big projects with many tests.

I would be very happy if this issue will be resolved and @cypress/grep will become compatible again.
Thanks in advance for your time.

[agoldis]

@Rikkun-1 thanks for reporting that issue - good catch, we'll issue a fix soon

[Rikkun-1]

I am using: cypress: 12.16.0 @cypress/grep: 3.1.5 sorry-cypress: 2.5.1 cypress-cloud: 1.8.2

Existing in repo test case is very simple and does not describe real world usage:

• 5d7f36d#diff-6bd6fd4f71bae5e0b20dfcc2a86a659b57a657f546d2bdf033b7825d985f9376R1-R17
• 5d7f36d#diff-5ba646cdcf865c7d8146e78da174190a6cc056645ad7ee916c4df28844b4ed0aR1-R5.

There is only one spec file and it satisfies grepTags. But if we have two spec files and one of them does not contain any case that satisfies grepTags then run will fail.

For example if we have following configuration where we have two files each with different tag:

// testTagA.spec.js
it('My test', { tags: ['@tagA'] } () => {
  // ...
});

// testTagB.spec.js
it('My test', { tags: ['@tagB'] } () => {
  // ...
});

then if we run it like this: npx cypress-cloud --env grepTags=@tagB ...

Then result will be like this:

Running: cypress/e2e/testTagA.spec.js (1/2)
@cypress/grep: filtering using tag(s) "@tagB"
Can't run because no spec files were found.

We searched for specs matching this glob pattern:

> /home/.../projects/frontend/cypress/e2e/testTagA.spec.js
WARNING  Cypress runner failed with message: "Could not find Cypress test run results"
WARNING  The following spec files will be marked as failed: 
- /home/.../projects/frontend/cypress/e2e/testTagA.spec.js

Reporting results and artifacts in background...  

I think thats because cy2 implementation tried to run all specs in one cypress run. So when one of specs is filtered out by @cypress/grep there is still another one that will be run.

But cypress-cloud works differently. It runs every spec file in separate cypress run command. Thus, there is only one spec file per run so when @cypress/grep filters out that single file, cypress fails with an error that there is no test cases that satisfy spec and tag filtering. Cypress programmed to fail if it can't find any test case in entire run. So if, instead of one big run with all specs, we create many runs with one spec, then all runs that don't match the grep pattern will fail.

I can't use @cypress/grep after I moved from cy2 to cypress-cloud. This issue makes @cypress/grep incompatible with cypress-cloud at the moment which is very sad for me. @cypress/grep is crucial for big projects with many tests.

I would be very happy if this issue will be resolved and @cypress/grep will become compatible again. Thanks in advance for your time.

I want to add a very important note to this comment.
I was wrong, the @cypress/grep plugin is not completely incompatible with cypress-cloud.
I forgot that I added the grepOmitFiltered option to my config file a long time ago.
Adding this will not only skip tests that don't match the tags, but it will remove those tests from the run entirely, resulting in the error described above.
@cypress/grep plugin works fine if you don't use grepFilterSpecs, grepOmitFiltered options.
Tests that don't match the grep pattern will simply be marked as skipped, and the runner will be fine even if all tests in the file are skipped.

P.S.
However, grepOmitFiltered is a very important option.
With it, those files that do not fit at all are completely discarded.
Without it, fetching a small set of tests with the filter is too slow, because instead of discarding completely unsuitable files, it first launches the browser and then skips all the cases for each individual file that usually would be discarded with grepOmitFiltered.

[agoldis]

That one was tricky!

First, the issue is only relevant when grepFilterSpecs is activated, grepOmitFiltered doesn't really matter.

@cypress/grep mutates config object - it modifies config.specPattern to include only the specs that match the tags. The order of plugins in setupNodeEvents matters, consider this:

async setupNodeEvents(on, config) {
      await currents(on, config);
      require("cypress-terminal-report/src/installLogsPrinter")(on);
      return require("@cypress/grep/src/plugin")(config);
},

• await currents(on, config); runs first and uses the original config.specPattern - it will include all the files defined in the original cypress configuration
• @cypress/grep runs afterwise, mutating the config, and excluding certain specs
• cypress-cloud will run cypress for spec files that should have been filtered, cypress activates @cypress/grep, excludes the intended file and fails because it has nothing to run

As a workaround, make sure to run require("@cypress/grep/src/plugin")(config); before await currents(on, config);, for example:

async setupNodeEvents(on, config) {
      require("cypress-terminal-report/src/installLogsPrinter")(on);
      require("@cypress/grep/src/plugin")(config);
      return currents(on, config);
    }

Not sure what to do with it because cypress-cloud is compatible, it's more of a configuration issue and probably a note in the documentation would be sufficient. @Rikkun-1 please let me know if the workaround worked for you.

[agoldis]

@Rikkun-1 ?

[Rikkun-1]

Sorry for the delay, @agoldis.
I was busy with work.

Your workaround works great.

I ran into another issue with @cypress/grep when tried to test it with grepFilterSpecs but it's easy to solve with this comment cypress-io/cypress#27216 (comment).

Thanks a lot for your workaround, it's really important for test suite management.

I think this issue can be closed.

[agoldis]

@Rikkun-1 awesome!
I updated to documentation with a reference to this issue: https://github.com/currents-dev/cypress-cloud/blob/main/.github/README.md#usage-with-cypressgrep