Specs json-reported 'pending' in Cypress 10.10.0 are now json-reported 'pending AND skipped' in Cypress 10.11.0

[RolandBurrows]

Current behavior

CONTEXT

• Cypress result shows skipped tests as pending #3092

• https://glebbahmutov.com/blog/cypress-test-statuses/

• pending - Slightly confusingly, if the user does not want an existing test to run, they can use the it.skip or xit syntax to skip the test or a block of tests. Cypress marks all these tests pending.

• skipped - The last test status is for tests that the user meant to run, but these tests were skipped due to some run-time error.

CURRENT CYPRESS 10.11.0 BEHAVIOR

• json reporter output shows pending tests as also skipped.

Desired behavior

Json reporter output behavior from Cypress v10.10.0, which shows pending tests as NOT ALSO skipped.

Test code to reproduce

$ npm install [email protected]

// cypress.config.js
const { defineConfig } = require("cypress");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      // implement node event listeners here
    },
    specPattern: '**/*.cy.js',
  },
  reporter: 'mochawesome',
  reporterOptions: {
    reportDir: 'cypress/results',
    overwrite: false,
    html: false,
    json: true
  }
});

// cypress/e2e/pass.cy.js
describe('pass', () => {
  it('pass', () => {
    cy.visit('https://example.cypress.io')
  })
})

// cypress/e2e/pending.cy.js
describe('pending', () => {
  it.skip('pending', () => {
    cy.visit('https://example.cypress.io')
  })
})

$ npm install [email protected]
$ ./node_modules/.bin/cypress run

       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ √  pass.cy.js                               00:03        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ √  pending.cy.js                             27ms        1        -        -        1        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    √  All specs passed!                        00:03        2        1        -        1        -

// cypress/e2e/pass.cy.js
  "stats": {
    "passes": 1,
    "pending": 0,
    "failures": 0,
    "passPercent": 100,
    "pendingPercent": 0,
    "skipped": 0,
    "hasSkipped": false
  }

// cypress/e2e/pending.cy.js
"stats": {
    "passes": 0,
    "pending": 1,
    "failures": 0,
    "passPercent": null,
    "pendingPercent": 100,
    "skipped": 0,
    "hasSkipped": false
  }

$ npm install [email protected]
$ ./node_modules/.bin/cypress run

       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ √  pass.cy.js                               00:02        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ √  pending.cy.js                             21ms        1        -        -        1        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    √  All specs passed!                        00:02        2        1        -        1        -

// cypress/e2e/pass.cy.js
"stats": {
    "passes": 1,
    "pending": 0,
    "failures": 0,
    "passPercent": 100,
    "pendingPercent": 0,
    "skipped": 0,
    "hasSkipped": false
  }

"stats": {
    "passes": 0,
    "pending": 1,
    "failures": 0,
    "passPercent": null,
    "pendingPercent": 100,
    "skipped": 1,           // UNEXPECTED BEHAVIOR
    "hasSkipped": true      // UNEXPECTED BEHAVIOR
  }

Cypress Version

10.11.0

Node version

v18.12.0

Operating System

Windows 10 Home 21H1

Debug Logs

No response

Other

No response

[emilyrohrbough]

@RolandBurrows Thank you for the detailed issue and reproducible example. I confirmed this behavior has changed from 10.10.0 to 10.11.0 and is still an issue in the latest v12 release.

I'm not really sure what change caused this problem, I assume my changes here: #24217. I opened a PR to fixed this behavior.

[cypress-bot]

The code for this is done in cypress-io/cypress#25264, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 12.3.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v12.3.0, please open a new issue.