Skipped tests due to a failure in before() are not reported

[mikeyeung123]

When you run this simple test:

describe("Sample test", () => {

  before("Fails here", () => {
    throw new Error("Manual failure in before()");
  });

  it("Should be skipped", () => {
    expect(true).to.be.true;
  });

  it("Should also be skipped", () => {
    expect(false).to.be.true;
  });
});

You get the following command line output:

     Spec                                              Tests  Passing  Failing  Pending  Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ×  SampleTest.spec.ts                       516ms        2        -        1        -        1 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
  ×  1 of 1 failed (100%)                     516ms        2        -        1        -        1

And the GUI looks like this:

Both the command line and the GUI reports one skipped test: “Should also be skipped”. But the XML report created using the JUnit reporter doesn’t list the test case as a skipped test; it just vanishes:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0" tests="2" failures="1">
  <testsuite name="" timestamp="2020-05-11T21:32:32" tests="0" file="cypress\integration\SampleTest.spec.ts" failures="0" time="0">
  </testsuite>
  <testsuite name="Root Suite.Sample test" timestamp="2020-05-11T21:32:32" tests="2" failures="1" time="0">
    <testcase name="&quot;before all&quot; hook: Fails here for &quot;Should be skipped&quot;" time="0" classname="Sample test">
      <failure message="Manual failure in before()

Because this error occurred during a `before all` hook we are skipping the remaining tests in the current suite: `Sample test`" type="Error"><![CDATA[Error: Manual failure in before()

Because this error occurred during a `before all` hook we are skipping the remaining tests in the current suite: `Sample test`
    at Context.<anonymous> (http://localhost:8087/__cypress/tests?p=cypress\integration\SampleTest.spec.ts-853:113:15)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

To Jenkins, the total number of tests is one. If the test didn’t fail in before(), the report would instead look like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.028" tests="2" failures="1">
  <testsuite name="" timestamp="2020-05-11T21:49:47" tests="0" file="cypress\integration\SampleTest.spec.ts" failures="0" time="0">
  </testsuite>
  <testsuite name="Root Suite.Sample test" timestamp="2020-05-11T21:49:47" tests="2" failures="1" time="0.028">
    <testcase name="Should be skipped" time="0.028" classname="Sample test">
    </testcase>
    <testcase name="Should also be skipped" time="0" classname="Sample test">
      <failure message="expected false to be true" type="AssertionError"><![CDATA[AssertionError: expected false to be true
    at Context.<anonymous> (http://localhost:8087/__cypress/tests?p=cypress\integration\SampleTest.spec.ts-665:119:28)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

You can see that if before() succeeds, we get two test cases, but if it fails, we get one.

These are my reporter options:

{
  "reporter": "junit",
  "reporterOptions": {
    "jenkinsMode": true,
    "mochaFile": "cypress-test-results/test-result-[hash].xml",
    "includePending": true
  }
}

I have tried turning on and off jenkinsMode, and have tried other reporters, including “json”, “spec”, and “mocha-jenkins-reporter”. None of them correctly report skipped tests.

I believe fixing this issue is important because the reported test results should match what the user sees in the command line or the GUI.

I am using Cypress 4.4.0 on Windows 10.

[jennifer-shehane]

When there is an error thrown within a before, there has to be a test to attribute the failure to. We attribute the failure to the first test. Any subsequent tests are skipped. The tests are in a failed state, so showing both of the tests as skipped would be inaccurate. There is no way to report failures outside of a test - there is no attribution within a report to some failure outside of a test state.

I can't speak to how junit creates their reports.

This is how we've implemented failures in hooks, so if there is a failure in a hook, you should expect one test to be attributed as a 'fail' and the rest as 'skip' within all of Cypress reporting. Outside reports are outside of our control.

[mikeyeung123]

This is the desired behavior: “If there is a failure in a hook, you should expect one test to be attributed as a ‘fail’ and the rest as ‘skip’ within all of Cypress reporting.”

The issue is that what Mocha reporters report are not that. As you see in my screenshots and snippets, Cypress correctly reports skipped tests in the command line output and the GUI, but the test reporters don’t report skipped tests. In this page, the Cypress docs say “Because Cypress is built on top of Mocha, that means any reporter built for Mocha can be used with Cypress.” What I suspect is simply that there is some inconsistency with how Mocha registers skipped tests and how Cypress does, and that inconsistency should be fixed so the reporting is correct.

This is not an issue in a third party reporter like JUnit, but rather how Cypress sends its test data to all Mocha reporters.

[mikeyeung123]

Hey Cypress team, any update on this? Thanks a lot!

[jennifer-shehane]

There is a difference in how Cypress designates skipped tests - see #3092

Duplicate of #3092