Vitest ignores coverage tresholds if reports fail to be created

[christian-bromann]

Describe the bug

Given I have a vitest.conf.js defined as following:

export default defineConfig({
  test: {
    include: ['tests/**/*.test.js'],
    exclude: [
      'dist', '.idea', '.git', '.cache',
      '**/node_modules/**', 'examples/**'
    ],
    coverage: {
      enabled: true,
      statements: 100,
      branches: 100,
      functions: 100,
      lines: 100
    }
  }
})

The config states that I am interested in asserting the test coverage for my tests and have them fail if they don't meet the treshold (which are in this example 100%).

Due to the recent issues with creating test coverage reports, e.g. in #3125, we found out that our tests were not validated against our given treshold causing pull requests that decreased the overall coverage in our test suite to just pass. After we fixed the issue by updating the dependencies of vitest and @vitest/c8-coverage, coverage reports were created again but we got surprised by the failing treshold validation.

I think it is a bug if Vitest should ignores the treshold validation due to missing coverage reports. If test coverage validation is enabled by providing a statements, branches, functions or lines property, Vitest should ensure that the validation happens and fails if for some reasons it can't find any coverage data.

Reproduction

Open with VS Code:

or manually check out: https://github.com/christian-bromann/vitest-coverage-bug

System Info

System:
    OS: macOS 13.3
    CPU: (10) arm64 Apple M1 Pro
    Memory: 45.38 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.15.0 - ~/.nvm/versions/node/v18.15.0/bin/node
    npm: 9.5.0 - ~/.nvm/versions/node/v18.15.0/bin/npm
  Browsers:
    Brave Browser: 111.1.49.120
    Chrome: 112.0.5615.121
    Chrome Canary: 114.0.5721.0
    Firefox Nightly: 114.0a1
    Safari: 16.4
  npmPackages:
    @vitest/coverage-c8: ^0.29.1 => 0.29.7
    vitest: ^0.30.1 => 0.30.1

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

In [email protected] the #3040 introduced an API mismatch bug where vitest was unable to load @vitest/coverage-c8 package if version of @vitest/coverage-c8 was not 0.29.8 as well. In 0.30.0 this is fixed. Your reproduction case is running into this old (already fixed) bug. It's using [email protected] and @vitest/[email protected].

As the whole coverage provider package fails to load we cannot do anything coverage related - checking thresholds here would require duplicating the logic from coverage package to vitest core package. But I think we can improve the situation by failing whole process if there is coverage.enabled: true and the coverageModule below is falsy. So instead of silently failing to load the coverage package (and disabling coverage) due to possible API mismatch bugs, we'll fail the whole process.

vitest/packages/vitest/src/integrations/coverage.ts

Lines 18 to 21 in ddfeb0c

	if (provider === 'c8' || provider === 'istanbul') {
	const { default: coverageModule } = await loader.executeId(CoverageProviderMap[provider])
	return coverageModule
	}

The error message could clearly indicate that @vitest/coverage-<x> failed to load, and maybe even recommend to check the package versions.

[christian-bromann]

Awesome, thanks @AriPerkkio for the quick turnaround!