Saving coverage data

[andrcuns]

Hei,
Encountered one issue, any suggestions would be welcome.

We are running jest tests using the knapsack pro jest runner and we would like to get coverage data for those runs as well using jest coverage reporters.

The issue is that coverage reporter always saves data to a single file and because knapsack executes tests in multiple chunks, coverage data get overwritten and we end up only we the one from last chunk execution.

[ArturT]

@andrcuns This is how @knapsack-pro/jest runs Jest:

https://github.com/KnapsackPro/knapsack-pro-jest/blob/master/src/knapsack-pro-jest.ts#L40

const {
    results: { success: isTestSuiteGreen, testResults },
  } = await jest.runCLI(
    {
      ...jestCLIOptions,
      runTestsByPath: true,
      _: testFilePaths,
    },
    [projectPath],
  );

How you generate a coverage report? Just with flag --coverage?
https://jestjs.io/docs/en/cli#--coverageboolean

1. I'm wondering how we could solve this problem. If we would be able to generate a unique name for each report generated for a batch of tests fetched from Queue then you would have many report files that you would have to later merge somehow. Today another person told me about https://codecov.io/ that could merge Jest reports.

2. We could figure out how to change Jest internally to somehow update the existing report on disk instead of overriding it.

3. An alternative solution would be to add Knapsack Pro Regular Mode to @knapsack-pro/jest and it would run Jest only once per parallel CI node so the coverage report would be generated only once.

Do you have any other ideas or suggestions on how to solve it? Thanks.

[andrcuns]

Yeah, with --coverage flag.

1. solution would be easiest. But for now it seems we are unable to generate unique file names: Pass options to coverageReporter jestjs/jest#4103. Though I am not sure if that would even help, I think reporter is not called each time, so even passing options with some function that would generate unique file name, quite possibly would be called only once.
   Merging those reports is not a problem, requires a simple enough script, I actually already did that to merge coverage from different CI nodes.

2. Not sure how to override that internal logic of jest to do that but it could be a workaround.

3. Regular node could be a solution, because I had a similar problem with jest-junit reporter but I was able to add a PR to reporter itself so it generates unique file names.

[ArturT]

Regarding 2.

We could figure out how to change Jest internally to somehow update the existing report on disk instead of overriding it.

@deser proposed the idea to do it in Jest repo jestjs/jest#9492

Regarding 3.

Regular node could be a solution, because I had a similar problem with jest-junit reporter but I was able to add a PR to reporter itself so it generates unique file names.

I added to my backlog a task to work on Regular Mode for @knapsack-pro/jest but it would be the best to have Jest coverage working with Queue Mode as Queue Mode gives better results of test suite splitting than Regular Mode.

@andrcuns Can you share your solution on how to configure jest-junit reporter and link to the issue? I could update my documentation so other people looking for the same problem could use your tips. You can send me this over email https://knapsackpro.com/contact to keep this thread to one topic. Thank you.

[ArturT]

Note: Here is @andrcuns idea to generate unique report name for jest-junit jest-community/jest-junit#108

I guess we have 2 options:

• update existing Jest report on the disk (that's proposed here Ability to configure updating coverage report rather than overriding it jestjs/jest#9492 )
• generate unique report names for each batch of tests executed by Jest inside of the single CI node similar to Generate unique xml file names jest-community/jest-junit#108

[ArturT]

I will try to implement a unique coverageDirectory as suggested here jestjs/jest#9492 (comment)

[ArturT]

I've released @knapsack-pro/jest 2.7.0 and here is PR that solves problem #13

Here you can find an example of how to generate code coverage for Jest with Knapsack Pro Jest in Queue Mode:
https://knapsackpro.com/faq/question/how-to-generate-code-coverage-for-jest-with-knapsack-pro-jest-in-queue-mode

[ArturT]

@andrcuns @deser Let me know if this works for you. For now, I'm closing this issue but feel free to post comments here.