Investigate code coverage discrepancies between Codacy, JaCoCo, and CodeCov

[nathanklick]

Description

The code coverage numbers reported by Codacy, CodeCov, and JaCoCo appear to all report different coverage figures which possibly indicates one or more configuration issues.

Additionally, there is a concern some code coverage is not being properly picked up or reported at all.

[jjohannes]

State after some investigation:

• Codecov upload broken right now Error: Codecov token not found.
• Both 'Codecov' and 'Codacy' use the JaCoCo XML reports as base. These are right now created per-project. Which only gives a "partial view" on the overall coverage. Code that is covered by tests (e.g. in API only projects of services like hedera-smart-contract-service) is counted as uncovered, because there are not tests directly in the API project covering them. I think it would be better to let Gradle aggregate a full report (one XML file) and then pass that to both 'Codecov' and 'Codacy'. Then we have the same input for both and can also generate a corresponding HTML report with JaCoCo/Gradle only locally. WiP PR: build: use 'jacoco-report-aggregation' Gradle plugin for a unified setup #13073

[jjohannes]

Investigation results

• With build: use 'jacoco-report-aggregation' Gradle plugin for a unified setup #13073, we make sure the same data is uploaded to both both Codecov and Codacy (as a single aggregated XML file)
• With build: use 'jacoco-report-aggregation' Gradle plugin for a unified setup #13073, there is also an aggregated report to view locally after running :reports:testCodeCoverageReport
• The numbers in the reports are different because:
  • The local report takes generated code (protobuf and dagger) into account. Therfore it sees more sources and the coverage percentage is lower.
  • IIUC, Codecov and Codacy take the code directly from the Git repository. So they cannot see the generated code. As a result the coverage percentage is higher. I experimented with excluding generated classes from the report aggregation in general and then the number in the local report and Codecov align.
  • The number in Codacy is always slightly higher. I am not totally sure why that is, but I assume the calculation, while based on the same data, works a bit different. What I suspect is that Codecov counts every line touched as "fully covered" (like here - 100% covered), while Codacy also takes "partial coverage" in case of "if-else" branching into account (liker here - same file, but 99.07% covered).