Propose colcon universal coverage result tool for python and cpp

[Briancbn]

Past work and tickets:

• Colcon extension for coverage results #110: lcov for cpp
• colcon-coveragepy-result package to collect coverage.py results  #349: coveragepy for python

These two all work wonders for packages that are purely cpp or python, but not for those packages that has both cpp and python package.

We can combine the two results and generate output in the following format that is available to both

• Terminal Display
• Cobertura xml format (which is good for codecov.io and jenkins)
• html (Good for debugging and improving coverage)
• json (for coveralls.io) format

There is a pycobertura, that can combine cobertura xml results. However, its API is said to be 'not stable' before the 1.0.0 version. We can either develop using its API or come out with our own ones.

With that it is much easier to

• generate and evaluate coverage results locally,
• implement code coverage for CI and integrate with reporting tools like codecov, coveralls, gitlab, jenkins etc.
• provide a universal report template, when new language like Java is complete supported, we can easily add them in by supporting the format.

The first two are especially for packages that has both python and cpp packages inside

Lastly about the library used, I would suggest to use

• gcovr instead of lcov
• coveragepy

gcovr supports a lot more report format compared to lcov and it's python based, which can directly be implemented into colcon using its API like coveragepy

I don't have a prototype yet, but please let me know your thoughts on whether this is needed and if there are design ideas or examples I should look more at.

[dirk-thomas]

Maybe the authors of these two extensions - @christophebedard for colcon-coveragepy-result and @jpsamper2009 for colcon-lcov-result - want to comment on this.

I don't know if or why both can't be used for the same package atm and what prevents them from doing so.

[christophebedard]

I thought about this too when I wrote colcon-coveragepy-result.

I also don't see how you couldn't run both on, say, a CMake package with C++ and Python code. However, I think it's more about combining the output so that C++ and Python (& more) coverage is combined into one report. So instead of getting an output like:

• Package A
  • C++ coverage data
• Package B
  • C++ corevage data
• Package A
  • Python coverage data
• Package B
  • Python corevage data
• etc. for other languages

we'd get an output like this:

• Package A
  • C++ & Python (& more) coverage data
• Package B
  • C++ & Python (& more) corevage data

The goal is to be able to support any and all languages, so combining the reports is a must, otherwise the coverage information is split up and is less readable. @Briancbn feel free to correct me or elaborate, of course.

[Briancbn]

@christophebedard Yes, this is exactly what I am referring to,

we'd get an output like this:

* Package A
  
  * C++ & Python (& more) coverage data

* Package B
  
  * C++ & Python (& more) corevage data

Does this mean separating the report based on meta packages (or git packages)?

My assumption was that the command simply will combine all the coverage data in a workspace and output them in one report that are easy to read and debug and also can be used by various coverage reporting services like codecov.io, coveralls.io, gitlab, jenkins etc.

[christophebedard]

Ah yes, you're right! Both colcon-coveragepy-result and colcon-lcov-result have an option to output coverage information per package, but of course by default they output a combined report with all packages.

[Briancbn]

I will come back with a feature list and a prototype soon, also try to make use of the existing code base as much as possible, so that we will have something more tangible to discuss here.