Option to output lcov.info format?

[nedbat]

Originally reported by Matthew Crenshaw (Bitbucket: sgtsquiggs, GitHub: sgtsquiggs)

---

So that we can use coverage.py with IDE plugins that read the standard lcov.info format and highlight covered/uncovered lines. Would be super great!

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/587

[nedbat]

Interesting idea. Do you have a link to information about the file format?

[nedbat]

Original comment by Matthew Crenshaw (Bitbucket: sgtsquiggs, GitHub: sgtsquiggs)

---

lcov format

   Following  is  a  quick  description of the tracefile format as used by
   genhtml, geninfo and lcov.

   A tracefile is made up of several human-readable lines of text, divided
   into sections. If available, a tracefile begins with the testname which
   is stored in the following format:

     TN:<test name>

   For each source file referenced in the .da file,  there  is  a  section
   containing filename and coverage data:

     SF:<absolute path to the source file>

   Following is a list of line numbers for each function name found in the
   source file:

     FN:<line number of function start>,<function name>

   Next, there is a list of execution counts for each  instrumented  func‐
   tion:

     FNDA:<execution count>,<function name>

   This  list  is followed by two lines containing the number of functions
   found and hit:

     FNF:<number of functions found>
     FNH:<number of function hit>

   Branch coverage information is stored which one line per branch:

     BRDA:<line number>,<block number>,<branch number>,<taken>

   Block number and branch number are gcc internal  IDs  for  the  branch.
   Taken  is either '-' if the basic block containing the branch was never
   executed or a number indicating how often that branch was taken.

   Branch coverage summaries are stored in two lines:

     BRF:<number of branches found>
     BRH:<number of branches hit>

   Then there is a list of execution counts  for  each  instrumented  line
   (i.e. a line which resulted in executable code):

     DA:<line number>,<execution count>[,<checksum>]

   Note  that  there  may be an optional checksum present for each instru‐
   mented line. The current geninfo implementation uses  an  MD5  hash  as
   checksumming algorithm.

   At  the  end of a section, there is a summary about how many lines were
   found and how many were actually instrumented:

     LH:<number of lines with a non-zero execution count>
     LF:<number of instrumented lines>

   Each sections ends with:

     end_of_record

   In addition to the main source code file there  are  sections  for  all
   #included files which also contain executable code.

   Note that the absolute path of a source file is generated by interpret‐
   ing the contents of the respective .bb file  (see  gcov  (1)  for  more
   information  on  this  file type). Relative filenames are prefixed with
   the directory in which the .bb file is found.

   Note also that symbolic links to the .bb file will be resolved so  that
   the  actual  file  path  is  used  instead  of the path to a link. This
   approach is necessary for the mechanism to  work  with  the  /proc/gcov
   files.

[nedbat]

I'm a bit nervous about the language semantics assumed by this file format (list of functions found?), but that seems to be par for the course for any of these formats.

[nedbat]

Original comment by Bojan Kogoj (Bitbucket: bojankogoj, GitHub: bojankogoj)

---

Any progress? I'd like to use vs code coverage plugin but it doesn't support current format

[nedbat]

I am not working on this at the moment, and probably will not be implementing this any time soon. There's no reason this has to be built-in to coverage.py. You can get data from coverage in XML format, or using the data API, and create a file like this with separate code.

[gpshead]

FWIW we've been using a patch for eons at work that has coverage generate LCOV files. Let me see about extricating that and making a PR.

[gpshead]

Example non-great code posted as a PR to help kick things off. :)

[ulfjack]

We've gotten repeated requests to support python coverage in Bazel (https://bazel.build/) and integrating it in a multi-language situation (e.g., coverage for Python + C++) requires being able to generate lcov format.

[gpshead]

I really haven't spend much time untangling this, are there any links to the specifics of the LCOV format or the specifics of what subset of that Bazel expects from LCOV files? (see also the comment thread on (#863).

[ulfjack]

Bazel supports multi-language coverage, e.g., getting coverage results from Java, Python, and C++ from a single test run. However, in order to generate a combined report, all of these need to be converted into a single unified format. We've looked at different formats, and it seems like every coverage tool / implementation comes with its own format, and none is particularly prevalent. Our main restriction is that it shouldn't be too verbose (so XML might be a problem). Whatever format we standardize on in Bazel, we need to convert from all other formats to that one.

Right now, we're using lcov as our "standard" format, which is fairly simple, reasonably compact, and it's the same format Google uses internally (full disclosure: I work for Google). For large code bases, we're seeing multi-GB lcov outputs, so less compact could be problematic.

Ideally, whatever format we use also supports advanced features like branch coverage or edge coverage, but it's more important to settle on something than to have the latest and greatest features.

Each tool has to decide what lines to mark as covered. I would strongly recommend that it at least correctly annotate on a per-file level; there are potential uses for automatically selecting affected tests based on coverage information from a previous test run to reduce CI resource usage.

One more thing: Python is an exception, but we otherwise don't allow access to source code from the test run. Bazel sandboxes execution, and - by design - does not support general source access. We collect lcov from the test runs, and then generate an HTML report as a post-process during which we allow source access. As a corollary, the coverage tool should not put absolute paths into the output, because the sandbox is in a different file system location. We can post-process the data if necessary, but it's not ideal. Ideal is to use relative paths to the current working directory.

[chaychoong]

Hope to see this being implemented soon!

If anyone needs a converting tool in the meantime, I created a simple converter here: https://github.com/chaychoong/coveragepy-lcov

[bradb423]

Hi @nedbat, we are currently looking at implementing code coverage for python in bazel. What do you think of a feature for coverage.py, that would act similarly to xmlreport.py and jsonreport.py, and output a file in LCOV format? Do you have thoughts on what such an implementation should look like?

[nedbat]

@bradb423 I know nothing about the LCOV format, but the code in @chaychoong's repo looks pretty simple. Perhaps adapting that to a new command in coverage.py is the way to go? What other guidance would you need?

[nedbat]

I should say: I'm torn between adding features like this to coverage.py (to keep things convenient and findable), and leaving them as separate addons that use supported APIs in coverage.py instead (to keep coverage.py light, and to foster innovation). Thoughts?

[TheCleric]

I should say: I'm torn between adding features like this to coverage.py (to keep things convenient and findable), and leaving them as separate addons that use supported APIs in coverage.py instead (to keep coverage.py light, and to foster innovation). Thoughts?

From what I've seen lcov seems to be a standard (non human readable) format that I've encountered across a few different projects/tech stacks. Specifically handy for me when combined with github actions to produce a simple coverage report on things like pull requests. I'd vote it should be core, but that just may be my bias/exposure to it. (And having to fork @chaychoong 's repo to support python < 3.8, lol).

[TLATER]

Istanbul's support list might be interesting to look at, since it has both a variety of output formats and supports using arbitrary npm packages as output plugins. I couldn't find any public plugins that provide formats not natively supported by Istanbul, so that list is probably fairly exhaustive as far as non-language-specific formats go.

The computer-readable formats listed there are:

• Clover
• Cobertura (a variant of which I believe coverage.py currently outputs as "XML"?)
• JSON (which I believe to be bespoke)
• lcov

grcov is also interesting to me because it's Rust and I like Rust ;)

• lcov
• Cobertura
• Covdir (Another JSON output that seems semi-bespoke)

These oddities also crop up there:

• Coveralls (https://coveralls.io/'s specific format, but they support lcov and others too so I don't really see why, probably more language-specific data?)
• ADE (AWS specific?)

I've had a skim through various other projects too, most other formats are specific to the coverage tool (e.g. Java's Jacoco, Ruby's SimpleCov), and consumers just implement parsing for their formats. Bazel actually implements its own formatter for Jacoco to get it to be lcov ;)

I think my personal opinion is that there simply aren't enough "generic" variants of these formats to warrant the overhead of a plugin system at the moment (Istanbul certainly doesn't seem to need it), but I recognize I'm a downstream user, mostly want convenience and share @TheCleric 's bias.

[bilderbuchi]

I think it's relevant to this discussion to also point to Gcovr - on the one hand it's written in Python and offers a nice set of features and customizability, on the other hand it generates nice coverage reports from gcc's gcov data (not lcov).

[nedbat]

I'm not sure where this conversation is going! :)

About LCOV: are there variants that would need to be supported, or is it well standardized? It does seem like a widely used format that could be supported in the core tool.

About plugins: there is definitely no need to write plugins for reporting. Reporting can happen in a separate tool after the fact if need be for unusual formats.

[TheCleric]

About LCOV: are there variants that would need to be supported, or is it well standardized? It does seem like a widely used format that could be supported in the core tool.

To my knowledge it's standardized. For example using my forked version of @chaychoong 's repo to out lcov data, I had no issues hooking it up to my lcov CI pipeline that also consumes lcov output from other sources such as Jest on a node app.

[bradb423]

We currently have a pull request open to provide some LCOV support. Currently this does not have branch coverage, but this feature is being worked on.

[TheCleric]

We currently have a pull request open to provide some LCOV support. Currently this does not have branch coverage, but this feature is being worked on.

@bradb423 Given that it's of a compatible license, would it make sense to just merge what @chaychoong has done as it does have branch coverage?

[TLATER]

would it make sense to just merge what @chaychoong has done as it does have branch coverage?

@TheCleric @chaychoong 's implementation does not do branch coverage either: https://github.com/chaychoong/coveragepy-lcov/blob/master/coveragepy_lcov/converter.py#L72

Branches being marked correctly in the genhtml output is incidental of the line coverage, but this is different from the BRDA/BRF/BRH records documented here.

[TheCleric]

would it make sense to just merge what @chaychoong has done as it does have branch coverage?

@TheCleric @chaychoong 's implementation does not do branch coverage either: https://github.com/chaychoong/coveragepy-lcov/blob/master/coveragepy_lcov/converter.py#L72

Branches being marked correctly in the genhtml output is incidental of the line coverage, but this is different from the BRDA/BRF/BRH records documented here.

Ahh, good to know. Thanks!

[TLATER]

That's not to say it might not be worth merging without fully implemented branch coverage :) lcov is quite flexible, a file with no branch/function records is valid. Most of what other tools use is in the line coverage records anyway.

[nedbat]

I really appreciate all the interest in this. I am still wondering about keeping this as a separate project instead of adding a command to coverage.py.

[TLATER]

I really appreciate all the interest in this. I am still wondering about keeping this as a separate project instead of adding a command to coverage.py.

I think we all know that adding more public API is always scary, it's hard to go back on in the future without breaking stuff for people - and there are already a handful of us who intend to build infrastructure on this.

On the other hand, I've looked through @bradb423 's patch, and the implementation doesn't seem too invasive. It uses internal API that already exists for the other reporters, so I think maintenance should be fairly minimal. lcov is also slow-moving (hasn't changed in over a decade, last commit to the format doc is from 12 years ago), and one of the simpler formats, so I don't think there's a major cost to having that command around either.

Ultimately this is the maintainers' dilemma, and I don't think we can help much with the decision, besides saying pretty please :) If you have particular concerns about the lcov format I think I'm reasonably versed on it at this point (much better than two weeks ago anyway)?

[nedbat]

@TLATER thanks. One (small) concern is the use of md5. I know it isn't a security concern the way it's used here, but there are dumb security scanners out there that flag things like that. Do we need the md5 fingerprints in the data file? What happens if we leave them out?

[TLATER]

@nedbat That checksum is only used to assert that any later analysis is actually running on the correct lines (and say, nobody has updated them in the mean time). If it's left out, tools just don't check the lines. It's nice for more complex use cases, but not essential.

The checksum is documented as not demanding a specific implementation, so we could use sha256 or such instead. I've not tried doing that before, so no idea what issues that will tickle out, I just know upstream uses md5.

[TLATER]

Hmm, yeah, looks like upstream doesn't actually support anything besides md5, despite that claim in the docs: https://github.com/linux-test-project/lcov/blob/master/bin/genhtml#L5305.

If the dumb scanners are a concern, I'd not be terribly sad to lose the checksum :)

[nedbat]

@bradb423's lcov implementation is merged as of 2e8c191.

[lazka]

Thanks @bradb423 @nedbat 🙏

[nedbat]

This is now released as part of coverage 6.3.