Any plans for a Dotty/Scala3 version of scoverage?

[gzoller]

Scoverage is the de-facto coverage tool for the Scala ecosystem. Sure sbt-jacoco exists, and I even managed to coerce it to work with Dotty, but operating just on class files does not yield ideal results. scoverage has always done very well.

Regaining scoverage as a key piece of Scala 3 ecosystem would be huge, as for now there really is nothing else.

Any plans?

[sksamuel]

Something I will investigate now that Scala 3 has been gone into feature freeze.
https://dotty.epfl.ch/blog/2020/11/09/scala3-m1.html

[gregghz]

Any update? Scala 3 is in RC1 now.

[smarter]

A prototype code coverage implementation for Dotty was implemented in a fork of the compiler a few years ago with promising results: the report is at http://guillaume.martres.me/code_coverage.pdf, the code at https://github.com/Qjaquier/dotty/tree/coverage with an example usage at https://github.com/Qjaquier/dotty-coverage-tools. I think it'd be great to get this merged into the compiler but we need a volunteer to rebase and complete the prototype.

[ironfish]

Any update on when this will be addressed?

[ckipp01]

I think it'd be great to get this merged into the compiler but we need a volunteer to rebase and complete the prototype.

@smarter if someone was to grab this, would the goal be to still have this as a standalone compiler plugin, or does what you wrote imply that there'd be a desire to have actually in the compiler?

[smarter]

I think it'd be great to have this built into the compiler under a flag yes.

[mdedetrich]

@smarter Is there a reason why its being integrated straight into the compiler rather than a plugin? To me it seems like having it as a plugin is more ideal since

1. You don't "bloat" the Scala3 compiler so it doesn't end up containing everything including the kitchen sink
2. Improvements to scoverage are now tied to Scala3 lifecycle/releases where as if its a plugin, incremental improvements can be made separately from the plugin

Just my 2 cents, not sure if there is more context behind including in the actual compiler itself.

[smarter]

• Compiler plugins need to be re-published for every compiler release since the API/ABI isn't stable, and every project that uses a compiler plugin has to wait for this to happen before they can upgrade the version of Scala they use, this already wasn't great in the Scala 2 days and it will be worse in Scala 3 now that we release a new version every six weeks, it also impedes the use of RCs and nightlies.
• If anything changes in the compiler AST representation, the coverage phase will have to be updated which will require some work, if it's in a compiler plugin that means it will take even longer for it to be available after a compiler release.
• If you look at https://github.com/Qjaquier/dotty/commits/coverage you'll see that it doesn't add much code to the compiler itself, it's mostly one self-contained compiler phase. And all the tooling around coverage like the way it generates html and the sbt integration can still be maintained in a separate repo.

[mdedetrich]

Thanks for the answer, didn't realize that Scala3 is expected to release so frequently. In this case then yes, it does make sense indeed.

[ckipp01]

So just a heads up that I've gone ahead and started on this. Thanks for the links @smarter, they're super helpful. One thing that popped up right away is the fact that not only the instrumentation is being done in that POC, but the full serialization to the XML report. Is this desired? When I was originally thinking about this I sort of thought that the compiler would produce the actual coverage files, but the logic to go from the instrumentation files to XML would actually be handled by scoverage or could be swapped our for whatever tooling the user wants. Any thoughts on this, or is it desirable to have the full XML writing be done by the compiler? Also, should I create an issue for this in the Dotty repo, or is having this in here fine enough? As soon as I wrap my head around it all and clean it up a bit I'll shoot in a draft PR.

[smarter]

Any thoughts on this

I don't have a particular opinion on that, I'd say whatever is easier to maintain is best, in particular I think it's important to be able to test that the coverage support is actually working from the dotty CI, but that could possibly be done by having scoverage in our community build.

should I create an issue for this in the Dotty repo, or is having this in here fine enough?

I think this issue is enough

[ckipp01]

For those of you following along, here is a little update. I'm getting pretty close to shooting in a PR to get some feedback. Here is a little demo of it working.

Can't get the video preview to work, so you can find it here: Scala 3 scoverage

There will be a -coverage flag that will need to be passed in along with the location you'd like to send the coverage file. The compiler will be in charge of producing the scoverage.coverage file which is then used by scoverage to produce various reports. This will work similarly to how it does with SemanticDB where the compiler produces what is needed by tools to utilize it.

At this stage the remaining things I need to work out are:

• Finish testing (I'm setting up testing somewhat similiar to the semanticDB tests where they'll be a coverage.expect file that we check against
• I'll also need split out some code in this repo from the actual compiler plugin package. That will be a full version bump, but it will ensure that the domain classes and serialization/deserialization stuff is completely separated from the plugin for 2.12/2.13 (we'll also remote support for 2.11 at this point).
• Clean up some stuff that you'll see in the current reports (for example printing out the entire tree)

EDIT: While working on a few things it also made sense to work on some long-standing issues since we'll be breaking compatibility anyways. You can follow along with those plans here in #368.

[Qjaquier]

Happy to see that my POC is somehow useful and that this topic is moving forward!

I went away from the topic lately, but let me know if I can do anything to help.
And in any way, looking forward to see the outcome, I will gladly have a look at the PR.

[ckipp01]

Your work has been incredibly helpful @Qjaquier!! I do have a question that I'm curious if you hit on or had put any thought into while doing your POC since something I'm stuck on at the moment.

In scoverage the runtime/invoker is basically re-created on every run. For example if you start sbt shell, run your test, it will create a new instance of Invoker which then keeps track of the ids that have been hit. Once finished, you maybe change something in a test, run it again, and you get a new instance of Invoker with new state.

In the POC and my implementation, if you did that same workflow, the Invoker is instantiated once, and the Map of ids are kept, meaning that even if a test changed, it willl delete the instrumentation files, but the map still retains the ids from the previous runs blocking new instrumentation files from being created.

Did you put any thought into how to "clear" that state, or create a new Invoker each time to avoid this problem? Let me know if what I explained doesn't make sense and I can give some more detailed example.

Thanks again for all the work you've put into this. I hope to have it finished soon, but have been quite swamped with other things lately.

[Qjaquier]

I remember that the code of the Invoker was simplified (probably over simplified ?) in order to suit the needs of the POC and that I tried a few experiments in order to reduce the impact on performance, but as far as I remember, nothing was done to answer the problem you describe.

[ckipp01]

Just to tie this all together, the initial pr has been created here for the support in Dotty.

• Add in initial support for code coverage scala/scala3#13880

You can also read about some of the other changes that was necessary in this repo here:

• https://github.com/scoverage/scalac-scoverage-plugin/releases/tag/v2.0.0-M2

[ckipp01]

Seeing that coverage support has now been merged into Dotty as of scala/scala3#13880 (thanks to the work of @TheElectronWill), I'll go ahead and close this. The V2 RC series will have another release shortly, and if nothing goes wrong, that will become 2.0. Support for coverage in Scala 3 will first be available in the 3.2.x series, and there is a pr in sbt-scoverage as well to ensure everything works smoothly scoverage/sbt-scoverage#429.

[ckipp01]

Now that 2.0.0 is released users are able to use this in conjunction with Scala 3.2.0-RC1 for code coverage support for Scala 3. 2.0.0 of sbt-scoverage has also been released supporting this, so the way you would set up a project with Scala 2 and Scala 3 should be exactly the same. Thanks for all the help everyone!

[soujiro32167]

@ckipp01 I just tried this plugin with Scala 3.1.3 and got no coverage information reported. Are you saying it only works with Scala 3.2.0-RC1 onward?

[ckipp01]

@ckipp01 I just tried this plugin with Scala 3.1.3 and got no coverage information reported. Are you saying it only works with Scala 3.2.0-RC1 onward?

Correct.