-
-
Notifications
You must be signed in to change notification settings - Fork 377
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Group line code-coverage by branch #964
Group line code-coverage by branch #964
Conversation
IMHO we should not build ranges like
I am not sure if this can solve every situation, imagine code like:
the if also please note
must not mark also
must not mark ping me once the PR is done, I can then comment on the impl. and find another test cases |
I think the opposite: line code-coverage is by definition wrong, but that's all we have. Starting from the AST is just a waste of time: let's go strait to lines without worrying about the tree.
What to you mean by "now",
That's obvious, why did you doubt it?
I don't get what you are trying to say: if you format the ternary correctly, we know which path has been taken, and we can mark as green only the taken path.
Let's be clear: line code-coverage is flawed by design. I couldn't care less about this example: if you are collecting the code-coverage with the line CC driver, it is required that you format your code correctly: try {
$var = fxA() . 'b';
$var .= 'c';
;} catch (\Throwable) {} If There is no solution that can fix every edge case. Let's focus on providing the best value for the sensible developers. |
all about the same topic - I agree it is a convention to code that is usually "friendly" to your approach, but I very strongly disagree we should do it that way.
Currently, for multiline statement/expression, we save only 1 coverage line/point when it is really emit by php/xdebug. This should stay IMHO. If something is a part of an expression/statement branch should be based on AST not an expr. start/end line, this has no physical meaning, no such data is emit by xdebug. Example:
you should build about this tree for it:
then you know variable assign op can be covered if and only if the inner expression aka branch has completed, so, the l2 and l5 can be marked as covered if l1 was covered but when the the code will be formatted like:
the tree should be:
and you cannot transfer any coverage information from assign l1 into l2/l5 as l1 source cannot be used as the coverage can be comming from unrelated statement |
I sincerely have no idea what you're talking about here. Ternary operator gets 4 line coverage from CC drivers: $var // 1
=
true // 1
?
'b' // 1
:
'c' // -1
; Assignment and if-condition are on the same logical branch, and then we have the 2 branches
Can you provide a real example? As far as I can tell, L1 and L2/L5 are coupled: you cannot execute one without the other. I am already taking into account |
1fb092f
to
7d6b533
Compare
@sebastianbergmann I am very disappointed by your review, the covered code percentage might look much better, but the reasoning behind is not done well and can mark more code as covered when in reality it is never executed. |
This change reverts some of infection#1773 by allowing the latest PHPUnit coverage tooling to be installed, for which @Slamdunk fixed coverage in regards of the `infection/infection` use-case. Ref: https://github.com/sebastianbergmann/php-code-coverage/releases/tag/9.2.20 Ref: sebastianbergmann/php-code-coverage#964 Ref: sebastianbergmann/php-code-coverage@71525ea
Excellent, thanks @sebastianbergmann! I sent a patch to re-enable Regarding the disagreement on precision@mvorisek as discussed above, constant expressions are "executed code" from our PoV. I suggest you propose a change in direction with your own patch/proposal, making it clear that there's a breakage in underlying understanding of coverage, and accompanied with documentation of how to get the more "precise" results (according to your own understanding). I am aware that @Slamdunk's patch leads to some edge cases in which precision is lost, but practically, the endgame for most people involved in this specific thread is mutation testing regardless, while coverage on its own is an entry point, but nowadays an insufficient quality metric. From our PoV, getting the mutation testing tooling running is much more important than this precision, at the moment, so some false-positive coverage is worse than the false-negative coverage before this patch. Please be aware of this trade-off. |
This change reverts some of infection#1773 by allowing the latest PHPUnit coverage tooling to be installed, for which @Slamdunk fixed coverage in regards of the `infection/infection` use-case. Ref: https://github.com/sebastianbergmann/php-code-coverage/releases/tag/9.2.21 Ref: sebastianbergmann/php-code-coverage#964 Ref: sebastianbergmann/php-code-coverage@71525ea
@Ocramius I understand this PoV, but still I am not convinced it is good and I am not sure if anyone done review on the actual analyser impl. Fixing the precision will require complete revert of this PR and basically redone this PR from scratch :( |
@mvorisek and that's fine: I encourage you to go for it, if you have the time/energy for it :) |
Why I? I spent quite a lot of time on writing the original tests. They were dropped by author of this PR. Shouldn't be he responsible for taking care of them? |
That's by design: that line is considered to be part of variable assignment now, not of the lambda. Lambda content is correctly reported as non executed indeed. |
Closing bracket has a meaning. If reached, it means the function has reached the end. When you for example call Actually any software that honors the standard meaning, can look at the function as covered and it also returns #954 bug to empty closures. |
Empty methods and functions are handled as expected. I'll be more than happy to fix any meaningful bug on closures, if you find one. |
I would be happy to hear feedback from @sebastianbergmann and @Ocramius too. I stand strongly by the fact that end bracket has meaning in line coverage. |
Please close the convo here and start a new one. If you have a test case on which to discuss it, I suggest adjusting that and showing a failure as a start. The closure declaration seems correctly covered there. |
…1777) This change reverts some of #1773 by allowing the latest PHPUnit coverage tooling to be installed, for which @Slamdunk fixed coverage in regards of the `infection/infection` use-case. Ref: https://github.com/sebastianbergmann/php-code-coverage/releases/tag/9.2.21 Ref: sebastianbergmann/php-code-coverage#964 Ref: sebastianbergmann/php-code-coverage@71525ea
Fix #959
ExecutableLinesFindingVisitor
fromRawCodeCoverageDataTest
to its own unit testline <=> branch index
line <=> branch index
association with CC driver's output to obtain the final CCEffects on real code bases
Before
After
As you can see, the difference can be quite high on line coverage: the code base in the shown here uses for example a lot of array declarations; prior to this PR, a single 1.000 array would have produced lines of CC only on non-scalar nodes; this PR instead marks all the 1.000 as executed (or not).
But as we know, line CC is convenient, quick and reliable only as a quantitative result.
Functions/Methods/Classes/Traits coverage has a much higher qualitative value, and this PR has very little effect on results. The difference in these cases are correct and expected: current
9.2.19
release has bugs that this PR fixes.Special cases
Some PHP structure cannot be deterministically handled well. For example:
This code will always depend on autoload runtimes, but in contrast to #892 this will also always produce 4 executable lines from CC drivers.
This PR favour determinism and stability over CC precision, so for the example provided a single branch is reported, and thus all the 4 lines are going to be marked as covered if
return
has any hit.If the user needs precise and reliable feedback over CC, only Mutation Testing can help; in the
match
example it would beMatchArmRemoval
mutators from Infection, see https://infection.github.io/guide/mutators.html#Removal-Mutators