Issues with skipped tests in summary and html report

[vmchukky]

I have been using karate for automated tests (starting with 0.6.0, in September 2017) and am very happy with this excellent tool. I pulled the latest 0.8.0 (for #418) and noticed couple of things that I would like to bring to your notice. Let me start of by saying this issue is related to karate.abort() and related summary/html reports (the undocumented / secret unreleased feature) and can be treated as enhancement. I did look at the open issues and feel this issue was not raised before.

Now here is the issue I would like to discuss before sending a PR. For the hypothetical test given below:

Scenario: test abort
  * def result = call read('some-test.feature')
  // skip tests below if some-test.feature failed for what ever reason
  * def check2Abort = (result.response == 'failed' ? karate.abort() : {} )
  // continue with rest of the steps in this scenario

The two issues are

1. summary shown after the test is executed doesn't show any skipped tests
2. and the cucumber html report doesn't show any skipped tests as well

Please note that the steps are actually skipped (as expected), but the report doesn't reflect these skipped steps. Let me point two one line changes that I was thinking about
Change needed in https://github.com/intuit/karate/blob/master/karate-core/src/main/java/com/intuit/karate/cucumber/KarateHtmlReporter.java#L236

else if ((StepResult.ABORTED.equals(result.getStatus())) || (Result.SKIPPED.getStatus()
        .equals(result.getStatus()))) {

And https://github.com/intuit/karate/blob/master/karate-core/src/main/java/com/intuit/karate/cucumber/KarateRuntime.java#L77 need to be modified as

Result result = aborted ? Result.SKIPPED : StepResult.PASSED;

This is the first time I was looking at karate source tree (so far I have been a passive user), so please bear with me if I missed the reason for proposed karate.abort() and feel free to shoot down the issue/suggested code change.

[ptrthomas]

@vmchukky this is one of those features that I added primarily for the mocks / test-doubles and it is why it is not advertised.

I don't think your proposed fix to line 77 will work because of line 93 - we are already hacking cucumber here. someone called @svarunbe also reported something but never followed up:
#416 (comment)

before making any change here I would like to ensure it doesn't break anything else. so you are free to work on this PR and submit if you can prove via test cases that this is working as expected. this is one of those areas I'd really like to see working examples of all the cases working.

you have a workaround - which is to move the conditional code into a "called" feature and then treat it as a block.

[vmchukky]

@ptrthomas thank you for your feedback.

I don't think your proposed fix to line 77 will work because of line 93

Am not sure, I have verified the fix. Line 93 would skip steps past karate.abort() in current scenario. And the fix I proposed to line# 77 will change the result to be marked as 'SKIPPED' instead of 'PASSED'.

before making any change here I would like to ensure it doesn't break anything else

I completely agree. I have created a sample test case to show the difference in reports (both summary as well as html), will be attaching it to this issue. Attached are the screenshots showing differences with current 0.8.0 and proposed fix on top of 0.8.0. And in both cases we can see the steps (after abort() are skipped, otherwise (match flag == 0 will fail when flag is set to 1).

Report generated when no abort

Report generated by current 0.8.0 release when aborted

Report generated after proposed fix is applied on 0.8.0 release when aborted

Please check and post your comments.

[vmchukky]

I couldn't attach the sample skeleton test project. But here is the one test file that I added to the skeleton project created by mvn.

Feature: sample karate test script

  Background:
    * def shouldAbort = function(){ return (Math.floor((Math.random() * 10) + 1) % 2);}

  Scenario: Abort based on a random (0/1) shouldAbort
    * def flag = shouldAbort()
    * print 'scenario-1 abort flag: ' + flag
    * def check2Abort = ( (flag == 1) ? karate.abort():{} )
    * print 'no abort as flag said so'
    * match flag == 0
    * def xyz = 'xyz'

  Scenario: scenario 2 should run irrespective of scenario-1 aborted or not
    * def abc = 'abc'
    * print 'scenario-2'

[ptrthomas]

@vmchukky 2 things. the above is a lot to go through, so I'd like an end to end sample or a PR: https://github.com/intuit/karate/wiki/Community-News

also, we need to ensure the Cucumber HTML reports are un-affected. the other issue talked about jenkins failing the build which I could not understand at all.

this is not a priority for me, so any effort from you or the community is going to make a difference.

[vmchukky]

@ptrthomas sure, I will try to work on it and raise a PR with a test case.

On a separate note: I am happy to see (from community news) that you will be presenting at Dev Conf in Bangalore (https://devconfin2018.sched.com/event/F749/karate-microservices-test-automation-made-simple), hoping to meet you in person.

[ptrthomas]

@vmchukky sounds great ! thanks for your patience :) you can imagine I really don't want to introduce a change that can break how Jenkins and CI works today - and the main problem here is - the code is already a hack to work around cucumber., I just don't feel good about the way abort() has been implemented for reporting. it works fine for the test-doubles.

[vmchukky]

@ptrthomas here is the PR #466 with proposed changes and corresponding unit test as well as functional test (attached to the PR comment).

Please review it at your convenience.

[ptrthomas]

re-opening and will close with next major version release.

thanks to @vmchukky for the PR - this fix is actually available in version 0.8.0.1 which was just released.

[ptrthomas]

0.9.0 released

[Adarshkrishna9099]

@ptrthomas seems like the issue with Karate.abort() still exist in V0.9.2.

[ptrthomas]

@Adarshkrishna9099 unlikely. open a new issue and make sure you follow this process please: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue