How to get pending steps to not be seen as failures? #129

SteveDonie · 2017-03-08T22:24:57Z

I've recently noticed that when a scenario has a step that returns 'pending' that the cucumber reports show the scenario as 'failed'. Is there a way to change that behavior? We use 'pending' to indicate that a certain dependency is not configured during a particular test run, and this is an expected condition.

I have tried setting the report plugin configuration "Number of pending steps" to a very high number (99999) and that does not seem to affect the result.

damianszczepanik · 2017-03-09T07:30:07Z

This is done by design. What exactly are your expectations?

SteveDonie · 2017-03-09T15:56:50Z

I would expect the scenario to report scenarios that have pending steps to report their overall status as pending.

The command-line reporting reports scenarios as failed, passed, undefined, or pending:

Failing Scenarios:
cucumber -p linux features/auditdb_and_reporting_mssql.feature:110 # Scenario: Verify a hammer insertSavedAuditData operation repeated, discrete H2 updates PROJECTS with exactly one row in MSSQL AuditDB (DVONE-4163)

3 scenarios (1 failed, 1 undefined, 1 passed)
171 steps (1 failed, 92 skipped, 1 undefined, 77 passed)
1m3.148s

Or in this case:

1 scenario (1 pending)
29 steps (27 skipped, 1 pending, 1 passed)
0m2.938s

The built-in html formatter also shows scenarios as passed, failed, or pending. I have attached a sample.
results.html.zip

I've also attached a screenshot showing what I am seeing with the cucumber-reports-plugin.

SteveDonie · 2017-03-09T16:08:46Z

Here's another html report with a single scenario that is pending.
results.html.zip

damianszczepanik · 2017-03-09T19:35:51Z

Sorry but I don't see string enough reason for changing default behave

SteveDonie · 2017-03-09T21:19:15Z

Thanks. I'll probably just change it in my fork then. Would you mind pointing me towards where the logic to calculate the overall results are computed?

damianszczepanik · 2017-03-09T21:34:02Z

This should be good start point
https://github.com/damianszczepanik/cucumber-reporting/blob/master/src/main/java/net/masterthought/cucumber/json/Element.java#L115

svsforsuccess · 2017-05-27T00:11:01Z

But am planning to skip few scenarios so it would be greatly helpful if you add column for pending in scenarios and also if you could tell me the way to modify plugin code also should be fine...but i need it badly...thanks

SteveDonie · 2017-06-13T15:27:37Z

I have made changes in my fork of cucumber-reporting that do almost all that is needed. The main reports now show pending and undefined steps/scenarios/features. The only thing I have not implemented is the trend reporting.

If you'd like to get the code, see pull request 636: damianszczepanik/cucumber-reporting#636

mpkorstanje · 2017-06-13T18:03:47Z

I've recently noticed that when a scenario has a step that returns 'pending' that the cucumber reports show the scenario as 'failed'. Is there a way to change that behavior? We use 'pending' to indicate that a certain dependency is not configured during a particular test run, and this is an expected condition.

When executing cucumber tests in the strict mode, pending steps will fail the test so rendering pending as failures not strictly speaking wrong. I guess what you're looking for is to mimic this strict mode in the report.

Semantically it is wrong to use Pending to mean not executed due to failing preconditions. However once cucumber-jvm 2.0.0 is out you'll be able to use Junit Assume to do just this.

Can't provide an eta. on that but we do have snapshot releases.

        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <repository>
            <id>sonatype-snapshots</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>

svsforsuccess · 2017-06-13T18:05:14Z

Thank you for update!!!

…

On Tue, Jun 13, 2017 at 2:03 PM mpkorstanje ***@***.***> wrote: I've recently noticed that when a scenario has a step that returns 'pending' that the cucumber reports show the scenario as 'failed'. Is there a way to change that behavior? We use 'pending' to indicate that a certain dependency is not configured during a particular test run, and this is an expected condition. When executing cucumber tests in the strict mode, pending steps will fail the test so rendering pending as failures not strictly speaking wrong. Semantically it is wrong to use Pending to mean not executed due to failing preconditions. However once cucumber-jvm 2.0.0 is out you'll be able to use Junit Assume <http://junit.org/junit4/javadoc/4.12/org/junit/Assume.html> to do just this. Can't provide an eta. on that but we do have snapshot releases. <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-junit</artifactId> <version>${cucumber.version}</version> <scope>test</scope> </dependency> <repository> <id>sonatype-snapshots</id> <url>https://oss.sonatype.org/content/repositories/snapshots</url> <snapshots> <enabled>true</enabled> </snapshots> </repository> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#129 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARXxMGMT4GeSJFHLATV5I72TEuBgXHkRks5sDs8NgaJpZM4MXY8d> .

svsforsuccess · 2017-06-14T00:17:03Z

I have done woraround for this from excel ,am not including that feature itself for run.

…

On Tue, Jun 13, 2017 at 2:03 PM mpkorstanje ***@***.***> wrote: I've recently noticed that when a scenario has a step that returns 'pending' that the cucumber reports show the scenario as 'failed'. Is there a way to change that behavior? We use 'pending' to indicate that a certain dependency is not configured during a particular test run, and this is an expected condition. When executing cucumber tests in the strict mode, pending steps will fail the test so rendering pending as failures not strictly speaking wrong. Semantically it is wrong to use Pending to mean not executed due to failing preconditions. However once cucumber-jvm 2.0.0 is out you'll be able to use Junit Assume <http://junit.org/junit4/javadoc/4.12/org/junit/Assume.html> to do just this. Can't provide an eta. on that but we do have snapshot releases. <dependency> <groupId>io.cucumber</groupId> <artifactId>cucumber-junit</artifactId> <version>${cucumber.version}</version> <scope>test</scope> </dependency> <repository> <id>sonatype-snapshots</id> <url>https://oss.sonatype.org/content/repositories/snapshots</url> <snapshots> <enabled>true</enabled> </snapshots> </repository> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#129 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARXxMGMT4GeSJFHLATV5I72TEuBgXHkRks5sDs8NgaJpZM4MXY8d> .

vitaliy-sharandin · 2017-06-27T15:40:50Z

This implementation seems quite reasonable, as Pending status tells us something was not yet implemented or is ommited on purpose. If something is wrong with stepdef(therefore scenario), there are failed and undefined statuses. So, +1 vote to see this in official implementation of plugin.

We have our own BDD framework and we didn't have Pending exception yet, so I'm implementing it right now and it kind of depends on this plugin what statuses will I pass to scenario level.
So, is there any chance this will be implemented or I should not hope for this useful feature?

svsforsuccess · 2017-06-27T16:10:16Z

My requirement was skipping the feature if I do not want,But I implemented it using pre requisite approach. If any one interested in approach ,pls ping me here. Thanks, Karthik

…

On Tue, Jun 27, 2017 at 11:40 AM maultosar ***@***.***> wrote: This implementation seems quite reasonable, as Pending status tells us something was not yet implemented or is ommited on purpose. If something is wrong with stepdef(therefore scenario), there are failed and undefined statuses. So, +1 vote to see this in official implementation of plugin. We have our own BDD framework and we didn't have Pending exception yet, so I'm implementing it right now and it kind of depends on this plugin what statuses will I pass to scenario level. So, is there any chance this will be implemented or I should not hope for this useful feature? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#129 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARXxMLt5z8E0mCOmJ7rMKQ4I-18LsaRxks5sISKGgaJpZM4MXY8d> .

damianszczepanik · 2017-06-29T21:12:27Z

Looks like there is need to add feature that was removed time ago because some problems with consistency. I need to think it over again

vitaliy-sharandin · 2017-06-30T13:00:09Z

It would be a very helpful feature. Thanks:)

LucasSq · 2017-07-06T11:12:46Z

This would be helpful for my project too :)

jess8147 · 2017-09-06T04:49:47Z

+1 for this feature requirement

KLBonn · 2017-10-16T08:49:03Z

+1 from me

The result view shown by @SteveDonie ( comment above / PR 636 ) would be exactly what we need.

"undefined" and "pending" should be differentiated from "failed":

"undefined" or "pending" is used for: planned "work in progress", no special action implied.
"failed" should be reserved for: Problems with application under test or test environment --> immediate action required!

Choosing between "undefined" and "pending" could be subject to discussion and local interpretation, but at least they should be visible up to feature level (instead of just "failed").

damianszczepanik · 2017-10-20T20:36:21Z

Would it be OK to add tow new flags: one for undefined and second for pending so it defines if the steps should be counted as passed or failed?

SteveDonie · 2017-10-20T21:06:02Z

@damianszczepanik I don't think that your suggestion of new flags meets my needs at all. I don't want a feature or a scenario to be limited to a passed or failed state - I actually like knowing that a scenario is 'pending' or 'undefined' - I see these as communicating more richly what the state of things is.

damianszczepanik · 2017-10-21T09:01:06Z

What when scenario has steps with different statuses?

KLBonn · 2017-10-21T10:58:30Z

Of course we have to have clearly defined aggregation rules. We should bring the statuses into an order, and then take the worst (min or max) value. E.g.: - 0: passed - 1: pending - 2: undefined - 3: failed The outcome of a scenario then would simply be maximum of step results. And same should hold for features which would be evaluated based on their worst scenario. Am 21. Oktober 2017 11:01:10 MESZ schrieb Damian Szczepanik <[email protected]>:

…

What when scenario has steps with different statuses? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #129 (comment)

SteveDonie · 2017-10-21T14:18:58Z

@KLBonn If you look at the diff in pull request 636, you'll see that is exactly what I have done.

See https://github.com/damianszczepanik/cucumber-reporting/pull/636/files#diff-8350411954262e85f9d3bee0c77b2879

svsforsuccess · 2017-10-21T14:34:18Z

Hi All, Am not sure this is right tag,but I would like to contribute following feature to this plugin ,please suggest When I execute examples in feature in parallel I have writtent an utility to merge all jsons according to this report and show the result appropriately. Thanks Karthik

…

On Sat, Oct 21, 2017 at 10:19 AM Steve Donie ***@***.***> wrote: @KLBonn <https://github.com/klbonn> If you look at the diff in pull request 636 <damianszczepanik/cucumber-reporting#636>, you'll see that is exactly what I have done. See https://github.com/damianszczepanik/cucumber-reporting/pull/636/files#diff-8350411954262e85f9d3bee0c77b2879 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#129 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARXxMIztZuodPsDS76QI4_ondavL9osWks5suf1XgaJpZM4MXY8d> .

damianszczepanik · 2017-10-21T15:35:20Z

Based on https://github.com/damianszczepanik/cucumber-reporting/blob/master/src/main/java/net/masterthought/cucumber/json/support/Status.java#L17-L19 I'm not really sure if this is clear to everybody what is the relation between those three. So I have concern if by introducing three new statuses for scenarios and features we introduce some doubts what is the different between pending and skipped for feature. It is not as clear as for steps which is very small unit.

KLBonn · 2017-10-23T12:45:39Z

Yes, looking at the code of PR 636, we seem to share the same idea here :-)

@damianszczepanik Maybe we could introduce a flag / feature toggle for that new behaviour?
So people could choose between "simple status" and "detailed status".

Then anyone confused by differentiated statuses can drop down to showing every non-passing scenario/feature as failed.

Of course, a description / definition of the meaning for each status should be part of the documentation, anyway.
(Anyone in need for a special "interpretation" could choose to "misuse" e.g. Pending status on his/her project, but the need to show that status even on feature level would stay the same)

kalyankp · 2017-11-20T08:32:37Z

Hi ,
@damianszczepanik - I am still confused, with the present cucumber-report plugin how does ""Number of pending steps" works? I have also tried giving 9999 in the "Number of pending steps" field , where as I have 7 pending steps in my Json. But still it overall status is failed.
Please suggest some quick solution for it.

Thank you.

damianszczepanik · 2017-11-20T08:45:59Z

Probably other statuses are set to zero while your report has more. Validate other statuses.

kalyankp · 2017-11-20T11:16:59Z

Hi,
@damianszczepanik - Here I am attaching cucumber-report screenshot and configuration screenshot, Please see and let me know what mistake I am doing in configuration.

damianszczepanik · 2017-11-20T21:31:31Z

OK, now I understand, this is duplicate of damianszczepanik/cucumber-reporting#636

UltimateGeek · 2018-08-09T17:44:37Z

The referenced pull request, damianszczepanik/cucumber-reporting#636 is going into the Datical fork, not jenkinsci. Why was this issue's status changed to closed?

damianszczepanik closed this as completed Mar 9, 2017

damianszczepanik reopened this Jun 29, 2017

damianszczepanik mentioned this issue Nov 9, 2017

How to set Over all Cucumber Status as Passed when it has Pending Status #199

Closed

damianszczepanik closed this as completed Nov 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to get pending steps to not be seen as failures? #129

How to get pending steps to not be seen as failures? #129

SteveDonie commented Mar 8, 2017

damianszczepanik commented Mar 9, 2017

SteveDonie commented Mar 9, 2017 •

edited

Loading

SteveDonie commented Mar 9, 2017

damianszczepanik commented Mar 9, 2017

SteveDonie commented Mar 9, 2017

damianszczepanik commented Mar 9, 2017

svsforsuccess commented May 27, 2017

SteveDonie commented Jun 13, 2017 •

edited

Loading

mpkorstanje commented Jun 13, 2017 •

edited

Loading

svsforsuccess commented Jun 13, 2017 via email

svsforsuccess commented Jun 14, 2017 via email

vitaliy-sharandin commented Jun 27, 2017

svsforsuccess commented Jun 27, 2017 via email

damianszczepanik commented Jun 29, 2017

vitaliy-sharandin commented Jun 30, 2017

LucasSq commented Jul 6, 2017

jess8147 commented Sep 6, 2017

KLBonn commented Oct 16, 2017 •

edited

Loading

damianszczepanik commented Oct 20, 2017

SteveDonie commented Oct 20, 2017

damianszczepanik commented Oct 21, 2017

KLBonn commented Oct 21, 2017 via email

SteveDonie commented Oct 21, 2017

svsforsuccess commented Oct 21, 2017 via email

damianszczepanik commented Oct 21, 2017

KLBonn commented Oct 23, 2017 •

edited

Loading

kalyankp commented Nov 20, 2017

damianszczepanik commented Nov 20, 2017

kalyankp commented Nov 20, 2017

damianszczepanik commented Nov 20, 2017

UltimateGeek commented Aug 9, 2018

How to get pending steps to not be seen as failures? #129

How to get pending steps to not be seen as failures? #129

Comments

SteveDonie commented Mar 8, 2017

damianszczepanik commented Mar 9, 2017

SteveDonie commented Mar 9, 2017 • edited Loading

SteveDonie commented Mar 9, 2017

damianszczepanik commented Mar 9, 2017

SteveDonie commented Mar 9, 2017

damianszczepanik commented Mar 9, 2017

svsforsuccess commented May 27, 2017

SteveDonie commented Jun 13, 2017 • edited Loading

mpkorstanje commented Jun 13, 2017 • edited Loading

svsforsuccess commented Jun 13, 2017 via email

svsforsuccess commented Jun 14, 2017 via email

vitaliy-sharandin commented Jun 27, 2017

svsforsuccess commented Jun 27, 2017 via email

damianszczepanik commented Jun 29, 2017

vitaliy-sharandin commented Jun 30, 2017

LucasSq commented Jul 6, 2017

jess8147 commented Sep 6, 2017

KLBonn commented Oct 16, 2017 • edited Loading

damianszczepanik commented Oct 20, 2017

SteveDonie commented Oct 20, 2017

damianszczepanik commented Oct 21, 2017

KLBonn commented Oct 21, 2017 via email

SteveDonie commented Oct 21, 2017

svsforsuccess commented Oct 21, 2017 via email

damianszczepanik commented Oct 21, 2017

KLBonn commented Oct 23, 2017 • edited Loading

kalyankp commented Nov 20, 2017

damianszczepanik commented Nov 20, 2017

kalyankp commented Nov 20, 2017

damianszczepanik commented Nov 20, 2017

UltimateGeek commented Aug 9, 2018

SteveDonie commented Mar 9, 2017 •

edited

Loading

SteveDonie commented Jun 13, 2017 •

edited

Loading

mpkorstanje commented Jun 13, 2017 •

edited

Loading

KLBonn commented Oct 16, 2017 •

edited

Loading

KLBonn commented Oct 23, 2017 •

edited

Loading