Improve report so that pending and undefined steps do not show scenarios and features as failed #636
Conversation
…es other than failed or passed. A feature or scenario can now have a status of 'pending' or 'undefined', similar to what the built-in cucumber runner does and what the built-in cucumber html formatter does.
…tatus' Conflicts: pom.xml src/test/java/net/masterthought/cucumber/generators/integrations/FeaturesOverviewPageIntegrationTest.java
|
Hello @SteveDonie! Are you still working on this one? Best, |
|
I have not done any work on it recently, and probably won't have any time
in the near future. I have built a version of the plugin that we are using
locally, and that version meets my needs. The main thing that is missing
from the existing code is support for the trending views, which we haven't
found useful.
…On Wed, Jun 14, 2017 at 8:03 AM, Codacy Bot ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/test/java/net/masterthought/cucumber/ReportableBuilder.java
<#636 (comment)>
:
> @@ -28,17 +32,21 @@
protected long duration;
- public ReportableBuilder(int passedFeatures, int failedFeatures, int totalFeatures,
- int passedScenarios, int failedScenarios, int totalScenarios,
+ public ReportableBuilder(int passedFeatures, int failedFeatures, int pendingFeatures, int undefinedFeatures, int totalFeatures,
[image: Codacy]
<https://camo.githubusercontent.com/37ddd55708a38212f4e9de4e3eac98358fefe597/68747470733a2f2f7777772e636f646163792e636f6d2f6173736574732f696d616765732f66617669636f6e2e706e67>
Issue found: Avoid long parameter lists.
<https://www.codacy.com/app/damianszczepanik/cucumber-reporting/pullRequest?prid=710123>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#636 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAk86tZvK1_Ss0ZAnXQ0JmwVNigbCfF-ks5sD9oegaJpZM4N4qZy>
.
|
|
Alright, thanks for the update.
Best,
Benjamin
Am 08.09.2017 um 21:02 schrieb Steve Donie ***@***.***>:
I have not done any work on it recently, and probably won't have any time
in the near future. I have built a version of the plugin that we are using
locally, and that version meets my needs. The main thing that is missing
from the existing code is support for the trending views, which we haven't
found useful.
On Wed, Jun 14, 2017 at 8:03 AM, Codacy Bot ***@***.***>
wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In src/test/java/net/masterthought/cucumber/ReportableBuilder.java
> <#636 (comment)>
> :
>
> > @@ -28,17 +32,21 @@
>
> protected long duration;
>
> - public ReportableBuilder(int passedFeatures, int failedFeatures, int totalFeatures,
> - int passedScenarios, int failedScenarios, int totalScenarios,
> + public ReportableBuilder(int passedFeatures, int failedFeatures, int pendingFeatures, int undefinedFeatures, int totalFeatures,
>
> [image: Codacy]
> <https://camo.githubusercontent.com/37ddd55708a38212f4e9de4e3eac98358fefe597/68747470733a2f2f7777772e636f646163792e636f6d2f6173736574732f696d616765732f66617669636f6e2e706e67>
> Issue found: Avoid long parameter lists.
> <https://www.codacy.com/app/damianszczepanik/cucumber-reporting/pullRequest?prid=710123>
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#636 (review)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAk86tZvK1_Ss0ZAnXQ0JmwVNigbCfF-ks5sD9oegaJpZM4N4qZy>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#636 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFgh8X8zFg04UMApAsOaIh40vn7uzJ6Qks5sgY9KgaJpZM4N4qZy>.
Benjamin Bischoff
[email protected]
|
|
@SteveDonie do you have a working version of the plugin? I have been struggling to get your fix built without much luck. I am unsure why pending or skipped scenarios are seen as failed if this is not the case with steps. |
|
Hi, I actually ended up writing my own open source report plugin that solves this issue. If you are interested just drop me a line. |
|
@laxersaz I would very much like to hear more about your report plugin. Does it work with Jenkins as well? |
|
Hi! It is a pure maven plugin. Here is the repo: https://github.com/trivago/cluecumber-report-plugin |
|
@laxersaz can i integrate this into Jenkins? I wanted something i can integrate into Jenkins. |
|
It is not a Jenkins Plugin. However, if your Jenkins job triggers a Maven build, then Cluecumber can be used inside the Maven POM after your test runs. We use it that way for all of our Jenkins test jobs. |
|
Hi, How can I use this commit done on pull 636 on Jenkins? It's doing exactly what I want! Thanks |
|
Hi! Cluecumber should not fail the scenario in this case. Could you give me an example json of your case so I can test it? |
|
Thanks, |
|
Thanks, @silviatejera ! In your case, the scenario contains pending and skipped steps - that's why Cluecumber could not handle it correctly. I opened a bug on my side that will be fixed in the next release: trivago/cluecumber#29 |
|
Thanks @laxersaz |
|
May i know, when this issue(Not marking "Pending" scenarios as "Failed" will be fixed? |
|
We need cucumber reporting Jenkins plugins as our test cases are running regularly on Jenkins server. Is Cluecumber available on Jenkins. Also does it has the ability to combine mutiple Json files created from prallel run and create 1 consolidated report? |
|
Cluecumber is not available as a native Jenkins plugin. However, it can easily be used in a Jenkins CI (we are using it on Jenkins as well) by invoking it directly via the command line like so: |
|
@laxersaz We are using gradle. Does cluecucumber-report supports gradle as well? |
|
Hello @kth13 , as I have too little gradle experience, I cannot really help you here (yet). In the future, I might consider porting Cluecumber over but at this moment, it is Maven only. |
damianszczepanik
force-pushed
the
master
branch
3 times, most recently
from
ff16211 to
6b5485d
Compare
…-reporting Conflicts: src/main/java/net/masterthought/cucumber/json/Feature.java src/test/java/net/masterthought/cucumber/generators/integrations/FeaturesOverviewPageIntegrationTest.java src/test/java/net/masterthought/cucumber/generators/integrations/TagsOverviewPageIntegrationTest.java
|
Could you please let me know here is there any further update on this. Can see this issue still although this thread is closed. If this is not getting fixed, is there any work around for this, something like not to fail the scenario if it has pending/skipped steps. |
|
Report new issue with details |
|
Cucumber v6 only supports strict mode now. Meaning pending and undefined steps are always a failure. So from v6 the report is always correct. |
|
What about skipped steps? Is there any way to prevent a skipped step from failing the entire scenario & feature? We use |
|
Suppose we have 2 scenarios - Scenarios A & Scenario B and I have a test requirement that if Scenario A is failed, then I dont want to run Scenario B since it depends on Scenario A. Is this going to be addressed or is there any workaround for it? |
|
Guys, this is just a report. It presents the state and it's up to you if you treat skipped or undefined steps as correct or incorrect case. However I'm not happy to make scenario as passed when it has not passed steps. |
|
Hello, I tried to configure cucumber_report in order to not fail the build if there are some skipped tests (skipped within jUnit Assume) perhaps did I not configure it correctly ? |
It doesn't need to be marked as passing, just not as failing, it didn't run, it was intentionally skipped |
|
+1. Current report is misleading where you already show some steps as "Skipped" but overall scenario is treated as "Failed". We should have separate column "Skipped" for Scenarios as well and over all status should be "Passed" as those tests were intentionally marked as skipped. |
|
@SteveDonie Do you have newer version of Jenkins plugin which was created from your forked branch? If so, can you please pass it on? |
|
I have left the company that was using the plugin, and haven't done any
work on it in years. The fork is still available at
https://github.com/Datical/cucumber-reporting
…On Wed, Sep 29, 2021 at 11:35 PM Viresh Shah ***@***.***> wrote:
@SteveDonie <https://github.com/SteveDonie> Do you have newer version of
Jenkins plugin which was created from your forked branch? If so, can you
please pass it on?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#636 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAETZ2XUCFK7DGM4E2CDJL3UEPSJPANCNFSM4DPCUZZA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
This is so annoying, my build doesn't fail but report generates failed scenarios |
|
The problem is not with the report which generates report based on the file you pass it but the step that makes test. But anyway you can get the result and decide yourself when the build should fail and when it should not |
|
i have the same issue as well, I launch a target that says 'dont execute tests with tags @ignore' and the reporter tells me these tests failed ........ well NO I didn't want to execute them they passed ! |
|
Encountering same issue as tests were intentionally skipped. In this case was using cypress-tags to run tests on a certain tag eg @staging, unfortunately the result contains any test in the file which does not have the @staging tag and marks them as skipped. Wondering if anyone has any workaround to prevent the build failing in Jenkins |
I had expected that when a scenario has pending or undefined steps, that the scenario as whole would also have the status 'undefined' or 'pending', but currently it shows these scenarios and the features they are in as 'failed'. The changes in this pull request implement that functionality for the main reports. The trend reports have not yet been improved to understand the concent of scenarios or features that are undefined or pending.
I came to this decision after looking at the other cucumber reporting tools, including the command line and the built-in html formatters.
The command-line reporting reports scenarios as failed, passed, undefined, or pending:
Or in this case:
The built-in html formatter also shows scenarios as passed, failed, or pending.