Introduce fail-fast feature for dynamic test execution

[sormuras]

Overview

This PR provides tools around the dynamic tests feature to accomplish a scenario-like way of expressing related tests. The new factory method DynamicTest.requiredTest(String displayName, Executable executable) creates dynamic test instances, that require to finish successfully. If the dynamic test execution loop detects an unsuccessful required test, it stops the loop immediatly. This can happen at any dynamic container level. If the execution loop is halted, no further dynamic test is registered and therefore not executed.

Example usage

Copied from DynamicTestsDemo.java with Log-in and third page visit being required dynamic test. They require to finish with a successful TestExecutionResult

dynamicTest("Visit page requiring authorization while not logged in", () -> {
	// attempt to visit page which requires that a user is logged in
	// assert user is redirected to login page
}),
requiredTest("Log-in", () -> {
	// submit login form with valid credentials
	// assert user is redirected back to previous page requiring authorization
	// assertTrue(false, "you shall not pass");
}),
dynamicContainer("Can access several pages while logged in",
	dynamicTest("Visit second page requiring authorization while logged in", () -> {
		// visit another page which requires that a user is logged in
		// assert user can access page
	}),
	requiredTest("Visit third page requiring authorization while logged in", () -> {
		// visit another page which requires that a user is logged in
		// assert user can access page
		// assertTrue(false, "exit here - no fourth page, not log out");
	}),
	dynamicTest("Visit fourth page requiring authorization while logged in", () -> {
		// visit another page which requires that a user is logged in
		// assert user can access page
	})
),
dynamicTest("Log-out", () -> {
	// visit logout URL
	// assert user has been logged out
})

• if all tests are successful, you see every test in the tree: https://travis-ci.org/junit-team/junit5/builds/229191437#L739
• if "Log-in" fails, the "access several pages while logged in" container and "Log-out" won't show up

├─ dynamicNodesWithRequiredTests() ✘
│  ├─ Visit page requiring authorization while not logged in ✔
│  └─ Log-in ✘ you shall not pass

• if "Visit third page..." fails, "Visit fourth page..." and "Log-out" won't show up

├─ dynamicNodesWithRequiredTests() ✘
│  ├─ Visit page requiring authorization while not logged in ✔
│  ├─ Log-in ✔
│  └─ Can access several pages while logged in ✘
│     ├─ Visit second page ... logged in ✔
│     └─ Visit third page ... logged in ✘ exit here - no fourth page, not log out

Related issues

• Introduce first-class support for scenario tests #48 "scenario tests" -- here the original idea of "lightweight scenarios" was posted by
• Introduce mechanism for terminating Dynamic Tests early #431 "terminating dynamic tests early"

Deliverables

• Discuss, if we want to add this requiredTest(...) feature - if yes, finish Definition of Done list below

---

I hereby agree to the terms of the JUnit Contributor License Agreement.

---

Definition of Done

• There are no TODOs left in the code
• Method preconditions are checked and documented in the method's Javadoc
• Coding conventions (e.g. for logging) have been followed
• Change is covered by automated tests
• Public API has Javadoc and @API annotations
• Change is documented in the User Guide and Release Notes
• All continuous integration builds pass

[nipafx]

If you end up not using merging this, there is a small side project I could recommend. 😉

[sbrannen]

Please note that the "blocking" feature you implemented in this PR is directly related to #431.

[mfulton26]

I think there needs to be a way to specify a blocking container named something like dynamicSteps, dynamicStepContainer, dynamicScenario, etc. e.g.:

    @TestFactory
    @DisplayName("Web Security Scenario")
    Stream<DynamicNode> scenarioTests() {
        return Stream.of(
                dynamicTest("Visit page requiring authorization while not logged in", () -> {
                    // attempt to visit page which requires that a user is logged in
                    // assert user is redirected to login page
                }),
                dynamicScenario("Log-in", Stream.of(
                    dynamicStep("Log-in button is disabled", () -> {
                        // assert user cannot submit login form without entering credentials
                    }),
                    dynamicStep("Log-in button becomes enabled after entering credentials", () -> {
                        // assert log-in button is enabled once credentials are entered
                    }),
                    dynamicStep("Visit third page requiring authorization while logged in", () -> {
                        // submit login form with valid credentials
                        // assert user is redirected back to previous page requiring authorization
                        // fail("you shall not pass");
                    })
                )),
                dynamicContainer("Can access several pages while logged in", Stream.of(
                    dynamicTest("Visit second page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    }),
                    dynamicTest("Visit third page requiring authorization while logged in", () -> {
                        // visit another page which requires that a user is logged in
                        // assert user can access page
                    })
                )),
                dynamicTest("Log-out", () -> {
                    // visit logout URL
                    // assert user has been logged out
                })
        );
    }

[sormuras]

Description updated.

@mfulton26 Does the new required strategy meet your needs?

[mfulton26]

I believe so. Thank you.

[sormuras]

DynamicTest.requiredTest(...) is not the best name. What other name could we use to describe "a dynamic test that has the power to halt the execution loop of the @TestFactory-annotated method"?

replace "dynamic" and leave "Test"

• DynamicTest.requiredTest
• DynamicTest.essentialTest
• DynamicTest.gatewayTest

leave "dynamic" and replace "Test"

• DynamicTest.dynamicCheckPoint
• DynamicTest.dynamicGuard
• DynamicTest.dynamicBarrier
• DynamicTest.dynamicStep

[mlevison]

Another alternative, a separate method called bool shouldContinue() It is called after each step is executed. Perhaps the quality of any of these names would easier to judge of there was a sample code fragment. Cheers Mark

…

On May 7, 2017 1:11 PM, "Christian Stein" ***@***.***> wrote: DynamicTest.requiredTest(...) is not the best name. What other name could we use to describe *"a dynamic test that has the power to halt the execution loop of the @TestFactory-annotated method"*? replace "dynamic" and leave "Test" - DynamicTest.requiredTest - DynamicTest.essentialTest - DynamicTest.gatewayTest leave "dynamic" and replace "Test" - DynamicTest.dynamicCheckPoint - DynamicTest.dynamicGuard - DynamicTest.dynamicBarrier - DynamicTest.dynamicStep — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#838 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAImZl2oWuoCoGFDIXOkorRiGoPmZrzJks5r3fsigaJpZM4NRnqa> .

[sormuras]

[...] It is called after each step is executed.

Mh. How do you pass the method (or any other callback handle) to the execution cycle? Can this "registerPredicateEvaluatedAfterEachDynamicTest(this::shouldContinue);" called more than once?

Perhaps the quality of any of these names would easier to judge of there
was a sample code fragment.

Look at the description text of this issue; it uses requiredTest.

[mlevison]

@christian - working from my phone is hazardous. Trying to do it while kids basketball skills session is in progress is worse. My solution is clearly not fantastic either. My mental challenge, even after reviewing the code, this whole thing feels awkward semantically. Your names for Guard and Barrier seem best, but it feels all very special case. Compared to rest of the JUnit 5 (and even JUnit 4) model this feels very different. I can't yet see a better approach, I just feel that I'm not seeing right now. Perhaps with the existing implementation someone could show us a sample test case - that illustrated the underlying problem. Time to drive kids home Cheers Mark

…

On Sun, May 7, 2017 at 3:43 PM, Christian Stein ***@***.***> wrote: [...] It is called after each step is executed. Mh. How do you pass the method (or any other callback handle) to the execution cycle? Can this "registerPredicateEvaluatedAfte rEachDynamicTest(this::shouldContinue);" called more than once? Perhaps the quality of any of these names would easier to judge of there was a sample code fragment. Look at the description text of this issue; it uses requiredTest. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#838 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAImZtB1c3nMWmY0ZpSW9HhmjfkV-PtLks5r3h7MgaJpZM4NRnqa> .

-- [image: headshot-square-300x300] <http://www.flickr.com/photos/36331075@N00/9674877046/> *Mark Levison* | 1 (877) 248-8277 | Twitter <https://twitter.com/mlevison> | LinkedIn <http://ca.linkedin.com/in/marklevison> | Facebook <https://www.facebook.com/agilepainrelief> Certified ScrumMaster Training: Vancouver <http://agilepainrelief.com/courses/vancouver> | Edmonton <http://agilepainrelief.com/courses/edmonton> | Ottawa <http://agilepainrelief.com/courses/ottawa> | Montreal <http://agilepainrelief.com/courses/montreal> | Toronto <http://agilepainrelief.com/courses/toronto> Certified Product Owner & Private Training also available ~ Our Training Schedule <http://agilepainrelief.com/courses/certified-scrum-agile-training> Agile Pain Relief Consulting <http://agilepainrelief.com/> | Notes from a Tool User <http://agilepainrelief.com/notesfromatooluser> Proud Sponsor of Agile Tour Gatineau Ottawa <http://goagiletour.ca/> and Agile Coach Camp Canada <http://agilecoachcampcanada.wordpress.com/>

[sormuras]

What about adding an optional parameter to DynamicTest.dynamicTest(String, Executable, Predicate) that is evaluated after the executable ran. If it evaluates to true the dynamic test execution proceeds, or otherwise terminates.

[mlevison]

Christian - to my limited tastes in this area is an improvement. I like the fact that it gets rid of something special which feels more inline with JUnit. I've never loved outparams, but at least all dynamicTest() methods are now the same. I also like that any test could be reason to terminate the sequence for any reason. Slightly better would more expressive enum. I'm guessing there are three states: continue; skip; halt. Skip might be used when the test should be skipped because a pre-condition isn't currently valid. Again lacking a concrete and specific example I'm struggling to make my own thinking clear. Cheers Mark

…

On Mon, May 8, 2017 at 3:47 AM, Christian Stein ***@***.***> wrote: What about adding an optional parameter to DynamicTest.dynamicTest(String, Executable, Predicate) is evaluated after the executable ran. If it evaluates to true the dynamic test execution proceeds, or otherwise terminates. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#838 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAImZjSoo5Zq9SGwS8rMsNs1JK0TyiKeks5r3siggaJpZM4NRnqa> .

-- [image: headshot-square-300x300] <http://www.flickr.com/photos/36331075@N00/9674877046/> *Mark Levison* | 1 (877) 248-8277 | Twitter <https://twitter.com/mlevison> | LinkedIn <http://ca.linkedin.com/in/marklevison> | Facebook <https://www.facebook.com/agilepainrelief> Certified ScrumMaster Training: Vancouver <http://agilepainrelief.com/courses/vancouver> | Edmonton <http://agilepainrelief.com/courses/edmonton> | Ottawa <http://agilepainrelief.com/courses/ottawa> | Montreal <http://agilepainrelief.com/courses/montreal> | Toronto <http://agilepainrelief.com/courses/toronto> Certified Product Owner & Private Training also available ~ Our Training Schedule <http://agilepainrelief.com/courses/certified-scrum-agile-training> Agile Pain Relief Consulting <http://agilepainrelief.com/> | Notes from a Tool User <http://agilepainrelief.com/notesfromatooluser> Proud Sponsor of Agile Tour Gatineau Ottawa <http://goagiletour.ca/> and Agile Coach Camp Canada <http://agilecoachcampcanada.wordpress.com/>

[sormuras]

Closing this PR with comment: "Back to the design board." Follow-up at #431