Introduce support for parameterized containers (classes, records, etc)

[jkschneider]

Overview

Currently, the target of @ParameterizedTest is constrained to methods. When creating technology compatibility kits, it would be awesome to be able to apply this (or a similar annotation) to the test class so that all tests in that class are parameterized the same way.

Proposal

Rather than:

class MyTest {
   @ParameterizedTest
   @ArgumentSource(...)
   void feature1() { ... }

   @ParameterizedTest
   @ArgumentSource(...)
   void feature2() { ... }
}

Something like:

@ParameterizedTest
@ArgumentSource(...)
class MyTest {
   @Test // ?
   void feature1() { ... }
   
   @Test
   void feature2() { ... }
}

Related Issues

• Introduce support for parameterized tests #14
• Introduce extension API for container templates #871
• Influence invocation order for TestTemplate executions #1141
• Parameterized BeforeEach or AfterEach only #3157

[marcphilipp]

I can imagine supporting something like

@ArgumentSource(...)
class MyTest {
   @ParameterizedTest
   void feature1() { ... }
   
   @ParameterizedTest
   void feature2() { ... }
}

Would that suit your needs?

[jkschneider]

I think that is a good solution.

[nipafx]

Related issues: #871, #853

[jkschneider]

For real-life examples, see GaugeTest and most of the other tests in its package.

[marcphilipp]

@jkschneider Do you have time to work on a PR?

[smoyer64]

If a parameter source annotation is placed on an individual method in a type that's been annotated like this would it make sense for the more specific annotation to take precedence?

[jkschneider]

Yes as a matter of determinism? I think in practice, this would not be the way I'd recommend folks structure their test.

[wrlyonsjr]

Is this feature on the roadmap? I can't migrate to 5 without it.

[wrlyonsjr]

...unless someone knows of a workaround.

[jkschneider]

@wrlyonsjr I flipped it around with Micrometer's TCK by defining an abstract class with my test methods, each of which takes an implementation of MeterRegistry and for which there are many implementations. The RegistryResolver extension injects the implementation into each method at any nesting level.

Then there is a corresponding test for each implementation of MeterRegistry that extends this base class.

The approach has the disadvantage that you can't run the TCK abstract class from the IDE and execute the tests for all implementations, but this is the best I could do.

[wrlyonsjr]

It seems like my problem could be solved with TestTemplate et al.

[sbrannen]

Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.

[dmitry-timofeev]

Let me share my experience with parameterized tests in the new JUnit. Hope that helps to clarify the uses cases and requirements for Parameterized classes.

I think that this feature will be universally useful, not for TCK only. I see its ultimate goal in bringing down the cost of defining multiple tests sharing the same parameters source. Currently, the following code is duplicated:

• Annotations, specifying the source (e.g., @MethodSource).
• Formatting strings for parameters (e.g., "[{index}] = {3}").
• Initialization & Clean-up code, if any. If it has to initialize multiple local variables (i.e., not extractable in a separate method as a whole), it has the highest cost in terms of LOCs.

Having to repeat that code discourages the users to write short, focused tests. On top of that, if developers do have a luxury of copy-pasting that code, reviewers and maintainers have to read all of it.

As an alternative, I've tried a dynamic test for a simple use case (little setup code, no teardown), but got both a personal impression and some reviews from colleagues that it's "overcomplicated".

Uses cases

A perfect use case for this feature, in my opinion, is the following:

1. A user writes a couple of parameterized tests which share the same parameters source and setup code.
2. When there are too many of them, a user extracts these tests in a nested parameterized class (ideally, an IDE inspection tells the user to do that).
   • Test method arguments become either constructor arguments, or injected with @Parameter (as in JUnit 4), or setup method (@BeforeEach) arguments.
   • Initialization code goes to the setup method. Any locals needed in tests become fields of the test class.
   • Clean-up code goes to the teardown method (@AfterEach).

[marcphilipp]

I think supporting something like @Parameter makes sense for fields and method parameters.

I'm out of ideas where you'd put formatting strings for parameters that are shared for all parameterized tests, though.

[dmitry-timofeev]

@marcphilipp , I can think of a class-level annotation with name attribute, or, if users need more flexibility, let them provide an instance method returning a test description + a method-level annotation, or a reference to the method in the class-level annotation (e.g., @Parameterized(name="#testDescription")).

If no one on the core team is planning to work on this issue soon, I may try to implement an MVP. I am new to the code base, and have a couple of questions about the requirements:

• Shall anything except @ParameterizedTest be supported in a class annotated with @ArgumentsSource (@Test, other @TestTemplates)? It might get tricky for one of the most compelling use cases for parameterized classes is extracting test template parameters to the fields, and if there is no extension to resolve the values of these fields, it won't work, will it?
• Shall the extension support a product of primary parameters (defined at the class level) and secondary parameters (defined at the method level, as in the current implementation), like this:

@CsvSource(/* primary parameters, injected into constructor/BeforeEach/fields */)
class FooTest {

  /** Invoked for each parameter in the source specified at the class level */
  @ParameterizedTest
  void foo() { }

  /** 
   * Invoked for the cartesian product of parameters from 
   * the source specified at the class level 
   * and parameters from the source at the method level.
   */
  @ParameterizedTest
  @ValueSource(/* secondary parameters for #bar only */)
  void bar(int secondaryParameter) { }
}

?

[andy-goryachev-oracle]

It would be nice to support a clearer migration path from junit4 to junit5 with class-level parameterized tests.

[sbrannen]

@andy-goryachev-oracle, this is scheduled for 5.12 M1, and there's a good chance it will make it into 5.12.

[sbernard31]

@sbrannen Is there a link where we can see how it will look like and/or how we will be able to use it ? (sorry, if answer is already given in previous comment, I didn't re-read the whole post 🙇‍♀️)

[marcphilipp]

It will probably look sth. like this:

@ParameterizedContainer
@ArgumentSource(...) // any `@...Source` annotation
class SomeTests {
	@Parameter(0) // Alternatively, constructor injection should be supported, e.g. for use with records as test classes
	String firstArg;
	@Parameter(1)
	int secondArg;

	// test methods as usual
}

[sbernard31]

Thx for quick answer 🙏 !
Any chances that you also go with something like : #3157 (comment) ?

it could help migration for people who use the workarround 1) from : #3157 (comment)

[marcphilipp]

Any chances that you also go with something like : #3157 (comment) ?

Are you referring to supporting resolving class-level parameters for @BeforeEach methods or something else?

[andy-goryachev-oracle]

@Parameter(0) // Alternatively, constructor injection should be supported, e.g. for use with records as test classes

I like that very much. Injection or the constructor.

The main idea is to minimize modification to the existing tests that use @RunWith(Parameterized.class)

[sbernard31]

Are you referring to supporting resolving class-level parameters for @beforeeach methods or something else?

I was talking about "injecting" parameter as method argument instead of class attribute.

@ParameterizedContainer(name = ...)
@ValueSource(strings = { "param1", "param2", "param3"})
public class TestSuite {
     
    @BeforeEach
    @Parameterized
    public void start(String param1) {
        objectToTest = new MyObjectToTest(param);
    }
    
    ... ...

ou

@ParameterizedContainer(name = ...)
@ValueSource(strings = { "param1", "param2", "param3"})
public class TestSuite {
     
    @BeforeEach
    public void start(@Parameter(0) String param) {
        objectToTest = new MyObjectToTest(param);
    }
    ... ..

[marcphilipp]

Is there a particular reason why you'd prefer an @BeforeEach method over a test class constructor?

[andy-goryachev-oracle]

so the problem is as follows: we have a bunch of junit4 tests which look like

@RunWith(Parameterized.class)
Test {
  public Test(params...)

  @Before

  @After

  @Test
}

An ideal situation would be to modify only a few lines - the class annotation, and the method which supplies the parameters, leaving the rest as is.

Whether the parameters are injected into fields or passed to the class constructor is not that important (I think).

edit: here is an example:
https://github.com/openjdk/jfx/blob/master/modules/javafx.controls/src/test/java/test/javafx/scene/control/skin/LabelSkinLayoutTest.java

[sbernard31]

Is there a particular reason why you'd prefer an @BeforeEach method over a test class constructor?

You mean something like :

@ParameterizedContainer(name = ...)
@ValueSource(strings = { "param1", "param2", "param3"})
public class TestSuite {
     
    public TestSuite(String param1) {
        objectToTest = new MyObjectToTest(param1);
    }

?

The reason is to avoid too much code modification from @ExtendWith(CustomParameterResolver.class) workaround.
Without that in mind, I guess using class constructor (like ☝️) or @beforeEach (like #878 (comment)) is more or less the same.

(Probably, not so important because, I guess there will be lot of code modification anyway 🤔 )

[quiram]

Is there a particular reason why you'd prefer an @BeforeEach method over a test class constructor?

Personally I'm happy with both options but the @BeforeEach approach has the advantage of being more easily composable: you can have multiple @BeforeEach methods, some of them inherited from other classes and/or interfaces, allowing you a staggered initialisation of tests. This level of composability is harder to achieve with the constructor solution (some people may argue that tests shouldn't be this complex but one complex scenarios come up it's good to have the flexibility).

[royteeuwen]

Just to be clear @sbrannen, the part that will make it (in 5.13 now instead of 5.12 I guess) would be to explicitely allow a clear migration path from @RunWith(Parametrized) on class level to junit 5?