Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JUnit 5 integration spike #137

Closed
bartoszmajsak opened this issue Nov 13, 2017 · 44 comments
Closed

JUnit 5 integration spike #137

bartoszmajsak opened this issue Nov 13, 2017 · 44 comments

Comments

@bartoszmajsak
Copy link
Member

Issue Overview

JUnit 5 Final has been around for a while and we see more demand to bring integration with Arquillian for the community.

Goal of this spike is to explore what needs to be implemented and plan following tasks based on this.

There has been a discussion on the forum and a PoC started by Payara folks.

@MatousJobanek
Copy link
Contributor

Based on my very short investigation, we will probably need to create our own TestEngine. The PoC you have referenced is using an approach of the extensions (callbacks) when you don't have the context of the whole test suite. If I understand it correctly, this is something similar to the current @Rules and @ClassRules
In addition, with the ability of the Launcher we should be able to finally bring the ability of sub-suite deployments. For more information take a look at the Advanced Topics paragraph.

@bartoszmajsak
Copy link
Member Author

bartoszmajsak commented Nov 13, 2017

Thanks for checking it out. What are the next steps then?

@MatousJobanek
Copy link
Contributor

MatousJobanek commented Nov 14, 2017

Definitely, spend more time with the JUnit 5 extensions - if there is some way of getting the whole test-suite context - eg. from the storage that is provided by JUnit. If not, then start moving the old runner logic to a new ArquillianTestEngine.
One of the first steps would be also taking a look at this PR: #107 as it contains logic that would allow using Arquillian as a JUnit @Rule. This work could be reused for new Arquillian JUnit 5 extensions - for cases when a user cannot or doesn't want to use the engine.
Both implementations the TestEngline and the extension approach should be done as one task as it will require complex refactoring and having in mind both approaches - if we agree on that implementing both.
When the current bits are refactored to support JUnit 5, then we can start working on the sub-suite deployment feature or something more.
Summing this up:

  1. finish the research
    a) investigate extensions & test engines more deeply
    b) take a look at already existing implementations
    c) take a look at PR Arquillian as JUnit rule instead of runner #107
    d) prepare a proposal of the final design
  2. agree on the concept/design we will want to implement (only one or both approaches)
  3. Implementation
    a) create a new module (if it is possible to keep it in one project)
    b) implement the support of JUni5
    c) execute all current Arquillian extension and container adapter test suites using JUnit5
  4. plan next steps
    a) sub-suite deployment
    b) ?? anything else ??

@dipak-pawar
Copy link
Contributor

Looks like as per Matous's above comment we don't have suite context if we decided to go through junit 5 extension. junit-team/junit5#1145.

Now if we decided to go with implementing custom TestEngine for Arquillian. We will be having lot of duplications from junit-jupiter-engine due to final & private variable.

Also I found this interesting comment - junit-team/junit5#20 (comment) about discovering all tests twice if you tries to use jupiter-engine as composition in your engine to avoid duplications

Now digging into test extension, to see any other possible ways.

@MatousJobanek
Copy link
Contributor

So, I've tried to create the integration using a combination of a new TestEngine implementation with JUnit 5 extension implementation.
The engine is necessary for having the context of the whole test suite and for starting and stopping of the containers. The extension is for managing the lifecycle on the level of test classes and test methods.

During the implementation I found a few obstacles:
The first problem was that the JUnit 5 surefire provider calls the engine for every test class separately. This has been fixed in the latest version r5.1.0-M1

So, we need to listen on the very beginning and then delegate the test execution to the standard JupiterTestEngine. This is pretty easy, the problematic part is providing an instance of the TestRunnerAdaptor to the extension implementation. TestRunnerAdaptor is used in Arquillian for firing events, managing lifecycle, keeping the scopes instances etc. In other words, without the adaptor (or any other similar class) we cannot share the context between the test engine and the extension, and thus, we are not able to do the Arquillian magic.

Technically speaking, that could be managed by overriding some Jupiter's internal classes and using delegation. It isn't easy to do, it would require several hacks, code duplications, and reflections, but is probably doable.

The last obstacle is that we need the junit-jupiter-engine dependency for the delegation of the test execution. But in the platform implementation, there are retrieved all TestEngine implementations that are on classpath. In our case it would mean that all classes would be executed twice - with our arquillian engine and with jupiter engine.
There is probably a way to workaround that using shaded jar and relocation with breaking inner jupiter's SPI, but this is nothing that I would like to distribute.

Summing it up. We can provide Arquillian functionality in the context of JUnit 5 extension, but not for the context of the whole test suite execution.
There is already a JUnit 5 issue that covers this missing functionality/SPI - it's that one that was referenced by @dipak-pawar : junit-team/junit5#1145

@MatousJobanek
Copy link
Contributor

As the Arquilian engine for JUnit 5 is not feasible for now, I've been trying to create an extension that would do similar work as JUnit 4 Arquillian Rules do.

JUnit 5 versions
I'm been working with JUnit 5 5.1.0-M1 and Platform Launcher 1.1.0-M1

Code
The resulting code can be found here: https://github.com/MatousJobanek/arquillian-core/tree/junit5-extension
The specific JUnit 5 module is here: https://github.com/MatousJobanek/arquillian-core/tree/junit5-extension/junit5

Dependencies:
standalone: org.jboss.arquillian.junit5:arquillian-junit5-standalone
container: org.jboss.arquillian.junit5:arquillian-junit5-container

Extension class
The JUnit 5 extension class: ArquillianExtension

Implementation
Standalone
The extension works in the standalone mode without any problem and as expected.

Container
Container startup/shutdown, as well as a deployment, is managed perfectly and without any obstacles. The problem comes with the in-container testing (client tests are OK). If any test should be executed inside of the container, then (as JUnit 5 doesn't provide any way to use my own test method executer) I need to use a workaround using nasty hacks and Java reflection. Using ExecutionCondition callback, I need to skip the in-container test methods (to prevent executing them on client) and then manually invoke them inside of the container. This works fine as well, but the main problem is with the results - as I had to skip them, then the JUnit 5 listeners marked those test methods as skipped. I'm able to partially change the restult, but some reports (surefire, IDE) can already contain the skipped one, so the reported number of the executed tests can be different than the actuall ones.

@OndroMih
Copy link

I've tried the approach of creating a JUnit5 extensions almost a year ago when I started this discussion (I'm the Payara folk mentioned above). I came to a similar conclusion as @MatousJobanek - it's not possible to intercept test execution so that it's routed to the test executed in the container. The only thing I could do is to run the test in the Arquillian container before it was executed by Junit5 outside the container. But I couldn't prevent JUnit5 running the tests outside the container just with a JUnit5 extension.

Therefore my conclusion is the same - we either need to build a JUnit5 test runner, possibly duplicating lots of code from the Jupiter engine, or wait until JUnit5 implements junit-team/junit5#1145 (which may take too long according to the comments)

My source code is here: OndrejM/arquillian-junit5-hacks, with the extendion class ArquillianExt.java. I didn't get as far as @MatousJobanek and the code is much simpler, but it demonstrates the same problem when you try running the BasicJunit5ArquillianTest - the test runs succesfully in the container but then it fails when running out of container because a managed resource isn't available (the test uses embedded Payara Server so it runs out of the box without external container running).

@MatousJobanek
Copy link
Contributor

Hi @OndrejM,
I saw your work. I was curious if there had been changed something in the JUnit 5 from the time you were trying it which would make the implementation feasible. There have been changed and fixed a lot of things, unfortunately, a possibility to provide a custom test method runner is still missing. I've created an issue for it: junit-team/junit5#1248
My extension is working and nothing is failing, the only thing that is weird are the test results when the in-container tests are used. If you want to use my implementation, feel free to take it ;-). If you would like to have it in the main repo (not my forked one) I can push it into a separated branch.

@keilw
Copy link

keilw commented Mar 28, 2018

@OndrejM @MatousJobanek Has there been any progress or is Arquillian still not working properly with JUnit5?

@bartoszmajsak
Copy link
Member Author

@keilw if you follow this thread and linked issues you will see that we are missing few pieces on JUnit 5 side to make this integration happen.

@keilw
Copy link

keilw commented Mar 28, 2018

@bartoszmajsak Thanks for the update. I tried to add the "Vintage" module which looks like it works fine for now. If JUnit 5 can solve this, we would be happy to use those new versions without the "Vintage Look" ;-)

@bartoszmajsak
Copy link
Member Author

@keilw we would love to make it happen. hopefully we can follow up on that any time soon. Another approach could be to use JUnit rules instead of runner we have introduced late last year http://arquillian.org/blog/2017/12/20/arquillian-core-1-2-0-Final/

@davidwittesz
Copy link

I had have a look at https://github.com/junit-team/junit5/issues/1248 seems that there is no progress up to now?

I Would be very happy about this feature.

Any guesses when it would be possible to use JU5 with arquillian?

@cardil
Copy link

cardil commented Jan 14, 2019

Arquillian's implicit behavior of running tests on remote container and then silently tunneling them to client JUnit was always hard to understand for developers. Maybe it is good opportunity to switch testing to more explicit solution.

I'm thinking of something like that:

@Test
void assertAtContainer(Arquillian arq) {
  // when
  int result = arg.runAtContainer(new ArquillianExecution<Integer>() {
    @Inject
    @Transactional
    Integer execute(MyDao dao) {
      return dao.countActiveUsers();
    }
  });

  // then
  assertThat(result).isGreatherThen(0);
}

@lordofthejars
Copy link
Member

@bartoszmajsak this would be also really great to have support.

@jameschensmith
Copy link

@bartoszmajsak, has there been any additional push for this? I'd love to be able to use JUnit-5 with Arquillian. What's the current status of this?

@bartoszmajsak
Copy link
Member Author

bartoszmajsak commented Feb 15, 2019

@james-r-smith nothing besides the PoC @MatousJobanek worked on back in the day. Two crucial feature requests in JUnit 5 are still not implemented:

You would probably be fine with JUnit5 Vintage Engine but I guess that's not what you are looking for.

@poikilotherm
Copy link

poikilotherm commented May 13, 2019

@bartoszmajsak You might have noticed that junit-team/junit5#157 has been implemented... 😄

@blabno
Copy link
Member

blabno commented Aug 6, 2019

@bartoszmajsak Hi! Any idea if junit5 will be supported in near future?

@gastaldi
Copy link
Contributor

gastaldi commented Sep 9, 2019

Is there anyone working on this feature? It would be really cool to adopt JUnit 5 in Arquillian projects

@bartoszmajsak
Copy link
Member Author

bartoszmajsak commented Sep 11, 2019

Is there anyone working on this feature? It would be really cool to adopt JUnit 5 in Arquillian projects

@gastaldi no one that I'm aware of. It's up for grabs.

@zforgo
Copy link
Contributor

zforgo commented Sep 29, 2019

Since Junit Jupiter 5.5.x has released intercepting test methods are available. I made an extension based on that.
https://github.com/zforgo/arquillian-junit5-extension

@tandraschko
Copy link

Nice! It would be great if it could make it into arquillian directly.

rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Nov 15, 2019
During his testing @manusa found out that integration tests are
not getting executed during their run on CircleCI. The reason was due
to incompatibility of Junit 5 with Arquillian. I've reverted to Junit 4
in kubernetes-itests/ module for now.

We can update this whenever this gets resolved:
    arquillian/arquillian-core#137
rohanKanojia added a commit to rohanKanojia/kubernetes-client that referenced this issue Nov 15, 2019
During his testing @manusa found out that integration tests are
not getting executed during their run on CircleCI. The reason was due
to incompatibility of Junit 5 with Arquillian. I've reverted to Junit 4
in kubernetes-itests/ module for now.

We can update this whenever this gets resolved:
    arquillian/arquillian-core#137
@arjantijms
Copy link
Contributor

Hi, just wondering. Any progress here?

@OndroMih
Copy link

I'm pretty sure the current solution is stable enough to become part of the Arquillian project. I'm not an Arquillian committer but I can submit this to the Arquillian project if nobody plans to do so. Another option would be to submit this as a separate project to the Eclipse foundation as an Arquillian extension.

I've rewritten the Jakarta Batch TCK to use Arquillian and JUnit 5, I wrote about it here: https://ondro.inginea.eu/index.php/possible-ways-to-use-arquillian-in-jakarta-ee-tcks/. I had no problems running the rewritten test suite against Payara Server with all tests passed. Therefore I believe that the current solution is stable enough. It would be good if some other people reported their success stories with this solution or tried to rewrite their current Arquillian tests to JUnit 5 and report how it went.

@lprimak
Copy link
Contributor

lprimak commented Sep 30, 2020

I would second that opinion. Something needs to be done with JUnit 5 and since there is no other solution, this would be a great addition

@bartoszmajsak
Copy link
Member Author

Hey @OndroMih, thanks for sharing your experience with JUnit 5 and Arquillian. I'm all for moving this forward so apologies there was not much traction on this issue - I have other things which are consuming both my professional and personal time.

That said I'm really happy to see people in the community involved and interested. I believe that should be part of arquillian-core as it is with junit4 and TestNG. Is there anything we should add/extend or porting @zforgo work is enough?

@dmatej
Copy link
Contributor

dmatej commented Oct 6, 2020

@bartoszmajsak There are still unmerged my two pull requests, but generally it works well.
https://github.com/zforgo/arquillian-junit5-extension/pulls

@arjantijms
Copy link
Contributor

Hope this can be moved forward. As I manage several projects that use junit and arquillian intensively, I'll try to test out something soon on these projects to see where we stand.

@OndroMih
Copy link

OndroMih commented Oct 6, 2020

Hi @bartoszmajsak, I'm pretty sure it's safe to start by porting @zforgo 's work. Then @dmatej can raise his pull requests against the ported code in arquillian-core to fix some known issues.

@bartoszmajsak
Copy link
Member Author

bartoszmajsak commented Oct 26, 2020

I integrated @zforgo work as a module in arquillian-core. Again thanks for this milestone!

You can find all the adjustments on the junit5_extension branch, including two PRs from @dmatej. If anyone can try it out against their code it would be just great.

The original code has no tests, so I don't feel comfortable with merging it yet. There are, however, some external integration tests which seems to be working fine with a few tweaks. Any contributions here would be more than awesome, but I will try to find some time this week to do a bit of test-after-development :)

@bartoszmajsak
Copy link
Member Author

@OndroMih @arjantijms do you have any larger projects you could give it a spin? If needed I can push snapshot artifact so that it's easier for you to consume.

@Thihup
Copy link
Contributor

Thihup commented Oct 29, 2020

I've got a try using the javaee-samples/jakartaee8-tck project and it seems to be working fine.
https://github.com/Thihup/jakartaee8-tck/commit/a503d3a30b232d0cfc34860439bf96378ed5efbf

@bartoszmajsak
Copy link
Member Author

Thanks for your feedback @Thihup!

@lprimak
Copy link
Contributor

lprimak commented Oct 29, 2020

Definitely please put out a snapshot

@lprimak
Copy link
Contributor

lprimak commented Nov 10, 2020

I can't get it to work...
every test throws an exception:

java.lang.ClassCastException: org.junit.jupiter.engine.descriptor.JupiterEngineDescriptor cannot be cast to org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor

Any ideas?

@lprimak
Copy link
Contributor

lprimak commented Nov 10, 2020

More context:

java.lang.ClassCastException: org.junit.jupiter.engine.descriptor.JupiterEngineDescriptor cannot be cast to org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor
	at org.jboss.arquillian.junit5.container.JUnitJupiterTestRunner.lambda$execute$0(JUnitJupiterTestRunner.java:41)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.lambda$applyPostDiscoveryFilters$3(EngineDiscoveryOrchestrator.java:122)
	at org.junit.platform.engine.TestDescriptor.accept(TestDescriptor.java:249)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.lambda$acceptInAllTestEngines$8(EngineDiscoveryOrchestrator.java:167)
	at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.acceptInAllTestEngines(EngineDiscoveryOrchestrator.java:167)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.applyPostDiscoveryFilters(EngineDiscoveryOrchestrator.java:128)
	at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:92)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:92)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:67)
	at org.jboss.arquillian.junit5.container.JUnitJupiterTestRunner.execute(JUnitJupiterTestRunner.java:49)
	at org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.executeTest(ServletTestRunner.java:139)
	at org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.execute(ServletTestRunner.java:117)
	at org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.doGet(ServletTestRunner.java:86)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1636)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:259)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)
	at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)
	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
	at java.lang.Thread.run(Thread.java:748)

@lprimak
Copy link
Contributor

lprimak commented Nov 10, 2020

I think it's a bug, when I check instanceof for the failure, my tests pass

@lprimak
Copy link
Contributor

lprimak commented Nov 11, 2020

The preceding PR #285 fixes the issue and gets my tests to pass

@lprimak
Copy link
Contributor

lprimak commented Nov 11, 2020

@bartoszmajsak Please check & merge

@lprimak
Copy link
Contributor

lprimak commented Nov 12, 2020

Not sure if the snapshot is available, but I have put the latest branch version up on Payara's Nexus:

Arquillian artifacts:

        <dependency>
            <groupId>org.jboss.arquillian.junit5</groupId>
            <artifactId>arquillian-junit5-container</artifactId>
            <version>1.7.0.Alpha5</version>
            <scope>test</scope>
        </dependency>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.jboss.arquillian</groupId>
                <artifactId>arquillian-bom</artifactId>
                <version>1.7.0.Alpha5</version>
                <type>pom</type>
                <scope>test</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

   <repositories>
        <!-- Once Arq 1.7.0 or later is released with JUnit 5 support, this can be removed -->
        <repository>
            <id>payara</id>
            <url>https://nexus.payara.fish/repository/payara-artifacts</url>
        </repository>
    </repositories>

Payara connector artifacts (example):

        <dependency>
            <groupId>fish.payara.arquillian</groupId>
            <artifactId>arquillian-payara-server-remote</artifactId>
            <version>2.3.1-arq-1.7.0.Alpha5</version>
            <scope>test</scope>
        </dependency>

Payara artifacts can be browsed here:
https://nexus.payara.fish/#browse/browse:payara-artifacts
under fish/payara/arquillian

Sample project can be found here:

https://github.com/flowlogix/flowlogix/tree/master/jakarta-ee/jee-examples

@bartoszmajsak
Copy link
Member Author

Thanks @lprimak. I'm actually about to cut the next alpha release today with all the changes (including yours) :)

@bartoszmajsak
Copy link
Member Author

bartoszmajsak commented Nov 13, 2020

1.7.0.Alpha5 is out with initial support for JUnit 5. Feel free to open an issue if you find anything and I'll be happy to address them. I'm closing this long thread.

Thank you all for your help! Much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests