DemoModulesTest.testSpec() is flaky

[lorban]

Jetty version(s)
12.1.x

Jetty Environment
ee10, ee11 only; ee9 looks stable

Description
DemoModulesTest.testSpec() sometimes fails with:

java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@7e3c79db::SocketChannelEndPoint@7a03e2c4[{l=/127.0.0.1:35542,r=localhost/127.0.0.1:42685,ISHUT,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}]->[HttpConnectionOverHTTP@7e3c79db(l:/127.0.0.1:35542 <-> r:localhost/127.0.0.1:42685,closed=false)=>HttpChannelOverHTTP@3f0173e0(exchange=HttpExchange@5467eea4{req=HttpRequest[POST /ee11-test-spec/test/xx HTTP/1.1]@c4c0b41[TERMINATED/null] res=HttpResponse[null 0 null]@160396db[PENDING/null]})[send=HttpSenderOverHTTP@131bfb9d(req=QUEUED,failure=null)[HttpGenerator@ef4aa84{s=START}],recv=HttpReceiverOverHTTP@6b047c5c(ex=HttpExchange@5467eea4{req=HttpRequest[POST /ee11-test-spec/test/xx HTTP/1.1]@c4c0b41[TERMINATED/null] res=HttpResponse[null 0 null]@160396db[PENDING/null]},rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]]

	at java.base/java.util.concurrent.CompletableFuture.wrapInExecutionException(CompletableFuture.java:345)
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:440)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2117)
	at org.eclipse.jetty.client.transport.HttpRequest.send(HttpRequest.java:707)
	at org.eclipse.jetty.tests.distribution.DemoModulesTests.testSpec(DemoModulesTests.java:386)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at java.base/java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:294)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at java.base/java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:294)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at java.base/java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:294)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at java.base/java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:294)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: java.io.EOFException: HttpConnectionOverHTTP@7e3c79db::SocketChannelEndPoint@7a03e2c4[{l=/127.0.0.1:35542,r=localhost/127.0.0.1:42685,ISHUT,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}]->[HttpConnectionOverHTTP@7e3c79db(l:/127.0.0.1:35542 <-> r:localhost/127.0.0.1:42685,closed=false)=>HttpChannelOverHTTP@3f0173e0(exchange=HttpExchange@5467eea4{req=HttpRequest[POST /ee11-test-spec/test/xx HTTP/1.1]@c4c0b41[TERMINATED/null] res=HttpResponse[null 0 null]@160396db[PENDING/null]})[send=HttpSenderOverHTTP@131bfb9d(req=QUEUED,failure=null)[HttpGenerator@ef4aa84{s=START}],recv=HttpReceiverOverHTTP@6b047c5c(ex=HttpExchange@5467eea4{req=HttpRequest[POST /ee11-test-spec/test/xx HTTP/1.1]@c4c0b41[TERMINATED/null] res=HttpResponse[null 0 null]@160396db[PENDING/null]},rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]]
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:544)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1797)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:321)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:85)
	at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
	at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:267)
	at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP$FillableCallback.succeeded(HttpConnectionOverHTTP.java:458)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:1575)

It seems that this PR has changes that are making the test fail more reliably, but even vanilla jetty-12.1.x can fail.

Running only ee10 or ee11 seems to make the test fail much less often.

Adding a Thread.sleep() just before client.POST(baseURI + "/test/xx").send() also seems to make the test fail much less often.

[janbartel]

@lorban, @sbordet this looks like a jetty client problem