[CI] RestClientSingleHostIntegTests.testGetWithBody failed on master #29558

dimitris-athanasiou · 2018-04-17T12:15:53Z

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=debian/2325/console

No reproduce line was printed.

Failures:

11:07:12 Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
11:07:12 ERROR   0.01s J0 | RestClientSingleHostIntegTests.testGetWithBody <<< FAILURES!
11:07:12    > Throwable #1: org.apache.http.ConnectionClosedException: Connection closed
11:07:12    > 	at __randomizedtesting.SeedInfo.seed([BA869D479D2D8EDD:60456C0CDC8F44A]:0)
11:07:12    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:718)
11:07:12    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
11:07:12    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
11:07:12    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:321)
11:07:12    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:312)
11:07:12    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.testGetWithBody(RestClientSingleHostIntegTests.java:216)
11:07:12    > 	at java.lang.Thread.run(Thread.java:748)
11:07:12    > Caused by: org.apache.http.ConnectionClosedException: Connection closed
11:07:12    > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344)
11:07:12    > 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
11:07:12    > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
11:07:12    > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
11:07:12    > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
11:07:12    > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
11:07:12    > 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
11:07:12    > 	... 1 more

The text was updated successfully, but these errors were encountered:

elasticmachine · 2018-04-17T12:15:55Z

Pinging @elastic/es-core-infra

javanna · 2018-04-17T13:03:33Z

I had a look, saw that there's a couple more recent failures that are similar. But it doesn't repro for me and I can't quite see at first glance what the problem could be, nor can I reproduce it when running multiple iterations of this test.

markharwood · 2018-04-27T11:01:48Z

Another example in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=linux/5/console

 Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
 ERROR   0.02s J0 | RestClientSingleHostIntegTests.testEncodeParams &lt;&lt;&lt; FAILURES!
    > Throwable #1: java.io.IOException: Connection reset by peer
    > 	at __randomizedtesting.SeedInfo.seed([8B1467B354CAACAB:353358761BB80C2A]:0)
    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728)
    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:177)
    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.testEncodeParams(RestClientSingleHostIntegTests.java:221)
    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    > 	at java.base/java.lang.Thread.run(Thread.java:844)
    > Caused by: java.io.IOException: Connection reset by peer
    > 	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    > 	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    > 	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:283)
    > 	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:250)

Again no reproduce line in logs and would not reproduce locally.
Mock HTTP server closing the connection?

bleskes · 2018-04-30T20:34:02Z

another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=linux/24/console

16:55:54 Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
16:55:54 ERROR   0.01s J0 | RestClientSingleHostIntegTests.testHeaders <<< FAILURES!
16:55:54    > Throwable #1: java.io.IOException: Connection reset by peer
16:55:54    > 	at __randomizedtesting.SeedInfo.seed([B67B0BD19953E74F:7CAB3B8E1061062F]:0)
16:55:54    > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728)
16:55:54    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
16:55:54    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
16:55:54    > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:177)
16:55:54    > 	at org.elasticsearch.client.RestClientSingleHostIntegTests.testHeaders(RestClientSingleHostIntegTests.java:177)
16:55:54    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:55:54    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:55:54    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:55:54    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
16:55:54    > 	at java.base/java.lang.Thread.run(Thread.java:844)
16:55:54    > Caused by: java.io.IOException: Connection reset by peer
16:55:54    > 	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
16:55:54    > 	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
16:55:54    > 	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:283)
16:55:54    > 	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:250)

it looks like the http server the test sets up crashed, no info as to why (I logged onto the server and couldn't find anything telling in the build folder)

spinscale · 2018-05-02T12:12:41Z

another one here https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=amazon/20/console

Suite: org.elasticsearch.client.RestClientSingleHostIntegTests
ERROR   0.01s J0 | RestClientSingleHostIntegTests.testDeleteWithBody <<< FAILURES!
   > Throwable #1: java.io.IOException: Connection reset by peer
   >    at __randomizedtesting.SeedInfo.seed([67C3230DCE755C5:8B6D096CBE2B9D58]:0)
   >    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
   >    at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:321)
   >    at org.elasticsearch.client.RestClientSingleHostIntegTests.bodyTest(RestClientSingleHostIntegTests.java:312)
   >    at org.elasticsearch.client.RestClientSingleHostIntegTests.testDeleteWithBody(RestClientSingleHostIntegTests.java:207)
   >    at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.io.IOException: Connection reset by peer
   >    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   >    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   >    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
   >    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
   >    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
   >    at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:204)
   >    at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136)
   >    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   >    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   >    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   >    ... 1 more

Wild assumption: I do somewhat remember that the default HttpServer does not have an executor configured (meaning it does not use one and uses the thread it used to bind to the port), so if anyone else connects to that port, I dont know how the HttpServer reacts.

jdconrad · 2018-05-07T21:10:35Z

Another one: (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/1039/consoleFull)

talevy · 2018-05-10T01:42:40Z

this has failed again:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux/2422/console

I cannot reproduce, but here would be the command:

./gradlew :client:rest:test   -Dtests.seed=D840989E2D6365FD   -Dtests.class=org.elasticsearch.client.RestClientSingleHostIntegTests   -Dtests.method="testDeleteWithBody"

It has failed 12 times in the last 30 days. I am AwaitsFix'ing it for master and 6.x.
It has also failed on 6.3, but I am not sure whether it warrants muting it there, if others disagree
I'll add that too

UPDATE: nevermind. cannot use AwaitsFix here since the client tests don't have the lucene test framework. The leading theory is that this is an infra issue, but not clear why

UPDATE2: I do not believe this has to do with any specific one of these test methods, but the suite in general. testDeleteWithBody,testHeaders,testEncodeParams,testGetWithBody have all failed on CI recently. @jasontedor (thanks!) pointed out that one suspicious test to blame may be testManyAsyncRequests.

jasontedor · 2018-05-11T23:14:50Z

This was failing many times per day, and now since #30504 it has not failed at all. This increases my level of hope that we have solved this one.

talevy · 2018-05-15T17:07:39Z

thanks @jasontedor, I've taken another look and it has continued passing since. I will go ahead and close this.

dimitris-athanasiou added >test Issues or PRs that are addressing/adding tests >test-failure Triaged test failures from CI :Clients/Java Low Level REST Client Minimal dependencies Java Client for Elasticsearch labels Apr 17, 2018

jdconrad mentioned this issue May 7, 2018

[CI] ConnectionClosedException in RestClientSingleHostIntegTests #30305

Closed

$@polyfractal$ polyfractal removed the >test Issues or PRs that are addressing/adding tests label May 9, 2018

talevy added v7.0.0 v6.4.0 labels May 10, 2018

jasontedor mentioned this issue May 10, 2018

Isolate REST client single host tests #30504

Merged

talevy closed this as completed May 15, 2018

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] RestClientSingleHostIntegTests.testGetWithBody failed on master #29558

[CI] RestClientSingleHostIntegTests.testGetWithBody failed on master #29558

dimitris-athanasiou commented Apr 17, 2018

elasticmachine commented Apr 17, 2018

javanna commented Apr 17, 2018

markharwood commented Apr 27, 2018 •

edited

Loading

bleskes commented Apr 30, 2018

spinscale commented May 2, 2018 •

edited

Loading

jdconrad commented May 7, 2018

talevy commented May 10, 2018 •

edited

Loading

jasontedor commented May 11, 2018

talevy commented May 15, 2018

[CI] RestClientSingleHostIntegTests.testGetWithBody failed on master #29558

[CI] RestClientSingleHostIntegTests.testGetWithBody failed on master #29558

Comments

dimitris-athanasiou commented Apr 17, 2018

elasticmachine commented Apr 17, 2018

javanna commented Apr 17, 2018

markharwood commented Apr 27, 2018 • edited Loading

bleskes commented Apr 30, 2018

spinscale commented May 2, 2018 • edited Loading

jdconrad commented May 7, 2018

talevy commented May 10, 2018 • edited Loading

jasontedor commented May 11, 2018

talevy commented May 15, 2018

markharwood commented Apr 27, 2018 •

edited

Loading

spinscale commented May 2, 2018 •

edited

Loading

talevy commented May 10, 2018 •

edited

Loading