ClickHouse exception, code: 1002, host: xxxx, port: xxx; xxxx:xxx failed to respond

[just-JL]

My ck-jdbc version is 0.2.6, this exception still occurs.Any help?

[zhicwu]

My ck-jdbc version is 0.2.6, this exception still occurs.Any help?

Have you tried the most recent version 0.3.1-patch? Could you please post the stack trace? Is it an insert query?

[just-JL]

My ck-jdbc version is 0.2.6, this exception still occurs.Any help?

Have you tried the most recent version 0.3.1-patch? Could you please post the stack trace? Is it an insert query?

I have not try 0.3.1-patch.And it is an insert operation.The exception stack is :
ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: xxxx, port: xxx; xxxx:xxx failed to respond
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.getException(ClickHouseExceptionSpecifier.java:91) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:55) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:24) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:761) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:196) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:166) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:161) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:156) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHouseStatementImpl.execute(ClickHouseStatementImpl.java:328) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.execute(ClickHousePreparedStatementImpl.java:134) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:382) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:362) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:291) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.clickhousewriter.ClickhouseWriter$Task.startWrite(ClickhouseWriter.java:96) [clickhousewriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
Caused by: org.apache.http.NoHttpResponseException: xxxx:xxx failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[httpclient-4.4.jar:4.4]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:735) ~[clickhouse-jdbc-0.2.6.jar:0.2.6]
... 13 common frames omitted

[zhicwu]

I see. I don't think new version of the driver will help. The problem is server somehow actively closed the connection leaving JDBC driver holding a broken connection. Below was the code to reproduce the issue:
https://github.com/ClickHouse/clickhouse-jdbc/blob/5e6f74b89354e60af7d0c9437eef0520ee9ffda1/clickhouse-jdbc/src/test/java/ru/yandex/clickhouse/util/ClickHouseHttpClientBuilderTest.java#L207-L223

The workaround being added for this is to simply retry idempotent operation(e.g. select query but not insert query). I'm sorry but currently I don't have a good solution for this except handling the error and retry at your end.

I think we probably need an option to stop using pooled connection(before switching to HttpURLConnection), which might be of use.

[just-JL]

I see. I don't think new version of the driver will help. The problem is server somehow actively closed the connection leaving JDBC driver holding a broken connection. Below was the code to reproduce the issue:
https://github.com/ClickHouse/clickhouse-jdbc/blob/5e6f74b89354e60af7d0c9437eef0520ee9ffda1/clickhouse-jdbc/src/test/java/ru/yandex/clickhouse/util/ClickHouseHttpClientBuilderTest.java#L207-L223

The workaround being added for this is to simply retry idempotent operation(e.g. select query but not insert query). I'm sorry but currently I don't have a good solution for this except handling the error and retry at your end.

I think we probably need an option to stop using pooled connection(before switching to HttpURLConnection), which might be of use.

So the next release，can this problem be fixed？Thanks a lot!

[zhicwu]

So the next release，can this problem be fixed？Thanks a lot!

Do you have a connection pool that can be configured to validate the connection each time borrowing a connection from the pool? Usually it's a test query like select 1. I think this is probably what you can count on at this point.

I'm sorry that switching to HttpUrlConnection or a different httpclient is not something I can add in next release(0.3.2) because 1) it won't eliminate the issue(when it's the server which killed connection); and 2) I need to close grpc client to unblock future work first.

[MephistoLynn]

same version and same problem

[just-JL]

So the next release，can this problem be fixed？Thanks a lot!

Do you have a connection pool that can be configured to validate the connection each time borrowing a connection from the pool? Usually it's a test query like select 1. I think this is probably what you can count on at this point.

I'm sorry that switching to HttpUrlConnection or a different httpclient is not something I can add in next release(0.3.2) because 1) it won't eliminate the issue(when it's the server which killed connection); and 2) I need to close grpc client to unblock future work first.

Sorry to reply so late. We use datax(https://github.com/alibaba/DataX) , the writer has no connection pool, and gets a new connection only once when a task init. So i think that i can not count on this point.

[zhicwu]

the writer has no connection pool, and gets a new connection only once when a task init. So i think that i can not count on this point.

I see. I'm in the middle of refactoring the JDBC driver. Let me see if I can use URLConnection to replace apache http client in next release.

[just-JL]

the writer has no connection pool, and gets a new connection only once when a task init. So i think that i can not count on this point.

I see. I'm in the middle of refactoring the JDBC driver. Let me see if I can use URLConnection to replace apache http client in next release.

Thanks so much!

[just-JL]

After adjust ck server keep_alive_timeout from 3s to 10s, our online datax task has not failed so far.In addition, according these solutions, the links are as follows. Enables the client to set keepAliveTimeout and validateAfterInactivity in ck jdbc url, we solve this exception in our test env as well , but it's not online yet.
https://blog.csdn.net/siantbaicn/article/details/80854528
https://helloword.blog.csdn.net/article/details/107256696?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.no_search_link&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.no_search_link

[zhicwu]

Thanks for sharing your findings. Although clickhouse-grpc-client is still experimental, I think it's probably another way around in terms of reliability and performance.

Speaking of HTTP client, I took HttpURLConnection as default implementation in clickhouse-http-client, and I'll add more using Apache HTTP client 5 and OkHttp later.

[chengbinghan]

check your jdbc server is busy with full gc

[zhicwu]

Two changes in 0.3.2 can mitigate the issue:

• Validate stale connection to fix the bug: failed to respond #760 for stale connection eviction
• and a brand new JDBC driver com.clickhouse.jdbc.ClickHouseDriver uses HttpURLConnection(JDK11 HttpClient seems has similar issue so it's not the default implementation)

[zhicwu]

I'm closing this as the issue does not exist in the new JDBC driver com.clickhouse.jdbc.ClickHouseDriver in 0.3.2 or above. Please upgrade driver as the legacy driver(ru.yandex.clickhouse.ClickHouseDriver) is deprecated and will be removed soon.

#760 has summarized all the attempts very well along with additional change trying to fix the issue. However, based on flaky test reported in trino, the issue still exists. As far as I know, the issue was raised from Apache HTTP client when it tried to issue request using a connection being closed by server. Purging stale connection may have timing issue(actual execution time could be different from scheduled due to uncertainty of context switching), so I'm not sure if the issue can be completely fixed or not.

Starting from 0.3.2, the driver has been rewritten and the old-school HttpURLConnection was used to replace Apache HTTP client with proved stability. On the other hand, it seems HttpClient in JDK 11+ has similar issue, that's why it's not enabled by default even it has better performance compare to HttpURLConnection.