[FEATURE] Performance issue in CloseableHttpClient

[zane-neo]

Is your feature request related to a problem?
When performing benchmark test with heavy load in remote inference case, there's a chance to encounter the Timeout to get connection issue like below:

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) ~[httpclient-4.5.14.jar:4.5.14]
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$DelegatingConnectionRequest.get(ClientConnectionRequestFactory.java:92) ~[apache-client-2.20.19.jar:?]
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$InstrumentedConnectionRequest.get(ClientConnectionRequestFactory.java:69) ~[apache-client-2.20.19.jar:?]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.14.jar:4.5.14]
        at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[apache-client-2.20.19.jar:?]
        at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) ~[apache-client-2.20.19.jar:?]
        at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) ~[apache-client-2.20.19.jar:?]
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) ~[apache-client-2.20.19.jar:?]
        at org.opensearch.ml.engine.algorithms.remote.AwsConnectorExecutor.lambda$invokeRemoteModel$0(AwsConnectorExecutor.java:83) ~[opensearch-ml-algorithms-2.11.0.0.jar:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.opensearch.ml.engine.algorithms.remote.AwsConnectorExecutor.invokeRemoteModel(AwsConnectorExecutor.java:82) ~[opensearch-ml-algorithms-2.11.0.0.jar:?]

The reason behind this is the default connection pool of CloseableHttpClient is 50, and if the remote model latency is high, a thread can occupy a connection for long time, under heavy load the 50 connections can be occupied very soon and the requests comes later needs to wait for connection from the pool, and since long latency the connections are not released, the waiting will be timed out.

What solution would you like?
Check if possible to add a configuration to control the connection pool size in CloseableHttpClient.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

[austintlee]

Does the HttpConnector also use the http client from AWS SDK?

[austintlee]

Would it be possible to capture session lengths to see if having a reasonable timeout would help?

[ylwu-amzn]

@zane-neo Can you help answer Austin's question?

[zane-neo]

Does the HttpConnector also use the http client from AWS SDK?

No, httpConnector use CloseableHttpClient directly not from aws sdk, but aws sdk internally use CloseableHttpClient.

[zane-neo]

Would it be possible to capture session lengths to see if having a reasonable timeout would help?

By session lengths you mean the a conversation session? If so, different user could have different session lengths, and even for one specific user, long session length doesn't mean the user would like to wait for a response that takes session length long.
I think allowing customer to configure the timeout could be the optimal solution, client can setup this value based on their different business use case.

[zane-neo]

Closing this as async http client is been used now, performance report here: #1839