ksqldb-api-client: Cannot retrieve more than 100 rows in pull/push queries using StreamedQueryResult

[nafeezmd]

I have a simple stream that contains 500+ records and I'm trying to query those records with some added filters in the push query. The result set should contain 200+ records easily but only 100 rows are being returned.

To reproduce:

• Create a test Ksql Table
• Insert 100+ records
• Try executing the pull/push query on this table using this.ksqlDBJavaClient.streamQuery(sql, Collections.singletonMap("auto.offset.reset", "earliest"));

@vcrfxia - Thanks for your contribution to the api client. I see you added the Request batch size limit here. Do you think this needs to be configurable?

Error stacktrace:

2021-02-28 23:27:07.736 ERROR 46923 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

java.lang.IllegalStateException: On wrong context or worker
	at io.confluent.ksql.util.VertxUtils.checkContext(VertxUtils.java:38) ~[ksqldb-common-6.1.0.jar:na]
	at io.confluent.ksql.reactive.BaseSubscriber.checkContext(BaseSubscriber.java:147) ~[ksqldb-common-6.1.0.jar:na]
	at io.confluent.ksql.reactive.BaseSubscriber.makeRequest(BaseSubscriber.java:122) ~[ksqldb-common-6.1.0.jar:na]
	at io.confluent.ksql.api.client.impl.PollableSubscriber.checkRequestTokens(PollableSubscriber.java:115) ~[ksqldb-api-client-6.1.0.jar:na]
	at io.confluent.ksql.api.client.impl.PollableSubscriber.poll(PollableSubscriber.java:90) ~[ksqldb-api-client-6.1.0.jar:na]
	at io.confluent.ksql.api.client.impl.StreamedQueryResultImpl.poll(StreamedQueryResultImpl.java:105) ~[ksqldb-api-client-6.1.0.jar:na]
	at io.confluent.ksql.api.client.impl.StreamedQueryResultImpl.poll(StreamedQueryResultImpl.java:86) ~[ksqldb-api-client-6.1.0.jar:na]
	at io.confluent.ksql.api.client.impl.StreamedQueryResultImpl.poll(StreamedQueryResultImpl.java:81) ~[ksqldb-api-client-6.1.0.jar:na]

[vcrfxia]

Hi @nafeezmd , thanks for your report. The REQUEST_BATCH_SIZE param you linked is not meant to be a limit on the number of records that may be returned. The expected behavior is for the client to request 100 records at a time until all records have been returned. Based on the stacktrace you shared, it appears there may be a bug in the implementation. I see @AlanConfluent is looking into this, so stay tuned.

[nafeezmd]

The expected behavior is for the client to request 100 records at a time until all records have been returned.

@vcrfxia: Thanks for your feedback. You mean, client is responsible in forming the query in a way that all 50000 records could be fetched in batches(100 at a time)?

On a side note, we have tried using ROWTIME to achieve this batching but seems like KSQL is returning the results out of order so using ROWTIME to achieve the batching did not work for us.

SELECT ....
WHERE ROWTIME >= ${start_time}
LIMIT 100;

[AlanConfluent]

I reproduced the error above. I have verified that on the CLI, it's able to return hundreds of rows, but using the java client, it hits this error. I'm not entirely sure what the issue is, but walking through the code a bit, it doesn't appear that the call to poll triggers the underlying methods to be invoked on the vertx context, so this may be the source of the bug.

[mrunmayee003]

Hi Team,

We are desperately waiting for this fix to be available. Our solutions are on hold because of this issue. @AlanConfluent can you please tell by when this patch will be available?
Or else, we might have to look for an alternatives if this 6.2.x patch is not available.
Thank you.

[vcrfxia]

Hi @mrunmayee003 , the fix has already been released with ksqlDB 0.17, and will also be released with Confluent ksqlDB 6.2.0 once that's out.