UnsupportedOperationException from ListSubscriber during mget

[arkadius]

Hi,

I've got this exception:

io.lettuce.core.RedisException: java.lang.UnsupportedOperationException
        at io.lettuce.core.LettuceFutures.awaitAll(LettuceFutures.java:88)
       (...)
Caused by: java.lang.UnsupportedOperationException: null
        at java.util.AbstractList.add(AbstractList.java:148)
        at java.util.AbstractList.add(AbstractList.java:108)
        at io.lettuce.core.output.ListSubscriber.onNext(ListSubscriber.java:47)
        at io.lettuce.core.output.KeyValueListOutput.set(KeyValueListOutput.java:56)
        at io.lettuce.core.protocol.RedisStateMachine.safeSet(RedisStateMachine.java:373)
        at io.lettuce.core.protocol.RedisStateMachine.decode(RedisStateMachine.java:124)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:624)
        at io.lettuce.core.protocol.CommandHandler.decode0(CommandHandler.java:599)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:585)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:542)
        at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:511)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

My environment:
Lettuce 5.0.1 (the same result with 5.0.2)
Redis 4.0.1

Sample code causing situation:

RedisURI uri = RedisURI.create(redisConnectionString);
RedisClient redisClient = RedisClient.create();
StatefulRedisConnection<String, String> asyncConnection = redisClient.connect(uri);
asyncConnection.setAutoFlushCommands(false);
List<Future<T>> futures = part.stream()
     .map(() -> asyncConnection.async().mget(keys))
     .collect(Collectors.toList());
asyncConnection.flushCommands();
LettuceFutures.awaitAll(timeoutInSecs, TimeUnit.SECONDS, futures.toArray(new Future[futures.size()]))

Unfortunately I can't reproduce this bug. It happens only on production environment. One idea that I have is that it can be connected with thing that I'm using the same connection for async.set() and there can be concurrent flushes of this commands. I've tried to prepare test which submit these commands concurrently but with no effect. Any other idea?

I've found out that there were similar issues in NestedMultiOutput (#589) so I've submitted MR with similar solution: #710

[mp911de]

Thanks a lot. Without a possibility to reproduce the issue its impossible to figure out that we're fixing a bug here. Depending on the load it would make sense to record some debug logs showing the payload, or even collect the response at …Output level to see how responses correlate.

flushCommands(…) internally synchronizes while buffered commands are drained. In any case, calling flushCommands(…) without external synchronization can cause application-side issues (non-flushed commands, etc.)

[arkadius]

I understand your point. You can close the issue and MR. When I'll find the way to reproduce the issue, I'll reopen it. Thank you for your help.

[mp911de]

Let's keep that one open for a couple of days as there is somewhere an issue, we just need to find it.

[mp911de]

One thing came to my mind, that maybe reconnect could be involved. Could you check whether there was a reconnect right before the exception?

[arkadius]

I don't see any log entry which could confirm that, but I had logging on INFO level. Couple of additional hints:

• That day, it happen deterministically - I have scheduled task that fetch some data from redis periodically - each fetch ended up with this error. It was on production so I haven't too much time for more debugging.
• I have two application nodes using sentinel connection - it happen on one of them, but it hadn't on the second one
• I had one shared connection for every opertion. After all I've switched to approach that I have separate connection for each operation (to avoid too often flushing of commands), and after that I haven't any problem with mget operation

[mp911de]

Closing this one as it seems we cannot do anything further here. Feel free to reopen the ticket once there is something we can do.

[mshinn]

I can reproduce this with the following sample code on version 6.3.2:

String key = "myKey";
redisClusterCommands.hset(key, "field1", "value1");
redisClusterCommands.hset(key, "field2", "value2");
Collection<KeyValue<String, String>> values = redisClusterCommands.hrandfieldWithvalues(key, 1);


If you change the last argument the hrandfieldWithvalues to 2, it works. I believe the bug is here in the multi method:
https://github.com/redis/lettuce/blob/main/src/main/java/io/lettuce/core/output/KeyValueListOutput.java#L77





If the count argument is one, and keys is null, the output field ends up as an immutable empty list and the UnsupportedOperationException is thrown when the code attempts to add an element to the list.