Redis Streams usability

[chrisckc]

After reading about Redis Streams i decided it would be a good fit for the problem i am trying to solve, i plan to also use Redis for caching so would rather make use of its capabilities to create a reliable queue than have to also introduce RabbitMQ etc.

Redis pub/sub would suffice initially, but i know i am going to need a proper queue with persistence and all the other good things that Redis Streams brings with it.

I started of with a simple proof of concept using pub/sub with this library, all was good as i could use:

subscriber.Subscribe("my-channel', async (channel, message) => { // do some work here...; });

which i could use to wire up incoming messages on the server side to web clients using "Server-Sent-Events" for example.

I then looked at how i could do this with Streams so i looked at the tests:
https://github.com/StackExchange/StackExchange.Redis/blob/master/tests/StackExchange.Redis.Tests/Streams.cs

But could i not find an example in the tests or anything in the source code of how to do this:
XREAD BLOCK 0 STREAMS my-stream $
which seems to be the Streams equivalent of SUBSCRIBE, the difference being that XREAD will need to be called in a loop because when a message is received on the stream, the XREAD command unblocks. This is fine because the loop will only fire when a message is received.

Looking further i then found out about the single (well 2 if using subscriptions) connection multiplexing...
#860 (comment)

So without going through all of the relevant source code so i may be wrong, it seems that a single extra connection is opened which covers all subscriptions, probably using PSUBSCRIBE * and messages for each channel that are seen coming in are routed to the correct Subscribe(<channel>,.. block.

So as it stands, the Streams support added to this library does not seem to cover what i think will be the main use case, waiting for new messages to arrive on the Stream. The only way to do this at the moment would be to simulate it by calling StreamRead / StreamReadAsync in a continuous loop with a small delay depending on how much latency is acceptable for the app.

Seeing as one of the main reasons for using Streams and the like would be to move away from a polling style architecture, it seems counter productive to have to introduce polling style code to make it work.

There doesn't seem to be any other library for .NET that support the Streams feature, however the ServiceStack client supports raw commands and uses connection pooling so that may be an option, or just write a basic library from scratch...

The easiest way that i can think of is the use both streams and Pub/Sub at the same time, which would be similar to using Pub/Sub and a List before Streams was available, but with additional benefits of Streams over Lists.

The process would be like this: Add a message to a Stream and then immediately publish a notification message (such as the stream entry id for the message) to a Pub/Sub channel sharing the same name. The subscriber can then use the Pub/Sub channel to listen for the notifications of messages (taking advantage of subscriber.Subscribe) that have been added to the stream and them pull them off the stream, storing the Id of the last message each time.

If the subscriber needs to go away and come back (service restart etc.), it can just re-subscribe to the Pub/Sub channel and also perform a one-off read on the stream using the last message Id to catch up on any missed messages while it was away.

The only issue i can foresee is how to handle a network dropout while listening to the pub/sub channel, according to the docs here, the subscription will auto re-establish itself, but the subscriber will also need to be informed of the dropout so it can immediately read the stream to capture any missed messages that would simply have been dropped by the Pub/Sub channel while the subscriber was offline. Without this, the subscriber would not see the missed messages until a new message arrives on the Pub/Sub channel which triggers a Stream read using the last message Id.

Has anyone else dealt this kind of issue with Streams yet?

[mgravell]

I think there's a lot of validity to what you're saying. In terms of the network dropout issue (which may or may not be somewhat mitigated by the RESP3 connection changes), I would expect any implementation based on pub/sub to do pub/sub plus periodic poll as fallback. But I do happen to agree that there's a compelling case for better blocking operation support. It just needs time to make it happen.

[ccerrato147]

Has there been any update on this request? I've also been looking for a subscribe to stream functionality without requiring a pooling option constantly checking if there's a new element in a stream.

[skuirrels]

Hi @mgravell do you know if there's any pencilled idea on when this may be implemented? There's quite some excitement over the stream blocking, although i understand it's anti the current architecture.