Redis error handling

[philippj]

Cannot publish to redis... giving up shows that a publish ultimately fails. However in case the Redis server is permanently unreachable, the Redis connection may never reestablish and the server basically hangs in a limbo.

Now a few cases came up where some instances never re-connected even after the Redis server came back online. While this is mostly a edge case scenario, I do think this requires appropriate error handling, potentially configurable.

Unless this is already planned, is there currently a way I am unaware of to monkey patch an exit handler in?

[miguelgrinberg]

in case the Redis server is permanently unreachable, the Redis connection may never reestablish and the server basically hangs in a limbo.

How is this possible? If Redis goes down permanently nothing is going to work. I'm open to changing how long it takes to give up and maybe even make this configurable, but I don't think there is anything else that can be done that makes sense in this case.

Now a few cases came up where some instances never re-connected even after the Redis server came back online.

Are you saying that the server was still retrying the Redis connection and it failed to reconnect? Or was the Redis service back up after the server gave up?

Unless this is already planned, is there currently a way I am unaware of to monkey patch an exit handler in?

I don't understand what you mean by "money patch an exit handler in".

[philippj]

How is this possible? If Redis goes down permanently nothing is going to work. I'm open to changing how long it takes to give up and maybe even make this configurable, but I don't think there is anything else that can be done that makes sense in this case.
Are you saying that the server was still retrying the Redis connection and it failed to reconnect? Or was the Redis service back up after the server gave up?

I am aware of the connection requirement. In my case, the Redis client disconnected as the network connection to the Redis cluster cut off, but it never managed to reconnect after it went back online with just a generic timeout error. All network connections were re-established correctly, and the issue resolved itself with a complete restart of the affected SocketIO server instances. As there is quite a bit of traffic going through, each instance generated a couple of gigabytes of logs with the SocketIO Redis warnings in the span of half an hour. Not all instances were affected by this, even though they have the same configuration and were residing on the same physical machine at the time of the event.

I don't understand what you mean by "money patch an exit handler in".

What I intend to achieve is to be able to stop the main loop in the event of a sustained error scenario. I have now monkey patched _redis_listen_with_retries to execute a restart script if it continuously fails to publish. However, a more elegant and built-in solution would be more appropriate.

If you have a suggestion for implementation, I would be happy to submit a PR of the finished thing.

[miguelgrinberg]

but it never managed to reconnect after it went back online with just a generic timeout error

Isn't this the issue then? Why ignore this and implement a workaround?

If you have a suggestion for implementation, I would be happy to submit a PR of the finished thing.

The only suggestion that I have is to try to figure out why the Redis reconnection logic does not always work. As I said above, making this process configurable makes sense (i.e. number of retries, how often, etc.). Logging only the first time there is a failure might also make sense, I do see the problem with lots of logs if Reids stays down for a while.

Also, monkey patching is almost always a bad idea. The client manager classes are designed so that your application can subclass them and provide additional functionality. If you want to make the server exit, you can implement this logic in a subclass, without having to monkey patch anything.

[philippj]

Well, it is the root issue causing my whole problem. I was not yet able to identify what leads to the reconnect issue as I can not consistently reproduce it. Having a solution for the symptoms seems to be a good course of action for me in the meantime.

Honestly, I did not think about subclassing, but I will transition to it and replace my current patch. Thanks for the input.