Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BZPOPMIN will hang for ever when a short disconnection happens and the port is different from 6379 #1888

Open
manast opened this issue Apr 25, 2024 · 1 comment

Comments

@manast
Copy link

manast commented Apr 25, 2024

This one was pretty weird but quite serious. If you issue a BZPOPMIN command, and there is a small disconnection while the command is blocking, then it will stay blocking forever, effectively hanging the app that uses the command.

Now, one of the insane things about this issue is that to reproduce it you must use a port different than the standard 6379, and also use docker network disconnect, it is not enough to just stop the docker container running Redis:

Try this code:
bug.mjs:

import Redis from "ioredis";

const connection = new Redis({
    host: "localhost",
    port: 6380,
});

connection.on("connect", () => console.log("Redis connected!"));
connection.on("ready", () => console.log("Redis ready!"));
connection.on("error", (err) => console.error("Redis error:", err));
connection.on("end", () => console.log("Redis connection ended"));
connection.on("reconnecting", () => console.log("Redis reconnecting..."));

async function test() {
    console.log("Gonna issue a BZPOPMIN command...")
    const result = await connection.bzpopmin("key", 0)
    console.log("Issued BZPOPMIN command!", result)
}

test().catch(console.error);

docker-compose.yml:

version: "3.7"
services:
  redis:
    image: redis:alpine
    ports:
      - "6380:6379"

Start the docker container with:

docker-compose up

Run the test with:

node bug.mjs

Results in:

Gonna issue a BZPOPMIN command...
Redis connected!
Redis ready!

Now in a different terminal, and depending on where you run the docker (I had my docker compose on directory ioredis-econnreset:

docker network disconnect ioredis-econnreset_default ioredis-econnreset-redis-1
sleep 3
docker network connect ioredis-econnreset_default ioredis-econnreset-redis-1

The Redis network was disconnected and connected again after 3 seconds. Open a Redis cli:

redis-cli -p 6380
> 127.0.0.1:6380> zadd key 10 test
(integer) 1
127.0.0.1:6380>

The terminal with the node app will stay the same, with no error or nothing. If you restart the app and without disconnecting and reconnecting you try again to ZADD to the key you will get instead:

Gonna issue a BZPOPMIN command...
Redis connected!
Redis ready!
Issued BZPOPMIN command! [ 'key', 'test', '10' ]

Let me know if you need more information.

@lukas-becker0
Copy link

Hi @manast,

I can not reproduce this with ioredis 5.4.1, node 20.14.0, the latest redis alpine version (7.2.5) and docker 26.1.3 (colima on macos).

The terminal with the node app will stay the same, with no error or nothing.

I do get connection errors and reconnecting output as expected when disconnecting
and after a few seconds reconnecting the docker network like you described ( I also tried it with less time between the disconnect and connect docker commands (1 second) and more than 3 seconds >=10 seconds but the result is the same).

I always get the Issued BZPOPMIN command! [ 'key', 'test', '10' ]
output in the apps terminal as expected even without restarting the app.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants