[Bug]: Uneven job distribution between workers

[nemphys]

Version

v5.4.3

Platform

NodeJS

What happened?

I am not sure if this is a bug, but I couldn't find a better category 😄

We are running a 5-node Redis sentinel setup, where each node is running both Redis and the application using BullMQ, therefore the node that is currently the master Redis instance has naturally the fastest connection to Redis.

What we observe is that this master node ends up executing the majority of jobs across queues, with the rest of the nodes usually picking up jobs only when this node has reached its worker concurrency limit.

I suppose this might be considered normal (I am not aware the mechanism that implements job picking, but it would make sense for it to be a on first-come-first-serve basis, with the node running the Redis master instance locally being the node that inherently arrives first), but I would like to know if there is some way to overcome this and get a more even job distribution between workers.

How to reproduce.

No response

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[mynameisvasco]

I'm also facing this issue on my project.
I have about 32 servers connected to a redis cluster. Usually there are some servers with 60-70% CPU usage while others are 5-15%.
Did you find any solution to this problem?

[nemphys]

@mynameisvasco no, I haven't.

As mentioned, I suppose it is caused by the latency difference between the workers and Redis but it would be nice to have a confirmation (and possible workarounds) by a maintainer!

[hhanesand]

Would also be interested in some insight here, we are seeing the same thing with our setup.

[manast]

Workers picks up jobs as soon as they are done with the previous job. If you have a high concurrency factor then I guess it could happen that the first worker that starts picks up a lot of jobs, and then if there are not enough jobs for the rest of the workers they will be idling more than the first one. But overtime I do expect all workers doing more or less the same amount of processing.

[manast]

Btw, it would be useful to know more about the setup, are they just standard jobs, delayed, prioritised?

[hhanesand]

Hey! Thanks 😄 I wouldn't describe this as essential for us, just what we're seeing surprised us a bit.

• At the moment we're running [email protected] & [email protected] and are about to update to 7.20.1 & 5.25.4
• We are using a redis cluster.
• We have a group id and job id on all our jobs, together with removeOnComplete and removeOnFail. That's it for options.
• We have ~100 queues running across 10 servers - so about 1000 workers.
• Worker concurrencies are adjusted frequently, depending on the number of jobs waiting.

These are the metrics we have. Average is typically below 10%. We can definitely scale down the amount of workers we're running, but are a tad concerned about the higher level of max utilization. Spikes may be caused by the adjustments we're doing, not sure.

[manast]

@hhanesand yes, it is strange, maybe it has to do with the shape of the incoming jobs, like if they come in bursts, then it could be that one eager worker picks a lot of jobs leaving much less to the rest, something like that. I will think more about it.

[nemphys]

In our setup, most jobs are standard jobs (not delayed/prioritized/etc, nothing special there).

Is my hypothesis correct regarding the latency between workers and redis? It could be the most simple explanation, but it would be nice to have some kind of solution for better distribution.

[manast]

I am playing with the idea of introducing an automatic concurrency mechanism, based on how utilised is the NodeJS event loop. So for example if it is below a certain value it can increase the number of concurrent jobs up until some defined maximum. Well, I am assuming that the peaks in those graphs are produced by some workers processing more jobs in parallel than their peers.

[hhanesand]

We do definitely have bursts of jobs. When we're enqueuing jobs we queue everything for one queue in an addBulk operation. Essentially we're doing

load jobs from outside datastore
split jobs by queue

for (jobs of jobs split by queue) {
    queue.addBulk(jobs)
}

I've tried using a flow producer for this so we can avoid O(# of queues) and instead have O(# of jobs / # bulk batch size), but ran into this issue.

Maybe adopting delayed jobs could help with this issue, but we're unable to predict all of the load - we have an API endpoint which adds jobs to queue unpredictably.

[manast]

@hhanesand I think addBulk may be the issue here, we have a PR to resolve this problem: #2841

[manast]

oh sorry, this would only affect if you are adding prioritised jobs not standard ones.

[manast]

ah no, even standard ones actually sorry for the confusion.

[nemphys]

As I mentioned, we are not using addBulk in our setup, will this still have a positive effect?

[manast]

As I mentioned, we are not using addBulk in our setup, will this still have a positive effect?

Not for you, but it seems like this could be the case for @hhanesand

[manast]

@nemphys it could be that what is happening in addBulk also happens with standard add in some scenarios, we will investigate it as well.

[nemphys]

Nice! Any hint to what is actually happening so that we can have a general idea?

[roggervalf]

hi @nemphys sorry for the delay. For awakening workers, we have a mechanism based on adding markers. For standard and prioritized jobs we add 1 marker with the same score. These markers only awakes a worker that consumes it and then it can start consuming other jobs. If these markers are consumed but there are still other jobs in waiting or prioritized state and there are other workers that couldn't consume this marker, they will be waiting for the next marker for at least 5 seconds (after this period workers, will check if there are pending jobs anyway).

[manast]

@nemphys the PR above should improve the situation in your case, please let me know how it goes.

[nemphys]

@manast thank you, I will try to test it in the first available slot, although I am stuck in an pretty old version due to issues upgrading repeatable jobs to the new format.

[manast]

@nemphys if you need help upgrading please let us know.