feat(queue): add global concurrency

[roggervalf]

ref #2465

[dhardtke]

Our team would greatly benefit from this feature as we have multiple backends that each spawn a worker but only one scheduled job should be executed at the same time.

Looking forward to this! <3

[manast]

This LGTM, thanks to the way we handle paused queues we get this almost for free. I think it will be quite important though to explain the documentation of this feature that if you choose a concurrency level in your workers it will not override the global one, it will just be the maximum jobs a given worker can process in parallel but never more than the global one.

[SnowMarble]

I really appreciate for implementing global concurrency feature. 🙇‍♂️

[faller]

What does this paragraph in the document mean？「It is not possible to achieve a global concurrency of at most 1 job at a time if you use more than one worker.」What if we setGlobalConcurrency to 1 when using multiple workers ?

[roggervalf]

hi @faller, we missed to change that statement now that we have global concurrency. And yes, you can set global concurrency as 1 when using múltiple workers

[faller]

hi @faller, we missed to change that statement now that we have global concurrency. And yes, you can set global concurrency as 1 when using múltiple workers

Thank you for your excellent work

[faller]

hi @roggervalf , If I set a queue with 10 global concurrency, when there are 2 wokers, should I set worker level concurrency at the same time or keep it default ?

[roggervalf]

hi @faller, as from our docs https://docs.bullmq.io/guide/queues/global-concurrency, global concurrency is the maximum value that any worker can handle. Worker concurrency can be set at any time independently. Also the default local concurrency value for a worker is 1. If you want to take advantage of your global concurrency, you should add a local concurrency with a value no more than 10.

[faller]

hi @faller, as from our docs https://docs.bullmq.io/guide/queues/global-concurrency, global concurrency is the maximum value that any worker can handle. Worker concurrency can be set at any time independently. Also the default local concurrency value for a worker is 1. If you want to take advantage of your global concurrency, you should add a local concurrency with a value no more than 10.

Thank you for your answer