Quickly emptying a large queue without stalling

[etnoy]

Hello.

I'm a contributor to the Immich project (https://github.com/immich-app/immich/) and we're using bullmq to manage our jobs. When importing assets from disk we create one job per asset to import, and this has worked well in the past.

We're now investigating bottlenecks our users are running into when importing large libraries. With millions of assets to import, we queue up millions of import jobs.

We discovered that calling drain() on a queue that has millions of jobs locks up redis, and in turn, bullmq. I'm observing that the lua code for drain (https://github.com/taskforcesh/bullmq/blob/master/src/commands/drain-4.lua) front-loads the job deletion by listing all jobs, then sending the whole list to ltrim (https://github.com/taskforcesh/bullmq/blob/master/src/commands/includes/removeListJobs.lua#L12)

In my testing, I find that instead using obliterate() has no problem clearing a queue, we can delete 4M jobs in about two minutes on my system. Outside of that, obliterate isn't really what we want to do so ideally the implementation of drain() would be as optimized as obliterate().

Job queues and redis is way outside my field of expertise, so I'm asking if my observations make sense, and if it is possible to make drain() have the same performance as obliterate().

Thanks!