Auto-removal of jobs with duplicate jobId

[nocodehummel]

The lazy auto-removal triggered when a new job fails or completes does not work in my use-case. The goal is to call an external API once per hour for all my Objects. Each object is added to the queue with the jobId set to the Object id.

My queue has completed jobs with an unique jobId for all my Objects that should expire after one hour. Since all Objects are in the queue it will not trigger completed or failed events. When I add an Object to the queue it triggers a duplicate event.

The duplicate event is a duplicate of an expired jobId, which should have been removed from the queue. I would prefer the new jobId to be queued and replace the expired job with the same jobId.

My preferred solution would be to include an age check into the duplicate check. When the job age has expired the expired job is replaced with the new job, and it does not trigger a duplicate event. The new job will get completed/failed which also triggers a lazy auto-removal of other expired jobs.

I think this features is important to prevent a queue to stop working. In my case where all jobs are in the queue, and new jobs added always have an existing jobId the queue no longer works.

[nocodehummel]

My workaround is to check for expired completed jobs in the duplicated event. This prevents the queue from getting stalled. The initial duplicate job is not added to the queue with this workaround.

queueEvents.on('duplicated', async ({ jobId }: { jobId: string }) => {
    console.info(queue.name.concat('/', jobId), 'is a duplicate.');
    const keep: KeepJobs = queue.jobsOpts.removeOnComplete as KeepJobs;
    if (keep && keep.age && keep.count) {
        const age = keep.age * 1000; // secs to milliseconds
        const res = await queue.clean(age, keep.count, 'completed');
        if (Array.isArray(res) && res.length > 0) {
            console.info(queue.name, 'removed', res.length, 'expired.');
        }
    }
});

[manast]

The ability to ignore a duplicate job or not depending on the age of the job seems to me like an interesting feature, however, restrictions must exist depending on which state the job is. For example, if the job is active we should not replace it with a new job to avoid side effects.

[nocodehummel]

@manast. I agree that it should be depending on the state. For my use case it would be sufficient to check for expired age with duplicates in state completed.

[roggervalf]

hi @nocodehummel, I recommend you to check our new debounce logic available since v5.11.0

[nocodehummel]

Great, thanks!

[nocodehummel]

@roggervalf I have implemented the debounce feature for a specific queue which improves duplicate job handling when the jobId is not set. However the problem with a stalled queue exists for flow jobs that use jobId. When all jobIds exist on the queue the removeOnComplete is never triggered since it always results in a duplicate event (before doing anything else). This deadlock situation occurs when all jobs are run within the removeOnComplete age.
For standard jobs the implementation could be refactored using the data property. As far as I know the data property is not available in the Flow producer.