Preventing Duplicate Task Creation in BullMQ with PM2 Cluster Mode

[850036165]

Discussed in #2741

^{Originally posted by 850036165 August 28, 2024}
I use multiple queues to structure my task flow. For example, if a task consists of two actions, I split the task into two queues: Queue A and Queue B. After an HTTP request, a job is created in Queue A. When Queue A completes, a new task is created in Queue B. This works fine during development. However, when I attempt to run multiple threads using PM2 in cluster mode, there is an issue. In Step 1, only one Node instance responds to my HTTP request and creates a task in Queue A, which is expected. But in the completion event of Queue A, BullMQ has multiple workers listening simultaneously, leading to multiple tasks being created in Queue B, which I want to avoid. How can I solve this problem?

@QueueEventsListener('archiveR0Queue')
export class ArchiveR0QueueEvents extends BaseQueueEvents {
  constructor(processService: ProcessService) {
    super(processService, ArchiveR0QueueEvents.name)
  }
  @OnQueueEvent('completed')
  async onCompleted(job: { jobId: string; returnvalue: any; prev?: string }) {
    const { orderId } = job.returnvalue
    if (orderId) {
      await this.processService.queueManagementService.handlePendingJob(
        orderId,
        job.returnvalue,
      )
    }
  }
}
```</div>