Bull error (stalled jobs)

[hardcodet]

Hi there

Version: 5.1.14

Noticed the following Bull error on our cluster that occurred during application startup.

Error: job stalled more than allowable limit
    at /app/node_modules/bullmq/dist/cjs/classes/worker.js:517:62
    at Array.forEach (<anonymous>)
    at WorkerPro.notifyFailedJobs (/app/node_modules/bullmq/dist/cjs/classes/worker.js:517:20)
    at WorkerPro.moveStalledJobsToWait (/app/node_modules/bullmq/dist/cjs/classes/worker.js:510:22)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async WorkerPro.checkConnectionError (/app/node_modules/bullmq/dist/cjs/classes/queue-base.js:118:20)
    at async WorkerPro.runStalledJobsCheck (/app/node_modules/bullmq/dist/cjs/classes/worker.js:494:17)
    ```
    
Looking at my logs for the last 30 days, I saw a burst of those on one single day, not sure what to make of that:

![image](https://user-images.githubusercontent.com/873009/226401815-c3bd2486-0669-4443-a584-c5bca3940680.png)

Any idea on how to triangulate this?

[manast]

These are jobs that have stalled more than the max stalled count setting (1 by default https://api.docs.bullmq.io/interfaces/WorkerOptions.html#maxStalledCount). In your case I suspect that you server was restarted several times in a small time span so that some jobs stalled more than once. If this is something expected to happen often you can increase the max stalled count, you should also consider graceful shutdowns to minimize this problem: https://docs.bullmq.io/guide/going-to-production#gracefully-shut-down-workers