[Bug]: (python) how to gracefully shutdown worker?

[klaemo]

Version

1.14.0

Platform

Python

What happened?

I think there is code missing to gracefully shutdown a worker. The Worker.close() method kills the redis connection directly and doesn't wait for pending jobs to finish. As a result the (eventually) computed results gets never reported back.

Maybe I'm holding it wrong but how do you shutdown a worker gracefully the official way?

My workaround looks like this:

class MyWorker(Worker):
    async def close(self, force: bool = False):
        self.closing = True
        if len(self.processing) > 0:
            await asyncio.wait(self.processing, return_when=asyncio.ALL_COMPLETED)
        return await super().close()
    def stop_on_signal(self, signum=signal.SIGINT):
        def _handler_sigint(signum, frame):
            print(' !!! SIGINT - GRACEFUL EXITING... !!!')
            self.closing = True
        signal.signal(signum, _handler_sigint)

async def main():
    worker = MyWorker("channel", handler, {'autorun': False})
    worker.stop_on_signal()
    try:
        print('STARTED')
        await worker.run()
    finally:
        await worker.close()
        print('DONE')

Thank you 🙏

How to reproduce.

No response

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[simonwindt]

The close() should wait for current jobs to finalise as per documentation;
https://api.docs.bullmq.io/classes/v4.Worker.html#close

This works for us. The problem we have is that close() stop emitting failed event. So jobs that fail after close() are never properly handled in our application.

[klaemo]

This should be fixed by #2545