-
Notifications
You must be signed in to change notification settings - Fork 11.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Database queue driver doesn't reconnect #15179
Comments
We should automatically reconnect for such errors. |
Hmm, seems we only do that for query executions, not acquiring transactions. Ping @taylorotwell. |
@jp7carlos
Can you provide a work around? :) |
+1 to get notified on this thread. Might be connected on an issue in my package yajra/laravel-oci8#191. Thanks! |
Seem to be encountering the same issue on 5.2 on PHP 7 The queues work for a day, then failed and sit in reserved. Seems It's also trying to log into the failed jobs table but that is also failing.
pdo here returns null |
@vaidas-lungis The workaround is probably to not use |
@vaidas-lungis Workarounds are ugly
If you have knowledge/time just fork laravel, fix this and send a PR |
Yea I used second option. |
@vaidas-lungis Probably. My personal opinion is that the daemon mode is an abomination. Php is meant to die. The daemon mode introduces complex scenarios where you need to handle dead database connections, memory limits and garbage collection, and other nasty things regarding long running php scripts. All these need to be handled. Basically, it adds complexity to save some cpu. The queue documentation has removed any mentions of the daemon workers in 5.3, and the supervisor configuration example no longer contains --daemon. I do not know if this was an oversight, or intentional.
Source: https://software-gunslinger.tumblr.com/post/47131406821/php-is-meant-to-die |
This might be what's causing my queue problems. |
I have supervisor set up to run
Is the cause that If it helps, I can run |
I'm running Laravel 5.3.9 with MySQL, and have the same issue as @nhowell . I'm also running several Laravel 5.1 projects, but none of them has such issue so far. Below is the stack trace.
|
This issue should be fixed by #15511, please ping me if it's not. |
@zhouyixiang I am running L 5.3.10 / MySQL 5.7.16-0ubuntu0.16.04.1, app is connecting without problems but when I run CLI php artisan queue:work I am getting same error, strange :/ any idea to fix? |
@pun-ky Updating to at least Laravel v5.3.19 should fix it. |
I still having this issue on Laravel 5.2. I'm running Queue Daemon using DB Queue driver, monitored by Supervisord. I'm just having 500 error messages like this everyday on my log: |
@iget-esoares There's your problem ;) Laravel 5.2 does not have the #15511 fix. |
@nhowell Do you know if this will be fixed on L5.2? If so I can do a PR. |
Laravel 5.2 is no longer supported. Can you upgrade to Laravel 5.3? |
@sisve, Yes I can, but it will take a few days to migrate and test since this is an old project that doesn't use unit testing. :( |
@iget-esoares I'm afraid you'll need to upgrade :) |
@themsaid Is the fix going to be rolled into the LTS version of Laravel 5.1? |
@njovin The mentioned pull request is already targetting 5.1. |
I am runnig Laravl 5.3.31 and I have the same problem with queue DB driver and connection lost. Shoul it by fixed in 5.3.18? Or where can I find the fix? |
@dusansalaj - same here, you find a solution? We use the latest version. |
@themsaid The issue still happening on 5.3.31 on some situations. Yesterday I've finally upgraded from 5.2 to 5.3 just to get this bug solved. Now, the error don't occour anymore on timeout, BUT, if I restart the mysql server, it cause infinite "MYSQL has gone away" again. |
@themsaid @GrahamCampbell Please take a look here
Note that the connection fail will only be caught if transactions count is zero. Since there's not a try-catch block here, the lost connections won't be detected and the Queue Woker will throw infinite errors on the log but will not reconnect. |
I know that its been a long time that this discussion had its last reply; we currently have the same issue and trying to investigate whats wrong... below i'll explain what we've seen so far. With the following setup (forge provisioned on DO): Only visible while dispatching through the horizon queue worker; like already said by other people it's not seen in horizon as an failed job but that's probably because of the architect of the queuing system itself. When failing (throwing an exception) laravel horizon will try the job again. What will happen on the background is as follows: Currently a fresh setup; no open connections to the database (no sleeps) except for the "show processlist" command via CLI:
also Sentry is empty and no errors are logged yet. Then while running a background tasks via the CLI which simply checks the database for any expired invite tokens to be removed the issue occurs. Inside Sentry the exception
So a new connection with PID 604 has been made and is idling right now... so when running the same script again we don't get an error in Sentry anymore since the system is using the same PID and refreshed the idling time as followed:
So for what i can see is that when there is no current active or idling thread in mysql (also if you kill it on purpose) the issue occurs. Of course horizon is passing because of the second try, at that moment there is an active connection been made which is idling). But if you kill it or don't have an connection available it will occur again... I saw some people talking about the framework which should reconnect if that happens but in my opinion we are never getting at that point in the code... or at least it looks like that... the exception has been thrown and is obvious being logged by Sentry and catched by horizon to retry again... so the framework wasn't involved at this moment. Also if this is an wider issue we should see it while connecting to the db in general via the framework but it only happens while queueing jobs and mostly when they are triggered by and CLI command inside Laravel. Maybe this help a little further in finding a solution for this issue... but it looks like that either the framework/horizon or mysql is remembering somehow the PID number from a last alive connection (which has expired in the meantime) or booting mysql connections from queued workers isn't working correctly... Either way i'll hope we can solve this issue some day. |
Hi,
The database queue driver doesn't seem to reconnect to the DB after losing connection.
Although I am able to work around this issue it's a big gotcha for other users of the queue daemon with similar settings.
Normal DB operations already have a tryAgainIfCausedByLostConnection method so the same logic could be used for this case: https://github.com/laravel/framework/blob/5.2/src/Illuminate/Database/Connection.php#L691
System:
Laravel: 5.2.45
PHP: 7.0.10
Mac: OS X 10.11.6
Steps to reproduce:
Stack trace
The text was updated successfully, but these errors were encountered: