Horizon process hangs after Redis reboot

[bartjuh4]

• Horizon Version: 3.7.1
• Laravel Version: 6.8.0
• PHP Version: 7.3.8
• Redis Driver & Version: phpredis
• Database Driver & Version:

Description:

Application is running on a webserver with separate database server. Horizon is running on the webserver and redis on the database server. When the database server reboots the horizon workers are stopped but the actual artisan horizon process is still running. Manually calling artisan supervisor:terminate doesn't do anything.

After having a closer look in the mechanism of Horizon and how it knows if it's running i notices that it's stored in Redis. But the Redis cache store is empty after a reboot so Horizon doesn't know which process to kill because Redis says no.

Made a small workaround for this with a cronjob command:

$repository = app(\Laravel\Horizon\Contracts\SupervisorRepository::class);

$supervisors = $repository->all();

if (empty($supervisors)) {

      exec('ps aux | grep "[h]orizon" | awk \'{print $2}\' | xargs kill -9 $1');

      Log::info('Horizon forced restart because of frozen or no inactive processes.');

}

This runs every 5 minutes and saved our lives many times after that.

Steps To Reproduce:

This part is the hardest thing, because i could not reproduce this on my development environment. When i kill Redis the horizon process get's terminated. Looks like something with a server which is running out of resources and is giving long response times when fetching data from Redis and then reboots.

When this issue occurs output of ps -aux is:
Feb18 0:28 /usr/local/bin/php artisan horizon

where you actually want this:

Feb18   0:28  /usr/local/bin/php artisan horizon
Feb18   0:26 /usr/local/php-7.3/bin/php artisan horizon:supervisor web01-baNb:supervisor-1 redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --balance=auto --max-processes=1 --min-processes=1 --nice=0
Feb18   0:11 /usr/local/php-7.3/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=web01-baNb:supervisor-1

Maybe this is fixable by adding a check in the main horizon process to verify if redis has the correct horizon keys stored. If not exit the process. In that case the supervisord will start a new instance.

When i have some free time i will look in to this and will try to make a PR to fix this in Horizon instead of my cronjob workaround.

[themsaid]

If the redis connection is down, we will re-attempt the connection until it's back up and Horizon will start working again. This was reported in #716 and was fixed in laravel/framework#30778.

As you said, it's not possible to replicate the issue which means something isn't right on your servers.

[Perh0rd]

As a general recommandation , redis should run alone in a server , and have dedicated ressources. Running Redis on a database server which consume lot of memory and io could result into strange behaviour.

We used to run redis along others services , and we had too many problems. Now we are using standalone server we never encountered any problem.

If you are in a scenario of high intensity of usage you should consider using redis along kubernetes and docker across multiples instances.

[driesvints]

@bartjuh4 can you try to upgrade to the latest 6.x version and see if the problem persists?

[mfn]

But the Redis cache store is empty after a reboot

Why is that so? Do you not persist your Redis server data?

[driesvints]

Closing this issue because it's inactive, already solved, old or not relevant anymore. Feel free to reply if you're still experiencing this issue and we'll re-open this issue.