[6.x] MySQL Aurora failover - DetectsLostConnections

[IndexZer0]

We recently experienced a MySQL Aurora failover on production and noticed that our queue workers didn't restart.

When running the following code inside a Laravel command and then starting a failover inside RDS dashboard we see these exception messages:

• SQLSTATE[HY000] [2002] Connection refused
• SQLSTATE[HY000]: General error: 1290 The MySQL server is running with the --read-only option so it cannot execute this statement (SQL: update brand_daily_summary set deposit_count = 23 where id = 1)

    public static function testConnection($attemptReconnect = true)
    {
        $end = Carbon::now()->copy()->addSeconds(120);

        $lastSecondRun = null;
        $count = 0;

        while (Carbon::now()->lessThan($end)) {
            $now = Carbon::now();

            if ($lastSecondRun === null || $lastSecondRun !== $now->second) {

                \Log::debug('second: ' . $now->second);

                $count++;

                try {
                    
                    $affected = \DB::table(DailySummary::TABLE)
                        ->where('id', 1)
                        ->update(['deposit_count' => $count]);

                    \Log::debug('affected: ' . $affected);

                } catch (\Exception $e) {
                    \Log::debug('Exception');
                    \Log::debug($e->getMessage());

                    if ($attemptReconnect) {
                        if (Str::contains($e->getMessage(), ['--read-only'])) {
                            \Log::debug('attempting to reconnect');
                            \DB::reconnect();
                        }
                    }
                }

            }

            $lastSecondRun = $now->second;
        }
    }

Maybe protected function causedByLostConnection is not the correct place to be handling the --read-only option exception message though as it is quite general?

Any thoughts on this?

[IndexZer0]

Here is full output of log file while command is running and failover is happening

[2020-02-20 09:50:58] macbook.DEBUG: second: 58 [] []
[2020-02-20 09:50:58] macbook.DEBUG: affected: 1 [] []
[2020-02-20 09:50:59] macbook.DEBUG: second: 59 [] []
[2020-02-20 09:50:59] macbook.DEBUG: affected: 1 [] []
[2020-02-20 09:51:00] macbook.DEBUG: second: 0 [] []
[2020-02-20 09:51:00] macbook.DEBUG: affected: 1 [] []
[2020-02-20 09:51:01] macbook.DEBUG: second: 1 [] []
[2020-02-20 09:51:01] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:01] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:02] macbook.DEBUG: second: 2 [] []
[2020-02-20 09:51:02] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:02] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:03] macbook.DEBUG: second: 3 [] []
[2020-02-20 09:51:03] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:03] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:04] macbook.DEBUG: second: 4 [] []
[2020-02-20 09:51:04] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:04] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:05] macbook.DEBUG: second: 5 [] []
[2020-02-20 09:51:05] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:05] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:06] macbook.DEBUG: second: 6 [] []
[2020-02-20 09:51:06] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:06] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:07] macbook.DEBUG: second: 7 [] []
[2020-02-20 09:51:07] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:07] macbook.DEBUG: SQLSTATE[HY000] [2002] Connection refused [] []
[2020-02-20 09:51:08] macbook.DEBUG: second: 8 [] []
[2020-02-20 09:51:08] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:08] macbook.DEBUG: SQLSTATE[HY000]: General error: 1290 The MySQL server is running with the --read-only option so it cannot execute this statement (SQL: update brand_daily_summary set deposit_count = 23 where id = 1) [] []
[2020-02-20 09:51:08] macbook.DEBUG: attempting to reconnect [] []
[2020-02-20 09:51:09] macbook.DEBUG: second: 9 [] []
[2020-02-20 09:51:09] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:09] macbook.DEBUG: SQLSTATE[HY000]: General error: 1290 The MySQL server is running with the --read-only option so it cannot execute this statement (SQL: update brand_daily_summary set deposit_count = 24 where id = 1) [] []
[2020-02-20 09:51:09] macbook.DEBUG: attempting to reconnect [] []
[2020-02-20 09:51:10] macbook.DEBUG: second: 10 [] []
[2020-02-20 09:51:10] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:10] macbook.DEBUG: SQLSTATE[HY000]: General error: 1290 The MySQL server is running with the --read-only option so it cannot execute this statement (SQL: update brand_daily_summary set deposit_count = 25 where id = 1) [] []
[2020-02-20 09:51:10] macbook.DEBUG: attempting to reconnect [] []
[2020-02-20 09:51:11] macbook.DEBUG: second: 11 [] []
[2020-02-20 09:51:11] macbook.DEBUG: Exception [] []
[2020-02-20 09:51:11] macbook.DEBUG: SQLSTATE[HY000]: General error: 1290 The MySQL server is running with the --read-only option so it cannot execute this statement (SQL: update brand_daily_summary set deposit_count = 26 where id = 1) [] []
[2020-02-20 09:51:11] macbook.DEBUG: attempting to reconnect [] []
[2020-02-20 09:51:12] macbook.DEBUG: second: 12 [] []
[2020-02-20 09:51:12] macbook.DEBUG: affected: 1 [] []
[2020-02-20 09:51:13] macbook.DEBUG: second: 13 [] []
[2020-02-20 09:51:13] macbook.DEBUG: affected: 1 [] []

[Manikandan85]

Is there any updates on this issue?. We are facing on the same one.
Will doing reconnect solves this issue?

[IndexZer0]

@Manikandan85 - This was merged and we have not had any issues with queue workers since. Though I'm not sure we've experienced a failover since.

[Manikandan85]

okay. Thank you @IndexZer0. I will replicate the failover and check whether it fixes it.