[feature] Reconstruct the status controller

[runkecheng]

Proposal: reconstruct the status controller

Summary

Today, Operator checks the health of mysql by remote connection to PODs.

There are some pain points in this solution:

1. Need a super user that allows remote access

2. When accessing the PODs through DNS, the latest podip may not be obtained in time (when the POD restarts, IP changes, DNS has cache)

3. When operator fails, the labels(role,healthy) of the PODs cannot be updated

Motivation

Goals

Give PODs a certain autonomy: Let PODs manage their own labels

Non-Goals

The following are non-goals for the current phase, but will be considered at a later phase.

The selection of the leader requires Operator to make judgments，guarantee that there are no multiple leader when the network partition between PODs

Proposal

Move the health checking and the role checking to the PODs.

Implementation

Implement a mysqlchecker, add it to the Percona image.

type MySQLChecker interface {

    CheckMySQLReadiness(role string) bool

    CheckMySQLLiveness() bool
    // k8s client, its used to get/update labels
    Client

}

The CheckMySQLReadiness()'s logic is the logic of updateEnodeStatus() in the current staus controller.

The rules of a healthy node

role/options	readonly	lagged	replicating
leader	no	-	no
follower	yes	no	yes
others	yes	-	-

The addtional step for leader

Turn off readonly

SET GLOBAL read_only=off;
SET GLOBAL super_read_only=off;

Role label

Implement a readiness probe of xenon, it only:

• Get its own role(xenoncli raft status)
• Patch label to it own pod

Rebuild

The logic of automatic rebuild can be migrated to readiness probe of the MySQL container, or keep in Operator

Future

...