Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

status: Skip the unavailable node and set default node status. #417 #418

Merged
merged 1 commit into from
Mar 17, 2022
Merged

status: Skip the unavailable node and set default node status. #417 #418

merged 1 commit into from
Mar 17, 2022

Conversation

runkecheng
Copy link
Collaborator

@runkecheng runkecheng commented Mar 17, 2022 •
edited
Loading

What type of PR is this?

/bug

Which issue(s) this PR fixes?

Fixes #417

What this PR does?

Summary:

  • Do not return err when operator cannot connect to node.
  • Provide a default status which is used to unavailable node.

Special notes for your reviewer?

for test:

helm upgrade demo radondb/mysql-operator --install --set manager.image=runkecheng/mysql-operator --set manager.tag=t0317b

@runkecheng runkecheng added the bug Something isn't working label Mar 17, 2022
@runkecheng runkecheng added this to the v2.1.3 milestone Mar 17, 2022
@runkecheng runkecheng self-assigned this Mar 17, 2022
@runkecheng
Copy link
Collaborator Author

runkecheng commented Mar 17, 2022 •
edited
Loading

Close the machine which pod sample-mysql-1 is located.

 Last Transition Time:  2022-03-17T02:25:37Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2022-03-17T03:03:56Z
    Message:               failed to ping host[http://sample-mysql-1.sample-mysql.default:6601/v1/xenon/ping], err: Get "http://sample-mysql-1.sample-mysql.default:6601/v1/xenon/ping": context deadline exceeded
    Status:                True
    Type:                  Error
  Nodes:
    Conditions:
      Last Transition Time:  2022-03-16T06:55:46Z
      Status:                False
      Type:                  Lagged
      Last Transition Time:  2022-03-17T03:04:35Z
      Status:                True
      Type:                  Leader
      Last Transition Time:  2022-03-17T03:04:35Z
      Status:                False
      Type:                  ReadOnly
      Last Transition Time:  2022-03-17T03:04:35Z
      Status:                False
      Type:                  Replicating
    Name:                    sample-mysql-2.sample-mysql.default
    Raft Status:
      Leader:  sample-mysql-2.sample-mysql.default:8801
      Nodes:
        sample-mysql-1.sample-mysql.default:8801
        sample-mysql-0.sample-mysql.default:8801
        sample-mysql-2.sample-mysql.default:8801
      Role:  LEADER
    Conditions:
      Last Transition Time:  2022-03-16T02:16:03Z
      Status:                False
      Type:                  Lagged
      Last Transition Time:  2022-03-16T06:16:39Z
      Status:                False
      Type:                  Leader
      Last Transition Time:  2022-03-16T06:16:39Z
      Status:                True
      Type:                  ReadOnly
      Last Transition Time:  2022-03-17T02:16:47Z
      Status:                True
      Type:                  Replicating
    Name:                    sample-mysql-0.sample-mysql.default
    Raft Status:
      Leader:  sample-mysql-2.sample-mysql.default:8801
      Nodes:
        sample-mysql-1.sample-mysql.default:8801
        sample-mysql-0.sample-mysql.default:8801
        sample-mysql-2.sample-mysql.default:8801
      Role:  FOLLOWER
    Conditions:
      Last Transition Time:  2022-03-17T03:04:35Z
      Status:                Unknown
      Type:                  Lagged
      Last Transition Time:  2022-03-17T03:04:06Z
      Status:                False
      Type:                  Leader
      Last Transition Time:  2022-03-17T03:04:35Z
      Status:                Unknown
      Type:                  ReadOnly
      Last Transition Time:  2022-03-17T03:04:35Z
      Status:                Unknown
      Type:                  Replicating
    Message:                 dial tcp 10.10.234.166:3306: i/o timeout
    Name:                    sample-mysql-1.sample-mysql.default
    Raft Status:
      Leader:   UNKNOWN
      Role:     UNKNOWN

kubectl get po -L healthy,role -w | grep sample
## initial state
sample-mysql-0                         3/3     Running            0          6m48s   yes       FOLLOWER
sample-mysql-1                         3/3     Running            0          6m4s    yes       LEADER
sample-mysql-2                         3/3     Running            0          4m6s    yes       FOLLOWER

## The following is a change process
sample-mysql-1                         3/3     Running            0          7m4s    no        UNKNOWN
sample-mysql-2                         3/3     Running            0          5m11s   yes       LEADER
sample-mysql-1                         3/3     Running            0          7m24s   no        UNKNOWN
sample-mysql-1                         0/3     Error              0          9m54s   no        UNKNOWN
sample-mysql-1                         0/3     Error              0          11m     no        UNKNOWN
sample-mysql-1                         0/3     Error              0          11m     no        UNKNOWN
sample-mysql-1                         0/3     Init:0/2           1          11m     no        UNKNOWN
sample-mysql-1                         0/3     Error              0          11m     no        UNKNOWN
sample-mysql-1                         0/3     Error              0          11m     no        UNKNOWN
sample-mysql-1                         0/3     Running            3          11m     no        UNKNOWN
sample-mysql-1                         1/3     Running            3          11m     no        UNKNOWN
sample-mysql-1                         2/3     Running            3          11m     no        UNKNOWN
sample-mysql-1                         3/3     Running            3          12m     no        UNKNOWN
sample-mysql-1                         3/3     Running            3          12m     no        FOLLOWER
sample-mysql-1                         3/3     Running            3          12m     yes       FOLLOWER

@runkecheng runkecheng force-pushed the bugfix_status_cannot_be_modify branch from 3113321 to c793cfe Compare March 17, 2022 07:47
@acekingke
Copy link
Contributor

LGTM

@andyli029 andyli029 mentioned this pull request Mar 17, 2022
Closed
@runkecheng runkecheng force-pushed the bugfix_status_cannot_be_modify branch from c793cfe to c0ae502 Compare March 17, 2022 08:39
@runkecheng runkecheng force-pushed the bugfix_status_cannot_be_modify branch from c0ae502 to dfe9dfb Compare March 17, 2022 08:41
@andyli029 andyli029 added the P0 High priority label Mar 17, 2022
andyli029
andyli029 reviewed Mar 17, 2022
View reviewed changes
Copy link
Contributor

@andyli029 andyli029 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@andyli029 andyli029 merged commit 9e6d0c4 into radondb:main Mar 17, 2022
@qianfen2021 qianfen2021 added ok to test test ok and removed ok to test test ok labels Mar 22, 2022
zhl003 pushed a commit to zhl003/radondb-mysql-kubernetes that referenced this pull request Aug 17, 2022
@andyli029
Merge pull request radondb#418 from runkecheng/bugfix_status_cannot_b…
2e5ffda 
…e_modify

status: Skip the unavailable node and set default node status. radondb#417
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andyli029 andyli029 andyli029 left review comments

@acekingke acekingke Awaiting requested review from acekingke

Assignees

@runkecheng runkecheng

Labels
bug Something isn't working P0 High priority
Projects
None yet
Milestone
v2.1.3
Development

Successfully merging this pull request may close these issues.

[bug] Node status and label are incorrect when the node`s xenoncli is not available.
4 participants
@runkecheng @acekingke @andyli029 @qianfen2021