Support graceful shutdown TiCDC node #4624
Conversation
Signed-off-by: Neil Shen <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
|
|
overvenus
requested review from
july2993 and
KanShiori
and removed request for
csuzhangxc
Codecov Report
@@ Coverage Diff @@
## master #4624 +/- ##
==========================================
+ Coverage 62.48% 72.39% +9.91%
==========================================
Files 184 188 +4
Lines 20454 23099 +2645
==========================================
+ Hits 12781 16723 +3942
+ Misses 6522 5213 -1309
- Partials 1151 1163 +12
|
Signed-off-by: Neil Shen <[email protected]>
| ns := tc.GetNamespace() | ||
| tcName := tc.GetName() | ||
|
|
||
| if tc.Spec.PD != nil && !tc.PDIsAvailable() { |
There was a problem hiding this comment.
If pd or tikv is not available, the status of cdc won't update, is it acceptable?
There was a problem hiding this comment.
Yes, TiCDC requires PD and TiKV service.
There was a problem hiding this comment.
Please note that with this change, all the operations, e.g. creation, scale, upgrade will be blocked. Is this expected?
There was a problem hiding this comment.
Thanks for reminder, I'll add some comments here.
Signed-off-by: Neil Shen <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
|
/run-pull-e2e-kind |
|
/test pull-e2e-kind pull-e2e-kind-serial |
1 similar comment
|
/test pull-e2e-kind pull-e2e-kind-serial |
| var resp drainCaptureResp | ||
| err = json.Unmarshal(body, &resp) | ||
| if err != nil { | ||
| // It is likely the TiCDC does not support the API, ignore. |
There was a problem hiding this comment.
Is this correct for the unmarshal error?
There was a problem hiding this comment.
Yes, we ignore the error because old ticdc does not support this API.
| @@ -219,7 +219,9 @@ func (c *defaultTidbClusterControl) updateTidbCluster(tc *v1alpha1.TidbCluster) | |||
| return err | |||
| } | |||
|
|
|||
| // works that should be done to make the ticdc cluster current state match the desired state: | |||
There was a problem hiding this comment.
Do you need to sync CDC before TiDB?
There was a problem hiding this comment.
No need, CDC only relies on TiKV and PD.
Signed-off-by: Neil Shen <[email protected]>
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: 8360cb0
|
|
/test pull-e2e-kind pull-e2e-kind-basic pull-e2e-kind-tngm |
|
/test pull-e2e-kind |
3 similar comments
|
/test pull-e2e-kind |
|
/test pull-e2e-kind |
|
/test pull-e2e-kind |
|
/test pull-e2e-kind-serial pull-e2e-kind-basic |
|
@overvenus: Your PR was out of date, I have automatically updated it for you. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
/test pull-e2e-kind-across-kubernetes pull-e2e-kind |
* Support graceful shutdown TiCDC node (#4624) * ticdc: support graceful upgrade TiCDC pods (#4647) * ticdc: wait TiCDC cluster becomes healthy after resigning owner (#4665) Signed-off-by: Neil Shen <[email protected]>
What problem does this PR solve?
Close #4623
What is changed and how does it work?
Before shutdown TiCDC node, we should first resign its ownership and move out its tables.
Code changes
Tests
Scale-in 3 node TiCDC to 1 node, works fine.
Side effects
Related changes
Release Notes
This PR add basic functionality of graceful shutdown, user can not use the feature without further PRs.