cdc: integrate two phase scheduler #5710

overvenus · 2022-06-01T12:39:19Z

What problem does this PR solve?

Issue Number: ref #4757

What is changed and how it works?

Integrate two-phase scheduler and fix two bug

Fix duplicate topic handlers panic
Add From to messages from scheduler
Fix missing processor epoch
Fix missing checkpoint ts in burstBalanceTask
Fix lost AddTable request during Commit

Check List

Tests

Unit test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

ti-chi-bot · 2022-06-01T12:39:21Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3AceShowHand

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: Neil Shen <[email protected]>

codecov-commenter · 2022-06-02T04:18:04Z

Codecov Report

❗ No coverage uploaded for pull request base (fb/latency@0b1c089). Click here to learn what that means.
The diff coverage is n/a.

Flag	Coverage Δ
cdc	`62.4713% <0.0000%> (?)`
dm	`52.0374% <0.0000%> (?)`
engine	`58.7516% <0.0000%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff               @@
##             fb/latency      #5710   +/-   ##
===============================================
  Coverage              ?   56.9524%           
===============================================
  Files                 ?        684           
  Lines                 ?      80425           
  Branches              ?          0           
===============================================
  Hits                  ?      45804           
  Misses                ?      30379           
  Partials              ?       4242

Signed-off-by: Neil Shen <[email protected]>

This reverts commit ebb0f2c.

* cdc: integrate two phase scheduler Signed-off-by: Neil Shen <[email protected]> * tp: fix duplicate topic handlers panic Signed-off-by: Neil Shen <[email protected]> * tp: add From to messages from scheduler Signed-off-by: Neil Shen <[email protected]> * tp: add more comments and fix typo Signed-off-by: Neil Shen <[email protected]> * cdc: fix missing table ID in table meta Signed-off-by: Neil Shen <[email protected]> * cdc: fix processor.tick panic Signed-off-by: Neil Shen <[email protected]> * tp: fix missing processor epoch Signed-off-by: Neil Shen <[email protected]> * tp: fix missing checkpoint ts in burstBalanceTask Signed-off-by: Neil Shen <[email protected]> * tp: fix lost AddTable request during Commit Signed-off-by: Neil Shen <[email protected]> * Revert "cdc: fix processor.tick panic" This reverts commit ebb0f2c.

overvenus added component/scheduler TiCDC inner scheduler component. area/ticdc Issues or PRs related to TiCDC. labels Jun 1, 2022

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 1, 2022

overvenus changed the title ~~Fb/latency testing~~ cdc: integrate two phase scheduler Jun 1, 2022

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 1, 2022

overvenus added 4 commits June 2, 2022 10:58

cdc: integrate two phase scheduler

e6fd2a8

Signed-off-by: Neil Shen <[email protected]>

tp: fix duplicate topic handlers panic

182ca9e

Signed-off-by: Neil Shen <[email protected]>

tp: add From to messages from scheduler

b2f44a6

Signed-off-by: Neil Shen <[email protected]>

tp: add more comments and fix typo

66096f6

Signed-off-by: Neil Shen <[email protected]>

overvenus force-pushed the fb/latency-testing branch from 7dbdb99 to 66096f6 Compare June 2, 2022 03:37

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 2, 2022

overvenus marked this pull request as ready for review June 2, 2022 03:37

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 2, 2022

overvenus requested a review from 3AceShowHand June 2, 2022 03:37

overvenus added 3 commits June 2, 2022 13:38

cdc: fix missing table ID in table meta

a079084

Signed-off-by: Neil Shen <[email protected]>

cdc: fix processor.tick panic

ebb0f2c

Signed-off-by: Neil Shen <[email protected]>

tp: fix missing processor epoch

c225353

Signed-off-by: Neil Shen <[email protected]>

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 2, 2022

overvenus added 3 commits June 2, 2022 15:45

tp: fix missing checkpoint ts in burstBalanceTask

7e5bef6

Signed-off-by: Neil Shen <[email protected]>

tp: fix lost AddTable request during Commit

569b745

Signed-off-by: Neil Shen <[email protected]>

Revert "cdc: fix processor.tick panic"

2df75bc

This reverts commit ebb0f2c.

3AceShowHand approved these changes Jun 2, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 2, 2022

3AceShowHand merged commit 39dc96f into pingcap:fb/latency Jun 2, 2022

overvenus mentioned this pull request Jun 6, 2022

Tracking issue for reducing high latency #4757

Closed

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cdc: integrate two phase scheduler #5710

cdc: integrate two phase scheduler #5710

overvenus commented Jun 1, 2022 •

edited

Loading

ti-chi-bot commented Jun 1, 2022 •

edited

Loading

codecov-commenter commented Jun 2, 2022

cdc: integrate two phase scheduler #5710

cdc: integrate two phase scheduler #5710

Conversation

overvenus commented Jun 1, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

ti-chi-bot commented Jun 1, 2022 • edited Loading

codecov-commenter commented Jun 2, 2022

Codecov Report

overvenus commented Jun 1, 2022 •

edited

Loading

ti-chi-bot commented Jun 1, 2022 •

edited

Loading