Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

load: load should not need metadata #1321

Merged
merged 5 commits into from
Dec 10, 2020
Merged

load: load should not need metadata #1321

merged 5 commits into from
Dec 10, 2020

Conversation

lance6716
Copy link
Collaborator

@lance6716 lance6716 commented Dec 2, 2020 •
edited
Loading

What problem does this PR solve?

part of #1320

What is changed and how it works?

since full mode doesn't not need REPLICATION related privilege and doesn't check it, mydumper/dumpling won't generate a valid metdata file containing "SHOW MASTER STATUS:" at then. And when loader parse it, load task will stop.

Now load should ignore that error

Check List

Tests

  • pass original test

Code changes

Side effects

Related changes

  • Need to cherry-pick to the release branch

@lance6716 lance6716 added the status/DNM Do not merge, test is failing or blocked by another PR label Dec 2, 2020
@lance6716
Copy link
Collaborator Author

Why didn't relay unit start sync from metadata's position, thus we need not unitTransWaitCondition 🤔

@GMHDBJD
Copy link
Collaborator

GMHDBJD commented Dec 3, 2020

Why didn't relay unit start sync from metadata's position, thus we need not unitTransWaitCondition 🤔

I think relay should be source level but not task level. So it may start from now, or start from pos/gtid which we specify

@lance6716 lance6716 changed the title pb, load: load should not need metadata load: load should not need metadata Dec 3, 2020
@lance6716 lance6716 added needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 priority/normal Minor change, requires approval from ≥1 primary reviewer status/PTAL This PR is ready for review. Add this label back after committing new changes type/bug-fix Bug fix and removed status/DNM Do not merge, test is failing or blocked by another PR labels Dec 3, 2020
GMHDBJD
GMHDBJD reviewed Dec 4, 2020
View reviewed changes
loader/loader.go
@@ -1307,6 +1307,11 @@ func (l *Loader) getMydumpMetadata() error {
metafile := filepath.Join(l.cfg.LoaderConfig.Dir, "metadata")
loc, _, err := dumpling.ParseMetaData(metafile, l.cfg.Flavor)
if err != nil {
if terror.ErrMetadataNoBinlogLoc.Equal(err) {
l.logger.Warn("dumped metadata doesn't have binlog location, it's OK if DM doesn't enter incremental mode")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about check task_mode here?
It is annoying to report errors after a long time import is completed

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're checking REPLICATION CLIENT in pre-check wrongly, (should make sure it's grants on *.*). So for all mode and incremental mode, pre-check could make sure there's enough privilege generating correct metadata. For full mode, it's no harm

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any other cases that may cause this error? 🤔

GMHDBJD
GMHDBJD reviewed Dec 9, 2020
View reviewed changes
Copy link
Collaborator

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@GMHDBJD GMHDBJD added status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Dec 9, 2020
csuzhangxc
csuzhangxc approved these changes Dec 9, 2020
View reviewed changes
Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csuzhangxc csuzhangxc added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Dec 9, 2020
@csuzhangxc csuzhangxc added this to the v2.0.1 milestone Dec 9, 2020
@lance6716 lance6716 merged commit 7e79861 into pingcap:master Dec 10, 2020
ti-srebot pushed a commit to ti-srebot/dm that referenced this pull request Dec 10, 2020
@lance6716 @ti-srebot
cherry pick pingcap#1321 to release-2.0
ca435fd 
Signed-off-by: ti-srebot <[email protected]>
@ti-srebot ti-srebot mentioned this pull request Dec 10, 2020
Merged
@ti-srebot
Copy link

cherry pick to release-2.0 in PR #1326

@ti-srebot ti-srebot added already-cherry-pick-2.0 The related PR is already cherry-picked to release-2.0. Add this label once the PR is cherry-picked and removed needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 labels Dec 10, 2020
csuzhangxc pushed a commit that referenced this pull request Dec 22, 2020
@ti-srebot @lance6716
load: load should not need metadata (#1321) (#1326)
a864d71 
Signed-off-by: ti-srebot <[email protected]>

Co-authored-by: lance6716 <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@GMHDBJD GMHDBJD GMHDBJD left review comments

@csuzhangxc csuzhangxc csuzhangxc approved these changes

Assignees
No one assigned
Labels
already-cherry-pick-2.0 The related PR is already cherry-picked to release-2.0. Add this label once the PR is cherry-picked priority/normal Minor change, requires approval from ≥1 primary reviewer status/LGT2 Two reviewers already commented LGTM, ready for merge type/bug-fix Bug fix
Projects
None yet
Milestone
v2.0.1
Development

Successfully merging this pull request may close these issues.
4 participants
@lance6716 @GMHDBJD @ti-srebot @csuzhangxc