Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

syncer: add detect server version for upstream and downstream database #1673

Merged
merged 5 commits into from
May 13, 2021

Conversation

lichunzhu
Copy link
Contributor

What problem does this PR solve?

DM didn't detect and print the versions for the upstream and downstream databases.

What is changed and how it works?

Add detect server version for syncer before we start the subtask.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Start a subtask and check the dm-worker's log:
[2021/05/11 12:12:51.380 +08:00] [INFO] [dumpling.go:292] ["found upstream SQL mode"] [task=testdm] [unit=dump] ["SQL mode"=ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION]
[2021/05/11 12:12:51.380 +08:00] [INFO] [dumpling.go:63] ["create dumpling"] [task=testdm] [unit=dump] [config="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":false,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":true,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"127.0.0.1\",\"Port\":3306,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"\",\"LogFile\":\"\",\"LogFormat\":\"\",\"OutputDirPath\":\"./dumped_data.testdm\",\"StatusAddr\":\"\",\"Snapshot\":\"\",\"Consistency\":\"auto\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\"\",\"CsvDelimiter\":\"\",\"Databases\":null,\"Where\":\"\",\"FileType\":\"\",\"ServerInfo\":{\"ServerType\":0,\"ServerVersion\":null},\"Rows\":200000,\"ReadTimeout\":0,\"TiDBMemQuotaQuery\":0,\"FileSize\":67108864,\"StatementSize\":1000000,\"SessionParams\":{},\"Tables\":null}"]
[2021/05/11 12:12:51.696 +08:00] [INFO] [config.go:599] ["detect server type"] [task=testdm] [unit="binlog replication"] [scope=upstream] [type=MySQL]
[2021/05/11 12:12:51.696 +08:00] [INFO] [config.go:618] ["detect server version"] [task=testdm] [unit="binlog replication"] [scope=upstream] [version=5.7.26-log]
[2021/05/11 12:12:51.697 +08:00] [INFO] [config.go:599] ["detect server type"] [task=testdm] [unit="binlog replication"] [scope=downstream] [type=TiDB]
[2021/05/11 12:12:51.697 +08:00] [INFO] [config.go:618] ["detect server version"] [task=testdm] [unit="binlog replication"] [scope=downstream] [version=4.0.11]
[2021/05/11 12:12:52.082 +08:00] [INFO] [checkpoint.go:701] ["create checkpoint schema"] [task=testdm] [unit="binlog replication"] [component="remote checkpoint"] [statement="CREATE SCHEMA IF NOT EXISTS `dm_meta`"]
[2021/05/11 12:12:52.272 +08:00] [INFO] [checkpoint.go:725] ["create checkpoint table"] [task=testdm] [unit="binlog replication"] [component="remote checkpoint"] [statements="[\"CREATE TABLE IF NOT EXISTS `dm_meta`.`testdm_syncer_checkpoint` (\\n\\t\\t\\tid VARCHAR(32) NOT NULL,\\n\\t\\t\\tcp_schema VARCHAR(128) NOT NULL,\\n\\t\\t\\tcp_table VARCHAR(128) NOT NULL,\\n\\t\\t\\tbinlog_name VARCHAR(128),\\n\\t\\t\\tbinlog_pos INT UNSIGNED,\\n\\t\\t\\tbinlog_gtid TEXT,\\n\\t\\t\\texit_safe_binlog_name VARCHAR(128) DEFAULT '',\\n\\t\\t\\texit_safe_binlog_pos INT UNSIGNED DEFAULT 0,\\n\\t\\t\\texit_safe_binlog_gtid TEXT,\\n\\t\\t\\ttable_info JSON NOT NULL,\\n\\t\\t\\tis_global BOOLEAN,\\n\\t\\t\\tcreate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,\\n\\t\\t\\tupdate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\\n\\t\\t\\tUNIQUE KEY uk_id_schema_table (id, cp_schema, cp_table)\\n\\t\\t)\"]"]
[2021/05/11 12:12:52.274 +08:00] [INFO] [subtask.go:223] ["start to run"] [subtask=testdm] [unit=Dump]
[2021/05/11 12:12:52.279 +08:00] [INFO] [config.go:599] ["detect server type"] [task=testdm] [unit=dump] [type=MySQL]
[2021/05/11 12:12:52.279 +08:00] [INFO] [config.go:618] ["detect server version"] [task=testdm] [unit=dump] [version=5.7.26-log]
[2021/05/11 12:12:52.279 +08:00] [INFO] [dump.go:82] ["begin to run Dump"] [task=testdm] [unit=dump] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":false,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":true,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"127.0.0.1\",\"Port\":3306,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"\",\"LogFile\":\"\",\"LogFormat\":\"\",\"OutputDirPath\":\"./dumped_data.testdm\",\"StatusAddr\":\"\",\"Snapshot\":\"\",\"Consistency\":\"flush\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\"\",\"CsvDelimiter\":\"\",\"Databases\":null,\"Where\":\"\",\"FileType\":\"sql\",\"ServerInfo\":{\"ServerType\":1,\"ServerVersion\":\"5.7.26-log\"},\"Rows\":200000,\"ReadTimeout\":0,\"TiDBMemQuotaQuery\":0,\"FileSize\":67108864,\"StatementSize\":1000000,\"SessionParams\":{},\"Tables\":null}"]

Related changes

  • Need to cherry-pick to the release branch

@ti-chi-bot ti-chi-bot requested review from GMHDBJD and lance6716 May 11, 2021 06:31
@lichunzhu lichunzhu added this to the v2.0.3 milestone May 11, 2021
@lichunzhu lichunzhu added needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 status/PTAL This PR is ready for review. Add this label back after committing new changes labels May 11, 2021
@lichunzhu lichunzhu modified the milestones: v2.0.3, v2.0.4 May 11, 2021
Copy link
Collaborator

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

return
}
dctx := dcontext.NewContext(tctx.Ctx, logger)
export.ParseServerInfo(dctx, versionInfo)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems GetFlavor and (*SourceConfig).AdjustFlavor could be replaced by this function 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these two are different functions and we don't need to change them for now.

@lance6716
Copy link
Collaborator

/lgtm

@ti-chi-bot ti-chi-bot added the status/LGT1 One reviewer already commented LGTM label May 13, 2021
Copy link
Collaborator

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • GMHDBJD
  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels May 13, 2021
@lance6716
Copy link
Collaborator

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 1ca2968

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #1693.

@lichunzhu lichunzhu deleted the syncerDetectServerVersion branch May 14, 2021 03:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 size/S status/can-merge status/LGT2 Two reviewers already commented LGTM, ready for merge status/PTAL This PR is ready for review. Add this label back after committing new changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants