Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VReplication Workflow: manage SQL_MODE as MySQL does in mysqldump #9505

Merged
merged 2 commits into from
Jan 16, 2022

Conversation

rohit-nayak-ps
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps commented Jan 12, 2022

Description

In VReplication we are tasked with taking a single logical table from a source and creating a logical copy of that on a target (typically partitioned/sharded in different ways on each side). In order to do this, we need to relax the sql_mode in order to ensure we can do so (the source could have had any sql_mode configured when the data was created).

Given that VReplication logically implements the same functionality in Vitess that mysqldump does in MySQL — generating SQL statements from a source that allow you to recreate database objects on a target — it makes sense for us to try and mimic MySQL (Vitess provides a single logical MySQL database as a keyspace). And what mysqldump does is this:

$ mysqldump -u root mydb dtest 2>/dev/null | head -n 15 ; mysqldump -u root mydb dtest | tail -n 10
-- MySQL dump 10.13  Distrib 8.0.27, for macos11.6 (arm64)
--
-- Host: localhost    Database: mydb
-- ------------------------------------------------------
-- Server version	8.0.27

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!50503 SET NAMES utf8mb4 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

-- Dump completed on 2022-01-13 13:29:28

So it saves the old sql_mode, sets it to 'NO_AUTO_VALUE_ON_ZERO' in the session used to recreate the database objects (CREATE, INSERT), then at the end it sets the sql_mode back to what it was (see source).

We can do the same thing in VReplication. That is what this PR sets out to do.

This then not only deals with the zero date issue, but any other sql_mode related issue. This is a simple approach and, because we're doing the same thing MySQL does (remember that Vitess offers MySQL as a logical database spread across N nodes), we would not need a flag or anything user visible.

Related Issue(s)

#7883
#6561

Checklist

  • Should this PR be backported? NO
  • Tests were added
  • Documentation not needed (it will now simply work as one would expect)

@mattlord mattlord self-requested a review January 12, 2022 21:54
@mattlord mattlord changed the title VReplication Workflow: allow zero dates in the source by modifying sql_mode in the target connection VReplication Workflow: manage SQL_MODE as MySQL does in mysqldump Jan 13, 2022
@mattlord mattlord force-pushed the rn-zero-dates branch 20 times, most recently from ec8483a to d9f821f Compare January 16, 2022 00:49
@mattlord mattlord added this to the v13.0 milestone Jan 16, 2022
@mattlord mattlord marked this pull request as ready for review January 16, 2022 01:36
@mattlord mattlord requested a review from deepthi as a code owner January 16, 2022 01:36
@mattlord mattlord force-pushed the rn-zero-dates branch 3 times, most recently from 2e5c838 to 356b44e Compare January 16, 2022 08:38
Copy link
Contributor

@shlomi-noach shlomi-noach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good! A few changes requested (bugfixes) and a couple suggestions (coding style, change at your discretion), see inline.

@mattlord
Copy link
Contributor

Overall looks good! A few changes requested (bugfixes) and a couple suggestions (coding style, change at your discretion), see inline.

THANK YOU for the great ideas/suggestions! ❤️ I've implemented all of them now.

@mattlord mattlord dismissed shlomi-noach’s stale review January 16, 2022 18:23

Requested changes were made

@mattlord mattlord force-pushed the rn-zero-dates branch 2 times, most recently from ead366a to 0d3a62e Compare January 16, 2022 18:42
rohit-nayak-ps and others added 2 commits January 16, 2022 14:30
We follow the same model that MySQL uses with mysqldump:
  1. Save the original sql_mode in use
  2. Set it to a permissive mode that will allow us to
     recreate any objects from the source
  3. Reset the sql_mode back to the original value

Add/update tests for permissive sql_mode in vrepl unit tests
and e2e tests.

Signed-off-by: Matt Lord <[email protected]>
@mattlord
Copy link
Contributor

mattlord commented Jan 16, 2022

⬆️ (rebased on vitessio/vitess/main to pick up upgrade/downgrade reparent flakiness fixes) ⬆️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants