-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initialize Tablet with super_read_only mode #12206
Initialize Tablet with super_read_only mode #12206
Conversation
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
…ead-only_final Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes in vtexplain
are hard to understand, and seem unrelated to the rest of the changes.
Signed-off-by: Rameez Sajwani <[email protected]>
…ead-only_final Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
…ead-only_final Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
@@ -112,7 +119,11 @@ func TestMain(m *testing.M) { | |||
tablet.VttabletProcess.ExtraArgs = commonTabletArg | |||
tablet.VttabletProcess.SupportsBackup = true | |||
|
|||
tablet.MysqlctlProcess = *cluster.MysqlCtlProcessInstance(tablet.TabletUID, tablet.MySQLPort, localCluster.TmpDirectory) | |||
mysqlctlProcess, err := cluster.MysqlCtlProcessInstance(tablet.TabletUID, tablet.MySQLPort, localCluster.TmpDirectory) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is due to refactoring done in go/test/endtoend/cluster/mysqlctl_process.go
"vt_app": [ | ||
"password" | ||
"VtAppPass" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changing these passwords so I can reuse GetPasswordUpdateSQL. This will make tests consistent as well ...
Signed-off-by: Rameez Sajwani <[email protected]>
…ead-only_final Signed-off-by: Rameez Sajwani <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost there 😄
Approving provisionally, the remaining changes are pretty small, but should be addressed before we merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work on this, @rsajwani ! I only had minor nits/comments/suggestions. We can discuss those and quickly get this thing merged already! 🙂
Signed-off-by: Rameez Sajwani <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! ❤️ I only had one insignificant nit. Up to you if you think it's worth changing.
@@ -57,6 +57,8 @@ func TestTabletInitialBackup(t *testing.T) { | |||
// - list the backups, remove them | |||
defer cluster.PanicHandler(t) | |||
|
|||
waitForReplicationToCatchup([]cluster.Vttablet{*replica1, *replica2}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a reminder that if vtorc is used in these tests, which tablets are replicas is no longer fixed. That's a bigger issue though, not related to this PR.
var timeoutDuration = time.Duration(5 * len(sqls)) | ||
ctx, cancel := context.WithTimeout(context.Background(), timeoutDuration*time.Second) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Non-blocking nit, but IMO it's more natural to do this all in the duration definition:
var timeoutDuration = time.Duration(5 * time.Second * len(sqls))
ctx, cancel := context.WithTimeout(context.Background(), timeoutDuration)
Description
As of today MySQL server which is controlled by VTTablet, starts in read-only mode. This is because Vitess controls the MySQL cluster topology and chooses one tablet to be the primary for each shard, and only that one MySQL server should be read-write. All other servers in that shard are replicas, and should not be taking any writes. However this doesn't prevent users like
root
andvt_dba
who haveSUPER
privileges from writing to the database on replicas and causing errant GTIDs. We want to leverage GLOBALsuper_read_only
configuration in order to protect against these scenarios. This will make sure that apart from primary no other component or offline system can mutate DB resulting in errant GTIDs that are then lying in wait to cause later failures, as you can see in #9312 and #10094Furthermore, it is possible that due to program bugs or unexpected failures during PRS/ERS, we end up with a replica that is not in read-only mode. With super-read-only change we will make sure that we cover those cases as well.
Tablet schema refactoring
This change is built on top of #11520, where instead of using
withddl
we use a declarative approach to initializing vttablet's sidecar schema. Using a declarative approach helps us condense all our schema changes to one place, which allows us to make changes to the code to applysuper-read-only
to the DB.The previous approach of using
withddl
meant that schema changes were scattered all across the code, which made it extremely difficult to do thesuper-read-only
change.Details
This PR usese #11520 as a building block. I am listing major changes below to help you review the PR.
OFF
temporarily in order to perform some mutations like creating necessary users and permissions.# {{custom_sql}}
inside init_db.sql file. This is because in our tests we modify this file and add custom SQL like (change password of DBA user etc). Now that we expect setting super-read-onlyON
to execute at the end, we need these custom sqls to be added before that last line. See example: go/test/endtoend/backup/vtbackup/main_test.goFor Reference
Some related work items done in the past
#11706
#10094
#9312
#10448
Unresolved Issues during code changes:
I have filed few issues which I will need to resolve once I check-in this PR.
Related Issue(s)
Fixes #10363
Fixes #12180
Fixes #12140
Test Results
vtctld command to show super_read_only state
Checklist
Deployment Notes