Schema Tracking Refactor: Merge schema-tracking in health-streamer into schema.Engine #13121

GuptaManan100 · 2023-05-19T17:35:31Z

Description

This PR refactors schema tracking to merge the schema tracking in healthstreamer into schema.Engine.

We still have the schema tracking in healthstreamer around because of backward compatibility, which we will remove in a later release. However, we have removed the ticker on which we would run the reload function and instead do it in response to a signal from the schema.Engine.

The schema tracking introduced in schema.Engine stores a little more information than what we had in healthstreamer to support the upcoming use case of #12967.

This PR also fixes the issue described in #13182. Now, we all the health streamer to write to the database only when the tablet is in a primary serving state. So, when the call to tm.QueryServiceControl.SetServingType succeeds, we can now guarantee that the health streamer will not be writing anything anymore.

Website docs - vitessio/website#1487

Related Issue(s)

Checklist

"Backport to:" labels have been added if this change should be back-ported
Tests were added or are not required
Did the new or modified tests pass consistently locally and on the CI
Documentation was added or is not required - SchemaTracker refactor documentation website#1487

Deployment Notes

Signed-off-by: Manan Gupta <[email protected]>

…in vttablets Signed-off-by: Manan Gupta <[email protected]>

…amer Signed-off-by: Manan Gupta <[email protected]>

…reload Signed-off-by: Manan Gupta <[email protected]>

…s as well Signed-off-by: Manan Gupta <[email protected]>

…it correctly Signed-off-by: Manan Gupta <[email protected]>

…ine instead of detecting the schema changes on its own Signed-off-by: Manan Gupta <[email protected]>

vitess-bot · 2023-05-19T17:35:34Z

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

Ensure that the Pull Request has a descriptive title.
If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

Is it really necessary to add this flag?
Flag names should be clear and intuitive (as far as possible)
Help text should be descriptive.
Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

Each item in Jobs should be named in order to mark it as required.
If the workflow should be required, the maintainer team should be notified.

Bug fixes

There should be at least one unit or end-to-end test.
The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

There should be some code comments as to why things are implemented the way they are.

New/Existing features

Should be documented, either by modifying the existing documentation or creating new documentation.
New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

Protobuf changes should be wire-compatible.
Changes to _vt tables and RPCs need to be backward compatible.
vtctl command output order should be stable and awk-able.
RPC changes should be compatible with vitess-operator
If a flag is removed, then it should also be removed from VTop, if used there.

…formation in a database Signed-off-by: Manan Gupta <[email protected]>

Signed-off-by: Manan Gupta <[email protected]>

go/vt/vttablet/tabletserver/tabletserver.go

go/vt/vttablet/tabletserver/tabletenv/config.go

Signed-off-by: Manan Gupta <[email protected]>

…at are removed Signed-off-by: Manan Gupta <[email protected]>

harshit-gangal · 2023-05-23T11:01:03Z

go/test/endtoend/cluster/cluster_process.go

@@ -929,6 +929,46 @@ func (cluster *LocalProcessCluster) StreamTabletHealth(ctx context.Context, vtta
 	return responses, nil
 }

+// StreamTabletHealthUntil invokes a HealthStream on a local cluster Vttablet and
+// returns the responses. It waits until a certain condition is met. The amount of time to wait is an input that it takes.
+func (cluster *LocalProcessCluster) StreamTabletHealthUntil(ctx context.Context, vttablet *Vttablet, timeout time.Duration, condition func(shr *querypb.StreamHealthResponse) bool) error {


there are existing usages of StreamTabletHealth can those also use this utility function?

There is an inherent difference between the two. StreamTabletHealth gets you specified number of stream responses back, which can then be processed. On the other hand StreamTabletHealthUntil is meant to be used where the test doesn't care if we consume more responses and only verifies if a certain check is ever true.
The tests currently using StreamTabletHealth actually can't really use StreamTabletHealthUntil because they want to run a test on the next packet, and not an eventual packet.

harshit-gangal · 2023-05-23T11:01:59Z

go/test/endtoend/cluster/vtctldclient_process.go

+func (vtctldclient *VtctldClientProcess) PlannedReparentShard(Keyspace string, Shard string, alias string) (err error) {
+	output, err := vtctldclient.ExecuteCommandWithOutput(
+		"PlannedReparentShard",
+		fmt.Sprintf("%s/%s", Keyspace, Shard),
+		"--new-primary", alias)
+	if err != nil {
+		log.Errorf("error in PlannedReparentShard output %s, err %s", output, err.Error())


there are other places as well we are doing this call, like InitializeShard can it be reused

We could, but eventually we want to move to using vtctldclient and not vtctlclient. So I thought it prudent to add a PRS call to vtctldclient. This is what we should be using going forward in all our tests.

go/test/endtoend/tabletmanager/tablet_health_test.go

go/vt/vttablet/tabletserver/health_streamer.go

go/vt/vttablet/tabletserver/schema/db_test.go

go/vt/vttablet/tabletserver/state_manager.go

go/vt/vttablet/tabletserver/schema/engine.go

Signed-off-by: Manan Gupta <[email protected]>

go/vt/vttablet/tabletserver/schema/engine_test.go

Signed-off-by: Manan Gupta <[email protected]>

harshit-gangal

looks good overall. this is a significant improvement on the schema engine and more cleaner code and also fixes the issue with view change tracking.

changelog/17.0/17.0.0/summary.md

Signed-off-by: Manan Gupta <[email protected]>

GuptaManan100 added 7 commits May 18, 2023 15:27

test: add test to verify the expectations of health stream

4e75fe9

Signed-off-by: Manan Gupta <[email protected]>

test: augment the test to also test the case where views are enabled …

122c5e6

…in vttablets Signed-off-by: Manan Gupta <[email protected]>

feat: use the schema engine notifier to run the reload in health-stre…

714a975

…amer Signed-off-by: Manan Gupta <[email protected]>

feat: deprecate the flag controlling the interval of health-streamer …

36758f2

…reload Signed-off-by: Manan Gupta <[email protected]>

test: augment test to verify schema tracking works for alters in view…

9923f76

…s as well Signed-off-by: Manan Gupta <[email protected]>

feat: add a view table type and augment loadTable function to assign …

a950d15

…it correctly Signed-off-by: Manan Gupta <[email protected]>

feat: change health streamer to use the tables got from the schemaEng…

69c3554

…ine instead of detecting the schema changes on its own Signed-off-by: Manan Gupta <[email protected]>

vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels May 19, 2023

github-actions bot added this to the v17.0.0 milestone May 19, 2023

GuptaManan100 added 2 commits May 22, 2023 20:13

feat: make the schema engine capable of storing the table and view in…

9a8202e

…formation in a database Signed-off-by: Manan Gupta <[email protected]>

feat: remove unused functions

b4a7f95

Signed-off-by: Manan Gupta <[email protected]>