schemadiff: validate views' referenced columns #12147

shlomi-noach · 2023-01-25T09:52:59Z

Description

Up till now, schemadiff views validation and analysis was limited to:

Obviously validate statement
Infer tables (or views) reverenced by a view
Validate those tables/views exist
Validate no dependency loop

As of this PR, schemadiff also validates columns used and returned by views:

Validate that columns in SELECT and in WHERE clause do in fact exist in tables/views referenced by the view. Error when an unknown column is encountered.
Allow unqualified table name where there is no ambiguity. Error on unqualified name with ambiguity.
Analyze view's column names, whether aliased or not.
Support cascaded views.
Support star expression, either qualified (create view v as select t.* from t, t2 ...) or unqualified (create view v as select * from t1,t2) ; infer list of columns ; support cascaded views based on star expression views.

Plenty tests added to validate behavior.

Joint with @dbussink and based on his preliminary work.

Related Issue(s)

Tracking issue: schemadiff: tracking issue #10203

Checklist

"Backport to:" labels have been added if this change should be back-ported
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-01-25T10:01:37Z

In a followup PR: optimize such that we only analyze columns tables/views that are referenced by some view in the first place; there is no need to analyze e.g. a table that is never read by a view. This will allow us to save some memory & cpu.

dbussink · 2023-01-25T10:36:50Z

go/vt/schemadiff/diff_test.go

@@ -667,6 +667,7 @@ func TestDiffSchemas(t *testing.T) {
 					// validate schema1 unaffected by Apply
 					assert.Equal(t, schema1SQL, schema1.ToSQL())

+					require.NotNil(t, schema2)


Is this check useful? We already check earlier that there's no error returned when schema2 is created?

dbussink · 2023-01-25T10:38:52Z

go/vt/schemadiff/schema.go

@@ -750,3 +759,258 @@ func (s *Schema) Apply(diffs []EntityDiff) (*Schema, error) {
 	}
 	return dup, nil
 }
+
+// TODO


What does this TODO reference to?

hehe, it references the fact that is is already done... removing...

dbussink · 2023-01-25T10:39:23Z

go/vt/schemadiff/schema.go

+// TODO
+
+func (s *Schema) ValidateViewReferences() error {
+	var allerrors error


Usually use errs as a the name for this kinda thing to collect errors.

dbussink · 2023-01-25T10:39:52Z

go/vt/schemadiff/schema.go

+	for _, e := range s.Entities() {
+		entityColumns, err := s.getEntityColumnNames(e.Name(), availableColumns)
+		if err != nil {
+			return err


Should this add to the error list instead? And then we continue through the rest of the entities? If this is a totally unexpected error, maybe good to add a comment about that and why it's safe to return immediately?

go/vt/schemadiff/schema.go

Signed-off-by: Shlomi Noach <[email protected]>

go/vt/schemadiff/schema.go

dbussink · 2023-01-25T11:00:10Z

go.mod

@@ -113,6 +113,7 @@ require (
 	github.com/kr/text v0.2.0
 	github.com/nsf/jsondiff v0.0.0-20210926074059-1e845ec5d249
 	github.com/openark/golib v0.0.0-20210531070646-355f37940af8
+	go.uber.org/multierr v1.8.0


Let's also update to 1.9.0 here since that's the latest?

Signed-off-by: Shlomi Noach <[email protected]>

systay · 2023-01-26T11:53:01Z

go/vt/schemadiff/schema.go

+		}
+		return true, nil
+	}, view.Select)
+	if err != nil {


FYI - sqlparser.Walk will only return the error you return from inside the visitor function, and since you don't return any errors from that, no errors will ever make it out.

@systay ah, great! Thank you. I'll keep the check as it is, for safety, but good to know!

@systay Right, but the errors are gathered here in errs? And those are at the end returned if the walker itself doesn't error (which is not expected).

@dbussink yeah, that was my point. Not really necessary to catch the returned error from sqlparser.Walk since that is not how we are dealing with the errors. When I know it will never return an error, I usually write.

_ = sqlparser.Walk(...)

OTOH - it's probably good defensive programming to do as @shlomi-noach is doing here and catching and checking the error anyway.

@systay Right, guess as someone now knowing the details too much about how Walk is implemented, I wouldn't know that it can't return an error and would still write it defensively 😄.

shlomi-noach · 2023-01-29T06:20:33Z

Looking for a 2nd review/approval

mattlord

I think we can/should avoid importing the external multierr package but it otherwise LGTM.

Not adding a formal approval as we should, IMO, get @systay or @harshit-gangal to review as they are in the query serving team and working on the overall Vitess View support work. This way we can ensure that we're moving forward together and all the pieces will fit together nicely.

@systay or @harshit-gangal , could you please review? Please note if you cannot and I can add a formal approval here if needed. Thanks!

mattlord · 2023-01-29T16:03:39Z

go/vt/schemadiff/schema.go

+	for _, e := range s.Entities() {
+		entityColumns, err := s.getEntityColumnNames(e.Name(), availableColumns)
+		if err != nil {
+			errs = multierr.Append(errs, err)


Why not use the Vitess concurrency package here along with vterrors.Aggregate? You can see that used throughout the code base if you look for .AggrError(vterrors.Aggregate). Between the concurrency and vterrors packages we should have whatever related functionality you need here for error handling. For example:

vitess/go/vt/wrangler/traffic_switcher.go

Lines 146 to 161 in 390ddad

func (ts *trafficSwitcher) ForAllSources(f func(source *workflow.MigrationSource) error) error {

var wg sync.WaitGroup

allErrors := &concurrency.AllErrorRecorder{}

for _, source := range ts.sources {

wg.Add(1)

go func(source *workflow.MigrationSource) {

defer wg.Done()

if err := f(source); err != nil {

allErrors.RecordError(err)

}

}(source)

}

wg.Wait()

return allErrors.AggrError(vterrors.Aggregate)

}

It looks like it does use locking which is unnecessary here. Dunno if we care about the overhead of that here? That logic seems designed for concurrent error gathering which isn't what we're doing here.

Using multierr seems simpler here and it's already an indirect dependency? Not a really strong opinion though, we can also use this but it seems a bit off for what it was designed for.

I think this logic here is temporary anyway, since once golang/go#53435 is available with Go 1.20 we probably want to switch to that anyway.

cc @dbussink

heh, race condition

How strongly do people feel about this? Looking at the two implementation multierr does seem to be more fitting to our purpose, but I don't feel strongly.

If you only need multiple errors you can use vterrors.Aggregate(), no need for the concurrency piece (as noted, geared toward N goroutines) like we do here e.g.:

vitess/go/vt/vttablet/tabletmanager/vreplication/vcopier.go

Lines 597 to 621 in 3f55e40

var terrs []error

for !empty {

select {

case result := <-resultCh:

switch result.state {

case vcopierCopyTaskCancel:

// A task cancelation probably indicates an expired context due

// to a PlannedReparentShard or elapsed copy phase duration,

// neither of which are error conditions.

case vcopierCopyTaskComplete:

// Get the latest lastpk, purely for logging purposes.

lastpk = result.args.lastpk

case vcopierCopyTaskFail:

// Aggregate non-nil errors.

terrs = append(terrs, result.err)

}

default:

empty = true

}

}

if len(terrs) > 0 {

terr := vterrors.Aggregate(terrs)

log.Warningf("task error in workflow %s: %v", vc.vr.WorkflowName, terr)

return fmt.Errorf("task error: %v", terr)

}

I don't feel strongly about it though. Up to you.

go/vt/schemadiff/errors.go

go.mod

Signed-off-by: Shlomi Noach <[email protected]>

…t views reading from DUAL Signed-off-by: Shlomi Noach <[email protected]>

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-01-31T04:42:26Z

Incorporated #12189 into this PR, as it deals with view analysis

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-02-06T06:44:52Z

I've switched away from multierr and into AllErrorsRecorder. I want to say multierr seems to have the better design over AllErrorsRecorder, and even over the new Go 1.20 error wrapping support.

shlomi-noach · 2023-02-06T06:45:12Z

Looking for another review/approval 🙏

shlomi-noach · 2023-02-08T06:22:43Z

ping @vitessio/query-serving for review

shlomi-noach · 2023-02-13T11:31:41Z

ping https://github.com/orgs/vitessio/teams/query-serving for review

GuptaManan100

Everything else LGTM!

GuptaManan100 · 2023-02-14T07:57:13Z

go.mod

@@ -187,7 +187,7 @@ require (
 	github.com/tidwall/pretty v1.2.0 // indirect
 	go.opencensus.io v0.24.0 // indirect
 	go.uber.org/atomic v1.10.0 // indirect
-	go.uber.org/multierr v1.9.0 // indirect


Is there a specific reason for this downgrade?

It was upgraded in a previous PR, that was eventually not using multierr. I just ran go mod tidy and it auto-downgraded, so... I don't know?

@shlomi-noach might be because the direct dependency was dropped? A go get -u go.uber.org/multierr probably would add this back correctly (it should be there, so good catch @GuptaManan100).

This is fixed.

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-02-20T15:10:39Z

Stalling this PR a bit as I'm to look into the go/vt/vtgate/semantics implementation of view dependency management.

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-03-08T05:45:22Z

Replaced by #12565 ; will close shortly.

…12551) * WIP: schemadiff, analyzing diff dependencies Signed-off-by: Shlomi Noach <[email protected]> * ongoing work; many tests added Signed-off-by: Shlomi Noach <[email protected]> * partitioning tests Signed-off-by: Shlomi Noach <[email protected]> * function refactor Signed-off-by: Shlomi Noach <[email protected]> * minor reordering of variables Signed-off-by: Shlomi Noach <[email protected]> * copyright year Signed-off-by: Shlomi Noach <[email protected]> * stages to be commited Signed-off-by: Shlomi Noach <[email protected]> * math: permutations Signed-off-by: Shlomi Noach <[email protected]> * mathutil Signed-off-by: Shlomi Noach <[email protected]> * sort table alter options on diff Signed-off-by: Shlomi Noach <[email protected]> * alter reordering: first drop foreign key, then drop key Signed-off-by: Shlomi Noach <[email protected]> * remove tags logic from EquivalenceRelation Signed-off-by: Shlomi Noach <[email protected]> * removed go/mathutil/permutations.go Signed-off-by: Shlomi Noach <[email protected]> * refactor/rename Signed-off-by: Shlomi Noach <[email protected]> * cleanup unused code Signed-off-by: Shlomi Noach <[email protected]> * OrderedDiffs Signed-off-by: Shlomi Noach <[email protected]> * added an impossible-order test Signed-off-by: Shlomi Noach <[email protected]> * clarify comment Signed-off-by: Shlomi Noach <[email protected]> * AllSequentialExecutionDeps() Signed-off-by: Shlomi Noach <[email protected]> * test: expected order of entities in ordered diffs Signed-off-by: Shlomi Noach <[email protected]> * function comments Signed-off-by: Shlomi Noach <[email protected]> * remove unused code Signed-off-by: Shlomi Noach <[email protected]> * code comments; privatize functions; remove unused code Signed-off-by: Shlomi Noach <[email protected]> * code comments. Privatize functoin Signed-off-by: Shlomi Noach <[email protected]> * test deferred till #12147 is merged Signed-off-by: Shlomi Noach <[email protected]> * code comments Signed-off-by: Shlomi Noach <[email protected]> * fix permutation logic, add tests Signed-off-by: Shlomi Noach <[email protected]> * unindent using early returns Signed-off-by: Shlomi Noach <[email protected]> * code comment Signed-off-by: Shlomi Noach <[email protected]> * reposition some functions in file (no actualy changes) Signed-off-by: Shlomi Noach <[email protected]> * relocated the functions back because the diff became really confusing Signed-off-by: Shlomi Noach <[email protected]> * grammar Signed-off-by: Shlomi Noach <[email protected]> * code comments, minor optimization Signed-off-by: Shlomi Noach <[email protected]> * typo Signed-off-by: Shlomi Noach <[email protected]> * Added tests, minor edits per review. changed language Signed-off-by: Shlomi Noach <[email protected]> * handling empty diff slice; improved comments Signed-off-by: Shlomi Noach <[email protected]> * Clarify test reasoning Signed-off-by: Shlomi Noach <[email protected]> * grammar Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added 3 commits January 24, 2023 14:50

validate table columns referenced by views

c4c213b

Signed-off-by: Shlomi Noach <[email protected]>

validate views' columns

3dac907

Signed-off-by: Shlomi Noach <[email protected]>

support star expression

e88d426

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach requested a review from dbussink January 25, 2023 09:52

shlomi-noach requested a review from deepthi as a code owner January 25, 2023 09:53

dbussink reviewed Jan 25, 2023

View reviewed changes

shlomi-noach added 5 commits January 25, 2023 12:49

removed 'TODO'

61de6b0

Signed-off-by: Shlomi Noach <[email protected]>

rename: errs

389872c

Signed-off-by: Shlomi Noach <[email protected]>

use mutierr

56c8421

Signed-off-by: Shlomi Noach <[email protected]>

excessive test, removed

11d6379

Signed-off-by: Shlomi Noach <[email protected]>

go mod tidy

f4fd673

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach requested review from ajm188, harshit-gangal, mattlord, rohit-nayak-ps, systay and frouioui as code owners January 25, 2023 10:58

dbussink reviewed Jan 25, 2023

View reviewed changes

go/vt/schemadiff/schema.go Outdated Show resolved Hide resolved

dbussink reviewed Jan 25, 2023

View reviewed changes

update to latest mutierr

a4ae98d

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving labels Jan 25, 2023

shlomi-noach added 2 commits January 26, 2023 11:19

merge main, resolve conflict

ef16bdf

Signed-off-by: Shlomi Noach <[email protected]>

avoid setting entity columns in case of error

54fb73a

Signed-off-by: Shlomi Noach <[email protected]>

dbussink approved these changes Jan 26, 2023

View reviewed changes

systay reviewed Jan 26, 2023

View reviewed changes

mattlord reviewed Jan 29, 2023

View reviewed changes

grammar

2d3e70b

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added 3 commits January 31, 2023 06:41

schemadiff: fix scenario where no tables exist in schema and with jus…

b69ede6

…t views reading from DUAL Signed-off-by: Shlomi Noach <[email protected]>

dual, not dual2

bc34369

Signed-off-by: Shlomi Noach <[email protected]>

stripped irrelevant comments

bc8799d

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach mentioned this pull request Jan 31, 2023

schemadiff: fix scenario where no tables exist in schema and with just views reading from DUAL #12189

Merged

3 tasks

shlomi-noach added 2 commits February 5, 2023 19:30

merge main, resolve conflict

dd1bc82

Signed-off-by: Shlomi Noach <[email protected]>

using AllErrorRecorder instead of multierr package

7f6fd13

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach requested a review from a team February 8, 2023 06:22

GuptaManan100 approved these changes Feb 14, 2023

View reviewed changes

multierr update

4939d01

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added a commit to planetscale/vitess that referenced this pull request Mar 5, 2023

test deferred till vitessio#12147 is merged

12df1ce

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach marked this pull request as draft March 6, 2023 07:25

This was referenced Mar 6, 2023

schemadiff: diffing two schemas generates a rich SchemaDiff object #12551

Merged

schemadiff: validate views' referenced columns via semantics #12565

Merged

shlomi-noach closed this Mar 21, 2023

shlomi-noach deleted the schemadiff-validate-view-columns-deps branch March 21, 2023 03:57

	func (ts trafficSwitcher) ForAllSources(f func(source workflow.MigrationSource) error) error {
	var wg sync.WaitGroup
	allErrors := &concurrency.AllErrorRecorder{}
	for _, source := range ts.sources {
	wg.Add(1)
	go func(source *workflow.MigrationSource) {
	defer wg.Done()

	if err := f(source); err != nil {
	allErrors.RecordError(err)
	}
	}(source)
	}
	wg.Wait()
	return allErrors.AggrError(vterrors.Aggregate)
	}

	var terrs []error
	for !empty {
	select {
	case result := <-resultCh:
	switch result.state {
	case vcopierCopyTaskCancel:
	// A task cancelation probably indicates an expired context due
	// to a PlannedReparentShard or elapsed copy phase duration,
	// neither of which are error conditions.
	case vcopierCopyTaskComplete:
	// Get the latest lastpk, purely for logging purposes.
	lastpk = result.args.lastpk
	case vcopierCopyTaskFail:
	// Aggregate non-nil errors.
	terrs = append(terrs, result.err)
	}
	default:
	empty = true
	}
	}
	if len(terrs) > 0 {
	terr := vterrors.Aggregate(terrs)
	log.Warningf("task error in workflow %s: %v", vc.vr.WorkflowName, terr)
	return fmt.Errorf("task error: %v", terr)
	}

schemadiff: validate views' referenced columns #12147

schemadiff: validate views' referenced columns #12147

Conversation

shlomi-noach commented Jan 25, 2023

Description

Related Issue(s)

Checklist

Deployment Notes

shlomi-noach commented Jan 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach commented Jan 29, 2023

mattlord left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbussink Jan 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord Jan 31, 2023 • edited Loading

Choose a reason for hiding this comment

shlomi-noach commented Jan 31, 2023

shlomi-noach commented Feb 6, 2023

shlomi-noach commented Feb 6, 2023

shlomi-noach commented Feb 8, 2023

shlomi-noach commented Feb 13, 2023

GuptaManan100 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach commented Feb 20, 2023

shlomi-noach commented Mar 8, 2023

dbussink Jan 30, 2023 •

edited

Loading

mattlord Jan 31, 2023 •

edited

Loading