schemachanger: defer catalog writes pre-commit #95330

postamar · 2023-01-17T00:29:29Z

Previously, in the declarative schema changer, each mutation stage would
be executed by visiting each of the stage's mutation ops to collect the
mutation state, followed by applying the mutations eagerly.

This commit defers the running of kv.Batch containing the catalog
mutations to the pre-commit phase. This change doesn't functionally
change anything but paves the way for undoing all pending catalog
mutations and re-planning the schema changes at pre-commit time.

Informs #88294.

Release note: None

cockroach-teamcity · 2023-01-17T00:29:42Z

This change is

Previously, it was impossible to execute EXPLAIN (DDL) COMMENT ON... This commit fixes this. Release note (bug fix): EXPLAIN (DDL) COMMENT ON is now possible.

This commit fixes a bug with this function which only becomes apparent when not using bazel. This only impacts testing with GoLand. Release note: None

In some cases, we define a set of almost-identical rules with the same name, purposefully. The name is what is used by the data-driven test output to sort the rule definitions, because these otherwise appear in the order that the init() functions were called, which is undefined. Sorting by name reduced the instability of the data-driven test when adding or removing init() blocks containing rule definitions, but did not altogether remove it. This commit should help further remove it by using a stable sort. In this way, the test output should be stable assuming that rule names are not re-used across init() blocks. Release note: None

This commit adds an underscore in the generated tests' name where there previously was one missing. Release note: None

This operation is no longer required. Release note: None

This commit fixes a bug where comments weren't handled correctly by the test state. This wasn't noticeable because the code wasn't actually exercised anywhere. This commit also fixes this. Release note: None

Previously, in the declarative schema changer, each mutation stage would be executed by visiting each of the stage's mutation ops to collect the mutation state, followed by applying the mutations eagerly. This commit defers the running of kv.Batch containing the catalog mutations to the pre-commit phase. This change doesn't functionally change anything but paves the way for undoing all pending catalog mutations and re-planning the schema changes at pre-commit time. Informs cockroachdb#88294. Release note: None

ajwerner

This could have been a few commits or PRs. I think the comment changes are welcome but maybe need some attention regarding their revertibility. The testutils change I don't get. In general, it looks good.

ajwerner · 2023-01-18T15:10:53Z

pkg/sql/opt/exec/execbuilder/testdata/show_trace

-commit sql txn  DelRange /Table/24/1/1/109
-commit sql txn  DelRange /Table/24/1/2/109
-commit sql txn  DelRange /Table/24/1/3/109
-commit sql txn  DelRange /Table/24/1/5/109


Do you have a clear understanding as to why this went away?

Yes, 9b8eb53 removes the RemoveAllTableComments op which was completely redundant and which generated those DelRanges.

ajwerner · 2023-01-18T15:15:41Z

pkg/ccl/schemachangerccl/BUILD.bazel

@@ -51,7 +51,7 @@ sctest_gen(
    ccl = True,
    new_cluster_func = "newCluster",
    package = "schemachangerccl",
-    suffix = "base",
+    suffix = "_base",


ajwerner · 2023-01-18T15:19:00Z

pkg/sql/schemachanger/testdata/explain_verbose/drop_table

+│           ├── • LogEvent
+│           │     Element:
+│           │       TableComment:
+│           │         comment: t has a comment
+│           │         tableId: 107
+│           │     EventBase:
+│           │       Authorization:
+│           │         UserName: root
+│           │       Statement: DROP TABLE ‹db›.‹sc›.‹t›
+│           │       StatementTag: DROP TABLE
+│           │       TargetMetadata:
+│           │         SourceElementID: 1
+│           │         SubWorkID: 1
+│           │     TargetStatus: 1


I can't decide whether this is a good thing to show. It's fine for now but we might end up generating a lot of events when dropping tables.

I'm working on this. The event logs are a bit of a nightmare to work with in the declarative schema changer. We should just roll out a new kind of event message.

In any case, the event for comments exists, for better or for worse. Removing logging of comment events is out of scope of this PR.

ajwerner · 2023-01-18T15:20:28Z

pkg/sql/schemachanger/testdata/explain_verbose/drop_column_basic.rollback_1_of_7

+            ├── • UpsertColumnComment
+            │     ColumnID: 2
+            │     Comment: j has a comment
+            │     PGAttributeNum: 2
+            │     TableID: 104
+            │
+            ├── • LogEvent
+            │     Element:
+            │       ColumnComment:
+            │         columnId: 2
+            │         comment: j has a comment
+            │         pgAttributeNum: 2
+            │         tableId: 104
+            │     EventBase:
+            │       Authorization:
+            │         UserName: root
+            │       Statement: ALTER TABLE ‹defaultdb›.public.‹t› DROP COLUMN ‹j›
+            │       StatementTag: ALTER TABLE
+            │       TargetMetadata:
+            │         SourceElementID: 1
+            │         SubWorkID: 1
+            │     TargetStatus: 2


I wonder about this. Perhaps we want to not remove the comment until we hit the non-revertible stage.

I'm working towards that, but this test output changed only because I added a comment, not because I changed any behaviour, so eager comment removal is already a thing right now.

ajwerner · 2023-01-18T15:21:28Z

pkg/testutils/datapathutils/data_path.go

@@ -64,7 +65,18 @@ func RewritableDataPath(t testing.TB, relative ...string) string {
 	if !ok {
 		t.Fatal("unable to get caller information")
 	}
-	cockroachWorkspace := filepath.Dir(filepath.Dir(filepath.Dir(thisFilePath)))
+	cockroachWorkspace := thisFilePath


what is this about? isn't testutils/datapathutils/data_path.go always at a fixed depth from the workspace dir?

It got broken when RewritableDataPath got moved to a new datapathutils package with a different fixed depth than the old fixed depth. I guess nobody noticed until now.

postamar · 2023-01-18T22:03:12Z

Closed in favour of #95467 which just has small fixes.

postamar force-pushed the split-validate-and-run branch from fff349b to 686240a Compare January 17, 2023 12:30

Marius Posta added 7 commits January 17, 2023 11:32

parser: make COMMENT ON explainable

d63d1f9

Previously, it was impossible to execute EXPLAIN (DDL) COMMENT ON... This commit fixes this. Release note (bug fix): EXPLAIN (DDL) COMMENT ON is now possible.

datapathutils: fix RewritableDataPath

115fe06

This commit fixes a bug with this function which only becomes apparent when not using bazel. This only impacts testing with GoLand. Release note: None

schemachangerccl: rename generated tests

6a84928

This commit adds an underscore in the generated tests' name where there previously was one missing. Release note: None

schemachanger: remove RemoveAllTableComments op

9b8eb53

This operation is no longer required. Release note: None

sctestdeps: fix comments handling

f65c14c

This commit fixes a bug where comments weren't handled correctly by the test state. This wasn't noticeable because the code wasn't actually exercised anywhere. This commit also fixes this. Release note: None

postamar force-pushed the split-validate-and-run branch from 686240a to 953dff4 Compare January 17, 2023 16:32

postamar marked this pull request as ready for review January 17, 2023 22:22

postamar requested a review from a team January 17, 2023 22:22

postamar requested review from a team as code owners January 17, 2023 22:22

postamar requested review from DrewKimball and removed request for a team and DrewKimball January 17, 2023 22:22

ajwerner reviewed Jan 18, 2023

View reviewed changes

postamar closed this Jan 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schemachanger: defer catalog writes pre-commit #95330

schemachanger: defer catalog writes pre-commit #95330

postamar commented Jan 17, 2023

cockroach-teamcity commented Jan 17, 2023

ajwerner left a comment

ajwerner Jan 18, 2023

postamar Jan 18, 2023 •

edited

Loading

ajwerner Jan 18, 2023

ajwerner Jan 18, 2023

postamar Jan 18, 2023

postamar Jan 18, 2023

ajwerner Jan 18, 2023

postamar Jan 18, 2023

ajwerner Jan 18, 2023

postamar Jan 18, 2023

postamar commented Jan 18, 2023

schemachanger: defer catalog writes pre-commit #95330

schemachanger: defer catalog writes pre-commit #95330

Conversation

postamar commented Jan 17, 2023

cockroach-teamcity commented Jan 17, 2023

ajwerner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

postamar Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

postamar commented Jan 18, 2023

postamar Jan 18, 2023 •

edited

Loading