pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed #106349

cockroach-teamcity · 2023-07-06T22:21:14Z

pkg/sql/schemachanger/schemachanger_test.TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed with artifacts on master @ 818aec861357579eb3a3e987cf5887f3cf112be4:

=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid3777751059
    test_log_scope.go:81: use -show-logs to present logs inline
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11122/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:1:
        setup [0 args]
        CREATE TABLE t (a INT NOT NULL)
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11122/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:5:
        stage-exec [2 args]
        INSERT INTO t VALUES($stageKey);
        INSERT INTO t VALUES($stageKey + 1);
        ----
        pq: INSERT has more expressions than target columns, 2 expressions for 1 targets
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11122/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:13:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11122/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:19:
        stage-exec [2 args]
        INSERT INTO t VALUES($stageKey);
        INSERT INTO t VALUES($stageKey + 1);
        ----
        pq: INSERT has more expressions than target columns, 2 expressions for 1 targets
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11122/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:27:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
=== CONT  TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid3777751059
--- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid (164.34s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage
    --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage (163.26s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_13_of_15
    cumulative.go:1724: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1724
        	            				GOROOT/src/runtime/asm_arm64.s:1172
        	Error:      	Received unexpected error:
        	            	dial tcp 127.0.0.1:44631: connect: connection refused
        	Test:       	TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_13_of_15
        --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_13_of_15 (4.65s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations _{This test on roachdash | Improve this report!

Jira issue: CRDB-29522}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2023-07-07T06:37:22Z

pkg/sql/schemachanger/schemachanger_test.TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed with artifacts on master @ 1137aee0089fc2fe161bab0f1c899d2ddc997c90:

=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/78a27721ee5ac1b54bee47886d5838a5/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid3029362930
    test_log_scope.go:81: use -show-logs to present logs inline
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:1:
        setup [0 args]
        CREATE TABLE t (a INT NOT NULL)
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:5:
        stage-exec [2 args]
        INSERT INTO t VALUES($stageKey);
        INSERT INTO t VALUES($stageKey + 1);
        ----
        pq: INSERT has more expressions than target columns, 2 expressions for 1 targets
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:13:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:19:
        stage-exec [2 args]
        INSERT INTO t VALUES($stageKey);
        INSERT INTO t VALUES($stageKey + 1);
        ----
        pq: INSERT has more expressions than target columns, 2 expressions for 1 targets
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/12563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:27:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
=== CONT  TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/78a27721ee5ac1b54bee47886d5838a5/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid3029362930
--- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid (144.04s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage
    --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage (142.80s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_9_of_15
    cumulative.go:1724: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1724
        	            				GOROOT/src/runtime/asm_amd64.s:1594
        	Error:      	Received unexpected error:
        	            	dial tcp 127.0.0.1:39099: connect: connection refused
        	Test:       	TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_9_of_15
        --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_9_of_15 (10.08s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

knz · 2023-07-07T10:03:37Z

here too https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/10819309?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

cockroach-teamcity · 2023-07-07T14:00:16Z

pkg/sql/schemachanger/schemachanger_test.TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed with artifacts on master @ 81b17ca3982aba16c64d7da9a337cd2a57013536:

=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid242614313
    test_log_scope.go:81: use -show-logs to present logs inline
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1757/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:1:
        setup [0 args]
        CREATE TABLE t (a INT NOT NULL)
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1757/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:5:
        stage-exec [2 args]
        INSERT INTO t VALUES($stageKey);
        INSERT INTO t VALUES($stageKey + 1);
        ----
        pq: INSERT has more expressions than target columns, 2 expressions for 1 targets
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1757/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:13:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1757/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:19:
        stage-exec [2 args]
        INSERT INTO t VALUES($stageKey);
        INSERT INTO t VALUES($stageKey + 1);
        ----
        pq: INSERT has more expressions than target columns, 2 expressions for 1 targets
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1757/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:27:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
=== CONT  TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid242614313
--- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid (153.85s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage
    --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage (151.90s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_14_of_15
    cumulative.go:1724: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1724
        	            				GOROOT/src/runtime/asm_arm64.s:1172
        	Error:      	Received unexpected error:
        	            	dial tcp 127.0.0.1:44525: connect: connection refused
        	Test:       	TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_14_of_15
        --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitPhase_14_of_15 (6.59s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

irfansharif · 2023-07-07T17:28:20Z

Driveby stress attempt, not very successful:

$ dev test pkg/sql/schemachanger -f TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage --stress
...
23090 runs so far, 0 failures, over 18m25s
23198 runs so far, 0 failures, over 18m30s
23305 runs so far, 0 failures, over 18m35s

It's hard to find failure logs for the runs above (internal discussion), but looking through another instance of TestValidateMixedVersionElements failure, I don't know that it's #106078. See #106315 (comment).

rafiss · 2023-07-07T21:10:53Z

We can't repro since after #106411 was merged (though I agree that it doesn't seem like the symptom that fixed is happening here..)

I'll keep this issue open and link any future occurrences to this.

cockroach-teamcity · 2023-07-10T20:07:58Z

pkg/sql/schemachanger/schemachanger_test.TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed with artifacts on master @ 7cd5444acd8f32ec62aa0077d43263f3087beed0:

        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_arm64.s:1172
        	            	Wraps: (2) query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	  |
        	            	  | got:
        	            	  | paused, NEW SCHEMA CHANGE, ALTER TABLE defaultdb.public.t ALTER PRIMARY KEY USING COLUMNS (a)
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitNonRevertiblePhase_2_of_4 (49.31s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitNonRevertiblePhase_4_of_4
    sql_runner.go:104: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:117
        	            				github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:312
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/end_to_end.go:488
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1632
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	Error:      	Received unexpected error:
        	            	query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:315
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
        	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:117
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:312
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.waitForSchemaChangesToFinish
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/end_to_end.go:488
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.executeSchemaChangeTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1632
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.ValidateMixedVersionElements.func5.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_arm64.s:1172
        	            	Wraps: (2) query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	  |
        	            	  | got:
        	            	  | running, NEW SCHEMA CHANGE, ALTER TABLE defaultdb.public.t ALTER PRIMARY KEY USING COLUMNS (a)
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitNonRevertiblePhase_4_of_4 (47.94s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

postamar · 2023-07-11T14:41:59Z

This is a more interesting error! I wouldn't be surprised to find out that we're polling the jobs table a tad too aggressively. Notice that the test sees the schema change as both paused and running in that error trace. Having read the code in ValidateMixedVersionElements there seems to be room for a few obvious potential improvements in light of this finding.

cockroach-teamcity · 2023-07-11T18:28:16Z

pkg/sql/schemachanger/schemachanger_test.TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed with artifacts on master @ bdaae6458a2388da03adbb979a25b5a07ea6edd9:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9415/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/schemachanger/schemachanger_test_/schemachanger_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/schemachanger/testdata/end_to_end/alter_table_alter_primary_key_drop_rowid/alter_table_alter_primary_key_drop_rowid.definition:27:
        stage-query [2 args]
        SELECT count(*)=$successfulStageCount*2 FROM t;
        ----
        true
=== CONT  TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid1413487098
--- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid (266.72s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage
    --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage (265.07s)
=== RUN   TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitNonRevertiblePhase_3_of_4
    sql_runner.go:104: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:117
        	            				github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:312
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/end_to_end.go:488
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1632
        	            				github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	Error:      	Received unexpected error:
        	            	query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:315
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
        	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:117
        	            	  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
        	            	  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:312
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.waitForSchemaChangesToFinish
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/end_to_end.go:488
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.executeSchemaChangeTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1632
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest.ValidateMixedVersionElements.func5.1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest/cumulative.go:1729
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1446
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_arm64.s:1172
        	            	Wraps: (2) query 'SELECT status, job_type, description FROM [SHOW JOBS] WHERE job_type IN ('SCHEMA CHANGE', 'TYPEDESC SCHEMA CHANGE', 'NEW SCHEMA CHANGE') AND status NOT IN ('succeeded', 'failed')': expected:
        	            	  |
        	            	  | got:
        	            	  | paused, NEW SCHEMA CHANGE, ALTER TABLE defaultdb.public.t ALTER PRIMARY KEY USING COLUMNS (a)
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        --- FAIL: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid/pause_upgrade_and_resume_at_each_stage/PostCommitNonRevertiblePhase_3_of_4 (48.65s)

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

DrewKimball · 2023-07-21T23:21:58Z

I failed to reproduce locally, but it seems like various of the TestValidateMixedVersionElements are flaking right now. I'll put up a skip.

Skip the `ValidateMixedVersionElements` tests, since most or all of them seem to be flaking right now. Informs: cockroachdb#106349 Informs: cockroachdb#106727 Informs: cockroachdb#107399 Release note: None

rafiss · 2023-07-24T21:28:02Z

Now that #106743 has merged, we'll see if this test has stabilized. Note that it was renamed, so if it's still flaky, the next failure will open a new issue. Keeping this open so we don't forget to check on it.

rafiss · 2023-07-25T19:06:56Z

After the test rename, #107552 is the new issue, so let's close this one.

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jul 6, 2023

cockroach-teamcity added this to the 23.2 milestone Jul 6, 2023

chengxiong-ruan mentioned this issue Jul 7, 2023

roachtest: schemachange/index/tpcc/w=800 failed [deadlock in kvflowhandle] #106078

Closed

exalate-issue-sync bot assigned postamar Jul 11, 2023

rafiss mentioned this issue Jul 11, 2023

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_drop_column_with_partial_index failed #106565

Closed

rafiss mentioned this issue Jul 13, 2023

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_drop_column_basic failed #106727

Closed

rafiss assigned Xiang-Gu and unassigned postamar Jul 20, 2023

DrewKimball mentioned this issue Jul 22, 2023

schemachanger: skip flakey tests #107406

Closed

This was referenced Jul 24, 2023

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_add_column_with_stored_family failed #107399

Closed

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_drop_column_computed_index failed #107429

Closed

rafiss mentioned this issue Jul 25, 2023

sql: flakes in TestBackupSuccessMixedVersion #107552

Closed

rafiss closed this as completed Jul 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed #106349

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed #106349

cockroach-teamcity commented Jul 6, 2023 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jul 7, 2023

knz commented Jul 7, 2023

cockroach-teamcity commented Jul 7, 2023

irfansharif commented Jul 7, 2023

rafiss commented Jul 7, 2023

cockroach-teamcity commented Jul 10, 2023

postamar commented Jul 11, 2023

cockroach-teamcity commented Jul 11, 2023

DrewKimball commented Jul 21, 2023

rafiss commented Jul 24, 2023

rafiss commented Jul 25, 2023

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed #106349

pkg/sql/schemachanger/schemachanger_test: TestValidateMixedVersionElements_alter_table_alter_primary_key_drop_rowid failed #106349

Comments

cockroach-teamcity commented Jul 6, 2023 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jul 7, 2023

knz commented Jul 7, 2023

cockroach-teamcity commented Jul 7, 2023

irfansharif commented Jul 7, 2023

rafiss commented Jul 7, 2023

cockroach-teamcity commented Jul 10, 2023

postamar commented Jul 11, 2023

cockroach-teamcity commented Jul 11, 2023

DrewKimball commented Jul 21, 2023

rafiss commented Jul 24, 2023

rafiss commented Jul 25, 2023

cockroach-teamcity commented Jul 6, 2023 •

edited by cockroach-jira-scripts

Loading