Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: version/mixed/nodes=5 failed #87687

Closed
cockroach-teamcity opened this issue Sep 9, 2022 · 14 comments
Closed

roachtest: version/mixed/nodes=5 failed #87687

cockroach-teamcity opened this issue Sep 9, 2022 · 14 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 9, 2022

roachtest.version/mixed/nodes=5 failed with artifacts on master @ a82711442c65cf14489c55041b45b11a1e38415b:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1777
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1822
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:906: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:906
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-19477

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 9, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Sep 9, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 389661e823c19f318fa07ec2278336262531692d:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ bc2e47da0523b347c28cf024707e80cd35d6c98a:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@renatolabs
Copy link
Contributor

ERROR: cockroach server exited with error: store node1,node1store1,store1=/mnt/data1/cockroach, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)

Shouldn't we be able to upgrade from a release version to a development version in this case? cc @dt, as this relates to #86345.

@dt
Copy link
Member

dt commented Sep 12, 2022

@renatolabs there was some worry that it was too easy to upgrade a production cluster to a master/alpha build inadvertently, given that such an upgrade is irreversible -- if you accidentally just started a master/alpha binary with default settings (no preserve downgrade), then within a few minutes you'd be permanently on dev/master versions and unable to later upgrade to any release build, including later releases.

To mitigate that, #87468 added the the env var COCKROACH_UPGRADE_TO_DEV_VERSION to opt-in to one of these one-way release->dev upgrades. So you are still able to do it, but you need to make it clear you want to, given irreversible nature.

Note that all builds from stabilized release branches, including nightlies, will count as release builds here; it is just builds from the development branch, where the upgrade process has not been finished, where such upgrades are untrusted and thus taint forever the cluster if they run.

@dt dt removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 12, 2022
@renatolabs
Copy link
Contributor

Thanks for the explanation! I'll update the test.

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 2cfa6b1779ead01508c04e228bf72b4e0e96d98c:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 95677eb5f8d006629b16024fb7d87d55344c1470:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 1f24e52aea4195ffe26b97160079bb29f264338a:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

srosenberg added a commit to srosenberg/cockroach that referenced this issue Sep 15, 2022
Recent changes to cockroach_versions (logic) now require setting
COCKROACH_UPGRADE_TO_DEV_VERSION environment variable in order to
allow upgrading a stable release data-dir to a dev. version.
The PR [1] which introduced this env. var. did not correctly backport
the change to all the roachtests which perform this type of upgrade.

Since all roachtests which exercise upgrade paths are intended to
test this upgrade scenario, we set COCKROACH_UPGRADE_TO_DEV_VERSION
by default in roachprod instead of polluting the roachtests with more
config. settings.

Consequently, MakeClusterSettings now returns the default
ClusterSettings which includes COCKROACH_UPGRADE_TO_DEV_VERSION;
generateStartCmd ensures it's passed into cockroach env. via start.sh.

Release note: None

Release justification: bug fix in roachtests.

Resolves:
  cockroachdb#87675
  cockroachdb#87687

[1] cockroachdb#87468
@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 9a05046ce19e7678340e82c70d61e928be95bc72:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ bd97ad5b8c9f537a89492a051574d867469bef33:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 3fa1a1600898d7b78b9e39d07132a387a2f9a1b6:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 87ed064dc23eab6948ee8a07e8507f150bda0e44:

		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1.2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:146
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	monitor.go:127,version.go:215,version.go:228,test_runner.go:917: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:215
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:228
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Sep 20, 2022
88005: roachprod: add COCKROACH_UPGRADE_TO_DEV_VERSION to DefaultEnvVars r=srosenberg a=srosenberg

Recent changes to cockroach_versions (logic) now require setting COCKROACH_UPGRADE_TO_DEV_VERSION environment variable in order to allow upgrading a stable release data-dir to a dev. version. The PR [1] which introduced this env. var. did not correctly backport the change to all the roachtests which perform this type of upgrade.

Since all roachtests which exercise upgrade paths are intended to test this upgrade scenario, we set COCKROACH_UPGRADE_TO_DEV_VERSION by default in roachprod instead of polluting the roachtests with more config. settings.

Consequently, MakeClusterSettings now returns the default ClusterSettings which includes COCKROACH_UPGRADE_TO_DEV_VERSION; generateStartCmd ensures it's passed into cockroach env. via start.sh.

Release note: None

Release justification: bug fix in roachtests.

Resolves:
  #87675
  #87687

[1] #87468

88031: admission: enable elastic cpu control by default r=irfansharif a=irfansharif

We want to have this bake through the 23.1 development cycle.

Release note: None (not being backported)

88083: stats: fix buckets for INT2 and INT4 r=yuzefovich a=yuzefovich

Previously, if we needed to create "outer" histogram buckets (which is the case when minimum and maximum values in the column weren't sampled yet they contributed to the distinct count) for INT2 and INT4 types, we would use the values that exceeded the supported range for those types. This could lead to incorrect estimation later on when those "outer" buckets are used during the costing as well as the histograms would need to be manually edited to be injected. This is now fixed by handling these two types separately.

Fixes: #76887.

Release note: None

88167: sql: pg_proc should respect context db for udf r=chengxiong-ruan a=chengxiong-ruan

`pg_proc` table should only return udf in the context db if it's not nil.

Release note: None
Release justification: low risk virtual table improvement.


Co-authored-by: Stan Rosenberg <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
Co-authored-by: Chengxiong Ruan <[email protected]>
blathers-crl bot pushed a commit that referenced this issue Sep 20, 2022
Recent changes to cockroach_versions (logic) now require setting
COCKROACH_UPGRADE_TO_DEV_VERSION environment variable in order to
allow upgrading a stable release data-dir to a dev. version.
The PR [1] which introduced this env. var. did not correctly backport
the change to all the roachtests which perform this type of upgrade.

Since all roachtests which exercise upgrade paths are intended to
test this upgrade scenario, we set COCKROACH_UPGRADE_TO_DEV_VERSION
by default in roachprod instead of polluting the roachtests with more
config. settings.

Consequently, MakeClusterSettings now returns the default
ClusterSettings which includes COCKROACH_UPGRADE_TO_DEV_VERSION;
generateStartCmd ensures it's passed into cockroach env. via start.sh.

Release note: None

Release justification: bug fix in roachtests.

Resolves:
  #87675
  #87687

[1] #87468
@rafiss
Copy link
Collaborator

rafiss commented Sep 20, 2022

resolved by #88005

@rafiss rafiss closed this as completed Sep 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Development

No branches or pull requests

4 participants