Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #78035

Closed
cockroach-teamcity opened this issue Mar 17, 2022 · 205 comments
Closed

roachtest: cluster_creation failed #78035

cockroach-teamcity opened this issue Mar 17, 2022 · 205 comments
Labels
branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-testeng TestEng Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 17, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on release-22.1 @ 2d49a7bd0d1ebeff93277233695411d25b2a5d39:

The test failed on branch=release-22.1, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:32:28Z},{Key=Name,Value=teamcity-4594125-1647494291-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script413225093 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping769912648 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:32:28Z},{Key=Name,Value=teamcity-4594125-1647494291-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script413225093 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping769912648 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-13932
@cockroach-teamcity cockroach-teamcity added branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 17, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 2d49a7bd0d1ebeff93277233695411d25b2a5d39:

The test failed on branch=release-22.1, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-14-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:33:32Z},{Key=Name,Value=teamcity-4594125-1647494291-14-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script880572208 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2964459769 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-14-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:33:32Z},{Key=Name,Value=teamcity-4594125-1647494291-14-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script880572208 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2964459769 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@erikgrinaker erikgrinaker removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 21, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

The test failed on branch=release-22.1, cloud=aws:
test tpccbench/nodes=3/cpu=16 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:19Z},{Key=Name,Value=teamcity-4698289-1648444778-10-n4cpu16-0004}] --user-data file:///tmp/aws-startup-script2540826881 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping17736695 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:19Z},{Key=Name,Value=teamcity-4698289-1648444778-10-n4cpu16-0004}] --user-data file:///tmp/aws-startup-script2540826881 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping17736695 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 28, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

The test failed on branch=release-22.1, cloud=aws:
test tpccbench/nodes=3/cpu=16/no-admission was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T12:49:18Z},{Key=Name,Value=teamcity-4698289-1648444778-13-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-13-n4cpu16},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1542903493 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3406362205 --output json: stderr: : exit status 254
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: aws
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T12:49:18Z},{Key=Name,Value=teamcity-4698289-1648444778-13-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-13-n4cpu16},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1542903493 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3406362205 --output json: stderr: 
		  | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
		Wraps: (5) exit status 254
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@jlinder jlinder added the T-testeng TestEng Team label Mar 28, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 87e23b5a6de65b0f184ad71e6706eebd65ec6303:

The test failed on branch=release-22.1, cloud=gce:
test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831899-1649395134-94-n5cpu96,lifetime=12h0m0s,created=2022-04-08t14_44_10z --metadata-from-file startup-script=/tmp/gce-startup-script1657207299 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831899-1649395134-94-n5cpu96,lifetime=12h0m0s,created=2022-04-08t14_44_10z --metadata-from-file startup-script=/tmp/gce-startup-script1657207299 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 15, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test backup/KMS/AWS/n3cpu4 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t05_50_55z,roachprod=true,cluster=teamcity-4897758-1649999866-25-n3cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script278218323 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t05_50_55z,roachprod=true,cluster=teamcity-4897758-1649999866-25-n3cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script278218323 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-25-n3cpu4-0001].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-25-n3cpu4-0002].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test backupTPCC was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_54_27z,roachprod=true,cluster=teamcity-4897758-1649999866-30-n3cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script140068770 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_54_27z,roachprod=true,cluster=teamcity-4897758-1649999866-30-n3cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script140068770 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cancel/tpch/distsql/queries=7,9,20,21,nodes=3 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-35-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_57_53z --metadata-from-file startup-script=/tmp/gce-startup-script3136905862 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-35-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_57_53z --metadata-from-file startup-script=/tmp/gce-startup-script3136905862 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-35-n3cpu4-0001].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-35-n3cpu4-0002].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/cloud-sink-gcs/rangefeed=true was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897758-1649999866-38-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_06_05z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script871450961 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897758-1649999866-38-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_06_05z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script871450961 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/crdb-chaos was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-43-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_11_54z --metadata-from-file startup-script=/tmp/gce-startup-script348927181 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-43-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_11_54z --metadata-from-file startup-script=/tmp/gce-startup-script348927181 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-43-n4cpu16-0003].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-43-n4cpu16-0004].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/initial-scan was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_13_04z,roachprod=true,cluster=teamcity-4897758-1649999866-44-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1396308951 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_13_04z,roachprod=true,cluster=teamcity-4897758-1649999866-44-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1396308951 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-44-n4cpu16-0002].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/sink-chaos was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_23_56z,roachprod=true,cluster=teamcity-4897758-1649999866-50-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4178551782 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_23_56z,roachprod=true,cluster=teamcity-4897758-1649999866-50-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4178551782 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-50-n4cpu16-0002].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-50-n4cpu16-0003].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-50-n4cpu16-0004].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test clearrange/checks=true was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-60-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_31_13z --metadata-from-file startup-script=/tmp/gce-startup-script2157506098 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-60-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_31_13z --metadata-from-file startup-script=/tmp/gce-startup-script2157506098 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test clearrange/checks=false was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-58-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_30_05z --metadata-from-file startup-script=/tmp/gce-startup-script3674307539 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-58-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_30_05z --metadata-from-file startup-script=/tmp/gce-startup-script3674307539 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-58-n10cpu16-0006].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/tpcc-1000 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-63-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_32_10z --metadata-from-file startup-script=/tmp/gce-startup-script514418152 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-63-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_32_10z --metadata-from-file startup-script=/tmp/gce-startup-script514418152 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-63-n4cpu16-0004].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test copy/bank/rows=1000000,nodes=9,txn=false was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-71-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_39_49z --metadata-from-file startup-script=/tmp/gce-startup-script3995681688 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-71-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_39_49z --metadata-from-file startup-script=/tmp/gce-startup-script3995681688 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test decommission/randomized was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-76-n6cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_03z --metadata-from-file startup-script=/tmp/gce-startup-script3998134501 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-76-n6cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_03z --metadata-from-file startup-script=/tmp/gce-startup-script3998134501 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test drain-and-decommission/nodes=9 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-77-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_11z --metadata-from-file startup-script=/tmp/gce-startup-script1738687397 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-77-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_11z --metadata-from-file startup-script=/tmp/gce-startup-script1738687397 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test follower-reads/mixed-version/single-region was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-2 --labels roachprod=true,cluster=teamcity-4897758-1649999866-82-n3cpu2,lifetime=12h0m0s,created=2022-04-15t06_44_51z --metadata-from-file startup-script=/tmp/gce-startup-script3153286598 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-2 --labels roachprod=true,cluster=teamcity-4897758-1649999866-82-n3cpu2,lifetime=12h0m0s,created=2022-04-15t06_44_51z --metadata-from-file startup-script=/tmp/gce-startup-script3153286598 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test drop/tpcc/w=100,nodes=9 was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-84-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_46_56z --metadata-from-file startup-script=/tmp/gce-startup-script3813097881 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-84-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_46_56z --metadata-from-file startup-script=/tmp/gce-startup-script3813097881 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-84-n9cpu4-0002].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-84-n9cpu4-0008].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-84-n9cpu4-0009].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test follower-reads/survival=region/locality=global/reads=bounded-staleness was skipped due to 	test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-89-n6cpu4-geo,lifetime=12h0m0s,created=2022-04-15t06_50_58z --metadata-from-file startup-script=/tmp/gce-startup-script134413619 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-89-n6cpu4-geo,lifetime=12h0m0s,created=2022-04-15t06_50_58z --metadata-from-file startup-script=/tmp/gce-startup-script134413619 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test sqlsmith/setup=seed-multi-region/setting=multi-region was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-6897515-1665552000-133-n4cpu4-geo,lifetime=12h0m0s,created=2022-10-12t11_54_07z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script142453087 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-6897515-1665552000-133-n4cpu4-geo,lifetime=12h0m0s,created=2022-10-12t11_54_07z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script142453087 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test tpcc/multiregion/survive=region/chaos=true was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-10-12t11_55_09z,roachprod=true,cluster=teamcity-6897515-1665552000-134-n10cpu4-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2741776384 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-10-12t11_55_09z,roachprod=true,cluster=teamcity-6897515-1665552000-134-n10cpu4-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2741776384 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test tpcc/multiregion/survive=zone/chaos=true was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-6897515-1665552000-136-n10cpu4-geo,lifetime=12h0m0s,created=2022-10-12t11_56_53z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script627839061 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-6897515-1665552000-136-n10cpu4-geo,lifetime=12h0m0s,created=2022-10-12t11_56_53z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script627839061 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test sysbench/oltp_read_only/nodes=3/cpu=32/conc=128 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels lifetime=12h0m0s,created=2022-10-12t11_59_16z,roachprod=true,cluster=teamcity-6897515-1665552000-140-n4cpu32 --metadata-from-file startup-script=/tmp/gce-startup-script3983952829 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels lifetime=12h0m0s,created=2022-10-12t11_59_16z,roachprod=true,cluster=teamcity-6897515-1665552000-140-n4cpu32 --metadata-from-file startup-script=/tmp/gce-startup-script3983952829 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test tpccbench/nodes=9/cpu=4/multi-region was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highmem-4 --labels roachprod=true,cluster=teamcity-6897515-1665552000-138-n12cpu4m-geo,lifetime=12h0m0s,created=2022-10-12t11_58_31z --metadata-from-file startup-script=/tmp/gce-startup-script1084163704 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highmem-4 --labels roachprod=true,cluster=teamcity-6897515-1665552000-138-n12cpu4m-geo,lifetime=12h0m0s,created=2022-10-12t11_58_31z --metadata-from-file startup-script=/tmp/gce-startup-script1084163704 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test sysbench/oltp_read_write/nodes=3/cpu=32/conc=128 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels lifetime=12h0m0s,created=2022-10-12t12_01_03z,roachprod=true,cluster=teamcity-6897515-1665552000-143-n4cpu32 --metadata-from-file startup-script=/tmp/gce-startup-script3954938546 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels lifetime=12h0m0s,created=2022-10-12t12_01_03z,roachprod=true,cluster=teamcity-6897515-1665552000-143-n4cpu32 --metadata-from-file startup-script=/tmp/gce-startup-script3954938546 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test sysbench/oltp_update_index/nodes=3/cpu=32/conc=128 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-148-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_02_41z --metadata-from-file startup-script=/tmp/gce-startup-script1802247405 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-148-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_02_41z --metadata-from-file startup-script=/tmp/gce-startup-script1802247405 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=128 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-149-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_02_50z --metadata-from-file startup-script=/tmp/gce-startup-script2086483432 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-149-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_02_50z --metadata-from-file startup-script=/tmp/gce-startup-script2086483432 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test ycsb/B/nodes=3/cpu=32 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels cluster=teamcity-6897515-1665552000-154-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_04_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1626835127 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels cluster=teamcity-6897515-1665552000-154-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_04_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1626835127 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test ycsb/C/nodes=3/cpu=32 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-155-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_04_42z --metadata-from-file startup-script=/tmp/gce-startup-script3873302125 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-155-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_04_42z --metadata-from-file startup-script=/tmp/gce-startup-script3873302125 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test ycsb/D/nodes=3/cpu=32 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels lifetime=12h0m0s,created=2022-10-12t12_06_12z,roachprod=true,cluster=teamcity-6897515-1665552000-160-n4cpu32 --metadata-from-file startup-script=/tmp/gce-startup-script4014392642 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels lifetime=12h0m0s,created=2022-10-12t12_06_12z,roachprod=true,cluster=teamcity-6897515-1665552000-160-n4cpu32 --metadata-from-file startup-script=/tmp/gce-startup-script4014392642 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test ycsb/E/nodes=3/cpu=32 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels created=2022-10-12t12_06_32z,roachprod=true,cluster=teamcity-6897515-1665552000-161-n4cpu32,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3169958050 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels created=2022-10-12t12_06_32z,roachprod=true,cluster=teamcity-6897515-1665552000-161-n4cpu32,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3169958050 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test zfs/ycsb/A/nodes=3/cpu=32 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-164-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_08_00z --metadata-from-file startup-script=/tmp/gce-startup-script1427013362 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-6897515-1665552000-164-n4cpu32,lifetime=12h0m0s,created=2022-10-12t12_08_00z --metadata-from-file startup-script=/tmp/gce-startup-script1427013362 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ e438c2f89282e607e0e6ca1d38b2e0a622f94493:

The test failed on branch=release-22.1, cloud=gce:
test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels created=2022-10-12t12_19_48z,roachprod=true,cluster=teamcity-6897515-1665552000-167-n5cpu96,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3001643136 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels created=2022-10-12t12_19_48z,roachprod=true,cluster=teamcity-6897515-1665552000-167-n5cpu96,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3001643136 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220927'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ c849cfb82a41b0da3cc2b3c3314e27757fe43e65:

The test failed on branch=release-22.1, cloud=gce:
test restoreTPCE10TB/nodes=10/cpus=8/pd-volume=1500GB was skipped due to 	test_runner.go:589,test_runner.go:261,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --create-disk type=pd-ssd,size=1500GB,auto-delete=yes --machine-type n1-highmem-8 --labels lifetime=12h0m0s,created=2022-12-13t12_49_46z,roachprod=true,cluster=teamcity-7937052-1670912359-94-n10cpu8m --metadata-from-file startup-script=/tmp/gce-startup-script3654895494 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
		  | [...repeated from below...]
		Wraps: (2) in provider: gce
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --create-disk type=pd-ssd,size=1500GB,auto-delete=yes --machine-type n1-highmem-8 --labels lifetime=12h0m0s,created=2022-12-13t12_49_46z,roachprod=true,cluster=teamcity-7937052-1670912359-94-n10cpu8m --metadata-from-file startup-script=/tmp/gce-startup-script3654895494 --project cockroach-ephemeral --boot-disk-size=32GB]
		  | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0001].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0002].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0003].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0004].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0005].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0006].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0007].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0009].
		  | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-7937052-1670912359-94-n10cpu8m-0010].
		  | WARNING: Some requests generated warnings:
		  |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
		  |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20221206'.
		  |
		  | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
		  |  - Quota 'SSD_TOTAL_GB' exceeded.  Limit: 200000.0 in region us-east1.
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 3855d706a038ce874840e5e8f4bc4cf0c24faac9:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:609).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner--BAXBtm0wnCU5u6KNpo1GipVn-0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-9182749-1679462459-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2023-03-22T17:11:34Z},{Key=Name,Value=teamcity-9182749-1679462459-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script546782782 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1423820867 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 3855d706a038ce874840e5e8f4bc4cf0c24faac9:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:609).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner--BAXBtm0wnCU5u6KNpo1GipVn-0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2023-03-22T17:13:01Z},{Key=Name,Value=teamcity-9182749-1679462459-14-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-9182749-1679462459-14-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1109469989 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3873657380 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Apr 12, 2023
Closed
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 9ab632f3bedfb653dacf571fb4b10e02de51ef77:

test kv0/enc=false/nodes=3/size=4kb was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-9554123-1681277535-84-n4cpu8,lifetime=12h0m0s,created=2023-04-12t11_14_41z --metadata-from-file startup-script=/tmp/gce-startup-script4116557719 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554123-1681277535-84-n4cpu8-0001].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 9ab632f3bedfb653dacf571fb4b10e02de51ef77:

test kv0/enc=false/nodes=3/splt=0 was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-9554123-1681277535-88-n4cpu8,lifetime=12h0m0s,created=2023-04-12t11_18_15z --metadata-from-file startup-script=/tmp/gce-startup-script1417798235 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 9ab632f3bedfb653dacf571fb4b10e02de51ef77:

test kv0/enc=true/nodes=3 was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-9554123-1681277535-92-n4cpu8,lifetime=12h0m0s,created=2023-04-12t11_21_44z --metadata-from-file startup-script=/tmp/gce-startup-script3024399674 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 9ab632f3bedfb653dacf571fb4b10e02de51ef77:

test kv50/rangelookups/relocate/nodes=8 was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-9554123-1681277535-95-n9cpu8,lifetime=12h0m0s,created=2023-04-12t11_24_05z --metadata-from-file startup-script=/tmp/gce-startup-script1138203935 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 9ab632f3bedfb653dacf571fb4b10e02de51ef77:

test kv50/rangelookups/split/nodes=8 was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-9554123-1681277535-98-n9cpu8,lifetime=12h0m0s,created=2023-04-12t11_26_25z --metadata-from-file startup-script=/tmp/gce-startup-script3524833300 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554123-1681277535-98-n9cpu8-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554123-1681277535-98-n9cpu8-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554123-1681277535-98-n9cpu8-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554123-1681277535-98-n9cpu8-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554123-1681277535-98-n9cpu8-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 366c8feea58d696c2ad017839b3049d45aa0b855:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels created=2023-04-14t11_39_13z,roachprod=true,cluster=teamcity-9599065-1681449501-85-n4cpu96,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2803440662 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-85-n4cpu96-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-85-n4cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 8000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 366c8feea58d696c2ad017839b3049d45aa0b855:

test kv95/enc=false/nodes=3/cpu=32/size=64kb was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-9599065-1681449501-91-n4cpu32,lifetime=12h0m0s,created=2023-04-14t11_43_06z --metadata-from-file startup-script=/tmp/gce-startup-script3703680115 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-91-n4cpu32-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-91-n4cpu32-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-91-n4cpu32-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 8000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.1 @ 366c8feea58d696c2ad017839b3049d45aa0b855:

test kv95/enc=false/nodes=3/cpu=32/splt=0 was skipped due to (test_runner.go:609).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --local-ssd interface=NVME --machine-type n1-highcpu-32 --labels roachprod=true,cluster=teamcity-9599065-1681449501-96-n4cpu32,lifetime=12h0m0s,created=2023-04-14t11_49_22z --metadata-from-file startup-script=/tmp/gce-startup-script4079896780 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-96-n4cpu32-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-96-n4cpu32-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9599065-1681449501-96-n4cpu32-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 8000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-testeng TestEng Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jlinder @erikgrinaker @srosenberg @cockroach-teamcity