Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #101289

Closed
cockroach-teamcity opened this issue Apr 12, 2023 · 251 comments
Closed

roachtest: cluster_creation failed #101289

cockroach-teamcity opened this issue Apr 12, 2023 · 251 comments
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 12, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test cdc/ledger was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554151-1681277484-43-n4cpu16,lifetime=12h0m0s,created=2023-04-12t09_45_00z, --metadata-from-file startup-script=/tmp/gce-startup-script596628129 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554151-1681277484-43-n4cpu16-0001 teamcity-9554151-1681277484-43-n4cpu16-0002 teamcity-9554151-1681277484-43-n4cpu16-0003 teamcity-9554151-1681277484-43-n4cpu16-0004]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-26894
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 12, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Apr 12, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 12, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test c2c/tpcc/warehouses=500/duration=10/cutover=5 was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-pro-fips-2004-focal-v20230302 --image-project ubuntu-os-pro-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554162-1681277516-40-n9cpu8,lifetime=12h0m0s,created=2023-04-12t09_45_35z, --metadata-from-file startup-script=/tmp/gce-startup-script1802429020 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554162-1681277516-40-n9cpu8-0001 teamcity-9554162-1681277516-40-n9cpu8-0002 teamcity-9554162-1681277516-40-n9cpu8-0003 teamcity-9554162-1681277516-40-n9cpu8-0004 teamcity-9554162-1681277516-40-n9cpu8-0005 teamcity-9554162-1681277516-40-n9cpu8-0006 teamcity-9554162-1681277516-40-n9cpu8-0007 teamcity-9554162-1681277516-40-n9cpu8-0008 teamcity-9554162-1681277516-40-n9cpu8-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-40-n9cpu8-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 12, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test cdc/crdb-chaos was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-pro-fips-2004-focal-v20230302 --image-project ubuntu-os-pro-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554162-1681277516-46-n4cpu16,lifetime=12h0m0s,created=2023-04-12t09_52_18z, --metadata-from-file startup-script=/tmp/gce-startup-script830702183 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554162-1681277516-46-n4cpu16-0001 teamcity-9554162-1681277516-46-n4cpu16-0002 teamcity-9554162-1681277516-46-n4cpu16-0003 teamcity-9554162-1681277516-46-n4cpu16-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-46-n4cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-46-n4cpu16-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test cdc/tpcc-1000 was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-pro-fips-2004-focal-v20230302 --image-project ubuntu-os-pro-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,lifetime=12h0m0s,created=2023-04-12t10_20_06z,roachprod=true,cluster=teamcity-9554162-1681277516-53-n4cpu16, --metadata-from-file startup-script=/tmp/gce-startup-script1105716752 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554162-1681277516-53-n4cpu16-0001 teamcity-9554162-1681277516-53-n4cpu16-0002 teamcity-9554162-1681277516-53-n4cpu16-0003 teamcity-9554162-1681277516-53-n4cpu16-0004]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 12, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test clearrange/checks=false/rangeTs=true was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,created=2023-04-12t10_19_04z,roachprod=true,cluster=teamcity-9554151-1681277484-49-n10cpu16,lifetime=12h0m0s, --metadata-from-file startup-script=/tmp/gce-startup-script2367122512 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554151-1681277484-49-n10cpu16-0001 teamcity-9554151-1681277484-49-n10cpu16-0002 teamcity-9554151-1681277484-49-n10cpu16-0003 teamcity-9554151-1681277484-49-n10cpu16-0004 teamcity-9554151-1681277484-49-n10cpu16-0005 teamcity-9554151-1681277484-49-n10cpu16-0006 teamcity-9554151-1681277484-49-n10cpu16-0007 teamcity-9554151-1681277484-49-n10cpu16-0008 teamcity-9554151-1681277484-49-n10cpu16-0009 teamcity-9554151-1681277484-49-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554151-1681277484-49-n10cpu16-0001].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test clearrange/checks=true/rangeTs=false was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554151-1681277484-52-n10cpu16,lifetime=12h0m0s,created=2023-04-12t10_28_09z, --metadata-from-file startup-script=/tmp/gce-startup-script225796789 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554151-1681277484-52-n10cpu16-0001 teamcity-9554151-1681277484-52-n10cpu16-0002 teamcity-9554151-1681277484-52-n10cpu16-0003 teamcity-9554151-1681277484-52-n10cpu16-0004 teamcity-9554151-1681277484-52-n10cpu16-0005 teamcity-9554151-1681277484-52-n10cpu16-0006 teamcity-9554151-1681277484-52-n10cpu16-0007 teamcity-9554151-1681277484-52-n10cpu16-0008 teamcity-9554151-1681277484-52-n10cpu16-0009 teamcity-9554151-1681277484-52-n10cpu16-0010]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test clearrange/checks=false/rangeTs=false was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-pro-fips-2004-focal-v20230302 --image-project ubuntu-os-pro-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554162-1681277516-56-n10cpu16,lifetime=12h0m0s,created=2023-04-12t10_24_46z, --metadata-from-file startup-script=/tmp/gce-startup-script3098620901 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554162-1681277516-56-n10cpu16-0001 teamcity-9554162-1681277516-56-n10cpu16-0002 teamcity-9554162-1681277516-56-n10cpu16-0003 teamcity-9554162-1681277516-56-n10cpu16-0004 teamcity-9554162-1681277516-56-n10cpu16-0005 teamcity-9554162-1681277516-56-n10cpu16-0006 teamcity-9554162-1681277516-56-n10cpu16-0007 teamcity-9554162-1681277516-56-n10cpu16-0008 teamcity-9554162-1681277516-56-n10cpu16-0009 teamcity-9554162-1681277516-56-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-56-n10cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-56-n10cpu16-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test clearrange/checks=false/rangeTs=true was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-pro-fips-2004-focal-v20230302 --image-project ubuntu-os-pro-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554162-1681277516-60-n10cpu16,lifetime=12h0m0s,created=2023-04-12t10_36_49z, --metadata-from-file startup-script=/tmp/gce-startup-script920528184 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554162-1681277516-60-n10cpu16-0001 teamcity-9554162-1681277516-60-n10cpu16-0002 teamcity-9554162-1681277516-60-n10cpu16-0003 teamcity-9554162-1681277516-60-n10cpu16-0004 teamcity-9554162-1681277516-60-n10cpu16-0005 teamcity-9554162-1681277516-60-n10cpu16-0006 teamcity-9554162-1681277516-60-n10cpu16-0007 teamcity-9554162-1681277516-60-n10cpu16-0008 teamcity-9554162-1681277516-60-n10cpu16-0009 teamcity-9554162-1681277516-60-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-60-n10cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554162-1681277516-60-n10cpu16-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test decommission/randomized was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-4 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554151-1681277484-67-n6cpu4,lifetime=12h0m0s,created=2023-04-12t11_09_44z, --metadata-from-file startup-script=/tmp/gce-startup-script2110029102 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554151-1681277484-67-n6cpu4-0001 teamcity-9554151-1681277484-67-n6cpu4-0002 teamcity-9554151-1681277484-67-n6cpu4-0003 teamcity-9554151-1681277484-67-n6cpu4-0004 teamcity-9554151-1681277484-67-n6cpu4-0005 teamcity-9554151-1681277484-67-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-9554151-1681277484-67-n6cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test decommissionBench/nodes=4/warehouses=1000 was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,lifetime=12h0m0s,created=2023-04-12t11_16_46z,roachprod=true,cluster=teamcity-9554151-1681277484-71-n5cpu16, --metadata-from-file startup-script=/tmp/gce-startup-script703758065 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554151-1681277484-71-n5cpu16-0001 teamcity-9554151-1681277484-71-n5cpu16-0002 teamcity-9554151-1681277484-71-n5cpu16-0003 teamcity-9554151-1681277484-71-n5cpu16-0004 teamcity-9554151-1681277484-71-n5cpu16-0005]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 12, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ c8a4703f9853a442ac676b44d074b43eb387f60c:

test decommissionBench/nodes=4/warehouses=1000/while-down was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n1-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-9554151-1681277484-75-n5cpu16,lifetime=12h0m0s,created=2023-04-12t11_25_20z, --metadata-from-file startup-script=/tmp/gce-startup-script3755719111 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-9554151-1681277484-75-n5cpu16-0001 teamcity-9554151-1681277484-75-n5cpu16-0002 teamcity-9554151-1681277484-75-n5cpu16-0003 teamcity-9554151-1681277484-75-n5cpu16-0004 teamcity-9554151-1681277484-75-n5cpu16-0005]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'CPUS' exceeded.  Limit: 7200.0 in region us-east1.
	metric name = compute.googleapis.com/cpus
	limit name = CPUS-per-project-region
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 84905b8396e7cd7ef999b264faaafe13110d631a:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9616167-1681536170-124-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-15t17_19_44z, --metadata-from-file startup-script=/tmp/gce-startup-script1149329961 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-9616167-1681536170-124-n9cpu8-geo-0007 teamcity-9616167-1681536170-124-n9cpu8-geo-0008 teamcity-9616167-1681536170-124-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6cce6c746150307a9ecf3b529dfd633a6985c110:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9627078-1681622387-128-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-16t17_21_24z, --metadata-from-file startup-script=/tmp/gce-startup-script3656377900 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-9627078-1681622387-128-n9cpu8-geo-0007 teamcity-9627078-1681622387-128-n9cpu8-geo-0008 teamcity-9627078-1681622387-128-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6cce6c746150307a9ecf3b529dfd633a6985c110:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,lifetime=12h0m0s,created=2023-04-17t06_01_44z,roachprod=true,cluster=teamcity-9637843-1681668600-125-n9cpu8-geo, --metadata-from-file startup-script=/tmp/gce-startup-script1745438865 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-9637843-1681668600-125-n9cpu8-geo-0007 teamcity-9637843-1681668600-125-n9cpu8-geo-0008 teamcity-9637843-1681668600-125-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 6cce6c746150307a9ecf3b529dfd633a6985c110:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,lifetime=12h0m0s,created=2023-04-17t17_34_45z,roachprod=true,cluster=teamcity-9638532-1681708911-132-n9cpu8-geo, --metadata-from-file startup-script=/tmp/gce-startup-script2768276040 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-9638532-1681708911-132-n9cpu8-geo-0004 teamcity-9638532-1681708911-132-n9cpu8-geo-0005 teamcity-9638532-1681708911-132-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs renatolabs removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Apr 17, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 12607684868d04f957b3385f142e4e8b9b55c799:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9658510-1681795204-120-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-18t17_29_38z, --metadata-from-file startup-script=/tmp/gce-startup-script4105505876 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-9658510-1681795204-120-n9cpu8-geo-0007 teamcity-9658510-1681795204-120-n9cpu8-geo-0008 teamcity-9658510-1681795204-120-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 2f86475f2e3d713b4c8c976c34dba99acce7c4c7:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9680367-1681881616-139-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-19t17_36_42z, --metadata-from-file startup-script=/tmp/gce-startup-script2520675249 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-9680367-1681881616-139-n9cpu8-geo-0001 teamcity-9680367-1681881616-139-n9cpu8-geo-0002 teamcity-9680367-1681881616-139-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ ad6ce866ea3b5c5bb47ba9a0ac19b721a0c98add:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9699489-1681968016-136-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-20t17_34_15z, --metadata-from-file startup-script=/tmp/gce-startup-script2782900443 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-9699489-1681968016-136-n9cpu8-geo-0001 teamcity-9699489-1681968016-136-n9cpu8-geo-0002 teamcity-9699489-1681968016-136-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 2c34f145b83f771667baf3f7ef8633e7e1cd2ad6:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9721250-1682054508-130-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-21t17_15_18z, --metadata-from-file startup-script=/tmp/gce-startup-script2384407304 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-9721250-1682054508-130-n9cpu8-geo-0007 teamcity-9721250-1682054508-130-n9cpu8-geo-0008 teamcity-9721250-1682054508-130-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 01f129a59ab66166d6ef7189a8930f2b37cced74:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:678).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n1-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-9738195-1682140836-127-n9cpu8-geo,lifetime=12h0m0s,created=2023-04-22t17_08_19z, --metadata-from-file startup-script=/tmp/gce-startup-script3950306230 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-9738195-1682140836-127-n9cpu8-geo-0001 teamcity-9738195-1682140836-127-n9cpu8-geo-0002 teamcity-9738195-1682140836-127-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 81e44ab558b20b0ff55020ff05621c0ef43b4241:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-11016945-1690004674-155-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-07-22t17_25_15z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3119740307 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-11016945-1690004674-155-n9cpu8-geo-0007 teamcity-11016945-1690004674-155-n9cpu8-geo-0008 teamcity-11016945-1690004674-155-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5fe2141d9f7c5e429e3e589a4dd722aa12840e53:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11023525-1690091012-143-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-07-23t17_07_22z --metadata-from-file startup-script=/tmp/gce-startup-script3446005013 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11023525-1690091012-143-n9cpu8-geo-0004 teamcity-11023525-1690091012-143-n9cpu8-geo-0005 teamcity-11023525-1690091012-143-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ 5fe2141d9f7c5e429e3e589a4dd722aa12840e53:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11030293-1690177540-145-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-07-24t17_16_44z --metadata-from-file startup-script=/tmp/gce-startup-script717505212 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11030293-1690177540-145-n9cpu8-geo-0001 teamcity-11030293-1690177540-145-n9cpu8-geo-0002 teamcity-11030293-1690177540-145-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv0/enc=false/nodes=3/cpu=32/size=4kb was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T05:52:30Z},{Key=Name,Value=teamcity-11045339-1690263864-09-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-09-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T05:52:30Z},{Key=Name,Value=teamcity-11045339-1690263864-09-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-09-n4cpu32}] --user-data file:///tmp/aws-startup-script690870317 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3487118636 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv0/enc=false/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T05:57:27Z},{Key=Name,Value=teamcity-11045339-1690263864-15-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-15-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T05:57:27Z},{Key=Name,Value=teamcity-11045339-1690263864-15-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-15-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script118400406 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2186025402 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv0/enc=false/nodes=3/cpu=32/splt=0 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-11045339-1690263864-18-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T05:58:45Z},{Key=Name,Value=teamcity-11045339-1690263864-18-n4cpu32-0001},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-11045339-1690263864-18-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T05:58:45Z},{Key=Name,Value=teamcity-11045339-1690263864-18-n4cpu32-0001},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script4221336871 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4247284379 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv0/enc=false/nodes=3/cpu=32/size=64kb was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:00:04Z},{Key=Name,Value=teamcity-11045339-1690263864-21-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-21-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:00:04Z},{Key=Name,Value=teamcity-11045339-1690263864-21-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-21-n4cpu32}] --user-data file:///tmp/aws-startup-script3259949382 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2685108228 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv0/enc=false/nodes=1/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11045339-1690263864-24-n2cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-24-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:01:26Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11045339-1690263864-24-n2cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-24-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:01:26Z}] --user-data file:///tmp/aws-startup-script183798551 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping373554539 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv0/enc=false/nodes=3/cpu=32/seq was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-27-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:02:44Z},{Key=Name,Value=teamcity-11045339-1690263864-27-n4cpu32-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-27-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:02:44Z},{Key=Name,Value=teamcity-11045339-1690263864-27-n4cpu32-0002}] --user-data file:///tmp/aws-startup-script164000634 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping495216309 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv95/enc=false/nodes=1/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-36-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:11:18Z},{Key=Name,Value=teamcity-11045339-1690263864-36-n2cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-36-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:11:18Z},{Key=Name,Value=teamcity-11045339-1690263864-36-n2cpu32-0001}] --user-data file:///tmp/aws-startup-script1301902765 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2647078236 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv95/enc=false/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-07-25T06:15:01Z},{Key=Name,Value=teamcity-11045339-1690263864-41-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-41-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-07-25T06:15:01Z},{Key=Name,Value=teamcity-11045339-1690263864-41-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-41-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script4033166900 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4050012321 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:16:21Z},{Key=Name,Value=teamcity-11045339-1690263864-44-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:16:21Z},{Key=Name,Value=teamcity-11045339-1690263864-44-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script3922847873 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1792035275 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv95/enc=false/nodes=3/cpu=32/seq was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:17:42Z},{Key=Name,Value=teamcity-11045339-1690263864-47-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-47-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:17:42Z},{Key=Name,Value=teamcity-11045339-1690263864-47-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-47-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2690412462 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping910718404 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv95/enc=false/nodes=3/cpu=32/size=64kb was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-50-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:19:06Z},{Key=Name,Value=teamcity-11045339-1690263864-50-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-50-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:19:06Z},{Key=Name,Value=teamcity-11045339-1690263864-50-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script1254886501 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1562274826 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test kv95/enc=false/nodes=3/cpu=32/splt=0 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-53-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:20:32Z},{Key=Name,Value=teamcity-11045339-1690263864-53-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-53-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T06:20:32Z},{Key=Name,Value=teamcity-11045339-1690263864-53-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script708519662 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2320032507 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:19:23Z},{Key=Name,Value=teamcity-11045339-1690263864-70-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-70-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:19:23Z},{Key=Name,Value=teamcity-11045339-1690263864-70-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-70-n4cpu32}] --user-data file:///tmp/aws-startup-script1052234032 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping945125113 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/B/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:20:51Z},{Key=Name,Value=teamcity-11045339-1690263864-73-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-73-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:20:51Z},{Key=Name,Value=teamcity-11045339-1690263864-73-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-73-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script729992781 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1163556800 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11045339-1690263864-76-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-76-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:22:16Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11045339-1690263864-76-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-76-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:22:16Z}] --user-data file:///tmp/aws-startup-script2814293789 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping896612649 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/C/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:23:37Z},{Key=Name,Value=teamcity-11045339-1690263864-79-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-79-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:23:37Z},{Key=Name,Value=teamcity-11045339-1690263864-79-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-79-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1990994614 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping705201537 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-82-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:25:02Z},{Key=Name,Value=teamcity-11045339-1690263864-82-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-82-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:25:02Z},{Key=Name,Value=teamcity-11045339-1690263864-82-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script1486186306 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping825396279 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/D/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-85-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:26:30Z},{Key=Name,Value=teamcity-11045339-1690263864-85-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-85-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:26:30Z},{Key=Name,Value=teamcity-11045339-1690263864-85-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script385791399 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2587804947 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/E/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-11045339-1690263864-88-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:27:58Z},{Key=Name,Value=teamcity-11045339-1690263864-88-n4cpu32-0002},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-11045339-1690263864-88-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:27:58Z},{Key=Name,Value=teamcity-11045339-1690263864-88-n4cpu32-0002},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2624485061 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping42668196 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:29:24Z},{Key=Name,Value=teamcity-11045339-1690263864-91-n4cpu32-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-91-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:29:24Z},{Key=Name,Value=teamcity-11045339-1690263864-91-n4cpu32-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-91-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2947442907 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1113326305 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/F/nodes=3/cpu=32 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:30:46Z},{Key=Name,Value=teamcity-11045339-1690263864-94-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-94-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:30:46Z},{Key=Name,Value=teamcity-11045339-1690263864-94-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11045339-1690263864-94-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3321295917 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping545158734 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cbKLGSiyqljtaWiSiJGv3Df6Qy4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-11045339-1690263864-97-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:32:10Z},{Key=Name,Value=teamcity-11045339-1690263864-97-n4cpu32-0001},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-11045339-1690263864-97-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-07-25T07:32:10Z},{Key=Name,Value=teamcity-11045339-1690263864-97-n4cpu32-0001},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3999714831 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2388596483 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1 @ fe6e361e4148bfee1d6e9ad12d0dddb9cd364dd0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11045338-1690263711-156-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-07-25t17_25_59z --metadata-from-file startup-script=/tmp/gce-startup-script3458303899 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-11045338-1690263711-156-n9cpu8-geo-0007 teamcity-11045338-1690263711-156-n9cpu8-geo-0008 teamcity-11045338-1690263711-156-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs renatolabs removed the C-test-failure Broken test (automatically or manually discovered). label Jul 25, 2023
@srosenberg
Copy link
Member

Superseded by #108533
Fixed by #108644

In future, this issue should not be tagged with C-test-failure or release-blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
4 participants
@srosenberg @tbg @cockroach-teamcity @renatolabs