Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #112528

Closed
cockroach-teamcity opened this issue Oct 17, 2023 · 122 comments
Closed

roachtest: cluster_creation failed #112528

cockroach-teamcity opened this issue Oct 17, 2023 · 122 comments
Labels
O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 17, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-A6IedK6eMVdN8kbAxlLBejH7ntE= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12211563-1697520384-32-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-17T07:53:36Z},{Key=Name,Value=teamcity-12211563-1697520384-32-n4cpu96-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12211563-1697520384-32-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-17T07:53:36Z},{Key=Name,Value=teamcity-12211563-1697520384-32-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script3874849969 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2405617638 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-32675
@cockroach-teamcity cockroach-teamcity added branch-release-23.1.12-rc O-roachtest O-robot Originated from a bot. X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels Oct 17, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Oct 17, 2023
@blathers-crl blathers-crl bot added the T-testeng TestEng Team label Oct 17, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-A6IedK6eMVdN8kbAxlLBejH7ntE= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12211563-1697520384-35-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-17T07:55:31Z},{Key=Name,Value=teamcity-12211563-1697520384-35-n4cpu96-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12211563-1697520384-35-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-17T07:55:31Z},{Key=Name,Value=teamcity-12211563-1697520384-35-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script3116539502 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3177665488 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 17, 2023
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-10-17t17_54_13z,roachprod=true,cluster=teamcity-12211562-1697520236-153-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3946742035 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12211562-1697520236-153-n9cpu8-geo-0004 teamcity-12211562-1697520236-153-n9cpu8-geo-0005 teamcity-12211562-1697520236-153-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 17, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230919'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 18, 2023
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-12229615-1697606800-139-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-18t17_36_05z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2149579712 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12229615-1697606800-139-n9cpu8-geo-0004 teamcity-12229615-1697606800-139-n9cpu8-geo-0005 teamcity-12229615-1697606800-139-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 18, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ eee56b2379446c0a115e6d2cd30735a7efe4fad0:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12247705-1697693067-162-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-19t18_22_29z --metadata-from-file startup-script=/tmp/gce-startup-script1679276946 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12247705-1697693067-162-n9cpu8-geo-0001 teamcity-12247705-1697693067-162-n9cpu8-geo-0002 teamcity-12247705-1697693067-162-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-10-20t17_41_26z,roachprod=true,cluster=teamcity-12265135-1697779461-145-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script1272863228 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12265135-1697779461-145-n9cpu8-geo-0001 teamcity-12265135-1697779461-145-n9cpu8-geo-0002 teamcity-12265135-1697779461-145-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12281311-1697865805-149-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-21t17_47_50z --metadata-from-file startup-script=/tmp/gce-startup-script2095478614 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12281311-1697865805-149-n9cpu8-geo-0001 teamcity-12281311-1697865805-149-n9cpu8-geo-0002 teamcity-12281311-1697865805-149-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-g3qrmkHkuQx0v3cwkmraiYOk7bs= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292145-1697952191-33-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T07:45:38Z},{Key=Name,Value=teamcity-12292145-1697952191-33-n4cpu96-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292145-1697952191-33-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T07:45:38Z},{Key=Name,Value=teamcity-12292145-1697952191-33-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script489957644 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3988375313 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-g3qrmkHkuQx0v3cwkmraiYOk7bs= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-12292145-1697952191-36-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292145-1697952191-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T07:47:10Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-12292145-1697952191-36-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292145-1697952191-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T07:47:10Z}] --user-data file:///tmp/aws-startup-script1836771269 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping18784360 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12292144-1697952192-155-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-22t17_56_01z --metadata-from-file startup-script=/tmp/gce-startup-script942333679 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12292144-1697952192-155-n9cpu8-geo-0001 teamcity-12292144-1697952192-155-n9cpu8-geo-0002 teamcity-12292144-1697952192-155-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12302427-1698038641-146-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-23t17_40_46z --metadata-from-file startup-script=/tmp/gce-startup-script2810805658 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12302427-1698038641-146-n9cpu8-geo-0004 teamcity-12302427-1698038641-146-n9cpu8-geo-0005 teamcity-12302427-1698038641-146-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231022'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-highcpu-96 --labels usage=roachtest,roachprod=true,cluster=teamcity-12302427-1698038641-167-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2023-10-23t18_14_24z --metadata-from-file startup-script=/tmp/gce-startup-script1111824263 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-12302427-1698038641-167-n4cpu96-0001 teamcity-12302427-1698038641-167-n4cpu96-0002 teamcity-12302427-1698038641-167-n4cpu96-0003 teamcity-12302427-1698038641-167-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-12302427-1698038641-167-n4cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-12302427-1698038641-167-n4cpu96-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-12302427-1698038641-167-n4cpu96-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231022'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test costfuzz/seed-multi-region was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --labels usage=roachtest,arch=amd64,created=2023-10-24t11_11_19z,roachprod=true,cluster=teamcity-12320959-1698125022-59-n9cpu4-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1519496983 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-c teamcity-12320959-1698125022-59-n9cpu4-geo-0005]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231022'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-c zone. Alternatively, you can try your request again with a
      different VM hardware configuration or at a later time. For more information,
      see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-c
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12320959-1698125022-150-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-24t17_31_07z --metadata-from-file startup-script=/tmp/gce-startup-script4128859880 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12320959-1698125022-150-n9cpu8-geo-0007 teamcity-12320959-1698125022-150-n9cpu8-geo-0008 teamcity-12320959-1698125022-150-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lNOExTzfRltEpR9bGpaaZOvRRAc= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T07:51:15Z},{Key=Name,Value=teamcity-12340167-1698211454-32-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12340167-1698211454-32-n4cpu96}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T07:51:15Z},{Key=Name,Value=teamcity-12340167-1698211454-32-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12340167-1698211454-32-n4cpu96}] --user-data file:///tmp/aws-startup-script3705971332 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2125098045 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:758).func5: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lNOExTzfRltEpR9bGpaaZOvRRAc= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-12340167-1698211454-35-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T07:52:44Z},{Key=Name,Value=teamcity-12340167-1698211454-35-n4cpu96-0001},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-12340167-1698211454-35-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T07:52:44Z},{Key=Name,Value=teamcity-12340167-1698211454-35-n4cpu96-0001},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script1416067086 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping917232618 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc @ 6aefe827ba7e2b62ca066d732e4db8ce4e6c485e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-12340166-1698211430-147-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-25t17_45_48z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script298392984 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12340166-1698211430-147-n9cpu8-geo-0004 teamcity-12340166-1698211430-147-n9cpu8-geo-0005 teamcity-12340166-1698211430-147-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

test django was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-12647671-1699856685-96-n1cpu16,lifetime=12h0m0s,arch=amd64,created=2023-11-13t12_57_01z --metadata-from-file startup-script=/tmp/gce-startup-script3231536517 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-12647671-1699856685-96-n1cpu16-0001]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231030'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231030'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-8 VM instance with 8 local SSD(s) is currently unavailable
      in the us-east1-b zone, because of local_ssd_availability root cause(s). Consider
      trying your request in the us-east1-c, us-east1-d zone(s), which currently has
      capacity to accommodate your request. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:8
      vmType: n2-standard-8
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: local_ssd_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-8 VM instance with 8 local SSD(s) is currently unavailable
      in the us-east1-b zone, because of local_ssd_availability root cause(s). Consider
      trying your request in the us-east1-c, us-east1-d zone(s), which currently has
      capacity to accommodate your request. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:8
      vmType: n2-standard-8
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: local_ssd_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=8

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:758).func5: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12647671-1699856685-234-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-11-13t17_31_17z --metadata-from-file startup-script=/tmp/gce-startup-script3326312915 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12647671-1699856685-234-n9cpu8-geo-0001 teamcity-12647671-1699856685-234-n9cpu8-geo-0002 teamcity-12647671-1699856685-234-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.12-rc.FROZEN @ b9be9093ed43f2ede41c5f0b7b5cce36e564e7c3:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-32 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-highcpu-32
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs
Copy link
Contributor

Branch doesn't exist anymore, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
No open projects
Test Engineering
Status: Done
Milestone
23.1
Development

No branches or pull requests
2 participants
@cockroach-teamcity @renatolabs