Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #108200

Closed
cockroach-teamcity opened this issue Aug 4, 2023 · 10 comments
Closed

roachtest: cluster_creation failed #108200

cockroach-teamcity opened this issue Aug 4, 2023 · 10 comments
Labels
branch-master Failures and bugs on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 4, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on master @ d87a04bf528cf3d8f242ceb539451cd14aa7a3cf:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-08-04t18_58_09z,roachprod=true,cluster=teamcity-11186090-1691128038-146-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script25840652 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11186090-1691128038-146-n9cpu8-geo-0001 teamcity-11186090-1691128038-146-n9cpu8-geo-0002 teamcity-11186090-1691128038-146-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-30360
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-dev-inf labels Aug 4, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 4, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 5, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 695d4ec85157a4da0c97237e6f16bf919eebf6c5:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-11200326-1691214430-153-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-05t19_10_28z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1459697257 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11200326-1691214430-153-n9cpu8-geo-0001 teamcity-11200326-1691214430-153-n9cpu8-geo-0002 teamcity-11200326-1691214430-153-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ f1623e0a2cffc81580ee3c3b71035efd854d89c9:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-08-06t19_27_08z,roachprod=true,cluster=teamcity-11207244-1691300832-152-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3795562881 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11207244-1691300832-152-n9cpu8-geo-0001 teamcity-11207244-1691300832-152-n9cpu8-geo-0002 teamcity-11207244-1691300832-152-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ f1623e0a2cffc81580ee3c3b71035efd854d89c9:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11213401-1691387291-154-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-07t19_31_44z --metadata-from-file startup-script=/tmp/gce-startup-script2516277930 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11213401-1691387291-154-n9cpu8-geo-0001 teamcity-11213401-1691387291-154-n9cpu8-geo-0002 teamcity-11213401-1691387291-154-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1382b26a97bf6f70a07d363dc319283c173359eb:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11229021-1691473676-149-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-08t19_08_59z --metadata-from-file startup-script=/tmp/gce-startup-script3799821623 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11229021-1691473676-149-n9cpu8-geo-0001 teamcity-11229021-1691473676-149-n9cpu8-geo-0002 teamcity-11229021-1691473676-149-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@jlinder jlinder added the T-testeng TestEng Team label Aug 9, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 9, 2023

cc @cockroachdb/test-eng

@jlinder jlinder removed the T-dev-inf label Aug 9, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ a3a389e464defd3d301a0b080435cf33502a02d2:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11245455-1691560191-150-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-09t19_49_31z --metadata-from-file startup-script=/tmp/gce-startup-script2577201407 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11245455-1691560191-150-n9cpu8-geo-0001 teamcity-11245455-1691560191-150-n9cpu8-geo-0002 teamcity-11245455-1691560191-150-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 10, 2023
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ a598a75435f6426236dde569191fbd95871f24ae:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-08-10t20_06_13z,roachprod=true,cluster=teamcity-11261527-1691646425-156-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script78236839 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11261527-1691646425-156-n9cpu8-geo-0001 teamcity-11261527-1691646425-156-n9cpu8-geo-0002 teamcity-11261527-1691646425-156-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6fa90e9e1003b0e0355c1ff42eb34b1b64ab36f2:

test restore/tpce/8TB/aws/nodes=10/cpus=8 was skipped due to (test_runner.go:725).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6i.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-cKZvlFc7kKgQ4-9DmL_-5hi-ems= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11277540-1691732912-32-n10cpu8-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11277540-1691732912-32-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-11T08:03:50Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11277540-1691732912-32-n10cpu8-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11277540-1691732912-32-n10cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-11T08:03:50Z}] --user-data file:///tmp/aws-startup-script2852330343 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3824076425 --output json: stderr: 
Parameter validation failed:
Unknown parameter in BlockDeviceMappings[0].Ebs: "Throughput", must be one of: DeleteOnTermination, Iops, SnapshotId, VolumeSize, VolumeType, KmsKeyId, Encrypted: exit status 252

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs
Copy link
Contributor

See cluster creation failure above please. cc @pavelkalinnikov @msbutler

@srosenberg srosenberg added X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue and removed C-test-failure Broken test (automatically or manually discovered). labels Aug 11, 2023
@srosenberg
Copy link
Member

Superseded by #108629
Fixed by #108644

In future, this issue should not be tagged with C-test-failure or release-blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
None yet
Milestone
23.2
Development

No branches or pull requests
4 participants
@jlinder @srosenberg @cockroach-teamcity @renatolabs