Failure during planning phase when node pool resources are exhausted

[coryodaniel]

When a machine type is exhausted in a zone, terraform plan fails making it impossible to destroy, taint, or change the nodepool.

To continue you have to delete the node pool in the GCP UI or CLI and then remove it from the terraform state.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

• google_container_node_pool

Terraform Configuration Files

Applies to any GCP machine type that is exhausted.

Debug Output

Error: Error reading NodePool "pool-worker-preemptible" from cluster "CLUSTER_NAME": Nodepool "pool-worker-preemptible" has status "RUNNING_WITH_ERROR" with message "asia-northeast1-a: Deploy error: Not all instances running in IGM after 4m4.752299662s. Expect 1. Current errors: [ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS]: Instance 'INSTANCE_NAME' creation failed: The zone 'projects/ox-delivery-prod/zones/asia-northeast1-a' does not have enough resources available to fulfill the request. '(resource type:compute)'."

Expected Behavior

The plan reads the current state and does not fail.

Actual Behavior

The plan reads and encounters an error on the resource and crashes.

Steps to Reproduce

1. terraform apply
2. Wait for the machine type to be exhausted in a region
3. terraform plan # this will crash

Important Factoids

Machine type is exhausted. Currently requesting n1-standard-96 pvms. These are easy to exhaust (and I know we shouldn't be using them), but the bug exists for other machine types as well.

References

• May be related to terraform import fails with the zone does not have enough resources available to fulfill the request #3304

[paddycarver]

I think this is definitely related to #3304. I think the crux of the problem is that we need to detect an error for when resources are exhausted, but we need to distinguish that from other errors, and I think that's trickier than it would appear at first blush.

[rileykarson]

Closing as dupe of #3304

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!