ec_deployment creation failure results in unmanaged resource

[marclop]

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

When creating a deployment with a valid configuration, if a deployment fails during creation, for example, due to lack of capacity or due to incorrect component settings, Terraform should still track the deployment and store at least the deployment ID in the state so the deployment can be cleaned up using Terraform and not have to manually issue an API call or use the UI to terminate the deployment.

Current Behavior

When creating a deployment with a valid configuration, if a deployment fails during creation, for example, due to lack of capacity or due to incorrect component settings, a subsequent terraform destroy won't remove the deployment from the ESS account. This results in the deployments "silently" being created, and not managed by terraform.

Terraform definition

this example will only work on the CFT Region

module "ec_deployment" {
  source                 = "github.com/elastic/apm-server//testing/infra/terraform/modules/ec_deployment?depth=1"
  deployment_name_prefix = "example_failure"
  region                 = "gcp-us-west2"
  docker_image_tag_override = {
    "elasticsearch" : "",
    "kibana" : "",
    "apm": "invalid_tag",
  }
}

Context

Caused a lot of waste in ESS to be created and not cleaned up.

Possible Solution

1. Call the POST /v1/deployment, and store the deployment in the state, or at least the deployment ID if the full state can't be stored at that point
2. Track the creation of the deployment.
3. Always read the deployment state regardless of the result of the plan change tracking.

Whether 1 is feasible or not is unclear to me, but after the POST /v1/deployment call has been issued, the deployment has most likely been created and allocated, at which point, terraform should know about it and keep track of it.

Your Environment

• Version used: 0.5.0
• Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: ESS
• Environment name and version (e.g. Go 1.9): N/A
• Server type and version: macOS
• Operating System and version: macOS
• Link to your project: https://github.com/elastic/apm-server