google_compute_url_map: cannot delete google_compute_backend_service

[JanMa]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.12.9

• provider.google v2.16.0

Affected Resource(s)

• google_compute_url_map
• google_compute_backend_service

Terraform Configuration Files

provider "google" {
  project = var.project
  region  = var.region
}

locals {
  instances = {
    # name: zone
    "a-vm" : "europe-west1-b",
    "b-vm" : "europe-west1-c",
    "c-vm" : "europe-west1-d",
    "d-vm" : "europe-west1-b",
    "e-vm" : "europe-west1-c",
  }
}

resource "google_compute_instance" "default" {
  for_each     = local.instances
  name         = each.key
  machine_type = "f1-micro"
  zone         = each.value

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-10"
    }
  }

  network_interface {
    network    = var.network
    subnetwork = var.subnetwork
  }
  metadata_startup_script = <<EOF
#! /bin/bash
apt-get update
apt-get install apache2 -y
a2ensite default-ssl
a2enmod ssl
vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://169.254.169.254/computeMetadata/v1/instance/name)"
echo "Page served from: $vm_hostname" | \
tee /var/www/html/index.html
systemctl restart apache2
EOF

}

resource "google_compute_instance_group" "default" {
  for_each = local.instances
  name     = each.key
  zone     = each.value

  instances = [google_compute_instance.default[each.key].self_link]

  named_port {
    name = "http"
    port = "80"
  }
}

resource "google_compute_backend_service" "default" {
  for_each = local.instances
  name     = each.key

  backend {
    group = google_compute_instance_group.default[each.key].self_link
  }

  health_checks = [google_compute_http_health_check.default[each.key].self_link]
  port_name     = "http"
}

resource "google_compute_http_health_check" "default" {
  for_each           = local.instances
  name               = "${each.key}-hc"
  request_path       = "/"
  check_interval_sec = 1
  timeout_sec        = 1
}


resource "google_compute_url_map" "default" {
  name            = "default"
  default_service = google_compute_backend_service.default["a-vm"].self_link

  dynamic "host_rule" {
    for_each = keys(local.instances)
    content {
      hosts        = ["${host_rule.value}.example.com"]
      path_matcher = host_rule.value
    }
  }

  dynamic "path_matcher" {
    for_each = keys(local.instances)
    content {
      name            = path_matcher.value
      default_service = google_compute_backend_service.default[path_matcher.value].self_link
    }
  }
}

Expected Behavior

When an element from the local.instances group gets removed, Terraform should remove all
corresponding resources and update the google_compute_url_map resource accordingly.

Actual Behavior

Instead of first updating the google_compute_url_map, Terraform tries to delete the google_compute_backend_service which causes the following error:

Error: Error reading BackendService: googleapi: Error 400: The backend_service resource 'projects/<project>/global/backendServices/d-vm' is already being used by 'projects/<project>/global/urlMaps/default', resourceInUse
ByAnotherResource                                                                                                                                                                                                                            

I tried various combinations of lifecycle { create_before_destroy = true } and explicit depends_on blocks but i can cannot get this working.

Note: Adding further entries to the local.instances list works just fine.

Steps to Reproduce

1. terraform apply

• make sure to pass all required variables for project, region, network and subnetwork

2. delete line "d-vm": "europe-west1-b"
3. terraform apply again

b/308569900

[JanMa]

So I have found a way around the issue by using the random provider that comes with Terraform.
You can do something like this

resource "random_id" "url_map" {
  keepers = {
    # Generate a new id each time the instance list changes
    instances = base64encode(jsonencode(local.instances))
  }

  byte_length = 1
}

resource "google_compute_url_map" "default" {
  name            = "default-${random_id.url_map.hex}"
  default_service = google_compute_backend_service.default["a-vm"].self_link

  dynamic "host_rule" {
    for_each = keys(jsondecode(base64decode(random_id.url_map.keepers.instances)))
    content {
      hosts        = ["${host_rule.value}.example.com"]
      path_matcher = host_rule.value
    }
  }

  dynamic "path_matcher" {
    for_each = keys(jsondecode(base64decode(random_id.url_map.keepers.instances)))
    content {
      name            = path_matcher.value
      default_service = google_compute_backend_service.default[path_matcher.value].self_link
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

This will cause the url-map to be recreated every time the local.instances map changes because the random_id resource will change and therefore the name of the url-map will also change.

It is not really solving the underlying issue of the wrong order of deletion of resources though.

[edwardmedia]

@JanMa your provider's version is old. Can you upgrade it and then try to see if you can repro the issue? If you still see the issue, please attach the full debug log and plan for both tf apply and tf delete. Thanks

[JanMa]

Hello @edwardmedia , the described issue still occurs with the latest provider version. After creating all resources with the attached config file and then trying to remove the line "b-vm" : "europe-west1-c", from the local.instances variable, I still get the following error:

Error: Error reading BackendService: googleapi: Error 400: The backend_service resource 'projects/<my-project>/global/backendServices/b-vm' is already being used by 'projects/<my-project>/global/urlMaps/default', resourceInUseByAnotherResource

If I have some more time I will add the full debug output of all commands.

[edwardmedia]

@JanMa waiting for your debug logs. Thanks

[edwardmedia]

@ndmckinley I can repro this issue on v3.20.0.

[nat-henderson]

Unfortunately this issue is a combination between a core terraform issue and a GCP issue, and is not resolvable. :(

The two issues are:

• Terraform does not allow a resource to reorder itself in the plan, which means it will always decide to act on the url map after the backend service.
• GCP does not provide a way to know which url maps are using backend services, so the url map cannot be updated in the backend service resource.

[zymotik]

Unfortunately this issue is a combination between a core terraform issue and a GCP issue, and is not resolvable. :(

The two issues are:

• Terraform does not allow a resource to reorder itself in the plan, which means it will always decide to act on the url map after the backend service.
• GCP does not provide a way to know which url maps are using backend services, so the url map cannot be updated in the backend service resource.

Hi @nat-henderson, do these two issues you mention still exist in 2023?

[nat-henderson]

@zymotik Hi there - unfortunately so; the former is unlikely to change, in my view.

For the latter, you can check https://cloud.google.com/compute/docs/reference/rest/v1/backendServices and see that there is no reference to url maps, and no listUsers or similar method on the left hand side of the page.

[45413]

There is an open feature request to get the required 'in use by' information added to the backendServices.get API request: https://issuetracker.google.com/issues/257459514. If you are interest please +1 or add a note to the ticket to hopefully motivate Google Product Team to get this implemented.

[x-yuri]

So unless I want to use the random trick and want to change e.g. load_balancing_scheme I need to comment out the following resources:

google_compute_global_forwarding_rule
google_compute_target_http_proxy
google_compute_url_map
google_compute_backend_service

terraform apply, then uncomment them, change the load balancing scheme and terraform apply again?

[btkelly]

The random id solution didn't change the error I receive as it's not allowing the deletion at all due to a backend service using it. In my use case, I am replacing a backend serverless NEG with a different one but for the same path via the URL map. I receive the same error as linked above seemingly caused by the same issue of order of operations. I'm not sure what a solution here is except manually going in and deleting items in GCP and then running Terraform to recreate them which is not ideal.

[derhally]

still happens with terraform 1.5.1 and TGP 5.26.0

[45413]

Unfortunately it is unlikely this will ever get resolved in the terraform provided until the API feature request is completed https://issuetracker.google.com/issues/257459514.

Our best option is to get as many people and companies to post on this feature request issue, and /or reach out to Google reps until it gets implemented.

[seils]

The response body for the backendServices.get method contains a list of URL maps that reference the backend service in usedBy[].

[jacobshirley]

With their latest update, I believe this could now be fixed

[brianwarsing]

Checked v5.29.1 and it's still an issue.

[gus]

Checked v5.29.1 and it's still an issue.

v5.33.0 as well. Noting that this was stated on 2024-05-14, which is exactly what I need atm:

in a few months there will be an update allowing seamless "EXTERNAL" -> "EXTERNAL_MANAGED" migration of Backend Servcies without the need of replacing them.

[natedogith1]

Unfortunately it is unlikely this will ever get resolved in the terraform provided until the API feature request is completed https://issuetracker.google.com/issues/257459514.

Our best option is to get as many people and companies to post on this feature request issue, and /or reach out to Google reps until it gets implemented.

It looks like this was implemented in May. Can someone look into updating the provider to make use of it?

[gitperr]

I am running into it in 6.7.0 as well.

[polinabee]

I'm also having this issue, on 6.6.0 but I guess it is an issue in the most recent ones too

[maxi-cit]

Hello folks, this dependencies issue are commonly solved by adding a lifecycle block. Random names should not be required.

I will open a PR soon.

[gitperr]

@maxi-cit did you end up writing a solution with lifecycle block somewhere? Because I could not make a solution with lifecycle block before, so I'm interested to see this.

Also, I tested with 6.7.0 today, and it is working as intended at the moment. I did not use lifecycle block either. It might be that there were some changes in google's api rather than the provider.

[Jojoooo1]

Also interested in the solution ?

[bahag-klickst]

@maxi-cit I would also be interested in a solution using lifecycle blocks as I was not able to solve it with any lifecycle block combination I have tried so far.

I also tested it with provider version 6.14.1 today and the issue still exists and gives us a hard time when it comes to removing backends from a load balancer configuration.