TagsLocationTagBinding fails to delete if its parent is removed

[mKeRix]

Checklist

• I did not find a related open issue.
• I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

When you create a TagsLocationTagBinding that binds to a resource (parentRef), and then remove that resource, the TagsLocationTagBinding cannot be deleted anymore. Instead, it fails with:

Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing TagsLocationTagBinding "<id>": googleapi: Error 403: The caller does not have permission

(more detailed error message below)

This is because removing the resource also removes the binding, but the Terraform code underneath does not interpret the error correctly. The related issue is hashicorp/terraform-provider-google#15928. I contributed a fix for this in GoogleCloudPlatform/magic-modules#11857, but the vendored copy of the provider kept in this project does not have those changes yet.

Additional Diagnostic Information

This issue happens when the resource is deleted - no matter if it was done externally, or via KCC, since KCC does not have the same dependency graph that Terraform has. This is especially relevant for us since we are currently working on auto-generating the tag bindings for KCC resources, and plan to solve the clean up via ownerReferences. Since those only trigger dependent resource deletion after the main resource was removed, we always end up with ghost tag binding resources that error out as described.

Kubernetes Cluster Version

v1.28.13-gke.1049000

Config Connector Version

1.123.1

Config Connector Mode

cluster mode

Log Output

Status on the tag binding resource after deletion:

status:
  conditions:
  - lastTransitionTime: "2024-10-10T11:36:49Z"
    message: |-
      Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing TagsLocationTagBinding "<id>": googleapi: Error 403: The caller does not have permission
      Details:
      [
        {
          "@type": "type.googleapis.com/google.rpc.ResourceInfo",
          "description": "permission [storage.buckets.listTagBindings] required (or the resource may not exist in this location)",
          "resourceName": "<one-platform-id>"
        }
      ]
    reason: DeleteFailed
    status: "False"
    type: Ready
  name: <id>
  observedGeneration: 3

Controller logs:

{"severity":"info","timestamp":"2024-10-10T11:37:47.225Z","logger":"tagslocationtagbinding-controller","msg":"starting reconcile","resource":{"name":"test-binding","namespace":"default"}}                                                                                                                                                                         
{"severity":"info","timestamp":"2024-10-10T11:37:47.237Z","logger":"tagslocationtagbinding-controller","msg":"finalizing resource deletion","resource":{"name":"test-binding","namespace":"default"}}                                                                                                                                                                     
{"severity":"error","timestamp":"2024-10-10T11:37:47.303Z","msg":"Reconciler error","controller":"tagslocationtagbinding-controller","controllerGroup":"tags.cnrm.cloud.google.com","controllerKind":"TagsLocationTagBinding","TagsLocationTagBinding":{"name":"test-binding","namespace":"default"},"namespace":"default","name":"test-binding","reconcileID":"3a025afb-63e7-45ad-b4b6-d0543bcbb14b","error":"Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing TagsLocationTagBinding \"<id>\": googleapi: Error 403: The caller does not have permission\nDetails:\n[\n  {\n    \"@type\": \"type.googleapis.com/google.rpc.ResourceInfo\",\n    \"description\": \"permission [storage.buckets.listTagBindings] required (or the resource may not exist in this location)\",\n    \"resourceName\": \"<one-platform-id>\"\n  }\n]"}

Steps to reproduce the issue

1. Install KCC with the alpha manifest for TagsLocationTagBinding.
2. Replace the placeholders in the YAML with information that works for your test environment.
3. Apply the YAML.
4. Delete the StorageBucket resource, wait for deletion to finish.
5. Try to delete the TagsLocationTagBinding, observe the errors.

YAML snippets

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: <project-id>
  name: <bucket-name>
  namespace: default
spec:
  location: europe-west1
  publicAccessPrevention: enforced
  uniformBucketLevelAccess: true

---

apiVersion: tags.cnrm.cloud.google.com/v1alpha1
kind: TagsLocationTagBinding
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: <project-id>
  name: test-binding
  namespace: default
spec:
  location: europe-west1
  parentRef:
    external: //storage.googleapis.com/projects/_/buckets/<bucket-name>
  tagValueRef:
    external: tagValues/<pre-existing-tag-value-id>