Bigtable: Avoid race between TableCreation and ModifyColumnFamily

[kevinsi4508]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

When creating a table, Terraform first invokes Bigtable table creation and then follow and then adds column families. If a table is replicated, adding column families can be expansive and can race with table creation. When that happens, you will see an error like this "Could not modify schema of ???, due to concurrent modifications.".

Workaround:

When creating replicated resources:

1. Create all resources in an un-replicated instance (with a single cluster).
2. Add a cluster to make the instance replicated.

When deleting replicated resources:

1. Remove a cluster so that the instance is un-replicated. Use "terraform apply" to remove the cluster.
2. Remove the rest of the un-replicated resources. Use "terraform destroy" to remove the rest of the resources.

The options to mitigate the issue:

1. Wait for table to "finish" before trying to add column families.
2. Don't wait for table to finish but keep retrying adding column families. This can result in running out of API quota if we don't backoff properly.
3. Combine table creation and adding column families into a single API call.

New or Affected Resource(s)

• google_bigtable_table

Potential Terraform Configuration

resource "google_bigtable_instance" "my-instance" {
  name                = "my-instance"
  
  cluster {
    cluster_id    = "my-cluster1"
    num_nodes     = "3"
    storage_type  = "HDD"
    zone          = "europe-west1-c"
  }

   cluster {
     cluster_id    = "my-cluster2"
     num_nodes     = "3"
     storage_type  = "HDD"
     zone          = "europe-west1-b"
   }
}

resource "google_bigtable_table" "my-table" {
  name          = "my-table"
  instance_name = google_bigtable_instance.my-instance.name
  column_family {
    family = "cf0"
  }
  column_family {
    family = "cf1"
  }
  column_family {
    family = "cf2"
  }
  column_family {
    family = "cf3"
  }
}

References

• #0000

[rileykarson]

@rileykarson to investigate for triage

[kevinsi4508]

Internal bug: b/244186196. This is for external tracking as requested.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.