Ignore order of BigQuery table columns

[paxorus]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.13.6
+ provider registry.terraform.io/hashicorp/google v3.32.0

Affected Resource(s)

• google_bigquery_table

Terraform Configuration Files

# First we apply this config:
resource "google_bigquery_table" "our_table" {
  dataset_id = google_bigquery_dataset.our_dataset.dataset_id
  table_id   = "our_table"
  schema     = <<EOF
  [
    { "mode": "REQUIRED", "name": "x", "type": "INTEGER" },
    { "mode": "REQUIRED", "name": "y", "type": "INTEGER" }
  ]
  EOF
}

# Then swap the columns, and plan:
resource "google_bigquery_table" "our_table" {
  dataset_id = google_bigquery_dataset.our_dataset.dataset_id
  table_id   = "our_table"
  schema     = <<EOF
  [
    { "mode": "REQUIRED", "name": "y", "type": "INTEGER" },
    { "mode": "REQUIRED", "name": "x", "type": "INTEGER" }
  ]
  EOF
}

Debug Output

We use Terraform Enterprise to plan and apply, and I'm not sure how to generate debug output with it.

Panic Output

None

Expected Behavior

The diff should be clean in v3.53.0, as it was in v3.39.0. We have confirmation from the GCP Support team that BigQuery columns are not guaranteed to stay in a particular order.

Actual Behavior

The plan shows that the google_bigquery_table resource needs to be changed, when using provider v3.53.0.

Steps to Reproduce

1. terraform init -upgrade to google = "~> 3.53.0".
2. terraform apply with the first TF config.
3. Switch to the second TF config.
4. terraform plan will show a diff (that we expect to be suppressed).
5. terraform apply will not do anything to the table. It will not re-order the columns, so it's a permadiff.
6. terraform init -upgrade to google = "~> 3.39.0".
7. terraform plan. The diff will now be clean.

Important Factoids

• We are using Terraform Enterprise.
• Sometimes our applications cause the BigQuery API to return the columns of a table in a scrambled order, even though the table schema was not modified in any way.

References

• The fix was introduced in v3.32.0 to resolve when updating BigQuery table, schema changes are always shown #4143. You can see the JSON deep-equals implementation sorts the columns by "name".
• A follow-up change to the fix was made in v3.39.0 to resolve JSONBytesEqual function go panic >= 3.32.0 for bigquery tables #7193, for the edge case where "name" is not present.
• The fix was removed recently in v3.53.0 to resolve bigquery table schema with field having missing mode attribute causes perma plan diff #7979, which adds a diff suppress for a separate issue and removes the sorting that was added earlier.

[edwardmedia]

Need to review this change

[afafara]

Combined with this introduced in 3.54.0, reordering causes table to be recreated on apply.

[rileykarson]

@ScottSuarez are you able to take a look and determine whether we should revert GoogleCloudPlatform/magic-modules#4400 or attempt to fix-forward? Also see #8258 (comment).

[ScottSuarez]

I'll /should/ be able to fix this. I wish we had a testcase for this scenario. I did not realize we could reorder the columns. Sorry about that folks ! We'll revert this for next release and work on proper fix for this scenario before checking in.

[rileykarson]

@paxorus: To be clear, you're seeing the same behaviour on both 3.32.0 and 3.54.0? What current provider version are you using?

Also, spoke with @ScottSuarez and our plan is to revert GoogleCloudPlatform/magic-modules#4400 for 3.55.0, attempt to fix the issues y'all have encountered, and then re-release in the future.

[paxorus]

3.53.0 behaves incorrectly, 3.39.0 behaves as expected, so different behavior. We have rolled back to and are currently on 3.39.0. We never tried 3.54.0.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!