No method to ignore changes in DynamoDB GSI

[hashibot]

This issue was originally opened by @cypai as hashicorp/terraform#13393. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform Version

$ terraform -v
Terraform v0.8.8

Affected Resource(s)

• aws_dynamodb_table

Terraform Configuration Files

resource "aws_dynamodb_table" "FooBar" {
    name = "FooBar"
    read_capacity = 10
    write_capacity = 10
    hash_key = "foo-id"
    attribute {
        name = "foo-id"
        type = "S"
    }
    attribute {
        name = "bar-id"
        type = "N"
    }
    global_secondary_index {
        name = "bar-index"
        hash_key = "bar-id"
        read_capacity = 10
        write_capacity = 10
        projection_type = "ALL"
    }
    lifecycle {
        ignore_changes = ["read_capacity", "write_capacity"]
    }
}

References

• provider/aws: Don't always update DynamoDB read/write capacity terraform#5617

We would like to ignore read/write capacity for the GSI as well, but this configuration only ignores the read/write capacity of the top-level table. We weren't able to find any usable workaround, since we can't interpolate variables in the lifecycle, and wildcards are not supported.

Technically we could hardcode the randomly generated number like ignore_changes = ["global_secondary_index.3291674533.read_capacity"], but that's obviously a terrible solution since it would change every time we make a change.

[lra]

Ignoring the GSI with its ID is not a work-around, it does ignore the *_capacity but stiff diffs the whole GSI =(

[mcwqy9]

I've got some time this week to take a stab at this. I've done some golang in the past, but this would be my first contribution to this project. Can anyone suggest what might need to be done, at a high level? Would a good approach be to remove a GSI's capacity from the hash function that determines the number in "global_secondary_index.3291674533.read_capacity"? That would make that number stable through capacity changes, and allow an ignore to be hardcoded. Stab in the dark...

[playdoz]

I've been testing with the following and it seems to be working quite well, terraform will ignore the GSI read and write capacity (and unfortunately projection_type) changes but will pick up other changes like name, hash_key etc

lifecycle {
    ignore_changes = [
      "global_secondary_index",
      "read_capacity",
      "write_capacity",
    ]
  }

Not an ideal solution.

[mcwqy9]

Interesting. Let me try that on my end. What version of Terraform and what version of the AWS provider are you testing with?

…

On Nov 23, 2017 5:14 PM, "kimgov" ***@***.***> wrote: I've been testing with the following and it seems to be working quite well, terraform will ignore the GSI read and write capacity changes but will pick up other changes like name, hash_key etc lifecycle { ignore_changes = [ "global_secondary_index", "read_capacity", "write_capacity", ] } — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#671 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABV4JBMt-c1fQVEvg4hDVcWrtbo05m0Cks5s5gpvgaJpZM4N5B0u> .

[icybin]

Any update on this topic? What is the best terraform version to get out of this issue? I'm ussing 0.9.6 and I always see dynamodb things updated when I run terraform apply :(

[armanshan12]

Any updates on this issue?

[rsalmond]

So apparently the above PR fixed this issue, does anyone know how to reference the read/write capacity of a GSI in the ignore_changes field to make it happen?

[cypai]

The comments in the above PR say that it is just a refactoring, only that it makes it easier to address this issue. I don't think it's fixed yet.

[rsalmond]

@cypai ah you're right, thanks for pointing that out! I read the wrong issue number. I'm getting by with the workaround @kimgov provided above in the meantime.

[philipl]

@kimgov, I tried ignoring changes on "global_secondary_index" but it completely ignores everything. I can't modify a hash key, or even add a new GSI. Terraform ignores all those changes. You are saying that these kinds of changes are planned by terraform?

[playdoz]

Hi @philipl, I left my workaround in months ago and for me haven't had to make much changes to the underlying dynamodb table itself since. I do note your issue though that it ignores everything, to get around that in the early days I just commented out the 'global secondary index' line, terraform apply the change and then add that line back in.

I know it's not an ideal solution at all, and I really don't like the way terraform manages dynamodbs and in particular GSIs.

[andresvia]

Maybe global secondary indexes should be a separate terraform resource type:

mock:

resource "aws_dynamodb_table" "table" {
  name = "table"
  // .. etc
}

resource "aws_dynamodb_global_secondary_index" "gsi" {
  name = "gsi"
  table = "${aws_dynamodb_table.table.name}"
  read_capacity = "100"
  write_capacity = "100"
  lifecycle {
    ignore_changes = ["read_capacity", "write_capacity"]
  }
 // ... etc
}

[emmekappa]

Having the same issue and still not found any workaround.

[davehowell]

Maybe global secondary indexes should be a separate terraform resource type:

@andresvia
Seems like a good idea, it would just have to depend on a dynamodb table given the prefix of a GSI arn includes the table's arn. Alternatively it might be more consistent to have its own lifecycle block inside the global_secondary_index block.

Either way, it is pretty standard to use autoscaling policies, and when you have them it is also good practice to apply it to both the table and the GSI, not just the table, especially for writes, given most table writes imply also writing to the GSI.

This big note at the top of the docs for aws_dynamodb_table resource seems to acknowledge half of that:

Note: It is recommended to use lifecycle ignore_changes for read_capacity and/or write_capacity if there's autoscaling policy attached to the table.

[ben-bourdin451]

I have just started to work on making GSIs into a new resource to fix this issue but I can't say how long it's going to take since I don't have a huge amount of spare time. I thought I would mention it here in case anyone was thinking of doing the same.

I would greatly appreciate some guidance on how to manage this kind of change from a maintainer. I'm not sure if I should leave the existing GSI code in the table as it is and support both for now much like security group rules or deprecate it completely as a new version of the provider.

[davehowell]

I can't speak for the maintainers but supporting both options for an interim period is a friendly way of deprecating features. Raise a warning that the old way will not be supported in the future gives people a chance to update their code. I've seen another issue that claims the timeout settings don't work for GSIs either, although I can't verify it. I know that timeout settings do work for table resources though. I'm guessing that the lifecycle, timeout and all the other common attributes would then be supported and those issues fixed by your PR. Thank you for working on it, the community will very much appreciate it.

[bevanbennett]

Just chiming in that this is an active and annoying issue for us, as our secondary index autoscaling is constantly at war with terraform. Splitting out secondary indexes as a separate resource seems reasonable, if refactor-inducng. I think we would prefer if we could add ignore_changes lifecycle blocks inside the GSI blocks.

[andresvia]

For people wanting to workaround this issue changing the table billing mode to PAY_PER_REQUEST may help in some cases. (warning: you may get charged more money depending on your dynamodb usage pattern)

[tinder-jlee]

What's the current status on this issue? We're having a difficulty as well.

[ben-bourdin451]

I opened a PR to add the concept of a GSI as a new resource however as it stands this would break existing table configuration states.

I was hoping that someone with more experience in updating existing resource configurations could point me in the right direction. Haven't had any feedback yet. Not sure what is the best way of chasing this up.

If anyone has any suggestions please let me know

[shijiesheng]

Same issue here and it's quite annoying.

[red8888]

bump

It would not be as bad if I could at least set a variable to ignore_changes = ["global_secondary_index"] so switching that on and off during updates could be a temp workaround but that is not possible: There is no support for conditionally modifying ignore_changes either hashicorp/terraform#3116

We have to manually make all GSI changes after the first terraform apply and try to manually reconcile with terraform config. Almost makes it not usable for our GSI tables

[ben-bourdin451]

I probably should have followed up on this a while ago but I still don't know how to migrate an attribute into its own resource without breaking changes.

I did find this really helpful doc a while ago: https://www.terraform.io/docs/extend/best-practices/deprecations.html but it doesn't talk about this use case.
I would happily open a proposal to add something to the site docs on how to approach this type of change if anyone thinks that would be useful. AWS security group and security group rules are the only other example of this type of change that can think of at the moment.

At this point in time I'm not sure what else to do apart from maybe:

• join slack/irc/gitter to ask for help
  can't find any info on how to do this atm
• open a topic on the community forum
  not sure that would help more than this issue
• open a proposal to migrate GSIs to their own resource
  again this would probably be more confusing and split the conversation that stemmed from this thread

If anyone has any other suggestions or thoughts please share! 😄

[kopatsy]

As a workaround, we, at Verkada, built a custom provider (https://github.com/verkada/terraform-provider-gsi) that takes ownership of global secondary indexes in a separate resource. As described in the documentation, it requires adding a lifecycle ignore_change on both attribute and global_secondary_index on the aws_dynamodb_table resource. Local secondary indexes will not work properly with this provider at the moment (verkada/terraform-provider-gsi#11).

To facilitate the migration, we also introduced an auto_import flag on the provider that will make the first apply act as an import if an index with the same name already exists. Extra attention must be given on the subsequent plan as you've probably been ignoring all changes for a while now due to this issue and the next plan might require a replace if any drift was introduced.

I would recommend trying it on a test account/ table to get acquainted with the provider's behavior as it is obviously a workaround with the caveats I just described. Contributions and comments welcome. Hopefully it will save you as much time as it is saving us.

[cmawhorter]

every time i get a new computer i hit this issue because i forget it exists and stupidly install the latest tf.

terraform v0.14 doesn't have this issue.

if you're not paying attention this can be a huge problem because your tables will essentially be unavailable while the index rebuilds -- which can take minutes to hours.

[justinretzolk]

Hey everyone 👋 Thank you very much for your interest and continued feedback on this. This is something we’re monitoring and discussing in order to find the best solution for. At the moment, we’re currently limited by the functionality of the underlying API (something we discussed in more detail over on the pull request that was recently opened in order to try to fix this (#22513, thanks again to @danquack for his work on that!). That discussion includes some of the ideas presented here around a CustomizedDiff or separate resource, which we unfortunately feel won’t fix this in a way that we could guarantee wouldn’t have unintentional side effects.

With that said, we’ve started the conversation with the folks at AWS so that we can try to make progress towards a resolution. When we have more information, we’ll make sure to update this issue again.

[ewbankkit]

Relates hashicorp/terraform#26359.

[yasntrk]

Damn. I really love coming across 3.5yo bugs that are still unresolved.

Double it and give it to next person.

Still experiencing this bug.

[tarfeef101]

this is a known problem, with known solutions. see: mongodb/terraform-provider-mongodbatlas#888

maybe steal inspiration from how this was fixed?

[schmittjoaopedro]

Come on, Starship development started in 2019 and they launched in 2023 and this bug is still here?

[tl-alex-nicot]

humanity will land on mars first

[jacob-tolar-bridg]

humanity will land on mars first

s/land on/terraform/ ? 🙃

---

As a workaround, if you can plan/apply your terraform changes during a period when autoscaling won't be running (to avoid any race conditions - e.g. you can pause autoscaling before and unpause it afterwards), something like this might work?

data "aws_dynamodb_table" "tableinfo" {
  name = "my_table"
}

locals {
  idx = [for i in data.aws_dynamodb_table.tableinfo.global_secondary_index: i if i.name == "my_index_name"][0]
}

resource "aws_dynamodb_table" "my_dynamo_table" {
  name           = "my_table"
  ...
  global_secondary_index {
    name            = "my_index_name"
    read_capacity   = local.idx.read_capacity
    write_capacity  = local.idx.write_capacity
    ...
  }
}

This does at least sort of work (my plan passed showing no changes), but I don't do enough Terraform to know if this is an actually-works bad idea or a definitely-broken bad idea.

[troberto-bkly]

Come on, Starship development started in 2019 and they launched in 2023 and this bug is still here?

In 2024, Starship is already maneuvering at the launch station and we still cannot create GSI in DynamoDB without affecting the existing ones.