IAM Policy Principals: Objects have changed outside of Terraform

[leonardoauribe]

After upgrading to TF v1.0.1, I began receiving the "refresh report" in the terraform plan/apply output. I've also updated to the latest AWS provider version, v3.55.0. I keep getting the output shown below for simple reordering of principals by AWS in our IAM/Key policies. I've ran "terraform apply -refresh-only" to no success.

Terraform Version

Terraform v1.0.1
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.55.0
+ provider registry.terraform.io/hashicorp/local v2.1.0

Terraform Configuration Files

resource "aws_kms_key" "tfstate" {
  lifecycle {
    ignore_changes = [
      policy
    ]
  }
  policy = data.aws_iam_policy_document.tfstate.json
}

data "aws_iam_policy_document" "tfstate" {
  policy_id = "key-consolepolicy-3"
  statement {
    actions = [
      "kms:*",
    ]
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = [
        "arn:aws-us-gov:iam::${var.account_id}:root",
      ]
    }
    resources = ["*"]
    sid       = "Enable IAM User Permissions"
  }
  statement {
    actions = [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:TagResource",
      "kms:UntagResource",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion",
    ]
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = [
        "arn:aws-us-gov:iam::${var.account_id}:role/role1",
        "arn:aws-us-gov:iam::${var.account_id}:role/role2",
        "arn:aws-us-gov:iam::${var.account_id}:role/role3",
        "arn:aws-us-gov:iam::${var.account_id}:role/role4",
      ]
    }
    resources = ["*"]
    sid       = "Allow access for Key Administrators"
  }
  statement {
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey",
    ]
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = [
        "arn:aws-us-gov:iam::${var.account_id}:role/role1",
        "arn:aws-us-gov:iam::${var.account_id}:role/role2",
        "arn:aws-us-gov:iam::${var.account_id}:role/role3",
        "arn:aws-us-gov:iam::${var.account_id}:role/role4",
      ]
    }
    resources = ["*"]
    sid       = "Allow use of the key"
  }
  statement {
    actions = [
      "kms:CreateGrant",
      "kms:ListGrants",
      "kms:RevokeGrant",
    ]
    condition {
      test     = "Bool"
      variable = "kms:GrantIsForAWSResource"
      values   = ["true"]
    }
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = [
        "arn:aws-us-gov:iam::${var.account_id}:role/role1",
        "arn:aws-us-gov:iam::${var.account_id}:role/role2",
        "arn:aws-us-gov:iam::${var.account_id}:role/role3",
        "arn:aws-us-gov:iam::${var.account_id}:role/role4",
      ]
    }
    resources = ["*"]
    sid       = "Allow attachment of persistent resources"
  }
}

Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.kms.aws_kms_key.tfstate has been changed
  ~ resource "aws_kms_key" "tfstate" {
        id                       = "5be68cce-d198-4f91-bccc-4ac45c12c191"
      ~ policy                   = jsonencode(
          ~ {
              ~ Statement = [
                    {
                        Action    = "kms:*"
                        Effect    = "Allow"
                        Principal = {
                            AWS = "arn:aws-us-gov:iam::384536020956:root"
                        }
                        Resource  = "*"
                        Sid       = "Enable IAM User Permissions"
                    },
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws-us-gov:iam::384536020956:role/role2",
                              - "arn:aws-us-gov:iam::384536020956:role/role4",
                              - "arn:aws-us-gov:iam::384536020956:role/role3",
                                "arn:aws-us-gov:iam::384536020956:role/role1",
                              + "arn:aws-us-gov:iam::384536020956:role/role3",
                              + "arn:aws-us-gov:iam::384536020956:role/role4",
                              + "arn:aws-us-gov:iam::384536020956:role/role2",
                            ]
                        }
                        # (4 unchanged elements hidden)
                    },
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws-us-gov:iam::384536020956:role/role2",
                              - "arn:aws-us-gov:iam::384536020956:role/role4",
                              - "arn:aws-us-gov:iam::384536020956:role/role3",
                                "arn:aws-us-gov:iam::384536020956:role/role1",
                              + "arn:aws-us-gov:iam::384536020956:role/role3",
                              + "arn:aws-us-gov:iam::384536020956:role/role4",
                              + "arn:aws-us-gov:iam::384536020956:role/role2",
                            ]
                        }
                        # (4 unchanged elements hidden)
                    },
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws-us-gov:iam::384536020956:role/role2",
                              - "arn:aws-us-gov:iam::384536020956:role/role4",
                              - "arn:aws-us-gov:iam::384536020956:role/role3",
                                "arn:aws-us-gov:iam::384536020956:role/role1",
                              + "arn:aws-us-gov:iam::384536020956:role/role3",
                              + "arn:aws-us-gov:iam::384536020956:role/role4",
                              + "arn:aws-us-gov:iam::384536020956:role/role2",
                            ]
                        }
                        # (5 unchanged elements hidden)
                    },
                ]
                # (2 unchanged elements hidden)
            }
        )
        tags                     = {}
        # (6 unchanged attributes hidden)
    }
  

Expected Behavior

Once state file has been refreshed the output should run clean.

Actual Behavior

Keep getting the same output shown above regardless of how many times i run "terraform apply -refresh-only".

References

[ewbankkit]

Potentially related: jen20/awspolicyequivalence#10.

[leonardoauribe]

Looks to be the same exact issue. Hoping for a solution soon.

[tedder]

I was working on creating an issue and ran across this. Here's my test file. I thought it might have been related to a change in the aws provider v3.38.0, but it isn't. I tried before that and what is current right now (3.64.0).

I thought that if I sorted the json list it might keep it from soft drift (requiring a refresh), but that's incorrect. After rerunning many times from scratch, the order of the principals list is deterministic at creation, not before. It makes me think that it's being stored by the IAM unique_id, not ARN. Or something.

I've tried it with a list of strings (eg "arn::...user1", "arn::...user2") and as a list of terraform resources. It's nondeterministic in both cases. It's also true for users, roles, pretty much anything stuffed in the identifiers list.

It's apparently not ordered by the IAM's unique_id, though. Here's the diff on a run, the internal order is "1 4 2 3".

                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws:iam::[...]:user/test4",
                                "arn:aws:iam::[...]:user/test1",
                              - "arn:aws:iam::[...]:user/test3",
                              + "arn:aws:iam::[...]:user/test4",
                                "arn:aws:iam::[...]:user/test2",
                              + "arn:aws:iam::[...]:user/test3",

I output the unique ids:

user1_uniqueid = "AIDAQKF6ZRELM2WOSBZJM"
user2_uniqueid = "AIDAQKF6ZRELAL6O4VS2A"
user3_uniqueid = "AIDAQKF6ZRELIEMFUM6B5"
user4_uniqueid = "AIDAQKF6ZRELI7VIR7PIW"

If those are alphasorted, the order is "2 4 3 1". Even reversed would be incorrect, "1 3 4 2".

--

Final note, if my experiment is revised to use unique_id instead of arn in the identifiers list, it will always hard drift, requiring an apply. Apply, plan, apply, plan, it will always show this drift:

                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws:iam::[...]:user/test1",
                              - "arn:aws:iam::[...]:user/test4",
                              - "arn:aws:iam::[...]:user/test2",
                              - "arn:aws:iam::[...]:user/test3",
                              + "AIDAQKF6ZRELOMN7J57RM",
                              + "AIDAQKF6ZRELKIKGPAFCS",
                              + "AIDAQKF6ZRELJG7TS4RKO",
                              + "AIDAQKF6ZRELDOMUHDWAG",

[tedder]

this is related to (or a dupe of) #11801.

Also similar to #20107, hashicorp/terraform#28803

[YakDriver]

Thanks for raised this problem. It should be resolved in #21969 so I'm going to close. If you're still experiencing any related problems, please re-open this or submit a new issue!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.