In-place update of aws_eks_cluster causes recreation of other resources

[jurajseffer]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.7

• provider.aws v2.26.0
• provider.kubernetes v1.9.0
• provider.random v2.2.0
• provider.template v2.1.2

Affected Resource(s)

• aws_eks_cluster

Terraform Configuration Files

Adding enabled_cluster_log_types to an existing EKS cluster.

resource aws_eks_cluster eks {
  name                      = var.name
  role_arn                  = aws_iam_role.control_plane.arn
  version                   = var.kubernetes_version
  enabled_cluster_log_types = ["api", "audit"]

  vpc_config {
    ...
  }

  depends_on = [
    ...
  ]
}

data aws_iam_policy_document describe_cluster {
  statement {
    actions = [
      "eks:DescribeCluster",
    ]
    resources = [aws_eks_cluster.eks.arn]
  }
}

resource aws_iam_role_policy developer {
  role   = aws_iam_role.developer.id
  policy = data.aws_iam_policy_document.describe_cluster.json
}

Expected Behavior

EKS cluster is updated in-place and logs enabled with no other changes planned.

Actual Behavior

Adding enabled_cluster_log_types shows in plan as in-place update of EKS cluster where arn nor id will be changing, only the logs directive. However, other resources such as aws_launch_configuration and aws_iam_role_policy which interpolate the ARN or id of the aws_eks_cluster resource want to be updated or recreated.

[bflad]

Hi @jurajseffer 👋 Sorry you are running into trouble. Can you please provide the full terraform plan output for triage?

[jurajseffer]

@bflad

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
+/- create replacement and then destroy
 <= read (data resources)

Terraform will perform the following actions:

  # module.eks.data.aws_iam_policy_document.describe_cluster will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "describe_cluster"  {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "eks:DescribeCluster",
            ]
          + resources = [
              + "arn:aws:eks:***:cluster/test",
            ]
        }
    }

  # module.eks.data.template_file.workers_userdata will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "workers_userdata"  {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = "#!/bin/bash\nset -o xtrace\n/etc/eks/bootstrap.sh --apiserver-endpoint '${cluster_endpoint}' --b64-cluster-ca '${certificate_authority_data}' '${cluster_name}'\n"
      + vars     = {
          + "certificate_authority_data" = "***"
          + "cluster_endpoint"           = "https://***..eks.amazonaws.com"
          + "cluster_name"               = "***"
        }
    }

  # module.eks.aws_autoscaling_group.workers will be updated in-place
  ~ resource "aws_autoscaling_group" "workers" {
        arn                       = "arn:aws:autoscaling:***:autoScalingGroup:***:autoScalingGroupName/test-20190819164056971500000004"
        availability_zones        = [
            "***",
            "***",
        ]
        default_cooldown          = 300
        desired_capacity          = 1
        enabled_metrics           = []
        force_delete              = false
        health_check_grace_period = 300
        health_check_type         = "EC2"
        id                        = "test-20190819164056971500000004"
      ~ launch_configuration      = "test-20190927140853940100000001" -> (known after apply)
        load_balancers            = []
        max_size                  = 1
        metrics_granularity       = "1Minute"
        min_size                  = 1
        name                      = "test-20190819164056971500000004"
        name_prefix               = "test-"
        protect_from_scale_in     = false
        service_linked_role_arn   = "arn:aws:iam::***:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        suspended_processes       = []
        target_group_arns         = []
        termination_policies      = []
        vpc_zone_identifier       = [
            "***",
            "***",
        ]
        wait_for_capacity_timeout = "10m"

        tag {
            key                 = "Name"
            propagate_at_launch = true
        }
        tag {
            key                 = "kubernetes.io/cluster/test"
            propagate_at_launch = true
        }
    }

  # module.eks.aws_eks_cluster.eks will be updated in-place
  ~ resource "aws_eks_cluster" "eks" {
        arn                       = "arn:aws:eks:***:cluster/test"
        certificate_authority     = [
            {
                data = "***"
            },
        ]
        created_at                = "2019-08-19 16:29:44 +0000 UTC"
      ~ enabled_cluster_log_types = [
          - "api",
          - "audit",
        ]
        endpoint                  = "https://***.eks.amazonaws.com"
        id                        = "test"
        name                      = "test"
        platform_version          = "eks.3"
        role_arn                  = "arn:aws:iam::***:role/test-eks-control-plane"
        status                    = "ACTIVE"
        version                   = "1.13"

        vpc_config {
            endpoint_private_access = true
            endpoint_public_access  = true
            security_group_ids      = [
                "***",
            ]
            subnet_ids              = [
                "***",
                "***",
            ]
            vpc_id                  = "***"
        }
    }

  # module.eks.aws_iam_role_policy.admin will be updated in-place
  ~ resource "aws_iam_role_policy" "admin" {
        id     = "test-eks-admin:terraform-20190819164055809600000001"
        name   = "terraform-20190819164055809600000001"
      ~ policy = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = "eks:DescribeCluster"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:eks:***:cluster/test"
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        role   = "test-eks-admin"
    }

  # module.eks.aws_iam_role_policy.developer will be updated in-place
  ~ resource "aws_iam_role_policy" "developer" {
        id     = "test-eks-developer:terraform-20190819164055809900000002"
        name   = "terraform-20190819164055809900000002"
      ~ policy = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = "eks:DescribeCluster"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:eks:***:cluster/test"
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        role   = "test-eks-developer"
    }

  # module.eks.aws_launch_configuration.workers must be replaced
+/- resource "aws_launch_configuration" "workers" {
        associate_public_ip_address      = false
      ~ ebs_optimized                    = false -> (known after apply)
        enable_monitoring                = true
        iam_instance_profile             = "test-eks-workers"
      ~ id                               = "test-20190927140853940100000001" -> (known after apply)
        image_id                         = "***"
        instance_type                    = "t3.medium"
      + key_name                         = (known after apply)
      ~ name                             = "test-20190927140853940100000001" -> (known after apply)
        name_prefix                      = "test-"
        security_groups                  = [
            "***",
        ]
      ~ user_data_base64                 = "***" -> (known after apply) # forces replacement
      - vpc_classic_link_security_groups = [] -> null

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      ~ root_block_device {
            delete_on_termination = true
          ~ encrypted             = false -> (known after apply)
          ~ iops                  = 0 -> (known after apply)
            volume_size           = 60
          + volume_type           = (known after apply)
        }
    }

Plan: 1 to add, 4 to change, 1 to destroy.

E.g., removing the EKS logs config: AWS launch config recreation is forced by Terraform thinking that EKS values in user data template will change. This is its config value:
user_data_base64 = base64encode(data.template_file.workers_userdata.rendered)

[erikselin]

I think we just hit this same bug but all we did was change a tag on an s3 bucket. However, we had a similar outcome in that id and json on an aws_iam_policy_document could only be "known after apply" which resulted in us having to apply an aws_iam_role_policy with a policy that also was only "known after apply". Exactly like the provided example.

[denissimonovski]

@bflad I would like to work on this. I will need time though, to research everything between EKS and launch configs.

[jurajseffer]

@denissimonovski I think this is a general Terraform problem, not specific to this provider. Any value being interpolated into a template causes updates to resources using the template and their dependencies even though the value interpolated into the template has not changed nor has the template output.

[jasonstitt]

I encountered what I think is another outcome of this issue which created a fatal error in plan. I use the Helm provider with EKS and use the token from a aws_eks_cluster_auth. A change to enabled_cluster_log_types made the Helm provider unable to refresh the token and the plan just crashes after that. When I used my own token, I could see it wanted to recreate the OIDC provider and everything dependent on it.

[jw-maynard]

@bflad I'm also experiencing a fatal error in plan that seems like it could be this issue. Trying to upgrade the version of an EKS cluster which should be an in-place operation changing nothing like cluster names or ARNs and the kubernetes provider chokes trying to plan for things that already exist in the cluster and have no changes. I posted much more detail here: hashicorp/terraform-provider-kubernetes#1131 (comment)

I'm happy to share debug logs of a succesful run (when I don't try to change the cluster) and an unsuccessful run. I'm not super experienced at reading them but in the failed log I see one call to eks/DescribeCluster and in the successful one I see two.

[jw-maynard]

@bflad Even more digging on this issue posted here: hashicorp/terraform-provider-kubernetes#1131 (comment)

[justinretzolk]

Hi all 👋 As @jurajseffer mentioned above, this is (or was) a general Terraform issue related to the template_file data source needing to be refreshed at apply time due to dependencies on other resources. Any resources that were then dependent on the data source would have values that were known after apply that may have triggered undesirable changes, as seen in this issue.

There's a pretty informative discussion about this over here that I'd recommend checking out if you'd like a bit more information. The gist of it is:

• There were quite a few improvements in Terraform 0.13 and 0.14 (perhaps more since) that should help with this.
• Terraform 0.12 and beyond has a templatefile function built in, which may also improve the behavior, and replaces the template_file data source.

Since this is (or was) ultimately a Terraform issue rather than an AWS provider issue, I'm going to close this issue out as well. If you feel I've done this in error, please do let me know.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.