A duplicate Security Group rule was found on...

[jaimehrubiks]

Description

This commit is causing me the following issue:

Terraform will perform the following actions:

  # module.eks.aws_security_group_rule.cluster_private_access_cidrs_source[0] will be created
  + resource "aws_security_group_rule" "cluster_private_access_cidrs_source" {
      + cidr_blocks              = [
          + "10.0.0.0/8",
          + "172.16.0.0/12",
          + "192.168.0.0/16",
        ]
      + from_port                = 443
      + id                       = (known after apply)
      + protocol                 = "tcp"
      + security_group_id        = "sg-0e241889ac74"
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 443
      + type                     = "ingress"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.eks.aws_security_group_rule.cluster_private_access_cidrs_source[0]: Creating...

Error: [WARN] A duplicate Security Group rule was found on (sg-0e241889ac74). This may be
a side effect of a now-fixed Terraform issue causing two security groups with
identical attributes but different source_security_group_ids to overwrite each
other in the state. See https://github.com/hashicorp/terraform/pull/2376 for more
information and instructions for recovery. Error message: the specified rule "peer: 10.0.0.0/8, TCP, from port: 443, to port: 443, ALLOW" already exists

  on .terraform/modules/eks/cluster.tf line 90, in resource "aws_security_group_rule" "cluster_private_access_cidrs_source":
  90: resource "aws_security_group_rule" "cluster_private_access_cidrs_source" {


Releasing state lock. This may take a few moments...

Versions

module version 17.0.2

Terraform v0.14.11

• provider registry.terraform.io/hashicorp/aws v3.40.0
• provider registry.terraform.io/hashicorp/cloudinit v2.2.0
• provider registry.terraform.io/hashicorp/helm v2.1.2
• provider registry.terraform.io/hashicorp/kubernetes v1.11.4
• provider registry.terraform.io/hashicorp/local v1.4.0
• provider registry.terraform.io/hashicorp/null v2.1.2
• provider registry.terraform.io/hashicorp/random v2.3.1
• provider registry.terraform.io/hashicorp/template v2.2.0
• provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Reproduction

Following config in a previous version (i.e. 16):

cluster_endpoint_public_access = false
cluster_endpoint_private_access = true
cluster_endpoint_private_access_cidrs = local.cluster_endpoint_private_access_cidrs
cluster_create_endpoint_private_access_sg_rule = true

the cidrs variable is a list of those 3 cidrs shown at the top of the issue.

Then, update to this module version 17

[barryib]

Is this rule already exists your previous states ? If yes, did you follow the breaking change notes which say you have to rename that resources in your state ?

Did you change your CDR list during this upgrade ?

[jaimehrubiks]

@barryib Sorry for that, I did not understand properly the breaking changes. I did not change the state as you mention, I thought that it meant something else.

I'll try to roll back and try again

--- Edit:

Still, cant figure out what the issue is.

I tried to run this:
❯ terraform0-14-11 state mv "module.eks.aws_security_group_rule.cluster_private_access[0]" "module.eks.aws_security_group_rule.cluster_private_access_cidrs_source[0]"

before the update, and still. I see the same issue.

Sorry if I am misunderstanding something

[barryib]

Did you change your CDR list during this upgrade ?

If yes, this probably related to #984

[jaimehrubiks]

No, I did not change it

[barryib]

Can you please share your plan output after your state manipulation ?

[jaimehrubiks]

16.2.0 after state change:
terraform0-14-11 state mv "module.eks.aws_security_group_rule.cluster_private_access[0]" "module.eks.aws_security_group_rule.cluster_private_access_cidrs_source[0]"

❯ terraform0-14-11 plan | grep module.eks
  # module.eks.aws_security_group_rule.cluster_private_access[0] will be created
  + resource "aws_security_group_rule" "cluster_private_access" {
      + cidr_blocks              = [
          + "10.0.0.0/8",
          + "172.16.0.0/12",
          + "192.168.0.0/16",
        ]
      + description              = "Allow private K8S API ingress from custom source."
      + from_port                = 443
      + id                       = (known after apply)
      + protocol                 = "tcp"
      + security_group_id        = "sg-0e2419108f9d9ac74"
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 443
      + type                     = "ingress"
    }

  # module.eks.aws_security_group_rule.cluster_private_access_cidrs_source[0] will be destroyed
  - resource "aws_security_group_rule" "cluster_private_access_cidrs_source" {
      - cidr_blocks       = [
          - "10.0.0.0/8",
          - "172.16.0.0/12",
          - "192.168.0.0/16",
        ] -> null
      - description       = "Allow private K8S API ingress from custom source." -> null
      - from_port         = 443 -> null
      - id                = "sgrule-2706511318" -> null
      - ipv6_cidr_blocks  = [] -> null
      - prefix_list_ids   = [] -> null
      - protocol          = "tcp" -> null
      - security_group_id = "sg-0e2419108f9d9ac74" -> null
      - self              = false -> null
      - to_port           = 443 -> null
      - type              = "ingress" -> null
    }

Plan: 1 to add, 0 to change, 1 to destroy.

(I did not run apply here)

Output after updating module to 17.0.2 and doing terraform init

  # module.eks.data.http.wait_for_cluster[0] will be read during apply
  # (config refers to values not yet known)
 <= data "http" "wait_for_cluster"  {
      ~ body             = "ok" -> (known after apply)
      ~ id               = "https://FASDF78S7DF878.gr7.us-east-1.eks.amazonaws.com/healthz" -> (known after apply)
      - insecure         = false -> null
      ~ response_headers = {
          - "Cache-Control"          = "no-cache, private"
          - "Content-Length"         = "2"
          - "Content-Type"           = "text/plain; charset=utf-8"
          - "Date"                   = "Fri, 28 May 2021 15:54:41 GMT"
          - "X-Content-Type-Options" = "nosniff"
        } -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.eks.aws_autoscaling_group.workers_launch_template[3] will be updated in-place
  ~ resource "aws_autoscaling_group" "workers_launch_template" {
        id                        = "ms-test-apps-windows20210520153227313800000015"
        name                      = "ms-test-apps-windows20210520153227313800000015"
        # (23 unchanged attributes hidden)


      ~ launch_template {
            id      = "lt-0d32de96ed566ad56"
            name    = "ms-test-apps-windows20210520153226695500000009"
          ~ version = "6" -> (known after apply)
        }

        # (17 unchanged blocks hidden)
    }

  # module.eks.aws_launch_template.workers_launch_template[3] will be updated in-place
  ~ resource "aws_launch_template" "workers_launch_template" {
        id                      = "lt-0d32de96ed566ad56"
      ~ latest_version          = 6 -> (known after apply)
        name                    = "ms-test-apps-windows20210520153226695500000009"
        tags                    = {
...
        }
        # (13 unchanged attributes hidden)

      ~ block_device_mappings {
          ~ device_name = "/dev/xvda" -> "/dev/sda1"

            # (1 unchanged block hidden)
        }







        # (8 unchanged blocks hidden)
    }

  # module.eks.aws_security_group_rule.cluster_private_access[0] will be created
  + resource "aws_security_group_rule" "cluster_private_access" {
      + cidr_blocks              = [
          + "10.0.0.0/8",
          + "172.16.0.0/12",
          + "192.168.0.0/16",
        ]
      + description              = "Allow private K8S API ingress from custom source."
      + from_port                = 443
      + id                       = (known after apply)
      + protocol                 = "tcp"
      + security_group_id        = "sg-0e2419108f9d9ac74"
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 443
      + type                     = "ingress"
    }

  # module.eks.aws_security_group_rule.cluster_private_access_cidrs_source[0] will be updated in-place
  ~ resource "aws_security_group_rule" "cluster_private_access_cidrs_source" {
      - description       = "Allow private K8S API ingress from custom source." -> null
        id                = "sgrule-2706511318"
        # (9 unchanged attributes hidden)
    }

  # module.eks.local_file.kubeconfig[0] must be replaced
-/+ resource "local_file" "kubeconfig" {
      ~ file_permission      = "0644" -> "0600" # forces replacement
      ~ id                   = "318828bf5966530ea5b846ccc4378b79c896ec17" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Here is where it fails if I apply

[jaimehrubiks]

Do these conflict?

resource "aws_security_group_rule" "cluster_private_access_cidrs_source" {
  count       = var.create_eks && var.cluster_create_endpoint_private_access_sg_rule && var.cluster_endpoint_private_access && var.cluster_endpoint_private_access_cidrs != null ? 1 : 0
  type        = "ingress"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = var.cluster_endpoint_private_access_cidrs

  security_group_id = aws_eks_cluster.this[0].vpc_config[0].cluster_security_group_id
}

resource "aws_security_group_rule" "cluster_private_access" {
  description = "Allow private K8S API ingress from custom source."
  count       = var.create_eks && var.cluster_create_endpoint_private_access_sg_rule && var.cluster_endpoint_private_access ? 1 : 0
  type        = "ingress"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = var.cluster_endpoint_private_access_cidrs

  security_group_id = aws_eks_cluster.this[0].vpc_config[0].cluster_security_group_id
}

cidr_blocks = var.cluster_endpoint_private_access_cidrs

Shouldn't second snippet be removed?

[barryib]

You're absolutely right. This was introduced during the merge conflict resolution.

[barryib]

@jaimehrubiks can you please review this #1412 ?

[jaimehrubiks]

All good now, completes without issue

Feel free to merge.

Thanks :)

PD: I tested both with and without issuing the terraform mv command. Both ways work. Without the mv, the rule is deleted and re-created on a different state name, but for me it is not an issue (it may cause a millisecond downtime to the api ?)

[barryib]

but for me it is not an issue (it may cause a millisecond downtime to the api ?)

Yep, or sometime more.

Thanks @jaimehrubiks for your help.

[barryib]

v17.0.3 is released

[bin-chen-techlabs]

I am still having the same issue with EKS version 17.1.0.
(I have two work stations, one run old version of EKS, now I try to sync on a different machine, it plans to do the following after pick up the latest version of the modules)

# module.eks_cluster.module.eks-cluster.aws_security_group_rule.cluster_private_access[0] will be destroyed
  - resource "aws_security_group_rule" "cluster_private_access" {
      - cidr_blocks       = [
          - "10.250.1.0/24",
          - "10.2.5.0/24",
          - "10.0.50.0/24",
          - "10.0.49.0/24",
          - "100.84.44.0/24",
        ] -> null
      - from_port         = 443 -> null
      - id                = "sgrule-3237738704" -> null
      - ipv6_cidr_blocks  = [] -> null
      - prefix_list_ids   = [] -> null
      - protocol          = "tcp" -> null
      - security_group_id = "sg-045876b1e9a550fe4" -> null
      - self              = false -> null
      - to_port           = 443 -> null
      - type              = "ingress" -> null
    }

  # module.eks_cluster.module.eks-cluster.aws_security_group_rule.cluster_private_access_cidrs_source[0] will be created
  + resource "aws_security_group_rule" "cluster_private_access_cidrs_source" {
      + cidr_blocks              = [
          + "10.250.1.0/24",
          + "10.2.5.0/24",
          + "10.0.50.0/24",
          + "10.0.49.0/24",
          + "100.84.44.0/24",
        ]
      + description              = "Allow private K8S API ingress from custom CIDR source."
      + from_port                = 443
      + id                       = (known after apply)
      + protocol                 = "tcp"
      + security_group_id        = "sg-045876b1e9a550fe4"
      + self                     = false
      + source_security_group_id = (known after apply)
      + to_port                  = 443
      + type                     = "ingress"
    }

[technotaff-nbs]

Hi everyone, for those having the same issue, please check your state file. These problems are often created by infection of create_before_destroy resources. Check where your security group is referenced in your state file dependencies (it might surprise you). Something in your module stack has create_before_destroy to true.

The create_before_destroy lifecycle override can have far-reaching "infection" on loosely-related resources, see:

https://github.com/hashicorp/terraform/blob/main/docs/destroying.md#create-before-destroy

[adelwin]

hi @technotaff-nbs , can you elaborate more about how this can contribute to the issue?
I'm seeing this issue right now, and i'm abit confused.

I'm using tf1.0 with EKS module 17.
The message says that it's a "now-fixed" terraform issue,
But I'm using quite up-to-date versions, but still see the issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.