migrating from aws_security_group to aws_security_group_rule

[bernadinm]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

 terraform -v
Terraform v0.11.11
+ provider.aws v1.54.0

Affected Resource(s)

• aws_security_group
• aws_security_group_rule

Terraform Configuration Files

This configuration shows that it went from defining all the rules from aws_security_group to aws_security_group_rule as well but removing the previously define rules from within the group, until independent rules.

Before

resource "aws_security_group" "internal" {
  name        = "dcos-${var.cluster_name}-internal-firewall"
  description = "Allow all internal traffic"
  vpc_id      = "${var.vpc_id}"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["${var.subnet_range}"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

After

resource "aws_security_group" "internal" {
  name        = "dcos-${var.cluster_name}-internal-firewall"
  description = "Allow all internal traffic"
  vpc_id      = "${var.vpc_id}"

}

resource "aws_security_group_rule" "internal_ingress_rule" {
  type            = "ingress"
  from_port       = 0
  to_port         = 0
  protocol        = "-1"
  cidr_blocks = ["${var.subnet_range}"]

  security_group_id = "${aws_security_group.internal.id}"
}

resource "aws_security_group_rule" "internal_egress_rule" {
  type            = "egress"
  from_port       = 0
  to_port         = 0
  protocol        = "-1"
  cidr_blocks     = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.internal.id}"
}

Debug Output

Error: Error applying plan:

2 error(s) occurred:

* module.dcos.module.dcos-infrastructure.module.dcos-security-groups.aws_security_group_rule.internal_ingress_rule: 1 error(s) occurred:

* aws_security_group_rule.internal_ingress_rule: [WARN] A duplicate Security Group rule was found on (sg-03f37cf438fef1b5d). This may be
a side effect of a now-fixed Terraform issue causing two security groups with
identical attributes but different source_security_group_ids to overwrite each
other in the state. See https://github.com/hashicorp/terraform/pull/2376 for more
information and instructions for recovery. Error message: the specified rule "peer: 172.12.0.0/16, ALL, ALLOW" already exists
* module.dcos.module.dcos-infrastructure.module.dcos-security-groups.aws_security_group_rule.internal_egress_rule: 1 error(s) occurred:

* aws_security_group_rule.internal_egress_rule: [WARN] A duplicate Security Group rule was found on (sg-03f37cf438fef1b5d). This may be
a side effect of a now-fixed Terraform issue causing two security groups with
identical attributes but different source_security_group_ids to overwrite each
other in the state. See https://github.com/hashicorp/terraform/pull/2376 for more
information and instructions for recovery. Error message: the specified rule "peer: 0.0.0.0/0, ALL, ALLOW" already exists

Panic Output

Expected Behavior

I've expected for it to review that the required updates were already made and there was nothing that was needed to do and no failure would show.

Actual Behavior

It thought that it needed to add these to the existing security group, even though it already existed.

  + module.dcos.module.dcos-infrastructure.module.dcos-security-groups.aws_security_group_rule.internal_egress_rule
      id:                       <computed>
      cidr_blocks.#:            "1"
      cidr_blocks.0:            "0.0.0.0/0"
      from_port:                "0"
      protocol:                 "-1"
      security_group_id:        "sg-03f37cf438fef1b5d"
      self:                     "false"
      source_security_group_id: <computed>
      to_port:                  "0"
      type:                     "egress"

  + module.dcos.module.dcos-infrastructure.module.dcos-security-groups.aws_security_group_rule.internal_ingress_rule
      id:                       <computed>
      cidr_blocks.#:            "1"
      cidr_blocks.0:            "172.12.0.0/16"
      from_port:                "0"
      protocol:                 "-1"
      security_group_id:        "sg-03f37cf438fef1b5d"
      self:                     "false"
      source_security_group_id: <computed>
      to_port:                  "0"
      type:                     "ingress"

Steps to Reproduce

1. Use first listed HCL, then terraform apply
2. Use second listed HCL, then terraform apply and observe halted failure

Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor documentation? For example:
--->

• #0000

[bflad]

Hi @bernadinm 👋

Each Terraform resource is independent in the sense that it:

• Is tracked independently in the Terraform state
• Does not attempt to assume management of infrastructure should a new Terraform resource be declared in your configuration without your explicit consent in the form of importing resources

The design philosophy here is to prevent you from attempting to manage the same infrastructure in two places.

In your case, Terraform is seeing that you are:

• No longer trying to manage ingress and egress rules inline in the internal instances of the aws_security_group resource
• Creating an internal_egress_rule instance of the aws_security_group_rule resource
• Creating an internal_egress_rule instance of the aws_security_group_rule resource

The last two have no knowledge of the first and Terraform design philosophies indicate that you need to terraform import them to bring them under the new type of management, e.g. something like

terraform import module.dcos.module.dcos-infrastructure.module.dcos-security-groups.aws_security_group_rule.internal_ingress_rule sg-03f37cf438fef1b5d_ingress_all_0_65536_172.12.0.0/16
terraform import module.dcos.module.dcos-infrastructure.module.dcos-security-groups.aws_security_group_rule.internal_egress_rule sg-03f37cf438fef1b5d_egress_all_0_65536_0.0.0.0/0

The import section of the aws_security_group_rule resource documentation goes into more detail about the expected format of these commands. After doing the above, the creation of the aws_security_group_rule resources should disappear during terraform plan and you will be able to manage ingress/egress rules via aws_security_group_rule resources in your Terraform configuration going forward.

The Terraform AWS provider (or any Terraform provider really) is not currently able to change or influence this behavior at this time without upstream enhancements in core code of Terraform since that code handles the relationship of multiple resources. If you would like to see an enhancement to provide a codified way to accomplish moving infrastructure management between resources or giving Terraform the ability to infer the correct behavior in a situation like this, you can file a new issue upstream or potentially check out this similar one: hashicorp/terraform#19354

Hope this helps!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!