Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Support secondary eips #1109

Closed
wants to merge 14 commits into from
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ No modules.
| [aws_default_vpc.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/default_vpc) | resource |
| [aws_egress_only_internet_gateway.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/egress_only_internet_gateway) | resource |
| [aws_eip.nat](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eip) | resource |
| [aws_eip.secondary](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eip) | resource |
| [aws_elasticache_subnet_group.elasticache](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticache_subnet_group) | resource |
| [aws_flow_log.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/flow_log) | resource |
| [aws_iam_policy.vpc_flow_log_cloudwatch](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
Expand Down Expand Up @@ -496,6 +497,7 @@ No modules.
| <a name="input_nat_eip_tags"></a> [nat\_eip\_tags](#input\_nat\_eip\_tags) | Additional tags for the NAT EIP | `map(string)` | `{}` | no |
| <a name="input_nat_gateway_destination_cidr_block"></a> [nat\_gateway\_destination\_cidr\_block](#input\_nat\_gateway\_destination\_cidr\_block) | Used to pass a custom destination route for private NAT Gateway. If not specified, the default 0.0.0.0/0 is used as a destination route | `string` | `"0.0.0.0/0"` | no |
| <a name="input_nat_gateway_tags"></a> [nat\_gateway\_tags](#input\_nat\_gateway\_tags) | Additional tags for the NAT gateways | `map(string)` | `{}` | no |
| <a name="input_number_of_secondary_eips_per_gateway"></a> [number\_of\_secondary\_eips\_per\_gateway](#input\_number\_of\_secondary\_eips\_per\_gateway) | how many secondary eips per NAT Gateway | `number` | `0` | no |
| <a name="input_one_nat_gateway_per_az"></a> [one\_nat\_gateway\_per\_az](#input\_one\_nat\_gateway\_per\_az) | Should be true if you want only one NAT Gateway per availability zone. Requires `var.azs` to be set, and the number of `public_subnets` created to be greater than or equal to the number of availability zones specified in `var.azs` | `bool` | `false` | no |
| <a name="input_outpost_acl_tags"></a> [outpost\_acl\_tags](#input\_outpost\_acl\_tags) | Additional tags for the outpost subnets network ACL | `map(string)` | `{}` | no |
| <a name="input_outpost_arn"></a> [outpost\_arn](#input\_outpost\_arn) | ARN of Outpost you want to create a subnet in | `string` | `null` | no |
Expand Down
19 changes: 19 additions & 0 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -1053,6 +1053,7 @@ resource "aws_route" "private_ipv6_egress" {
locals {
nat_gateway_count = var.single_nat_gateway ? 1 : var.one_nat_gateway_per_az ? length(var.azs) : local.max_subnet_length
nat_gateway_ips = var.reuse_nat_ips ? var.external_nat_ip_ids : aws_eip.nat[*].id
seips_suffixs = [for num in range(0, var.number_of_secondary_eips_per_gateway) : "s${num + 1}"]
}

resource "aws_eip" "nat" {
Expand All @@ -1074,6 +1075,22 @@ resource "aws_eip" "nat" {
depends_on = [aws_internet_gateway.this]
}

resource "aws_eip" "secondary" {
for_each = toset(flatten([for nat in aws_eip.nat : [for suffix in local.seips_suffixs : "${nat.tags.Name}-${suffix}"]]))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we cannot have computed values as keys in maps

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, could you explain why please and if you could recommend an alternative approach?

I did consider doing this with count like the other arrays were doing, however if you changed the number of AZs or number of ips per az it would start trying to move the ips from 1 natgateway to the other which then wants to destroy and re-create the natgateway which isn't ideal. Using keys seem to make the solution much more solid.

Or is it the use of the nat.tags.Name that is the issue. This does seem like it could be flaky 🤔 I guess we could use a range based on
local.create_vpc && var.enable_nat_gateway && !var.reuse_nat_ips ? local.nat_gateway_count : 0
As the first part of the composite key, this would be much more solid thinking about it 🚀

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

@AlexisColes AlexisColes Aug 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, but a fresh plan with the current code does not result in unknown values being passed to the for_each argument as the Name tag can be determined.

I added 2 ips per gateway to the complete example and a fresh plan is good.

  # module.vpc.aws_eip.nat[0] will be created
  + resource "aws_eip" "nat" {
      + tags                 = {
          + "Example"    = "ex-complete"
          + "GithubOrg"  = "terraform-aws-modules"
          + "GithubRepo" = "terraform-aws-vpc"
          + "Name"       = "ex-complete-eu-west-1a"
        }
    }

  # module.vpc.aws_eip.secondary["ex-complete-eu-west-1a-s1"] will be created
  + resource "aws_eip" "secondary" {
        ......
    }

  # module.vpc.aws_eip.secondary["ex-complete-eu-west-1a-s2"] will be created
  + resource "aws_eip" "secondary" {
         ......
    }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexisColes I tested your solution and got this errors:

module.vpc.aws_eip.secondary["nebula-paastocaas-eks-vpc-nprd-eu-central-1c-s2"]: Creation complete after 0s [id=eipalloc-0b8add66c71b6ab16]
module.vpc.aws_eip.secondary["nebula-paastocaas-eks-vpc-nprd-eu-central-1b-s2"]: Creation complete after 0s [id=eipalloc-08c1ad57da35acdf2]

│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include

│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" changed the planned action from
│ Update to DeleteThenCreate.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .network_interface_id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .public_ip: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .association_id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .private_ip: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .secondary_private_ip_address_count: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[1] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .secondary_private_ip_addresses: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ Warning: Argument is deprecated

│ with module.eks.aws_eks_addon.this["kube-proxy"],
│ on .terraform/modules/eks/main.tf line 400, in resource "aws_eks_addon" "this":
│ 400: resolve_conflicts = try(each.value.resolve_conflicts, "OVERWRITE")

│ The "resolve_conflicts" attribute can't be set to "PRESERVE" on initial
│ resource creation. Use "resolve_conflicts_on_create" and/or
│ "resolve_conflicts_on_update" instead

│ (and 3 more similar warnings elsewhere)

│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" changed the planned action from
│ Update to DeleteThenCreate.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .network_interface_id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .public_ip: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .association_id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .private_ip: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .secondary_private_ip_address_count: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[0] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .secondary_private_ip_addresses: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" changed the planned action from
│ Update to DeleteThenCreate.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .network_interface_id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .public_ip: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .secondary_private_ip_addresses: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .association_id: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .private_ip: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.


│ Error: Provider produced inconsistent final plan

│ When expanding the plan for module.vpc.aws_nat_gateway.this[2] to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .secondary_private_ip_address_count: was known, but now unknown.

│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Error: Process completed with exit code 1.

Copy link

@flaviomoringa flaviomoringa Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above errors happened running the code in a previously created VPC (with the 5.13.0 version of the module) with the default of 3 NATGW (1 per az), and now running against your version changing the secondary IP's from 0 to 2.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm testing the following:

Adding to locals:
seips_names = flatten([for nat_index in range(0, local.nat_gateway_count) : [for suffix in local.seips_suffixs : "${aws_eip.nat[nat_index].tags.Name}-${suffix}"]])

and replacing the for_each with:
for_each = toset(local.seips_names)

Would that help?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

guess not... same errors "Error: Provider produced inconsistent final plan" when going from 0 secondary IP's to 2 :-(

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flaviomoringa I have done some testing and it seems a provider issue indeed. The only way to add secondary IPs with terraform is on NAT Gateway creation. Any further change to that results in a re-creation of the NAT Gateway. However, if you have created it without secondary IPs you get the inconsistent plan error. I believe that has to do with how it associates the private IPs to the NAT Gateway interfaces but it doesn't allow manual association.

I tested with this very basic example:

locals {
  nat_gateway_count                    = 2
  public_subnets                       = ["subnet-xxxxxx", "subnet-xxxxxx"]
  number_of_secondary_eips_per_gateway = 0
}

resource "aws_eip" "nat" {
  count = local.nat_gateway_count

  domain = "vpc"
}

resource "aws_eip" "secondary" {
  count = local.nat_gateway_count * local.number_of_secondary_eips_per_gateway

  domain = "vpc"
}

resource "aws_nat_gateway" "this" {
  count = local.nat_gateway_count

  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = local.public_subnets[count.index]

  secondary_allocation_ids = slice(aws_eip.secondary[*].id, count.index * local.number_of_secondary_eips_per_gateway, (count.index + 1) * local.number_of_secondary_eips_per_gateway)
}

I think this should be an issue on the aws provider repo if there is not one already.

Copy link

@velkovb velkovb Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


domain = "vpc"

tags = merge(
{
"Name" = each.key,
},
var.tags,
var.nat_eip_tags,
)

depends_on = [aws_internet_gateway.this]
}

resource "aws_nat_gateway" "this" {
count = local.create_vpc && var.enable_nat_gateway ? local.nat_gateway_count : 0

Expand All @@ -1086,6 +1103,8 @@ resource "aws_nat_gateway" "this" {
var.single_nat_gateway ? 0 : count.index,
)

secondary_allocation_ids = [for suffix in local.seips_suffixs : aws_eip.secondary["${aws_eip.nat[count.index].tags.Name}-${suffix}"].allocation_id]

tags = merge(
{
"Name" = format(
Expand Down
6 changes: 6 additions & 0 deletions variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -1222,6 +1222,12 @@ variable "external_nat_ips" {
default = []
}

variable "number_of_secondary_eips_per_gateway" {
description = "how many secondary eips per NAT Gateway"
type = number
default = 0
}

variable "nat_gateway_tags" {
description = "Additional tags for the NAT gateways"
type = map(string)
Expand Down