Load Balancer Not Created due to AddTags issue

[jls-appfire]

Describe the bug
Load balancers are not getting created from properly configured services.

Steps to reproduce

1. Launch an EKS Cluster (I specifically am using 1.28)
2. Create the IAM policy per the instructions (I am using the latest one that includes the AddTags)
3. Create the LB service account per the instructions
4. helm install the LB controller
5. Create a service that requests an LB

Expected outcome
An NLB should be created.

Environment

• AWS Load Balancer controller version - 2.6.1
• Kubernetes version - 1.28
• Using EKS (yes/no), if so version? Yes, 1.28

Additional Context:
I have three clusters, on one the LB controller works great, on the other two it does not. All three clusters were built using the same Terraform module with slightly different inputs. The TF module creates the IAM policy for the LB controller and the policy template includes the AddTag permission and this has not changed since July.

All three clusters are in separate AWS accounts as well.

The first cluster was launched as a 1.27 cluster to which the LB controller was deployed and running fine. This was launched in May / June of this year. The initial LB controller version was 2.5.x (I don't recall the exact version and it is now running 2.6.1). About a week ago EKS was upgraded from 1.27 to 1.28. The LB controller on this cluster works fine.

The second two clusters were built yesterday with Terraform. The LB controller policy was created by Terraform and a shell script was used to create the service account referencing the role ARN. Helm was then used to deploy the LB controller (version 2.6.1) to the cluster:

helm upgrade -i \
  --wait \
  aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=virginia-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

A service was then created requesting a load balancer, however we receive the following in the logs:

{
  "level": "error",
  "ts": "2023-10-14T00:44:13Z",
  "msg": "Reconciler error",
  "controller": "service",
  "object": {
    "name": "public-nlb-us-dts-one",
    "namespace": "ingress"
  },
  "namespace": "ingress",
  "name": "public-nlb-us-dts-one",
  "reconcileID": "e2454283-xxxx-xxxx-xxxx-3b024b13c587",
  "error": "AccessDenied: User: arn:aws:sts::1234567890:assumed-role/dts-appfire-virginia-cluster-lbcontroller-role/1697243906603341273 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-east-1:1234567890:targetgroup/k8s-ingress-publicnl-74fc3a3aab/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 29835544-xxxx-xxxx-xxxx-c3d50d0d09e9"
}

I used the same YAML to create a test service on all three clusters. The first one creates an LB fine, and the second two do not. I feel like I am missing something in my config, but the policy associated to the IAM role and the IAM role trust policy should be the only factor causing this issue.

I have confirmed that the policy contains the following on all three clusters:

        {
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
        },
        {
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },

I also verified the OIDC configuration in the Trust policy and it matches the OIDC identity provider. I ran the same tests, and came to the same conclusion as the person who reported #3399

Essentially if I remove the "aws:RequestTag/elbv2.k8s.aws/cluster": "true", condition, everything works perfectly.

I did recently remove some of the "legacy" tags that I thought were previously needed for clusters before 1.15 per https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html but are listed as no longer needed. I didn't see (but may have missed) any notes about related tagging requirements in the LB controller docs.

I also read #3383 and #2692 but those both call out that users upgraded the LB controller via Helm without updating the IAM policy and thus were missing the AddTags permission required by AWS in September. I have that permission in place and just that aws:RequestTag condition seems to be a problem.

[oliviassss]

@jls-appfire, thanks for the details. I think what you might be missing is the permission here: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/install/iam_policy.json#L199, this is the fix we had to grant explicitly the AddTags permission to the creation of LB and TG.

Can you please double-check your IAM policies? Thanks

[oliviassss]

Closing this issue for now, please feel free to reach out if there's any question, thanks.