AWS EKS Cluster with 1.29 -application load balancer fails with "error":"AccessDenied: User: arn:aws:sts::XXXXXXX:assumed-role/aws-load-balancer-controller/XXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-east-2:xxxx:targetgroup/k8s-istiosys-istioing-xxxx/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 30780d8e-569f-4a86-8b1b-ce70e0ce8ee3"}

[Sandeepisebauz]

Describe the bug
User is not able to provision application loadbalancer on AWS EKS Cluster with 1.29 version

Steps to reproduce
When I provision an EKS cluster using terraform, we notice that "AmazonEKS_AWS_Load_Balancer_Controller-20240310223824038200000002" being provsioned. The JSON content for this is as below
{ "Statement": [ { "Action": "iam:CreateServiceLinkedRole", "Condition": { "StringEquals": { "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com" } }, "Effect": "Allow", "Resource": "*" }, { "Action": [ "elasticloadbalancing:DescribeTargetHealth", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetGroupAttributes", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:DescribeSSLPolicies", "elasticloadbalancing:DescribeRules", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeLoadBalancerAttributes", "elasticloadbalancing:DescribeListeners", "elasticloadbalancing:DescribeListenerCertificates", "ec2:GetCoipPoolUsage", "ec2:DescribeVpcs", "ec2:DescribeVpcPeeringConnections", "ec2:DescribeTags", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeNetworkInterfaces", "ec2:DescribeInternetGateways", "ec2:DescribeInstances", "ec2:DescribeCoipPools", "ec2:DescribeAvailabilityZones", "ec2:DescribeAddresses", "ec2:DescribeAccountAttributes" ], "Effect": "Allow", "Resource": "*" }, { "Action": [ "wafv2:GetWebACLForResource", "wafv2:GetWebACL", "wafv2:DisassociateWebACL", "wafv2:AssociateWebACL", "waf-regional:GetWebACLForResource", "waf-regional:GetWebACL", "waf-regional:DisassociateWebACL", "waf-regional:AssociateWebACL", "shield:GetSubscriptionState", "shield:DescribeProtection", "shield:DeleteProtection", "shield:CreateProtection", "iam:ListServerCertificates", "iam:GetServerCertificate", "cognito-idp:DescribeUserPoolClient", "acm:ListCertificates", "acm:DescribeCertificate" ], "Effect": "Allow", "Resource": "*" }, { "Action": [ "ec2:RevokeSecurityGroupIngress", "ec2:CreateSecurityGroup", "ec2:AuthorizeSecurityGroupIngress" ], "Effect": "Allow", "Resource": "*" }, { "Action": "ec2:CreateTags", "Condition": { "Null": { "aws:RequestTag/elbv2.k8s.aws/cluster": "false" }, "StringEquals": { "ec2:CreateAction": "CreateSecurityGroup" } }, "Effect": "Allow", "Resource": "arn:aws:ec2:*:*:security-group/*" }, { "Action": [ "ec2:DeleteTags", "ec2:CreateTags" ], "Condition": { "Null": { "aws:RequestTag/elbv2.k8s.aws/cluster": "true", "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } }, "Effect": "Allow", "Resource": "arn:aws:ec2:*:*:security-group/*" }, { "Action": [ "ec2:RevokeSecurityGroupIngress", "ec2:DeleteSecurityGroup", "ec2:AuthorizeSecurityGroupIngress" ], "Condition": { "Null": { "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } }, "Effect": "Allow", "Resource": "*" }, { "Action": [ "elasticloadbalancing:CreateTargetGroup", "elasticloadbalancing:CreateLoadBalancer" ], "Condition": { "Null": { "aws:RequestTag/elbv2.k8s.aws/cluster": "false" } }, "Effect": "Allow", "Resource": "*" }, { "Action": [ "elasticloadbalancing:DeleteRule", "elasticloadbalancing:DeleteListener", "elasticloadbalancing:CreateRule", "elasticloadbalancing:CreateListener" ], "Effect": "Allow", "Resource": "*" }, { "Action": [ "elasticloadbalancing:RemoveTags", "elasticloadbalancing:AddTags" ], "Condition": { "Null": { "aws:RequestTag/elbv2.k8s.aws/cluster": "true", "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } }, "Effect": "Allow", "Resource": [ "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*", "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*", "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*" ] }, { "Action": [ "elasticloadbalancing:RemoveTags", "elasticloadbalancing:AddTags" ], "Effect": "Allow", "Resource": [ "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*", "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*", "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*", "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*" ] }, { "Action": [ "elasticloadbalancing:SetSubnets", "elasticloadbalancing:SetSecurityGroups", "elasticloadbalancing:SetIpAddressType", "elasticloadbalancing:ModifyTargetGroupAttributes", "elasticloadbalancing:ModifyTargetGroup", "elasticloadbalancing:ModifyLoadBalancerAttributes", "elasticloadbalancing:DeleteTargetGroup", "elasticloadbalancing:DeleteLoadBalancer" ], "Condition": { "Null": { "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } }, "Effect": "Allow", "Resource": "*" }, { "Action": [ "elasticloadbalancing:RegisterTargets", "elasticloadbalancing:DeregisterTargets" ], "Effect": "Allow", "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*" }, { "Action": [ "elasticloadbalancing:SetWebAcl", "elasticloadbalancing:RemoveListenerCertificates", "elasticloadbalancing:ModifyRule", "elasticloadbalancing:ModifyListener", "elasticloadbalancing:AddListenerCertificates" ], "Effect": "Allow", "Resource": "*" } ], "Version": "2012-10-17" }

Expected outcome
Now when we apply below test yaml for ingress ALB
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: test-alb namespace: istio-system annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/certificate-arn: arn:aws:iam::XXXXXX:server-certificate/CSC alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":31390}]' alb.ingress.kubernetes.io/backend-protocol: HTTPS alb.ingress.kubernetes.io/actions.ssl-redirect: | { "Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "XXXX", "StatusCode": "HTTP_301" } } alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready alb.ingress.kubernetes.io/healthcheck-port: "XXXX" alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true alb.ingress.kubernetes.io/group.name: XXXXX spec: rules: - http: paths: - backend: service: name: ssl-redirect port: name: use-annotation path: / pathType: Prefix - backend: service: name: istio-ingressgateway port: number: XXXX path: / pathType: Prefix
we notice following error under AWS loadbalancer pod
{"level":"info","ts":1710119559.829732,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"upf-alb-sit-ingress-group","resourceID":"istio-system/test-alb-istio-ingressgateway:XXXX"} {"level":"error","ts":1710119559.8391085,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"upf-alb-sit-ingress-group","namespace":"","error":"AccessDenied: User: arn:aws:sts::XXXXX:assumed-role/aws-load-balancer-controller/1710119424124753004 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-east-2:002483744614:targetgroup/k8s-istiosys-istioing-445b612369/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 30780d8e-569f-4a86-8b1b-ce70e0ce8ee3"}

Environment

• AWS Load Balancer controller version
• Kubernetes version: 1.29
• Using EKS (yes/no), if so version?: Yes

Additional Context:
This used to work when we used AWS EKS with 1.26 version and right now this breaking our CI-CD autoamtion,
A temp manual workaround is remove the below from the ALB
"Condition": { "Null": { "aws:RequestTag/elbv2.k8s.aws/cluster": "true", "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" }

But there is no automation to do this. It would be helpful if we can get this fixed ASAP.

[M00nF1sh]

@Sandeepisebauz
Not sure what's wrong. What's the current AWS tags on arn:aws:elasticloadbalancing:us-east-2:002483744614:targetgroup/k8s-istiosys-istioing-445b612369/*?

[M00nF1sh]

@Sandeepisebauz
Nvm, the issue should be same as #2692
TLDR: ELB team did a change last year for their behind the Create*** API which requires IAM policy change.
We have updated our sample IAM policy to coordinate their change, please update your policy according to this file: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/install/iam_policy.json#L200 (this line also indicates the missing permissions)