Failing to create target group with default policy #2692

patrickmscott · 2022-06-16T14:53:28Z

Describe the bug
The controller is failing to create a target group based on the provided policy.

User: arn:aws:sts::123:assumed-role/AWSALBIngressController_3f2bb898eae5ea79ebdb9cb3514f5ec6/1655374570081000102 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-west-2:123:targetgroup/foo/*

This error occurs after trying to create the target group and before reconciliation.

Based on my understanding of the policy, AddTags requires a null cluster tag in the request but also requires a non-null resource tag. I'm not sure how that can be possible during target group creation. Especially since the request has the cluster tag and presumably the target group has no tags on creation.

Steps to reproduce

Expected outcome
The target group should be created with expected tags

Environment

AWS Load Balancer controller version 2.2.3
Kubernetes version v1.17.12-eks-7684af
Using EKS (yes/no), if so version? yes, v1.17.12-eks-7684af

Additional Context:

The text was updated successfully, but these errors were encountered:

berry2012 · 2022-06-20T11:31:11Z

@croachrose do you mind sharing a little more details about your EKS environment?
Also what does the IAM policy used by the role AWSALBIngressController_3f2bb898eae5ea79ebdb9cb3514f5ec6 looks like?
Look for block like this

        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },

patrickmscott · 2022-06-20T13:22:28Z

@berry2012 Not sure what other details would be helpful.

Our policy looks exactly like that. We applied the policy from docs/install.

In order to fix this, I temporarily removed

            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }

patrickmscott · 2022-06-20T13:24:44Z

@berry2012 It's almost like AWS changed the implementation of CreateTargetGroup to invoke AddTags if tags are part of the request. In that case, aws:ResourceTag/elbv2.k8s.aws/cluster is null and aws:RequestTag/elbv2.k8s.aws/cluster is not null.

berry2012 · 2022-06-20T15:13:21Z

@croachrose from a more recent alb controller, e.g. v2.4.1 with EKS 1.22, I see tags as part of the request so that explains it.

I reproduced this in the environments below without altering the default IAM policy of the alb controller in the documentation and did not encounter the error.

Env 1
AWS Load Balancer controller version 2.3.1
EKS 1.21
Endpoint: Public

Env 2
AWS Load Balancer controller version 2.4.2
EKS 1.22
Endpoint: Public

Your workaround make sense as well for your specific environment. Below works as well

            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }

patrickmscott · 2022-06-20T16:37:15Z

@berry2012 Any idea what is different about our environment? Is it the controller version? For what it's worth, this was working just fine and we changed nothing. It just stopped working suddenly.

patrickmscott · 2022-06-21T11:48:17Z

@berry2012 update - AWS got back to us and our clusters were upgraded automatically to 1.18 when this started happening. Could k8s 1.18 have issues with controller 2.2.3?

chanpark · 2022-06-24T16:45:52Z

My EKS version is 1.20 (not fresh, but upgraded from 1.15 -> 1.16 -> 1.17 -> 1.18 -> 1.19 -> 1.20) with ALB Ingress Controller v 1.1.5.

When upgrading it to:

AWS Load Balancer Controller v2.3.1
No error with the default IAM policy of the alb controller. So, I will keep this version for my environment.
AWS Load Balancer Controller v2.4.2
I had to apply @croachrose's workaround to get rid of the error.

In order to fix this, I temporarily removed

            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }

Below didn't work either.

            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }

Glaaj · 2022-07-27T14:37:22Z

Any update on this issue?

kishorj · 2022-09-01T00:00:44Z

@croachrose, if you ran the aws-alb-ingress-controller previously, then you'd need to be on v1.1.3 or later before upgrading to the aws-load-balancer-controller v2.x. You'd also need additional IAM permissions. With appropriate IAM permissions, controller v2.2.3 works with k8s 1.18.

In your case, is the target group an existing one, or controller is trying to create a new target group?

patrickmscott · 2022-09-01T01:26:28Z

@kishorj We were happily running v2.2.3 for months with no issue and no changes. We were using the exact policy from the install docs (with the tag conditions). It just stopped working one day and removing those conditions fixed it. The target group was not an existing group.

Glaaj · 2022-09-06T08:28:48Z

@kishorj We were happily running v2.2.3 for months with no issue and no changes. We were using the exact policy from the install docs (with the tag conditions). It just stopped working one day and removing those conditions fixed it. The target group was not an existing group.

We also ran into the same problem with existing target groups across multiple EKS clusters.

kishorj · 2022-09-14T22:29:59Z

@Glaaj, @croachrose, do you have any automation removing tags on resources? Please share the cloudtrail events from the failure. You can email to k8s-alb-controller-triage AT amazon.com

Glaaj · 2022-11-02T15:16:38Z

I believe we do not have any automation for that.

Glaaj · 2023-01-19T12:08:48Z

bump

patrickmscott · 2023-01-19T14:17:56Z

We also do not have any automation to remove tags

kishorj · 2023-01-19T20:01:50Z

@Glaaj, were you able to get the cloudtrail events? Also share the policy permissions from the role. You can email them to k8s-alb-controller-triage AT amazon.com

msf-woven · 2023-02-08T11:31:28Z

We've run into this issue (or something with very similar symptoms) on a brand new eks cluster using k8s 1.24 and the latest release of the controller. we are still investigating

marcinswigon · 2023-02-08T15:42:49Z

We've also run into that issue Today. Surprisingly that happened to only one of our clusters that was running fine for over 4 months. We are currently using EKS 1.22 and aws-lb-controller v2.4.2 (we tried also to update to v2.4.6, it didn't help though). Only a cluster in ap-northeast-1 is affected.

Event record

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXX",
        "accountId": "XXXXXXXXXXXX",
        "accessKeyId": "XXXXXXXXXXXXXXXXXXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "XXXXXXXXXXXXXXXXXXXXX",
                "arn": "arn:aws:iam::XXXXXXXXXXXX:role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "accountId": "XXXXXXXXXXXX",
                "userName": "role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX"
            },
            "webIdFederationData": {
                "federatedProvider": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "attributes": {}
            },
            "attributes": {
                "creationDate": "2023-02-08T02:58:05Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2023-02-08T03:40:24Z",
    "eventSource": "elasticloadbalancing.amazonaws.com",
    "eventName": "CreateTargetGroup",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "XXX.XXX.XXX.XXX",
    "userAgent": "elbv2.k8s.aws/v2.4.2 aws-sdk-go/1.42.27 (go1.17.10; linux; amd64)",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/* because no identity-based policy allows the elasticloadbalancing:AddTags action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "eventID": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "elasticloadbalancing.ap-northeast-1.amazonaws.com"
    }
}

For now we have set up elasticloadbalancing:AddTags without restrictions.

nepalevov · 2023-02-09T17:25:20Z

Same issue as described above, only one of ap-southeast-1 clusters affected, happed for the first time today.

aws-load-balancer-controller v2.4.4
EKS 1.24, Platform version eks.3

Policy attached to the role

{
    "Statement": [
        {
            "Action": "iam:CreateServiceLinkedRole",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeTags",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeInstances",
                "ec2:DescribeCoipPools",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeAddresses",
                "ec2:DescribeAccountAttributes"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "wafv2:GetWebACLForResource",
                "wafv2:GetWebACL",
                "wafv2:DisassociateWebACL",
                "wafv2:AssociateWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:GetWebACL",
                "waf-regional:DisassociateWebACL",
                "waf-regional:AssociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:DeleteProtection",
                "shield:CreateProtection",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:CreateSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": "ec2:CreateTags",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                },
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Sid": ""
        },
        {
            "Action": [
                "ec2:DeleteTags",
                "ec2:CreateTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Sid": ""
        },
        {
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:CreateLoadBalancer"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:DeleteRule",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:CreateListener"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ],
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeleteLoadBalancer"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

event with error

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "XXXXXXXXXXXX",
        "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/XXX-aws-load-balancer-controller/fXXXXXXXXXXXXXXXXXXX",
        "accountId": "XXXXXXXXXXXX",
        "accessKeyId": "XXXXXXXXXXXXXXXXXXXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "XXXXXXXXXXXXXXXXXXXXXX",
                "arn": "arn:aws:iam::XXXXXXXXXXXX:role/XXX-aws-load-balancer-controller",
                "accountId": "XXXXXXXXXXXX",
                "userName": "XXX-aws-load-balancer-controller"
            },
            "webIdFederationData": {
                "federatedProvider": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.ap-southeast-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "attributes": {}
            },
            "attributes": {
                "creationDate": "2023-02-09T14:00:28Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2023-02-09T14:39:00Z",
    "eventSource": "elasticloadbalancing.amazonaws.com",
    "eventName": "CreateTargetGroup",
    "awsRegion": "ap-southeast-1",
    "sourceIPAddress": "XXX.XXX.XXX.XXX",
    "userAgent": "elbv2.k8s.aws/v2.4.4 aws-sdk-go/1.42.27 (go1.18.6; linux; amd64)",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/XXX-aws-load-balancer-controller/XXXXXXXXXXXXXXXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:ap-southeast-1:XXXXXXXXXXXX:targetgroup/XXXXXXXXXXXXXXXXXXXX/* because no identity-based policy allows the elasticloadbalancing:AddTags action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "eventID": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "elasticloadbalancing.ap-southeast-1.amazonaws.com"
    }
}

doy-materialize · 2023-02-09T21:37:20Z

i also just started seeing this today when spinning up a new eks cluster on kubernetes 1.24 in us-east-1, running aws-load-balancer-controller 2.4.2, using the policy from the docs (https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.2/docs/install/iam_policy.json)

nickzelei · 2023-02-09T23:55:28Z

We also just started seeing this issue today. Using v2.4.4 (via helm chart v1.4.5) on EKS 1.23
We installed 1 cluster and it worked as expected. Then we installed a second and third cluster and it was failing with the AccessDenied error.

        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": [
                        "true"
                    ],
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": [
                        "false"
                    ]
                }
            }
        },

Removing the Condition block allowed the ALB Controller to function again.
This is so strange because this has been working for us for literally months and this is the first time we've run into it.

sjones512 · 2023-02-10T00:10:29Z

We're having the same issue with a new cluster running kubernetes 1.24 and aws-load-balancer-controller 2.4.6 with the policy from https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json.

Following for advice.

seungmun · 2023-02-10T08:04:00Z

The same problems occurs on Cluster 1.24 eks.4 alb 2.4.6.

It works if only aws:RequestTag/elbv2.k8s.aws/cluster line is removed, but this seems like a temporary solution.

Hopefully there will be a quick update. 🥲

vinixaavier · 2023-02-10T14:14:53Z

Same for me here: EKS 1.24 and aws-load-balancer-controller 2.4.2

I'll remove the Condition block temporary.

Apollorion · 2023-02-10T17:25:11Z

This is also happening in GovCloud FWIW.
Removing the aforementioned condition fixes this problem.

kishorj · 2023-02-10T18:40:26Z

This is due to recent behavior change on the requirement of explicit elasticloadbalancing:AddTags permission for create resources, and potentially affect new AWS accounts. Please add the following additional IAM policies if you encounter this error:

{
  "Effect": "Allow",
  "Action": [
    "elasticloadbalancing:AddTags"
  ],
  "Resource": [
    "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
    "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
    "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
  ],
  "Condition": {
    "StringEquals": {
      "elasticloadbalancing:CreateAction": [
        "CreateTargetGroup",
        "CreateLoadBalancer"
      ]
    },
    "Null": {
      "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
    }
  }
}

We will update our recommended policies and documentation.

askalski-tcg · 2023-02-10T19:26:27Z

The above policy change does not work for me unless I also add CreateListener to the elasticloadbalancing:CreateAction list.

Versions: aws-load-balancer-controller v2.4.1, EKS 1.22

M00nF1sh · 2023-02-11T01:07:18Z

ELB team have temporarily rolled back their change regards IAM permissions

dynajoe · 2023-06-08T13:31:56Z

I ran into this issue as well. Deleting and creating the service resolved it even without changing the IAM Policy.

I had done this many times before without issue.

The issue occurred on EKS 1.24 ap-southeast-1, controller v2.4.6

hrabect · 2023-06-13T14:58:25Z

I have exactly the same issue on EKS 1.26, controller v2.4.7.

Anhui-tqhuang · 2023-06-15T08:24:05Z

we meet the same error today with v2.4.0

YikaiHu · 2023-06-19T06:37:40Z

We met the same error today with EKS v1.24.0 and controller v2.4.1 in us-east-1

YikaiHu · 2023-06-19T08:28:48Z

For anyone who has encountered the same issue, we have found a solution in CDK.

The key to solving the problem is to explicitly declare the policy of the ALB controller and assign it when creating the cluster.

Here is the example:

const document = {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "iam:CreateServiceLinkedRole"
        ],
        "Resource": "*",
        "Condition": {
          "StringEquals": {
            "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
          }
        }
      },
      {
        "Effect": "Allow",
        "Action": [
          "ec2:DescribeAccountAttributes",
          "ec2:DescribeAddresses",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeVpcs",
          "ec2:DescribeVpcPeeringConnections",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeInstances",
          "ec2:DescribeNetworkInterfaces",
          "ec2:DescribeTags",
          "ec2:GetCoipPoolUsage",
          "ec2:DescribeCoipPools",
          "elasticloadbalancing:DescribeLoadBalancers",
          "elasticloadbalancing:DescribeLoadBalancerAttributes",
          "elasticloadbalancing:DescribeListeners",
          "elasticloadbalancing:DescribeListenerCertificates",
          "elasticloadbalancing:DescribeSSLPolicies",
          "elasticloadbalancing:DescribeRules",
          "elasticloadbalancing:DescribeTargetGroups",
          "elasticloadbalancing:DescribeTargetGroupAttributes",
          "elasticloadbalancing:DescribeTargetHealth",
          "elasticloadbalancing:DescribeTags"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "cognito-idp:DescribeUserPoolClient",
          "acm:ListCertificates",
          "acm:DescribeCertificate",
          "iam:ListServerCertificates",
          "iam:GetServerCertificate",
          "waf-regional:GetWebACL",
          "waf-regional:GetWebACLForResource",
          "waf-regional:AssociateWebACL",
          "waf-regional:DisassociateWebACL",
          "wafv2:GetWebACL",
          "wafv2:GetWebACLForResource",
          "wafv2:AssociateWebACL",
          "wafv2:DisassociateWebACL",
          "shield:GetSubscriptionState",
          "shield:DescribeProtection",
          "shield:CreateProtection",
          "shield:DeleteProtection"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:RevokeSecurityGroupIngress"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ec2:CreateSecurityGroup"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ec2:CreateTags"
        ],
        "Resource": "arn:aws:ec2:*:*:security-group/*",
        "Condition": {
          "StringEquals": {
            "ec2:CreateAction": "CreateSecurityGroup"
          }
        }
      },
      {
        "Effect": "Allow",
        "Action": [
          "ec2:CreateTags",
          "ec2:DeleteTags"
        ],
        "Resource": "arn:aws:ec2:*:*:security-group/*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:RevokeSecurityGroupIngress",
          "ec2:DeleteSecurityGroup"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:CreateLoadBalancer",
          "elasticloadbalancing:CreateTargetGroup"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:CreateListener",
          "elasticloadbalancing:DeleteListener",
          "elasticloadbalancing:CreateRule",
          "elasticloadbalancing:DeleteRule"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:AddTags",
          "elasticloadbalancing:RemoveTags"
        ],
        "Resource": [
          "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
          "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
          "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:AddTags",
          "elasticloadbalancing:RemoveTags"
        ],
        "Resource": [
          "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
          "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
          "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
          "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:ModifyLoadBalancerAttributes",
          "elasticloadbalancing:SetIpAddressType",
          "elasticloadbalancing:SetSecurityGroups",
          "elasticloadbalancing:SetSubnets",
          "elasticloadbalancing:DeleteLoadBalancer",
          "elasticloadbalancing:ModifyTargetGroup",
          "elasticloadbalancing:ModifyTargetGroupAttributes",
          "elasticloadbalancing:DeleteTargetGroup"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:RegisterTargets",
          "elasticloadbalancing:DeregisterTargets"
        ],
        "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticloadbalancing:SetWebAcl",
          "elasticloadbalancing:ModifyListener",
          "elasticloadbalancing:AddListenerCertificates",
          "elasticloadbalancing:RemoveListenerCertificates",
          "elasticloadbalancing:ModifyRule"
        ],
        "Resource": "*"
      }
    ]
}

const cluster = new eks.Cluster(this, 'EKSCluster', {
    vpc: props.workshopVpc,
    vpcSubnets: [{ subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }],
    mastersRole: clusterAdminRole,
    defaultCapacity: 1,
    defaultCapacityInstance: new ec2.InstanceType('m5.large'),
    version: eks.KubernetesVersion.V1_24,
    kubectlLayer: new KubectlV24Layer(this, 'Kubectlv24Layer'),
    albController: {
        version: eks.AlbControllerVersion.V2_4_1,
        policy: document
    },
    clusterName: `Workshop-Cluster`,
    endpointAccess: eks.EndpointAccess.PUBLIC,
});

aleonsan · 2023-06-21T10:23:23Z

Same error today with

EKS 1.23
aws-load-balancer-controller 2.4.0

IAM policies applied to aws-alb-controller Role coming from https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.0/docs/install/iam_policy.json

0xlen · 2023-06-22T13:11:38Z

For anyone who got this issue and suddently see it works, you might be able to check if there is any message is detected in your AWS Health Dashboard to allow you still can use deprecating policy to create target groups.

To make sure you won't be impacted by this API change, please make sure you are applying the latest IAM Policy [1] for your controller. Your policy generally needs to include:

        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        }

If you still have any issue, feel free to open a technical support case in account basis.

Hope this helps!

[1] https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

rcosperec · 2023-07-03T16:37:31Z

Thank you @0xlen for the hint about AWS Health Dashboard

I can confirm the issue is still around yet inconsistent, I had no problem with two 1.23 clusters in eu-west-1 and us-east-1 yet when we tried to create a new one in 1.23 the issue spawned.

Here is the resolution recommended by AWS

"We recommend you update your IAM policy configuration to allow access to 'elasticloadbalancing:AddTags' when using 'Create*' API calls. You can find the most recent Global [2], China [3], or GovCloud [4] IAM policy on the AWS Load Balancer Controller repository, which contains the required permission for these API calls. For more information on control permission to tag resources during creation, please refer to the 'ABAC with Elastic Load Balancing' documentation [5]. "

tirelibirefe · 2023-12-01T21:27:00Z

The same problem is stil exists, eks 1.27 & AWS lb controller 2.6.2
Removing "conditions" works.

sjones512 mentioned this issue Feb 10, 2023

no identity-based policy allows the elasticloadbalancing:AddTags #3044

Closed

jdn5126 mentioned this issue Feb 10, 2023

Update IAM policy template #3045

Closed

6 tasks

kishorj mentioned this issue Feb 10, 2023

update IAM policy template #3046

Merged

6 tasks

jdn5126 mentioned this issue Feb 20, 2023

update recommended IAM policy template #3068

Merged

6 tasks

k8s-ci-robot closed this as completed in #3068 Feb 23, 2023

Reasonably mentioned this issue Mar 2, 2023

Add CreateListner API to AddTags condition #3086

Closed

6 tasks

armujahid mentioned this issue Jun 26, 2023

no identity-based policy allows the elasticloadbalancing:AddTags aws-ia/terraform-aws-eks-blueprints#1426

Closed

yukkes mentioned this issue Jul 7, 2023

Failing to create target group with default policy aws-ia/terraform-aws-eks-blueprints-addons#202

Closed

1 task

ciiiii mentioned this issue Jul 10, 2023

Fix aws lb iam policy streamnative/terraform-managed-cloud#27

Merged

yardenw-terasky mentioned this issue Jul 11, 2023

aws-load-balancer-controller not authorized to addTags on Target groups aws-ia/terraform-aws-eks-blueprints#1683

Closed

mcharriere mentioned this issue Jul 18, 2023

Update IAM Policy for AWS LoadBalancer Controller giantswarm/roadmap#2661

Closed

agershman referenced this issue Jul 19, 2023

update recommended IAM policy template (#3068)

9333126

surajkota mentioned this issue Aug 15, 2023

Update Alb policy to include AddTag permissions awslabs/kubeflow-manifests#777

Merged

2 tasks

KlausVii mentioned this issue Sep 12, 2023

LB controller spontaneously loses permission to add tags #3383

Closed

dkliss mentioned this issue Sep 25, 2023

Failed deploy model due to AccessDenied: because no identity-based policy allows the elasticloadbalancing:AddTags action status code: 403, request i #3399

Closed

jls-appfire mentioned this issue Oct 14, 2023

Load Balancer Not Created due to AddTags issue #3428

Closed

alebedev87 mentioned this issue Oct 17, 2023

OCPBUGS-19757: bump controller iam policy to v2.4.7 openshift/aws-load-balancer-operator#114

Merged

barroco mentioned this issue Dec 21, 2023

[terraform-aws] Update policy following API change interuss/dss#980

Merged

emalinowski mentioned this issue Feb 7, 2024

fix(alb-policy): policy fix uc-cdis/cloud-automation#2459

Merged

kurktchiev mentioned this issue Feb 28, 2024

todo: aws policy note back-stack/backstack.dev#9

Open

Failing to create target group with default policy #2692

Failing to create target group with default policy #2692

Comments

patrickmscott commented Jun 16, 2022

berry2012 commented Jun 20, 2022

patrickmscott commented Jun 20, 2022

patrickmscott commented Jun 20, 2022

berry2012 commented Jun 20, 2022 • edited Loading

patrickmscott commented Jun 20, 2022

patrickmscott commented Jun 21, 2022

chanpark commented Jun 24, 2022 • edited Loading

Glaaj commented Jul 27, 2022

kishorj commented Sep 1, 2022

patrickmscott commented Sep 1, 2022

Glaaj commented Sep 6, 2022

kishorj commented Sep 14, 2022

Glaaj commented Nov 2, 2022

Glaaj commented Jan 19, 2023

patrickmscott commented Jan 19, 2023

kishorj commented Jan 19, 2023

msf-woven commented Feb 8, 2023

marcinswigon commented Feb 8, 2023

nepalevov commented Feb 9, 2023

doy-materialize commented Feb 9, 2023

nickzelei commented Feb 9, 2023 • edited Loading

sjones512 commented Feb 10, 2023

seungmun commented Feb 10, 2023

vinixaavier commented Feb 10, 2023

Apollorion commented Feb 10, 2023 • edited Loading

kishorj commented Feb 10, 2023

askalski-tcg commented Feb 10, 2023

M00nF1sh commented Feb 11, 2023

dynajoe commented Jun 8, 2023 • edited Loading

hrabect commented Jun 13, 2023

Anhui-tqhuang commented Jun 15, 2023

YikaiHu commented Jun 19, 2023 • edited Loading

YikaiHu commented Jun 19, 2023 • edited Loading

aleonsan commented Jun 21, 2023

0xlen commented Jun 22, 2023 • edited Loading

rcosperec commented Jul 3, 2023

tirelibirefe commented Dec 1, 2023

berry2012 commented Jun 20, 2022 •

edited

Loading

chanpark commented Jun 24, 2022 •

edited

Loading

nickzelei commented Feb 9, 2023 •

edited

Loading

Apollorion commented Feb 10, 2023 •

edited

Loading

dynajoe commented Jun 8, 2023 •

edited

Loading

YikaiHu commented Jun 19, 2023 •

edited

Loading

YikaiHu commented Jun 19, 2023 •

edited

Loading

0xlen commented Jun 22, 2023 •

edited

Loading