Update IAM Policy for AWS LoadBalancer Controller

[mcharriere]

In upstream, the policy has been recently updated to include the following block:

{
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        }

We need to update the policy in aws-operator.

See for more details kubernetes-sigs/aws-load-balancer-controller#2692
Full policy: https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

The error associated with the missing permission is:

aws-load-balancer-controller- aws-load-balancer-controller {"level":"error","ts":1689690117.7841887,"logger":"controller-runtime.manager.controller.service","msg":"Reconciler error","name":"nginx-ingress-nlb-controller-app-internal","namespace":"kube-system","error":"AccessDenied: User: arn:aws:sts::xxxx:assumed-role/gs-xxxx-ALBController-Role/ is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:eu-central-1:xxxx:targetgroup/k8s-kubesyst-nginxing-xxx/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 1ac329c7-9e43-43c4-9ce7-6b5ee93d7815"}

[T-Kukawka]

PR: giantswarm/aws-operator#3578

[T-Kukawka]

@mcharriere could we test this last time if it is really needed based on Pau's findings from last week?

[mcharriere]

I think this is needed. A fresh cluster with the app installed without any configuration fails with the following error:

aws-load-balancer-controller-c9bc69bdf-qshjl aws-load-balancer-controller {"level":"error","ts":1690192611.0681303,"logger":"controller-runtime.manager.controller.service","msg":"Reconciler error","name":"test-public","namespace":"kube-system","error":"AccessDenied: User: arn:aws:sts::270935918670:assumed-role/gs-cg56w-ALBController-Role/1690192604397817548 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:eu-central-1:270935918670:targetgroup/k8s-kubesyst-testpubl-b971d4e2b1/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 70b5db90-8901-4e58-b951-3e81bffb7f78"}

the test service is:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-type: external
  labels:
    app: test2
  name: test-public
  namespace: kube-system
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Local
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    nodePort: 32456
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 30929
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    k8s-app: nginx-ingress-public-controller-app
  sessionAffinity: None
  type: LoadBalancer

Not sure what tests Pau did.

[whites11]

I did it on a v19 cluster WITHOUT the change and it worked like a charm

[T-Kukawka]

so what is the difference between @whites11 and @mcharriere test? 🤯

[mcharriere]

I found this in aws-ia/terraform-aws-eks-blueprints-addons#200

https://user-images.githubusercontent.com/3725386/251379248-d37a6e60-42ea-47b6-834f-0af83e6d0070.png

It seems that this is a API change from AWS and it doesn't affect all the installations. I'm not sure what the pattern is.

[whites11]

@T-Kukawka I have the fix ready and tested, but it's pointless to backport to v19.0.0 because it requires kicking the cloud formation stack by hand. Do you want me to release anyway?

[T-Kukawka]

From the ticket I undestand that: "After August these calls will start failing"

[T-Kukawka]

What i have in mind.

We could backport this and not force roll the CF stack, but let it roll in one of the processes, either upgrade or new cluster creation. Accordingly to the shared ticket the issue affects only new LBs hence the patch would apply when:

• The new LB is created at most of the cases on a new cluster
• (For the particular issue we saw in the past) the new LB was recreated or created during upgrade

WDYT? my reasoning makes sense? @whites11 @mcharriere

[whites11]

ok let's do it