Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use EKS IAM role #1024

Closed
jdziat opened this issue Aug 15, 2021 · 19 comments
Closed

Unable to use EKS IAM role #1024

jdziat opened this issue Aug 15, 2021 · 19 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@jdziat
Copy link

jdziat commented Aug 15, 2021

/kind bug

What happened?
When using IAM role and policy with a helm chart I get the below error:

1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-test", UID:"090eba72-a253-424c-9753-f4a65ed2f820", APIVersion:"v1", ResourceVersion:"115104014", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc-fast": rpc error: code = Internal desc = Could not create volume "pvc-090eba72-a253-424c-9753-f4a65ed2f820": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message:

Same Policy works when using a kubernetes secret with the aws key and secret.

What you expected to happen?
The role should be used successfully when attempting to perform the operation.

How to reproduce it (as minimally and precisely as possible)?
Attempt to create and use the IAM role auth policy using the helm chart and terraform module: terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc

Anything else we need to know?:

Environment

  • Kubernetes version (use kubectl version): v1.20.7-eks-d88609
  • Driver version: helm chart: 2.1.0
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 15, 2021
@nirmalaagash
Copy link
Contributor

@jdziat Can you please share the input values that you used when creating using terraform and can you confirm the policy that you used?

@vtomasr5
Copy link

Hi,

I see the same problem.
I'm using:

  • EKS 1.21
  • Helm chart aws-ebs-csi-driver v2.1.0

Role policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:AttachVolume",
                "ec2:DetachVolume",
                "ec2:ModifyVolume",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:CreateVolume",
                "ec2:DeleteVolume",
                "ec2:DeleteSnapshot",
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "*"
        }
    ]
}

I also tried the example https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/example-iam-policy.json with the same result.

@vtomasr5
Copy link

BTW, using the helm chart 1.2.x it works fine with either IAM policy.

@jdziat
Copy link
Author

jdziat commented Aug 25, 2021

Thanks @vtomasr5

@nirmalaagash i can make a gist of the terraform module i've created it's not the most condensed thing. Also the secret name should be customizable since all the aws-csi charts use that same name it leads to conflict and possible collisions.

@nirmalaagash
Copy link
Contributor

@vtomasr5 @jdziat I tried creating the csi driver (both helm version v1.2.4 and v2.1.0) following the steps link both the helm and manifest ways in step 3. I was not facing any issue while performing dynamic provisioning example. Please make sure your IAM service account is created right, the role is tagged to it and also annotate the service account with IAM role. Let me know if I went wrong anywhere.

@jdziat
Copy link
Author

jdziat commented Aug 27, 2021

@nirmalaagash i was using: https://github.com/terraform-aws-modules/terraform-aws-iam/tree/v4.3.0/modules/iam-assumable-role-with-oidc to do the role creation piece of this. I have added the necessary annotations in the chart as well.

@nirmalaagash
Copy link
Contributor

@jdziat Can you share the template(.yaml) that you were trying to apply in the cluster?

@jdziat
Copy link
Author

jdziat commented Aug 27, 2021

serviceAccount:
  controller:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: ${arn}
  snapshot:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: ${arn}
  node:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: ${arn}
enableVolumeResizing: ${enable_volume_resizing}
enableVolumeSnapshot: ${enable_volume_snapshot}

@nirmalaagash
Copy link
Contributor

@jdziat I tried creating the iam role using the terraform registry module. I did not face any issue in creating a volume.
Please ensure you provide these required inputs for the terraform,

  • provider_url <OIDC Provider URL from the Cluster>
  • role_policy_arns <ARN of the policy that you want to attach>
  • oidc_fully_qualified_subjects <["system:serviceaccount:kube-system:ebs-csi-controller-sa"]>
  • oidc_fully_qualified_audiences <["sts.amazonaws.com"]>
  • create_role <true>
  • role_name <ROLE_NAME>
  • number_of_role_policy_arns <NUMBER OF ROLE POLICY ARNS>

Once after you create the role using terraform, ensure that you annotate the service account 'ebs-csi-controller-sa' with the ARN of the role that you just created.

kubectl annotate serviceaccount ebs-csi-controller-sa -n kube-system eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>

Restart the pods using

kubectl delete pods -n kube-system -l=app=ebs-csi-controller

Let me know if you face any issues. In case you face any issues, please share the input values that you are passing through terraform.

@Kaelten
Copy link

Kaelten commented Sep 1, 2021

I was having a similar sounding issue, I was able to use this values file with the helm chart to get things working.

image:
  repository: 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/aws-ebs-csi-driver
enableVolumeResizing: 'true'
enableVolumeSnapshot: 'true'
controller:
  serviceAccount:
    create: 'true'
    name: ebs-csi-controller-sa
    annotations:
      eks.amazonaws.com/role-arn: <role arn>
storageClasses:
 - name: ebs-gp3
   annotations:
     storageclass.kubernetes.io/is-default-class: "true"
   volumeBindingMode: WaitForFirstConsumer
   reclaimPolicy: Retain
   parameters:
     encrypted: "true"
     type: "gp3"
 - name: ebs-io2
   volumeBindingMode: WaitForFirstConsumer
   reclaimPolicy: Retain
   parameters:
     allowAutoIOPSPerGBIncrease: "true"
     encrypted: "true"
     iopsPerGB: "10"
     type: "io2"

The role arn is the result of running this CFN snippet...

EBSDriverServiceRole:
    Type: AWS::IAM::Role
    Properties:
      # this kinda hurts me inside, but it's really he only way to parameterize keys
      AssumeRolePolicyDocument: !Sub
        - |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Federated": "arn:aws:iam::${AWS::AccountId}:oidc-provider/${OIDC_PROVIDER}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                  "StringEquals": {
                    "${OIDC_PROVIDER}:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa",
                    "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
                  }
                }
              }
            ]
          }
        - OIDC_PROVIDER: !ImportValue EKSClusterOIDCProvider
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - ec2:CreateSnapshot
                  - ec2:AttachVolume
                  - ec2:DetachVolume
                  - ec2:ModifyVolume
                  - ec2:DescribeAvailabilityZones
                  - ec2:DescribeInstances
                  - ec2:DescribeSnapshots
                  - ec2:DescribeTags
                  - ec2:DescribeVolumes
                  - ec2:DescribeVolumesModifications
                Resource: '*'
              - Effect: Allow
                Action:
                  - ec2:CreateTags
                Resource:
                  - arn:aws:ec2:*:*:volume/*
                  - arn:aws:ec2:*:*:snapshot/*
                Condition:
                  StringEquals:
                    ec2:CreateAction:
                      - CreateVolume
                      - CreateSnapshot
              - Effect: Allow
                Action:
                  - ec2:DeleteTags
                Resource:
                  - arn:aws:ec2:*:*:volume/*
                  - arn:aws:ec2:*:*:snapshot/*
              - Effect: Allow
                Action:
                  - ec2:CreateVolume
                Resource: '*'
                Condition:
                  StringLike:
                    aws:RequestTag/ebs.csi.aws.com/cluster: "true"
              - Effect: Allow
                Action:
                  - ec2:CreateVolume
                Resource: '*'
                Condition:
                  StringLike:
                    aws:RequestTag/CSIVolumeName: '*'
              - Effect: Allow
                Action:
                  - ec2:DeleteVolume
                Resource: '*'
                Condition:
                  StringLike:
                    ec2:ResourceTag/CSIVolumeName: '*'
              - Effect: Allow
                Action:
                  - ec2:DeleteVolume
                Resource: '*'
                Condition:
                  StringLike:
                    ec2:ResourceTag/ebs.csi.aws.com/cluster: "true"
              - Effect: Allow
                Action:
                  - ec2:DeleteSnapshot
                Resource: '*'
                Condition:
                  StringLike:
                    ec2:ResourceTag/CSIVolumeSnapshotName: '*'
              - Effect: Allow
                Action:
                  - ec2:DeleteSnapshot
                Resource: '*'
                Condition:
                  StringLike:
                    ec2:ResourceTag/ebs.csi.aws.com/cluster: "true"

@vtomasr5
Copy link

vtomasr5 commented Oct 5, 2021

You're right @nirmalaagash

In my case, I was missing the ServiceAccount annotation for the role. It works now with the v2.3.0

Thank you!

@jdziat
Copy link
Author

jdziat commented Oct 5, 2021

In my case the kubernetes service-account-name had to match the role that I created.

@jdziat jdziat closed this as completed Oct 5, 2021
@mathisve
Copy link

Anybody struggling with this, I recommend you restart the CSI controllers.

kubectl delete pods -n kube-system -l=app=ebs-csi-controller

If you annotate the serviceAccount, it doesn't utilize the assigned role until rebooted.

@jdziat
Copy link
Author

jdziat commented Feb 17, 2022

On configuration changes it should do a rolling deployment. At least depending on configuration of the deployment. I can't remember if i enabled Wave for this though.

@shturcite
Copy link

shturcite commented Aug 25, 2022

I had to delete the "${OIDC_PROVIDER}:aud": "sts.amazonaws.com" part from the condition in the IAM role policy to make it work, when using EKS 1.23 with AWS managed EBS CSI addon. Otherwise the ServiceAccount didn't assumed the IAM role and caused UnauthorizedOperation: You are not authorized to perform this operation.

@leftluoyi
Copy link

Anybody struggling with this, I recommend you restart the CSI controllers.

kubectl delete pods -n kube-system -l=app=ebs-csi-controller

If you annotate the serviceAccount, it doesn't utilize the assigned role until rebooted.

Saved my day!!

@cande1gut
Copy link

Is this a bug, intended or any other sort of problem? I have been struggling with this for weeks

Anybody struggling with this, I recommend you restart the CSI controllers.

kubectl delete pods -n kube-system -l=app=ebs-csi-controller

If you annotate the serviceAccount, it doesn't utilize the assigned role until rebooted.

@kyeongjun-dev
Copy link

@mathisve you saved my time so much Thank you... !!

@OneideLuizSchneider
Copy link

To make this work, just add to the EKS Cluster role the permission to create volumes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests