-
Notifications
You must be signed in to change notification settings - Fork 807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to use EKS IAM role #1024
Comments
Hi, I see the same problem.
Role policy:
I also tried the example https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/example-iam-policy.json with the same result. |
BTW, using the helm chart 1.2.x it works fine with either IAM policy. |
Thanks @vtomasr5 @nirmalaagash i can make a gist of the terraform module i've created it's not the most condensed thing. Also the secret name should be customizable since all the aws-csi charts use that same name it leads to conflict and possible collisions. |
@vtomasr5 @jdziat I tried creating the csi driver (both helm version v1.2.4 and v2.1.0) following the steps link both the helm and manifest ways in step 3. I was not facing any issue while performing dynamic provisioning example. Please make sure your IAM service account is created right, the role is tagged to it and also annotate the service account with IAM role. Let me know if I went wrong anywhere. |
@nirmalaagash i was using: https://github.com/terraform-aws-modules/terraform-aws-iam/tree/v4.3.0/modules/iam-assumable-role-with-oidc to do the role creation piece of this. I have added the necessary annotations in the chart as well. |
@jdziat Can you share the template(.yaml) that you were trying to apply in the cluster? |
|
@jdziat I tried creating the iam role using the terraform registry module. I did not face any issue in creating a volume.
Once after you create the role using terraform, ensure that you annotate the service account 'ebs-csi-controller-sa' with the ARN of the role that you just created.
Restart the pods using
Let me know if you face any issues. In case you face any issues, please share the input values that you are passing through terraform. |
I was having a similar sounding issue, I was able to use this values file with the helm chart to get things working. image:
repository: 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/aws-ebs-csi-driver
enableVolumeResizing: 'true'
enableVolumeSnapshot: 'true'
controller:
serviceAccount:
create: 'true'
name: ebs-csi-controller-sa
annotations:
eks.amazonaws.com/role-arn: <role arn>
storageClasses:
- name: ebs-gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
parameters:
encrypted: "true"
type: "gp3"
- name: ebs-io2
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
parameters:
allowAutoIOPSPerGBIncrease: "true"
encrypted: "true"
iopsPerGB: "10"
type: "io2" The role arn is the result of running this CFN snippet... EBSDriverServiceRole:
Type: AWS::IAM::Role
Properties:
# this kinda hurts me inside, but it's really he only way to parameterize keys
AssumeRolePolicyDocument: !Sub
- |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${AWS::AccountId}:oidc-provider/${OIDC_PROVIDER}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OIDC_PROVIDER}:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa",
"${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
}
}
}
]
}
- OIDC_PROVIDER: !ImportValue EKSClusterOIDCProvider
Policies:
- PolicyName: root
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ec2:CreateSnapshot
- ec2:AttachVolume
- ec2:DetachVolume
- ec2:ModifyVolume
- ec2:DescribeAvailabilityZones
- ec2:DescribeInstances
- ec2:DescribeSnapshots
- ec2:DescribeTags
- ec2:DescribeVolumes
- ec2:DescribeVolumesModifications
Resource: '*'
- Effect: Allow
Action:
- ec2:CreateTags
Resource:
- arn:aws:ec2:*:*:volume/*
- arn:aws:ec2:*:*:snapshot/*
Condition:
StringEquals:
ec2:CreateAction:
- CreateVolume
- CreateSnapshot
- Effect: Allow
Action:
- ec2:DeleteTags
Resource:
- arn:aws:ec2:*:*:volume/*
- arn:aws:ec2:*:*:snapshot/*
- Effect: Allow
Action:
- ec2:CreateVolume
Resource: '*'
Condition:
StringLike:
aws:RequestTag/ebs.csi.aws.com/cluster: "true"
- Effect: Allow
Action:
- ec2:CreateVolume
Resource: '*'
Condition:
StringLike:
aws:RequestTag/CSIVolumeName: '*'
- Effect: Allow
Action:
- ec2:DeleteVolume
Resource: '*'
Condition:
StringLike:
ec2:ResourceTag/CSIVolumeName: '*'
- Effect: Allow
Action:
- ec2:DeleteVolume
Resource: '*'
Condition:
StringLike:
ec2:ResourceTag/ebs.csi.aws.com/cluster: "true"
- Effect: Allow
Action:
- ec2:DeleteSnapshot
Resource: '*'
Condition:
StringLike:
ec2:ResourceTag/CSIVolumeSnapshotName: '*'
- Effect: Allow
Action:
- ec2:DeleteSnapshot
Resource: '*'
Condition:
StringLike:
ec2:ResourceTag/ebs.csi.aws.com/cluster: "true" |
You're right @nirmalaagash In my case, I was missing the ServiceAccount annotation for the role. It works now with the v2.3.0 Thank you! |
In my case the kubernetes service-account-name had to match the role that I created. |
Anybody struggling with this, I recommend you restart the CSI controllers.
If you annotate the serviceAccount, it doesn't utilize the assigned role until rebooted. |
On configuration changes it should do a rolling deployment. At least depending on configuration of the deployment. I can't remember if i enabled Wave for this though. |
I had to delete the |
Saved my day!! |
Is this a bug, intended or any other sort of problem? I have been struggling with this for weeks
|
@mathisve you saved my time so much Thank you... !! |
To make this work, just add to the |
/kind bug
What happened?
When using IAM role and policy with a helm chart I get the below error:
Same Policy works when using a kubernetes secret with the aws key and secret.
What you expected to happen?
The role should be used successfully when attempting to perform the operation.
How to reproduce it (as minimally and precisely as possible)?
Attempt to create and use the IAM role auth policy using the helm chart and terraform module:
terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc
Anything else we need to know?:
Environment
kubectl version
): v1.20.7-eks-d88609The text was updated successfully, but these errors were encountered: