-
Notifications
You must be signed in to change notification settings - Fork 995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Could not launch node, launching instances, with fleet error(s), UnauthorizedOperation: You are not authorized to perform this operation. #1488
Comments
For better understanding I will provide the context of my cluster: EKS Cluster with IRSA integration
Karpenter IRSA
From your documentation I have to use The question would be:
|
@cradules you can get get role_name like: However, I am getting same error. |
Thank you @gopiio! I have already tried that, before opening the issue. In my personal view, the right code would look like this:
As an attached policy I have:
But I have the same error. I even edit manually the role, temporarily, and I did add administrator policy, and I have the same error. A part of the decoded message is :
From what I can see, the karpenter is not allowed to The policy attached to the role would be:
|
Update: By manually assigning the AutoScalingFullAccess to
I did attach the
|
Hey @cradules , |
Edit: |
Similar issue on my end. Using eks module (latest version) and iam-for-service-accounts-eks module. I will say while troubleshooting, I can get it working by removing the below condition from the controller policy if that's any help.
|
You can try this Terraform EKS Accelerator example for Karpenter if it helps. |
I managed to workaround by adding IAM policy manually using aws_iam_role_policy. With |
I had some time to look into this further. We have two gaps which surfaced as part of this issue. First, Karpenter Getting Started docs are geared towards older versions of the EKS module (<18). It would be worthwhile to update the docs to support version 18 and greater. I believe the answer to the specific question about what role should be used for the Karpenter node instance profile is either to use
Second, the Karpenter IRSA Terraform module seems to be incorrect. Unless I'm mistaken, I think we should look into having this condition removed from the Karpenter Controller IAM policy: https://github.com/terraform-aws-modules/terraform-aws-iam/blob/master/modules/iam-role-for-service-accounts-eks/policies.tf#L456-L460 (thanks to @shane-snyder for finding this). Any thoughts on these two items? |
@dewjam |
That's a good callout @shane-snyder . I'll look into it further. I'm seeing something slightly different than you are, however. From what I can tell, the failure to launch a new instance is due to the condition applied to the We also will not be able to terminate instances or delete launch templates unless the |
Sorry, I did not have the time to work on this issue, for the last few days. I hope next week, not to be so busy and I will have the chance to dig some more, especially since I have some starting points that emerged from @shane-snyder input, and I want to thank him for that! |
FYI, it seems that Karpenter are not applying default tags to AWS resources per the documentation outlined here. This is relevant as the condition in the Terraform Karpenter IRSA module could instead target one of our default tags. I'm working on a fix now. |
I've re-opened #1332 again which is related - any support on getting this across the line from the maintainers would be appreciated |
Thanks @bryantbiggs . It seems I was missing some context when looking into this issue. As far as your PR goes, I think we were waiting for your changes to be tested. I've commented on the PR as well. |
#1332 was merged yesterday, which updates the Karpenter Terraform Getting Started guide so it uses v18+ of the Terraform EKS module. Also, the Karpenter IRSA modules has been updated with some fixes. Thanks to @bryantbiggs for his hard work on this PR! @cradules, |
I am having a similar error: ERROR controller.provisioning Launching node, creating cloud provider machine, with fleet error(s), UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: **** The decoded message is as follows:
|
@dewjam I'm using the latest version of the documentation (preview) -> https://karpenter.sh/preview/getting-started/getting-started-with-terraform/ |
Hey @midestefanis , |
Also, did you provision the cluster using the getting Terraform Started guide? - Yes Provisioner spec:
|
Hey @midestefanis , The "tags" key in your Provisioner spec must be the same, otherwise the condition will not match. If you modify your provisioner to instead use the below, it should work as expected:
@bryantbiggs , |
@dewjam I can definitely add that - however, that wouldn't have resolved the issue because the tag is written incorrectly. What would it take to bake in a tag that I can set on the IRSA module as the default (users could still override) - https://karpenter.sh/v0.8.2/aws/provisioning/#tags (meaning, users don't have to remember to set the tag on the provisioner) I'll open a PR to make the IAM statement condition for the tag configurable to opt out of, but the default will still be scoped to the condition so that users are starting with a preferred security practice |
@dewjam That is not an option because the tags must be customizable. I use multiple clusters in the same account and VPC, therefore I cannot tag in that way since that key is unique. It should be possible to do that last thing you mention. |
ah I see - that makes sense. one sec |
@midestefanis if you update your IRSA role to the following, this should start working for your use case now: module "karpenter_irsa" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
version = "4.19.0" # <= available starting in 4.19.0
role_name = "karpenter-controller-${local.cluster_name}"
attach_karpenter_controller_policy = true
karpenter_tag_key = "karpenter.sh/discovery/${module.eks.cluster_id}" # <= this
karpenter_controller_cluster_id = module.eks.cluster_id
karpenter_controller_node_iam_role_arns = [
module.eks.eks_managed_node_groups["initial"].iam_role_arn
]
oidc_providers = {
ex = {
provider_arn = module.eks.oidc_provider_arn
namespace_service_accounts = ["karpenter:karpenter"]
}
}
} |
@bryantbiggs Thank you! |
It seems as if the original purpose of this issue has been resolved. I'm going to go ahead and close this out, but please feel free to reopen if you disagree :). |
@dewjam and @bryantbiggs, I got same / similar problems. Refer to the issue, #2519. Can you please take a look? |
@garyyang6 please open a new issue and provide a reproduction plus the error you are seeing |
@bryantbiggs You may find all the details at #2519. Do I need to create another one? |
For us the problem was a version mismatch between the cloudformation stack and the controller running in the cluster. Make sure
|
I was getting the same error whist trying to install Karpenter |
Is an existing page relevant?
https://karpenter.sh/v0.6.5/getting-started/getting-started-with-terraform/
What karpenter features are relevant?
aws_iam_instance_profile
How should the docs be improved?
From the above example, I have to assign for instance profile worker's IAM role name. The problem is, that in the latest EKS module version
worker_iam_role_name
output does not exist anymore.I am not sure what value I have to use here, because I don't know what permission need karpenter for aws_iam_instance_profile
Any suggestions?
Community Note
The text was updated successfully, but these errors were encountered: