Design Doc on AWS Provider LaunchTemplate evolution #1270
Conversation
✅ Deploy Preview for karpenter-docs-prod ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
| 1. In order to override [the root volume](https://github.com/aws/karpenter/issues/885), you’d need to author an entire launch template containing security groups, userData etc. | ||
| 2. In order to just have all statically provisioned [volumes be encrypted](https://github.com/aws/karpenter/issues/933), you need to author an entire launch template. | ||
|
|
||
| While some users do like complete control over the launch template since they customize everything (kOps), most only need to override a few settings and would therefore like those to be present as knobs on the provisioner itself, and for all other settings to be figured out by Karpenter. |
There was a problem hiding this comment.
The challenge here is really that karpenter assumes EKS and anything that is not EKS need different launch templates. Rancher, OKD etc all will have this requirement. Had Karpenter been made to work firstly with kOps, EKS would have this requirement.
I don't think it is feasible that Karpenter truly can manage LTs that will work across all these installers, and appreciate that for EKS this is needed since there is no tooling for managing LTs. But other installers do provide that tooling. So certainly no kOps user will want to manage their own LTs. In fact, kOps do not allow custom LTs, and I haven't seen a single request for such a feature (our approach is somewhat similar to approach 2 below). End goal being sort of the same as this aspect of Karpenter: hide the complexity of LTs.
There was a problem hiding this comment.
This suggests to me that we probably need to make provider: Kops or provider: AWSKops implementation, since they vary so much.
bwagner5
changed the title
|
|
||
| ### Recommendation | ||
|
|
||
| I recommend we continue with Approach 1, and give everyone the ability to customize a few select, commonly used fields in the provisionerSpec and ask them to configure all others via a complete LaunchTemplate. I think this keeps our API easy to use. The fields I expect us to add to the Provider in the short term are _BlockDeviceMappings, ImageId / AMIFamily and UserData._ The use cases for other fields either don’t apply to Karpenter, or they’re quite niche where it probably makes sense for the customer to manage all other aspects of the launch template in any case. |
There was a problem hiding this comment.
Personally, I am more in favor of Approach 3 as it is more flexible (allows for more use-cases) without having to change karpenter's code base again and again. I am looking at this from an operator's (not developer's) point-of-view, in which it is often frustrating and annoying that developers do not understand that operator's (customer's) need for flexible tools (and not just opinionated and limited tools that don't get the job done).
There was a problem hiding this comment.
I try and call this out early in the doc, but the core reason I'm recommending the first approach is because we often see a need to only override one or two of the AWS Provider's opinionated defaults, so defining an entire launch template is too heavy handed.
But if you've got more advanced needs that frequently change, maintaining an entire launch template shouldn't be too cumbersome and likely similar to embedding all of that information within the provisioner.
|
I'd like to see some discussion around the complexity of the bootstrap.sh script in eksoptimized ami, as well as partial user data. e.g., bottlerocket userdata is pretty easy to share, since it's a toml file that can be merged or appended easily. EKSOptimizedAMI is much more challenging since the bootstrap script may have user args (iprefix) and karpenter args (labels/taints) in the same arg string. |
Agreed. I'm going to add a separate doc to flesh out the different options on what the support of AMI and UserData is going to look like. Sorry about the delay on adding that in here. |
1. Issue, if available:
N/A
2. Description of changes:
We've had a lot of requests around launch template functionality. This document goes over some options we can consider for the AWS Provider.
3. How was this change tested?
N/A
4. Does this change impact docs?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.