Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(eks): managed nodegroup support #6759

Merged
merged 36 commits into from
Mar 25, 2020
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
1686081
initial nodegroup support
pahud Mar 17, 2020
d28b856
- update README
pahud Mar 17, 2020
0fd3398
update integ test
pahud Mar 17, 2020
17a34ac
Update packages/@aws-cdk/aws-eks/lib/cluster.ts
pahud Mar 17, 2020
7662429
Update packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
pahud Mar 17, 2020
3075be0
Update packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
pahud Mar 17, 2020
055a918
Update packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
pahud Mar 17, 2020
29d5907
Update packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
pahud Mar 17, 2020
f060878
Update packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
pahud Mar 17, 2020
c5e77b6
Update packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
pahud Mar 17, 2020
f52094d
Merge branch 'master' into managed-ng
pahud Mar 17, 2020
433ce36
minor fix
pahud Mar 18, 2020
df44c52
remove duplicate
pahud Mar 18, 2020
a098055
add unit tests
pahud Mar 18, 2020
77e7871
update integ json
pahud Mar 18, 2020
bd6ae41
Merge branch 'master' into managed-ng
pahud Mar 18, 2020
1f11ccc
- create a nodegroup as the defaultCapacity instead of an autoscaling…
pahud Mar 19, 2020
694613c
Merge branch 'managed-ng' of github.com:pahud/aws-cdk into managed-ng
pahud Mar 19, 2020
4d9ac49
Merge branch 'master' into managed-ng
pahud Mar 19, 2020
cbbc499
- the cluster default capacity is nodegroup now
pahud Mar 21, 2020
22e0619
fix unit tests error
pahud Mar 22, 2020
edc50ee
Merge branch 'master' into managed-ng
pahud Mar 22, 2020
a982289
fix trailing space
pahud Mar 22, 2020
75aefae
Merge branch 'managed-ng' of github.com:pahud/aws-cdk into managed-ng
pahud Mar 22, 2020
36a3eaa
support multiple instance types
pahud Mar 22, 2020
bff7265
minor fix on the multiple instance types support
pahud Mar 22, 2020
b3abaac
Revert "support multiple instance types"
pahud Mar 23, 2020
b1a3b5f
Revert "minor fix on the multiple instance types support"
pahud Mar 23, 2020
36eae5d
revert
pahud Mar 23, 2020
257b028
update README with additional notes of NodeGroup and default capacity
pahud Mar 23, 2020
a3ba7ca
minor fix
pahud Mar 23, 2020
c267cb1
Merge branch 'master' into managed-ng
pahud Mar 25, 2020
70471fd
Add `defaultCapacityType` swith
pahud Mar 25, 2020
5c6a74b
merge branch 'managed-ng' of github.com:pahud/aws-cdk into managed-ng
pahud Mar 25, 2020
e13286d
add defaultNodegroup support and update the test and README
pahud Mar 25, 2020
9f7df9a
Merge branch 'master' into managed-ng
mergify[bot] Mar 25, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions packages/@aws-cdk/aws-eks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ manifests within EKS clusters.

This example defines an Amazon EKS cluster with the following configuration:

- 2x **m5.large** instances (this instance type suits most common use-cases, and is good value for money)
- Managed nodegroup with 2x **m5.large** instances (this instance type suits most common use-cases, and is good value for money)
- Dedicated VPC with default configuration (see [ec2.Vpc](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ec2-readme.html#vpc))
- A Kubernetes pod with a container based on the [paulbouwer/hello-kubernetes](https://github.com/paulbouwer/hello-kubernetes) image.

Expand All @@ -49,7 +49,7 @@ cluster.addResource('mypod', {

### Capacity

By default, `eks.Cluster` is created with x2 `m5.large` instances.
By default, `eks.Cluster` is created with a managed nodegroup with x2 `m5.large` instances.
pahud marked this conversation as resolved.
Show resolved Hide resolved

```ts
new eks.Cluster(this, 'cluster-two-m5-large');
Expand All @@ -71,18 +71,12 @@ To disable the default capacity, simply set `defaultCapacity` to `0`:
new eks.Cluster(this, 'cluster-with-no-capacity', { defaultCapacity: 0 });
```

The `cluster.defaultCapacity` property will reference the `AutoScalingGroup`
The `cluster.defaultCapacity` property will reference the `NodeGroup`
pahud marked this conversation as resolved.
Show resolved Hide resolved
resource for the default capacity. It will be `undefined` if `defaultCapacity`
is set to `0`:

```ts
const cluster = new eks.Cluster(this, 'my-cluster');
cluster.defaultCapacity!.scaleOnCpuUtilization('up', {
targetUtilizationPercent: 80
});
```

You can add customized capacity through `cluster.addCapacity()` or
You can add `AutoScalingGroup` resource as customized capacity through `cluster.addCapacity()` or
`cluster.addAutoScalingGroup()`:

```ts
Expand All @@ -93,6 +87,25 @@ cluster.addCapacity('frontend-nodes', {
});
```

### Managed Node Groups

Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances)
for Amazon EKS Kubernetes clusters. By default, `eks.Nodegroup` create a nodegroup with x2 `t3.medium` instances.

```ts
new eks.Nodegroup(stack, 'nodegroup', { cluster });
pahud marked this conversation as resolved.
Show resolved Hide resolved
```

You can add customized node group through `cluster.addNodegroup()`:

```ts
cluster.addNodegroup('nodegroup', {
instanceType: new ec2.InstanceType('m5.large'),
minSize: 4,
});
```


### Fargate

AWS Fargate is a technology that provides on-demand, right-sized compute
Expand Down
23 changes: 21 additions & 2 deletions packages/@aws-cdk/aws-eks/lib/cluster.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { FargateProfile, FargateProfileOptions } from './fargate-profile';
import { HelmChart, HelmChartOptions } from './helm-chart';
import { KubernetesPatch } from './k8s-patch';
import { KubernetesResource } from './k8s-resource';
import { Nodegroup, NodegroupOptions } from './managed-nodegroup';
import { spotInterruptHandler } from './spot-interrupt-handler';
import { renderUserData } from './user-data';

Expand Down Expand Up @@ -312,7 +313,7 @@ export class Cluster extends Resource implements ICluster {
* The auto scaling group that hosts the default capacity for this cluster.
* This will be `undefined` if the default capacity is set to 0.
*/
public readonly defaultCapacity?: autoscaling.AutoScalingGroup;
public readonly defaultCapacity?: Nodegroup;

/**
* If this cluster is kubectl-enabled, returns the `ClusterResource` object
Expand Down Expand Up @@ -422,7 +423,7 @@ export class Cluster extends Resource implements ICluster {
const minCapacity = props.defaultCapacity === undefined ? DEFAULT_CAPACITY_COUNT : props.defaultCapacity;
if (minCapacity > 0) {
const instanceType = props.defaultCapacityInstance || DEFAULT_CAPACITY_TYPE;
this.defaultCapacity = this.addCapacity('DefaultCapacity', { instanceType, minCapacity });
this.defaultCapacity = this.addNodegroup('DefaultCapacity', { instanceType, minSize: minCapacity } );
}

const outputConfigCommand = props.outputConfigCommand === undefined ? true : props.outputConfigCommand;
Expand Down Expand Up @@ -470,6 +471,24 @@ export class Cluster extends Resource implements ICluster {
return asg;
}

/**
* Add managed nodegroup to this Amazon EKS cluster
*
* This method will create a new managed nodegroup and add into the capacity.
*
* @see https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html
* @param id The ID of the nodegroup
* @param options options for creating a new nodegroup
*/
public addNodegroup(id: string, options?: NodegroupOptions): Nodegroup {
// initialize the awsAuth for this cluster
this._awsAuth = this._awsAuth ?? this.awsAuth;
return new Nodegroup(this, `Nodegroup${id}`, {
cluster: this,
...options,
});
}

/**
* Add compute capacity to this EKS cluster in the form of an AutoScalingGroup
*
Expand Down
3 changes: 2 additions & 1 deletion packages/@aws-cdk/aws-eks/lib/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ export * from './fargate-profile';
export * from './helm-chart';
export * from './k8s-patch';
export * from './k8s-resource';
export * from './fargate-cluster';
export * from './fargate-cluster';
export * from './managed-nodegroup';
271 changes: 271 additions & 0 deletions packages/@aws-cdk/aws-eks/lib/managed-nodegroup.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,271 @@
import { InstanceType, ISecurityGroup, SubnetSelection } from '@aws-cdk/aws-ec2';
import { IRole, ManagedPolicy, Role, ServicePrincipal } from '@aws-cdk/aws-iam';
import { Construct, IResource, Resource } from '@aws-cdk/core';
import { Cluster } from './cluster';
import { CfnNodegroup } from './eks.generated';

/**
* NodeGroup interface
*/
export interface INodegroup extends IResource {
/**
* Name of the nodegroup
* @attribute
*/
readonly nodegroupName: string
}

/**
* The AMI type for your node group. GPU instance types should use the `AL2_x86_64_GPU` AMI type, which uses the
* Amazon EKS-optimized Linux AMI with GPU support. Non-GPU instances should use the `AL2_x86_64` AMI type, which
* uses the Amazon EKS-optimized Linux AMI.
*/
export enum NodegroupAmiType {
AL2_X86_64 = 'AL2_x86_64',
AL2_X86_64_GPU = 'AL2_x86_64_GPU',
}

/**
* The remote access (SSH) configuration to use with your node group.
*
* @see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-eks-nodegroup-remoteaccess.html
*/
export interface NodegroupRemoteAccess {
/**
* The Amazon EC2 SSH key that provides access for SSH communication with the worker nodes in the managed node group.
*/
readonly sshKeyName: string;
/**
* The security groups that are allowed SSH access (port 22) to the worker nodes. If you specify an Amazon EC2 SSH
* key but do not specify a source security group when you create a managed node group, then port 22 on the worker
* nodes is opened to the internet (0.0.0.0/0).
*
* @default - port 22 on the worker nodes is opened to the internet (0.0.0.0/0)
*/
readonly sourceSecurityGroups?: ISecurityGroup[];
}

/**
* The Nodegroup Options for addNodeGroup() method
*/
export interface NodegroupOptions {
/**
* Name of the Nodegroup
*
* @default - resource ID
*/
readonly nodegroupName?: string;
/**
* The subnets to use for the Auto Scaling group that is created for your node group. By specifying the
* SubnetSelection, the selected subnets will automatically apply required tags i.e.
* `kubernetes.io/cluster/CLUSTER_NAME` with a value of `shared`, where `CLUSTER_NAME` is replaced with
* the name of your cluster.
*
* @default - private subnets
*/
readonly subnets?: SubnetSelection;
/**
* The AMI type for your node group.
*
* @default AL2_x86_64
*/
readonly amiType?: NodegroupAmiType;
/**
* The root device disk size (in GiB) for your node group instances.
*
* @default 20
*/
readonly diskSize?: number;
/**
* The current number of worker nodes that the managed node group should maintain. If not specified,
* the nodewgroup will initially create `minSize` instances.
*
* @default 2
*/
readonly desiredSize?: number;
/**
* The maximum number of worker nodes that the managed node group can scale out to. Managed node groups can support up to 100 nodes by default.
*
* @default - desiredSize
*/
readonly maxSize?: number;
/**
* The minimum number of worker nodes that the managed node group can scale in to. This number must be greater than zero.
*
* @default 1
*/
readonly minSize?: number;
/**
* Force the update if the existing node group's pods are unable to be drained due to a pod disruption budget issue.
* If an update fails because pods could not be drained, you can force the update after it fails to terminate the old
* node whether or not any pods are
* running on the node.
*
* @default true
*/
readonly forceUpdate?: boolean;
/**
* The instance type to use for your node group. Currently, you can specify a single instance type for a node group.
* The default value for this parameter is `t3.medium`. If you choose a GPU instance type, be sure to specify the
* `AL2_x86_64_GPU` with the amiType parameter.
*
* @default t3.medium
*/
readonly instanceType?: InstanceType;
/**
* The Kubernetes labels to be applied to the nodes in the node group when they are created.
*
* @default - None
*/
readonly labels?: { [name: string]: string };
/**
* The IAM role to associate with your node group. The Amazon EKS worker node kubelet daemon
* makes calls to AWS APIs on your behalf. Worker nodes receive permissions for these API calls through
* an IAM instance profile and associated policies. Before you can launch worker nodes and register them
* into a cluster, you must create an IAM role for those worker nodes to use when they are launched.
*
* @default - None. Auto-generated if not specified.
*/
readonly nodeRole?: IRole;
/**
* The AMI version of the Amazon EKS-optimized AMI to use with your node group (for example, `1.14.7-YYYYMMDD`).
*
* @default - The latest available AMI version for the node group's current Kubernetes version is used.
*/
readonly releaseVersion?: string;
/**
* The remote access (SSH) configuration to use with your node group. Disabled by default, however, if you
* specify an Amazon EC2 SSH key but do not specify a source security group when you create a managed node group,
* then port 22 on the worker nodes is opened to the internet (0.0.0.0/0)
*
* @default - disabled
*/
readonly remoteAccess?: NodegroupRemoteAccess;
/**
* The metadata to apply to the node group to assist with categorization and organization. Each tag consists of
* a key and an optional value, both of which you define. Node group tags do not propagate to any other resources
* associated with the node group, such as the Amazon EC2 instances or subnets.
*
* @default - None
*/
readonly tags?: { [name: string]: string };
}

/**
* NodeGroup properties interface
*/
export interface NodegroupProps extends NodegroupOptions {
/**
* Cluster resource
* [disable-awslint:ref-via-interface]"
*/
readonly cluster: Cluster;
}

/**
* The Nodegroup resource class
*/
export class Nodegroup extends Resource implements INodegroup {
/**
* Import the Nodegroup from attributes
*/
public static fromNodegroupName(scope: Construct, id: string, nodegroupName: string): INodegroup {
class Import extends Resource implements INodegroup {
public readonly nodegroupName = nodegroupName;
}
return new Import(scope, id);
}
/**
* ARN of the nodegroup
*
* @attribute
*/
public readonly nodegroupArn: string;
/**
* Nodegroup name
*
* @attribute
*/
public readonly nodegroupName: string;
/**
* the Amazon EKS cluster resource
*
* @attribute ClusterName
*/
public readonly cluster: Cluster;
/**
* IAM role of the instance profile for the nodegroup
*/
public readonly role: IRole;

private readonly desiredSize: number;
private readonly maxSize: number;
private readonly minSize: number;

constructor(scope: Construct, id: string, props: NodegroupProps ) {
super(scope, id, {
physicalName: props.nodegroupName
});

this.cluster = props.cluster;

this.desiredSize = props.desiredSize ?? props.minSize ?? 2;
this.maxSize = props.maxSize ?? this.desiredSize;
this.minSize = props.minSize ?? 1;

if (this.desiredSize > this.maxSize) {
throw new Error(`Desired capacity ${this.desiredSize} can't be greater than max size ${this.maxSize}`);
}
if (this.desiredSize < this.minSize) {
throw new Error(`Minimum capacity ${this.minSize} can't be greater than desired size ${this.desiredSize}`);
}

if (!props.nodeRole) {
const ngRole = new Role(this, 'NodeGroupRole', {
assumedBy: new ServicePrincipal('ec2.amazonaws.com')
});

ngRole.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSWorkerNodePolicy'));
ngRole.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('AmazonEKS_CNI_Policy'));
ngRole.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ContainerRegistryReadOnly'));
this.role = ngRole;
} else {
this.role = props.nodeRole;
}

const resource = new CfnNodegroup(this, 'Resource', {
clusterName: this.cluster.clusterName,
nodegroupName: props.nodegroupName,
nodeRole: this.role.roleArn,
subnets: this.cluster.vpc.selectSubnets(props.subnets).subnetIds,
amiType: props.amiType,
diskSize: props.diskSize,
forceUpdateEnabled: props.forceUpdate ?? true,
instanceTypes: props.instanceType ? [props.instanceType.toString()] : undefined,
pahud marked this conversation as resolved.
Show resolved Hide resolved
labels: props.labels,
releaseVersion: props.releaseVersion,
remoteAccess: props.remoteAccess ? {
ec2SshKey: props.remoteAccess.sshKeyName,
sourceSecurityGroups: props.remoteAccess.sourceSecurityGroups ?
props.remoteAccess.sourceSecurityGroups.map(m => m.securityGroupId) : undefined
} : undefined,
scalingConfig: {
desiredSize: this.desiredSize,
maxSize: this.maxSize,
minSize: this.minSize
},
tags: props.tags
});

// As managed nodegroup will auto map the instance role to RBAC behind the scene and users don't have to manually
// do it anymore. We don't need to print out the instance role arn now.

this.nodegroupArn = this.getResourceArnAttribute(resource.attrArn, {
service: 'eks',
resource: 'nodegroup',
resourceName: this.physicalName,
});
this.nodegroupName = this.getResourceNameAttribute(resource.ref);
}

}
Loading