Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws clusters: existing clusters to use eks managed addons and gp3 storage class with volume expansion by default #5129

Merged
merged 3 commits into from
Nov 18, 2024
Merged

aws clusters: existing clusters to use eks managed addons and gp3 storage class with volume expansion by default #5129

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a2decc
aws clusters: transition to eks managed addons from deprecated eksctl…
consideRatio Nov 18, 2024
8d78c9b
docs: update documentation on EKS add-ons upgrades
consideRatio Nov 18, 2024
f684d07
aws docs: volume expansion is setup by default and require no action …
consideRatio Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 9 additions & 15 deletions docs/howto/upgrade-cluster/aws.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,26 +101,20 @@ now upgraded.

#### 4.2. Upgrade EKS add-ons

As documented in `eksctl`'s documentation[^1], we also need to upgrade three EKS
add-ons managed by `eksctl` (by EKS these are considered self-managed add-ons),
and one declared in our config (by EKS this is considered a managed add-on).

These upgrades are believed to briefly disrupt networking.
As documented in `eksctl`'s documentation[^2], we also need to upgrade EKS
add-ons. This upgrade is believed to very briefly disrupt networking.

```bash
# upgrade the kube-proxy daemonset (takes ~5s)
eksctl utils update-kube-proxy --config-file=$CLUSTER_NAME.eksctl.yaml --approve

# upgrade the aws-node daemonset (takes ~5s)
eksctl utils update-aws-node --config-file=$CLUSTER_NAME.eksctl.yaml --approve

# upgrade the coredns deployment (takes ~5s)
eksctl utils update-coredns --config-file=$CLUSTER_NAME.eksctl.yaml --approve

# upgrade the aws-ebs-csi-driver addon's deployment and daemonset (takes ~60s)
# upgrade all EKS addons (takes up to a few minutes)
eksctl update addon --config-file=$CLUSTER_NAME.eksctl.yaml
```

```{note}
Since November 2024, the add-ons are systematically installed as EKS managed
add-ons and not self-managed as they were before when `eksctl` installed them
without involving EKS.
```

#### 4.3. Repeat to upgrade multiple minor versions

If you need to upgrade multiple minor versions, repeat the previous steps
Expand Down
14 changes: 0 additions & 14 deletions docs/hub-deployment-guide/new-cluster/new-cluster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -712,17 +712,3 @@ kubectl get node
It should show you the provisioned node on the cluster if everything works out ok.
````
`````

## AWS only: Expandable storage class

The default storage class that is created when we deploy a cluster to AWS does permit auto-expansion of persistent volumes.
This can cause problems when we want to expand the size of a disk, say used by Prometheus to store metrics data.
We will therefore patch the default storage class to permite auto-expansion.

```bash
# Gain k8s access to the cluster
deployer use-cluster-credentials $CLUSTER_NAME

# Patch the storage class
kubectl patch storageclass gp2 --patch '{\"allowVolumeExpansion\": true}'
```
58 changes: 45 additions & 13 deletions eksctl/2i2c-aws-us.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,10 @@ local daskNodes = [
name: "2i2c-aws-us",
region: clusterRegion,
version: "1.30",
tags+: {
"ManagedBy": "2i2c",
"2i2c.org/cluster-name": $.metadata.name,
},
},
availabilityZones: masterAzs,
iam: {
Expand All @@ -223,20 +227,48 @@ local daskNodes = [
// eksctl create addon --config-file=2i2c-aws-us.eksctl.yaml
//
addons: [
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: 'aws-ebs-csi-driver',
version: "latest",
wellKnownPolicies: {
ebsCSIController: true,
{ version: "latest", tags: $.metadata.tags } + addon
for addon in
[
{ name: "coredns" },
{ name: "kube-proxy" },
{
// vpc-cni is a Amazon maintained container networking interface
// (CNI), where a CNI is required for k8s networking. The aws-node
// DaemonSet in kube-system stems from installing this.
//
// Related docs: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
// https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
//
name: "vpc-cni",
attachPolicyARNs: ["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"],
# FIXME: enabling network policy enforcement didn't work as of
# August 2024, what's wrong isn't clear.
#
# configurationValues ref: https://github.com/aws/amazon-vpc-cni-k8s/blob/HEAD/charts/aws-vpc-cni/values.yaml
configurationValues: |||
enableNetworkPolicy: "false"
|||,
},
},
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: "aws-ebs-csi-driver",
wellKnownPolicies: {
ebsCSIController: true,
},
# configurationValues ref: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/HEAD/charts/aws-ebs-csi-driver/values.yaml
configurationValues: |||
defaultStorageClass:
enabled: true
|||,
},
]
],
nodeGroups: [
n + {clusterName: $.metadata.name} for n in
Expand Down
58 changes: 45 additions & 13 deletions eksctl/catalystproject-africa.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ local daskNodes = [];
name: "catalystproject-africa",
region: clusterRegion,
version: "1.30",
tags+: {
"ManagedBy": "2i2c",
"2i2c.org/cluster-name": $.metadata.name,
},
},
availabilityZones: masterAzs,
iam: {
Expand All @@ -88,20 +92,48 @@ local daskNodes = [];
// eksctl create addon --config-file=catalystproject-africa.eksctl.yaml
//
addons: [
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: 'aws-ebs-csi-driver',
version: "latest",
wellKnownPolicies: {
ebsCSIController: true,
{ version: "latest", tags: $.metadata.tags } + addon
for addon in
[
{ name: "coredns" },
{ name: "kube-proxy" },
{
// vpc-cni is a Amazon maintained container networking interface
// (CNI), where a CNI is required for k8s networking. The aws-node
// DaemonSet in kube-system stems from installing this.
//
// Related docs: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
// https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
//
name: "vpc-cni",
attachPolicyARNs: ["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"],
# FIXME: enabling network policy enforcement didn't work as of
# August 2024, what's wrong isn't clear.
#
# configurationValues ref: https://github.com/aws/amazon-vpc-cni-k8s/blob/HEAD/charts/aws-vpc-cni/values.yaml
configurationValues: |||
enableNetworkPolicy: "false"
|||,
},
},
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: "aws-ebs-csi-driver",
wellKnownPolicies: {
ebsCSIController: true,
},
# configurationValues ref: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/HEAD/charts/aws-ebs-csi-driver/values.yaml
configurationValues: |||
defaultStorageClass:
enabled: true
|||,
},
]
],
nodeGroups: [
n + {clusterName: $.metadata.name} for n in
Expand Down
55 changes: 42 additions & 13 deletions eksctl/earthscope.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ local daskNodes = [
version: "1.30",
tags+: {
"ManagedBy": "2i2c",
"2i2c.org/cluster-name": $.metadata.name,
"earthscope:application:name": "geolab",
"earthscope:application:owner": "research-onramp-to-the-cloud"
},
Expand All @@ -130,20 +131,48 @@ local daskNodes = [
// eksctl create addon --config-file=earthscope.eksctl.yaml
//
addons: [
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: 'aws-ebs-csi-driver',
version: "latest",
wellKnownPolicies: {
ebsCSIController: true,
{ version: "latest", tags: $.metadata.tags } + addon
for addon in
[
{ name: "coredns" },
{ name: "kube-proxy" },
{
// vpc-cni is a Amazon maintained container networking interface
// (CNI), where a CNI is required for k8s networking. The aws-node
// DaemonSet in kube-system stems from installing this.
//
// Related docs: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
// https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
//
name: "vpc-cni",
attachPolicyARNs: ["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"],
# FIXME: enabling network policy enforcement didn't work as of
# August 2024, what's wrong isn't clear.
#
# configurationValues ref: https://github.com/aws/amazon-vpc-cni-k8s/blob/HEAD/charts/aws-vpc-cni/values.yaml
configurationValues: |||
enableNetworkPolicy: "false"
|||,
},
},
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: "aws-ebs-csi-driver",
wellKnownPolicies: {
ebsCSIController: true,
},
# configurationValues ref: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/HEAD/charts/aws-ebs-csi-driver/values.yaml
configurationValues: |||
defaultStorageClass:
enabled: true
|||,
},
]
],
nodeGroups: [
n + {clusterName: $.metadata.name} for n in
Expand Down
58 changes: 45 additions & 13 deletions eksctl/gridsst.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,10 @@ local daskNodes = [
name: "gridsst",
region: clusterRegion,
version: "1.29",
tags+: {
"ManagedBy": "2i2c",
"2i2c.org/cluster-name": $.metadata.name,
},
},
availabilityZones: masterAzs,
iam: {
Expand All @@ -84,20 +88,48 @@ local daskNodes = [
// eksctl create addon --config-file=gridsst.eksctl.yaml
//
addons: [
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: 'aws-ebs-csi-driver',
version: "latest",
wellKnownPolicies: {
ebsCSIController: true,
{ version: "latest", tags: $.metadata.tags } + addon
for addon in
[
{ name: "coredns" },
{ name: "kube-proxy" },
{
// vpc-cni is a Amazon maintained container networking interface
// (CNI), where a CNI is required for k8s networking. The aws-node
// DaemonSet in kube-system stems from installing this.
//
// Related docs: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
// https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
//
name: "vpc-cni",
attachPolicyARNs: ["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"],
# FIXME: enabling network policy enforcement didn't work as of
# August 2024, what's wrong isn't clear.
#
# configurationValues ref: https://github.com/aws/amazon-vpc-cni-k8s/blob/HEAD/charts/aws-vpc-cni/values.yaml
configurationValues: |||
enableNetworkPolicy: "false"
|||,
},
},
{
// aws-ebs-csi-driver ensures that our PVCs are bound to PVs that
// couple to AWS EBS based storage, without it expect to see pods
// mounting a PVC failing to schedule and PVC resources that are
// unbound.
//
// Related docs: https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html
//
name: "aws-ebs-csi-driver",
wellKnownPolicies: {
ebsCSIController: true,
},
# configurationValues ref: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/HEAD/charts/aws-ebs-csi-driver/values.yaml
configurationValues: |||
defaultStorageClass:
enabled: true
|||,
},
]
],
nodeGroups: [
n + {clusterName: $.metadata.name} for n in
Expand Down
Loading