Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Manually create Equinix cluster #2

Merged
merged 3 commits into from
Nov 16, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 122 additions & 0 deletions docs/create_equinix_cluster.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Create Equinix Metal cluster using CAPI / CAPEM

## Overview

- We are using machines contributed to the CNCF by Equinix Metal.
- These steps manually create a cluster using CAPI / CAPEM using a local
Kind cluster for the management cluster.
- These docs are based on the Equinix CAPI [guide](https://deploy.equinix.com/developers/guides/kubernetes-cluster-api/).
- Using [Podman](https://github.com/kubernetes-sigs/cluster-api-provider-packet/issues/496)
is recommended by Equinix for MacOS.
- We plan to automate these steps later using GitHub Actions and an IaC tool.

## Create Cluster

- Log in to the Equinix Metal [console](https://console.equinix.com/) and get
the project API key from the project settings. If it doesn't exist create it.

```sh
export PACKET_API_KEY="<PROJECT_API_KEY>"
```

- Install the CAPI controllers using [clusterctl](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl).

```sh
clusterctl init --infrastructure packet
```

- Set env vars with cluster config.

```sh
# Get the project ID from the project settings in the console
export PROJECT_ID="<PROJECT_ID>"

# Use Paris metro (Equinix region)
export METRO="pa"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to consider carbon intensity when deciding which metro to use?

I chose Paris since France has a low average carbon intensity and this metro has a good selection of node types.

Full list is here and we can easily change if I'm over thinking this :)
https://deploy.equinix.com/developers/docs/metal/locations/metros/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very important callout! We can hardcode Paris for now and make the testing carbon-aware in the future. Would you like to create a new issue for this to add it to our backlog? 🌟

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes sure I'll create an issue for it. 👍


# Use Ubuntu 22.04 with cGroup v2 for Kepler
export NODE_OS="ubuntu_22_04"

# The pod and service CIDRs for the new cluster
export POD_CIDR="192.168.0.0/16"
export SERVICE_CIDR="172.26.0.0/16"

# Use node type with Intel CPUs for RAPL support
export CONTROLPLANE_NODE_TYPE="m3.small.x86"
export WORKER_NODE_TYPE="m3.small.x86"
Copy link
Contributor Author

@rossf7 rossf7 Oct 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its safest to use a node type with an Intel CPU. I tested with an AMD CPU and Kepler couldn't access RAPL.

Screenshot 2023-10-18 at 11 15 34

I think this node type has good enough resources for now. WDYT?

See https://deploy.equinix.com/product/bare-metal/servers/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for using an Intel CPU and starting with small! 👍


# SSH key to use for access to nodes
export SSH_KEY="<YOUR_SSH_KEY>"

# Kubernetes version to install
export KUBERNETES_VERSION="v1.28.2"
```

- Generate cluster manifests.

```sh
clusterctl generate cluster wg-green-reviews \
--kubernetes-version $KUBERNETES_VERSION \
--control-plane-machine-count=1 \
--worker-machine-count=1 \
> wg-green-reviews.yaml
```
rossf7 marked this conversation as resolved.
Show resolved Hide resolved

- Apply cluster manifests.

```sh
kubectl apply -f wg-green-reviews.yaml
```

- Get kubeconfig and store it securely.

```sh
clusterctl get kubeconfig wg-green-reviews > wg-green-reviews.kubeconfig
```

- Set `KUBECONFIG` env var so following commands are run on the cluster.

```sh
export KUBECONFIG=wg-green-reviews.kubeconfig
```

- Install Cilium as CNI.

```sh
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.14.2 --namespace kube-system
```

- SSH to each cluster node and ensure Kepler dependencies are installed
(user is named `root`).

```sh
apt update
apt install linux-headers-$(uname -r)
apt install linux-modules-extra-$(uname -r)
modprobe intel_rapl_common
```
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't actually need to run these steps but I think its safest to include them since they are requirements for Kepler. WDYT?

Copy link
Contributor

@nikimanoledaki nikimanoledaki Oct 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to validate that the kernel headers are there, I always double-check this as well 👍


- Install Kepler.

```sh
helm repo add kepler https://sustainable-computing-io.github.io/kepler-helm-chart
helm install kepler kepler/kepler --namespace kepler --create-namespace
```

- Check Kepler container metrics are non-zero.

```sh
kubectl exec -ti -n kepler daemonset/kepler \
-- bash -c "curl localhost:9102/metrics" | grep 'kepler_container_package_joules_total'
```

## Delete Cluster

- If the Kind cluster still exists it can be used to delete the Equinix cluster.

```sh
kubectl delete cluster wg-green-reviews
```

- Otherwise delete both servers and the elastic IP via the Equinix console.