-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: Manually create Equinix cluster #2
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
# Create Equinix Metal cluster using CAPI / CAPEM | ||
|
||
## Overview | ||
|
||
- We are using machines contributed to the CNCF by Equinix Metal. | ||
- These steps manually create a cluster using CAPI / CAPEM using a local | ||
Kind cluster for the management cluster. | ||
- These docs are based on the Equinix CAPI [guide](https://deploy.equinix.com/developers/guides/kubernetes-cluster-api/). | ||
- Using [Podman](https://github.com/kubernetes-sigs/cluster-api-provider-packet/issues/496) | ||
is recommended by Equinix for MacOS. | ||
- We plan to automate these steps later using GitHub Actions and an IaC tool. | ||
|
||
## Create Cluster | ||
|
||
- Log in to the Equinix Metal [console](https://console.equinix.com/) and get | ||
the project API key from the project settings. If it doesn't exist create it. | ||
|
||
```sh | ||
export PACKET_API_KEY="<PROJECT_API_KEY>" | ||
``` | ||
|
||
- Install the CAPI controllers using [clusterctl](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). | ||
|
||
```sh | ||
clusterctl init --infrastructure packet | ||
``` | ||
|
||
- Set env vars with cluster config. | ||
|
||
```sh | ||
# Get the project ID from the project settings in the console | ||
export PROJECT_ID="<PROJECT_ID>" | ||
|
||
# Use Paris metro (Equinix region) | ||
export METRO="pa" | ||
|
||
# Use Ubuntu 22.04 with cGroup v2 for Kepler | ||
export NODE_OS="ubuntu_22_04" | ||
|
||
# The pod and service CIDRs for the new cluster | ||
export POD_CIDR="192.168.0.0/16" | ||
export SERVICE_CIDR="172.26.0.0/16" | ||
|
||
# Use node type with Intel CPUs for RAPL support | ||
export CONTROLPLANE_NODE_TYPE="m3.small.x86" | ||
export WORKER_NODE_TYPE="m3.small.x86" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think its safest to use a node type with an Intel CPU. I tested with an AMD CPU and Kepler couldn't access RAPL. I think this node type has good enough resources for now. WDYT? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 for using an Intel CPU and starting with |
||
|
||
# SSH key to use for access to nodes | ||
export SSH_KEY="<YOUR_SSH_KEY>" | ||
|
||
# Kubernetes version to install | ||
export KUBERNETES_VERSION="v1.28.2" | ||
``` | ||
|
||
- Generate cluster manifests. | ||
|
||
```sh | ||
clusterctl generate cluster wg-green-reviews \ | ||
--kubernetes-version $KUBERNETES_VERSION \ | ||
--control-plane-machine-count=1 \ | ||
--worker-machine-count=1 \ | ||
> wg-green-reviews.yaml | ||
``` | ||
rossf7 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- Apply cluster manifests. | ||
|
||
```sh | ||
kubectl apply -f wg-green-reviews.yaml | ||
``` | ||
|
||
- Get kubeconfig and store it securely. | ||
|
||
```sh | ||
clusterctl get kubeconfig wg-green-reviews > wg-green-reviews.kubeconfig | ||
``` | ||
|
||
- Set `KUBECONFIG` env var so following commands are run on the cluster. | ||
|
||
```sh | ||
export KUBECONFIG=wg-green-reviews.kubeconfig | ||
``` | ||
|
||
- Install Cilium as CNI. | ||
|
||
```sh | ||
helm repo add cilium https://helm.cilium.io/ | ||
helm install cilium cilium/cilium --version 1.14.2 --namespace kube-system | ||
``` | ||
|
||
- SSH to each cluster node and ensure Kepler dependencies are installed | ||
(user is named `root`). | ||
|
||
```sh | ||
apt update | ||
apt install linux-headers-$(uname -r) | ||
apt install linux-modules-extra-$(uname -r) | ||
modprobe intel_rapl_common | ||
``` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't actually need to run these steps but I think its safest to include them since they are requirements for Kepler. WDYT? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good to validate that the kernel headers are there, I always double-check this as well 👍 |
||
|
||
- Install Kepler. | ||
|
||
```sh | ||
helm repo add kepler https://sustainable-computing-io.github.io/kepler-helm-chart | ||
helm install kepler kepler/kepler --namespace kepler --create-namespace | ||
``` | ||
|
||
- Check Kepler container metrics are non-zero. | ||
|
||
```sh | ||
kubectl exec -ti -n kepler daemonset/kepler \ | ||
-- bash -c "curl localhost:9102/metrics" | grep 'kepler_container_package_joules_total' | ||
``` | ||
|
||
## Delete Cluster | ||
|
||
- If the Kind cluster still exists it can be used to delete the Equinix cluster. | ||
|
||
```sh | ||
kubectl delete cluster wg-green-reviews | ||
``` | ||
|
||
- Otherwise delete both servers and the elastic IP via the Equinix console. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to consider carbon intensity when deciding which metro to use?
I chose Paris since France has a low average carbon intensity and this metro has a good selection of node types.
Full list is here and we can easily change if I'm over thinking this :)
https://deploy.equinix.com/developers/docs/metal/locations/metros/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a very important callout! We can hardcode Paris for now and make the testing carbon-aware in the future. Would you like to create a new issue for this to add it to our backlog? 🌟
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes sure I'll create an issue for it. 👍