Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kepler installation guide #1

Merged
merged 7 commits into from
Apr 28, 2023
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
## Contributing
If you would like to contribute to this repository, please fork the repository and submit a pull request with your changes. You can also open an issue to report bugs or suggest new experimental conditions.
40 changes: 38 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,38 @@
# gitops-envsus-benchmark-tests
A project to do energy benchmark tests on different GitOps tools and patterns with Kepler.
# Energy Benchmarking Experimental Setup for GitOps
This GitHub repository contains energy-related experimental benchmark tests for different GitOps tools and patterns with Kepler.

## Motivation
GitOps has rapidly gained attraction as a promising software delivery approach that can bring numerous benefits, including faster deployment times, increased reliability, and improved scalability. However, there is currently limited research on the energy consumption of GitOps architectures and tools.

This repository aims to fill this gap by collecting energy-related metrics such as energy consumption that can help to evaluate the energy efficiency of GitOps solutions under various experimental conditions and to identify ways to reduce their environmental impact.

## Contents
The repository includes the following contents:

- docs: This directory contains documentation and instructions on how to run the experiments.
- experiments: This directory contains experimental scripts that can be used to run the benchmark tests.
- images: This directory contains screenshots and other images that are used in this repository.

## Requirements
To run the experimental benchmark tests in this repository, you will need the following:

- [Kubernetes cluster](https://github.com/kubernetes/minikube)
- [Docker](https://docs.docker.com/engine/install/ubuntu)
- [Kepler](https://github.com/sustainable-computing-io/kepler)
- GitOps tools such [Argo CD](https://github.com/argoproj/argo-cd) or [Flux CD](https://github.com/fluxcd/flux2)

Detailed installation and configuration instructions for these tools are provided in the docs directory.

## Running the Experiments
To run the benchmark tests, follow these steps:

- Install the required tools and configure the environment as described in the docs directory.
- Clone this repository to your local machine.
- Navigate to the experiments directory.
- Run the experimental scripts using the commands provided in the docs directory.

## License
This repository is licensed under the Apache-2.0 License. See the [LICENSE](LICENSE) file for more details.

## Acknowledgments
This repository was inspired by the work of the GitOps & Environmental Sustainability Subgroup. We would like to thank the members of the community for their contributions and support.
49 changes: 49 additions & 0 deletions docs/01-utilities.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Utilities
The following utilities need to be installed in order to follow Kepler installation guide:

- Go
- Make
- Docker
- kubectl
- Helm

# Installation Steps

## Installing Go

```bash
sudo apt install golang-go
```

## Installing Make

```bash
sudo apt install make
```

## Installing Docker

```bash
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
nikimanoledaki marked this conversation as resolved.
Show resolved Hide resolved
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
```

## Installing kubectl

```bash
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256) kubectl" | sha256sum --check
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
```

## Installing Helm

```bash
tar -zxvf helm-v3.11.0-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
```
100 changes: 100 additions & 0 deletions docs/02-kepler-installation-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Kepler Installation Guide

This guide will show you how to install Kepler, a lightweight pod-level power consumption metrics exporter, on a Minikube cluster running on an Ubuntu 22.04 host.

## Prerequisites

Before you begin, you'll need to have the following:

- Kernel 4.18+
- Docker installed
- Make and go packages installed
- kubectl command-line tool installed
- Helm command-line tool installed

## Installation steps

Start by installing Minikube on your machine. You can follow the instructions in the [Minikube documentation](https://minikube.sigs.k8s.io/docs/start/).
I ran the following commands:

```bash
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
sudo usermod -aG docker $USER && newgrp docker
```

Once you have Minikube installed, start a new cluster using the following command (CPUs and Memory values can be customized):

```bash
# check CPU cores
lscpu | grep "CPU(s)"
# start the cluster
minikube start --cpus 4 --memory 4096
nikimanoledaki marked this conversation as resolved.
Show resolved Hide resolved
```

Next, install the Prometheus Operator on your cluster by running the following commands:

```bash
git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus/
kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/
cd ../
```

Clone Kepler repository and generate manifest files by running the following commands:

```bash
git clone https://github.com/sustainable-computing-io/kepler.git -b v0.4
cd kepler/
# to configure Prometheus to scrape Kepler-exporter endpoints, Kepler exporter servicemonitor object is required
make build-manifest OPTS="PROMETHEUS_DEPLOY"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you guys applied make on mac? I tried to install it via brew but it is not working for me

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strange, brew install make should do it.. 🤔 did you get any errors?

# update the NODE_NAME to localhost in the deployment manifest when using Minikube
nikimanoledaki marked this conversation as resolved.
Show resolved Hide resolved
```

Deploy Kepler v0.4 to your cluster by running the following command:

```bash
kubectl apply -f _output/generated-manifest/deployment.yaml
cd ../
```

Once Kepler is deployed, verify that Kepler is working by running the following commands:

```bash
# get Kepler's pod name
kubectl get pods -n kepler
# check Kepler's logs
kubectl logs kepler-exporter-<xxxxx> -n kepler -f
# get kepler related metrics
kubectl exec -ti -n kepler daemonset/kepler-exporter -- bash -c "curl localhost:9102/metrics"
Comment on lines +79 to +82
Copy link
Owner

@nikimanoledaki nikimanoledaki Apr 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the steps don't work with my minikube configuration on MacOs with the Docker driver.

exporter logs:

k logs kepler-exporter-5l6vg -n kepler -f
I0413 09:23:19.309510       1 gpu.go:43] Failed to init nvml, err: could not init nvml: error opening libnvidia-ml.so.1: libnvidia-ml.so.1: cannot open shared object file: No such file or directory
I0413 09:23:19.311530       1 exporter.go:149] Kepler running on version: eaa17c7
I0413 09:23:19.311579       1 config.go:169] using gCgroup ID in the BPF program: true
I0413 09:23:19.311617       1 config.go:170] kernel version: 5.15
I0413 09:23:19.311776       1 rapl_msr_util.go:129] input/output error
I0413 09:23:19.311863       1 power.go:64] Not able to obtain power, use estimate method
modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.15.49-linuxkit
chdir(/lib/modules/5.15.49-linuxkit/build): No such file or directory
I0413 09:23:20.779625       1 bcc_attacher.go:64] failed to attach the bpf program: <nil>
I0413 09:23:20.779689       1 bcc_attacher.go:124] failed to attach perf module with options [-DNUM_CPUS=6 -DSET_GROUP_ID]: failed to attach the bpf program: <nil>, not able to load eBPF modules
I0413 09:23:20.779714       1 exporter.go:184] failed to start : failed to attach bpf assets: failed to attach the bpf program: <nil>
I0413 09:23:20.779960       1 exporter.go:209] Started Kepler in 1.468330908s

it looks like it fails to find the kernel headers with linuxkit used as the node image. in your setup, did minikube setup the nodes using this image or a different one?

the kepler metric endpoint could not be reached:

curl http://localhost:9102/metrics

curl: (52) Empty reply from server

server logs:

Forwarding from 127.0.0.1:9102 -> 9102
Forwarding from [::1]:9102 -> 9102
Handling connection for 9102
E0413 11:27:39.880892    4870 portforward.go:409] an error occurred forwarding 9102 -> 9102: error forwarding port 9102 to pod 61a3c66c5ebc3e82d48151386d7d26cc37e7dbd9d971593e5f792bce305b0920, uid : exit status 1: 2023/04/13 09:27:39 socat[182461] E connect(5, AF=2 127.0.0.1:9102, 16): Connection refused
error: lost connection to pod

I will also try with a VM-based driver in case anything changes and get back to you!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thank you for the thorough write-up 👏 Left some Qs as I'm having some issues (similar to the ones I had before) when running this on MacOs 🤔

Thank you! :) It's my pleasure!
Let's try to make it work 🛠️

Copy link
Owner

@nikimanoledaki nikimanoledaki Apr 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I managed to get Kepler working with driver=hyperkit ! 🎉
Will add documentation for MacOS users in a separate PR.

Virtualbox was not an option because it breaks with the latest MacOS upgrade: kubernetes/minikube#15274

What do you think about the fact that we will be using different drivers? 🤔 It's good to have a dev env option available for MacOS. I will be home on Wednesday and will be able to run things on my Linux machine if needed. Maybe we can try cross-referencing the tests in both environments. Either way, I think it's good to start getting some numbers, and we can be upfront about the different environments that we used. This is more of a POC than a scientific paper. And either way, the ultimate test would be to run these tests in a consistent environment in the cloud rather than locally, right? :)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Argh. No. The energy data for the node is 0 so it's not reading it properly. 🤔

# HELP kepler_node_energy_stat Several labeled node metrics
# TYPE kepler_node_energy_stat counter
kepler_node_energy_stat{cpu_architecture="Sky Lake",node_block_devices_used="0",node_curr_bytes_read="0",node_curr_bytes_writes="0",node_curr_cache_miss="0",node_curr_container_cpu_usage_seconds_total="0",node_curr_container_memory_working_set_bytes="0",node_curr_cpu_cycles="0",node_curr_cpu_instr="0",node_curr_cpu_time="0",node_curr_energy_in_core_joule="0",node_curr_energy_in_dram_joule="0",node_curr_energy_in_gpu_joule="0",node_curr_energy_in_other_joule="0",node_curr_energy_in_pkg_joule="0",node_curr_energy_in_uncore_joule="0",node_name="minikube"} 0

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right!
I think for now the more variants of drivers, OSs, etc., the better for cross-checking and results validation.
I agree, the best would be spinning up a cluster in the cloud for the tests.

If it doesn't work on MacOS, I think we can still use Linux for the tests.

I have opened another PR for the first experiment scenario which involves deploying two applications (one at a time) using both, Argo CD and Flux CD, in addition to performing a rolling update and a rollback.

I am thinking of using the following query:

sum(rate(kepler_container_joules_total{container_namespace=~"argocd|flux-system"}[1m])) by (container_namespace)

# or
kubectl port-forward kepler-exporter-<xxxxx> 9102:9102 -n kepler
# run the following command in another window while keeping port-forward running
curl http://localhost:9102/metrics
```

View Kepler metrics that have been exported to Prometheus and visualized with Grafana [Kepler's dashboard](https://github.com/sustainable-computing-io/kepler/blob/main/grafana-dashboards/Kepler-Exporter.json) by running the following commands:

```bash
# to view grafana dashboard, port-forward grafana svc and import Kepler's dashboard
kubectl port-forward svc/grafana 3000:3000 -n monitoring
# to view metrics in prometheus UI
kubectl port-forward svc/prometheus-k8s 9090:9090 -n monitoring
```

## Conclusion

That's it! You should now have Kepler up and running on your Minikube cluster with power consumption metrics being exported.

![Grafana dashboard screenshot](../images/grafana-dashboard.png)

## Clean up

To remove kepler related objects, run the following commands:
```bash
kubectl delete ns kepler
kubectl delete clusterrole kepler-clusterrole
kubectl delete clusterrolebinding kepler-clusterrole-binding
```
Binary file added images/grafana-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.