decouple platform charts from application charts

Signed-off-by: Fotis Nikolaidis <[email protected]>

CHANGELOG.md

@@ -3,6 +3,8 @@
 ## Changes Since Last Release
 
 ### Changed defaults / behaviours
+- Moved charts from `charts/{category}` to `examples/apps`. This allows to have the apps and the test-cases on the same directory. Additionally, that
+  means that chart releasing is no longer part of the frisbee release -- which shouldn't have been the case in the first place.
 - ...
 
 ### New Features & Functionality

README.md

@@ -52,36 +52,16 @@ To learn more about Frisbee, check the **[QuickStart](https://frisbee.dev/docs/q
 our **[Website](https://frisbee.dev)**.
 
 
-## Use-Cases and Testing Patterns
-
-In declarative testing, a test scenario focuses on what to accomplish rather than on the imperative details of how to manipulate the state of an application under test and verify the final application state against an expected state. 
-
-This approach not make tests more *readable, maintainable, and reproducible*, but it also help devops in identifying testing patterns.
-
-Here, you can see some testing patterns we have identified across different application domains.
-
-👉 [Databases](charts/databases)
-
-👉 [Federated Learning](charts/federated-learning)
-
-👉 [Filesystems](charts/filesystems)
-
-👉 [HPC](charts/hpc)
-
-👉 [Networking](charts/networking)
-
-
 ## Getting Started
 
 
 ### Step 1 – Prerequsities:
 
-To run Frisbee you must access to a running Kubernetes cluster and that [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
-and  [Helm](https://helm.sh/docs/intro/install/) are installed on your system. If so, skip to the next step.
+To run Frisbee you must have access to a running Kubernetes cluster and ensure that [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
+and  [Helm](https://helm.sh/docs/intro/install/) are installed on your system. 
 
 
-For quick testing, you can use [microk8s](https://microk8s.io/docs/getting-started). In that case, make sure you have
-added the addons and created aliases to the commands.
+For quick testing, you can use [microk8s](https://microk8s.io/docs/getting-started) to deploy a local Kubernetes cluster on your machine. 
 
 
 ```shell
@@ -102,54 +82,38 @@ git clone [email protected]:CARV-ICS-FORTH/frisbee.git
 cd frisbee
 ```
 
-Now, you can inspect the `examples` and `charts` directories.
 
-* [examples](examples): contains a list of test-cases.
-* [charts](charts): contains Helm charts that provide templates used in the test-cases.
-
-
-#### Install Frisbee CLI on your system.
 The above will download the project, but not the executables.
-For that, you have two options: download the binaries and compile the binaries.
+For that, you have two options: download the precompile binaries and or compile the binaries on your machine.
 
-* From binary:
+* **From binary:** 
 ```shell
-# Download CLI
 ./install.sh
-# Load CLI binary of the PATH.
-export PATH=/usr/local/bin
+export PATH=$PATH:/usr/local/bin
 ```
 
-* From source: 
+* **From source:**  
 ```shell
-# Compile CLI
 make run
-# Load CLI binary of the PATH.
 export PATH=$PATH:$(pwd)/bin
 ```
 
 
-#### Install Frisbee platform on K8s cluster.
+
 The next step is to install Frisbee in your Kubernetes cluster. You can two options:
 
 * **Development mode:** In this configuration, the Frisbee controller must run manually on the local node. 
 The development mode is convenient for developing new functionality on the controller. 
 
 ```shell
-# Install Frisbee controller outside the k8s cluster.
 kubectl-frisbee install development ./ <public address>
-
-# Run the controller manually on the local node.
 make run
 ```
 
 * **Production mode:** The production mode will install the Frisbee controller at yet another container within the cluster.
 
 ```shell
-# Install Frisbee controller within the k8s cluster.
-kubectl frisbee install production
-
-# Follow the controller's output:
+kubectl-frisbee install production
 kubectl logs -l control-plane=frisbee-operator -n frisbee --follow
 ```
 
@@ -175,22 +139,29 @@ To submit a testing job, the general syntax is:
 kubectl-frisbee submit test <testName> <scenario.yaml> <path to dependent charts>
 ```
 
+Conventionally, we separate the test-cases from the templates of the system under evaluation. 
+
+* [examples](examples): contains a list of test-cases.
+* [charts](charts): contains Helm charts that provide templates used in the test-cases.
+
+
 Let's try to run a scenario from the tutorial. 
 
 ```shell
-kubectl-frisbee submit test demo- ./examples/tutorial/15.performance-monitoring.yml ./charts/system/ ./charts/networking/iperf2/
+kubectl-frisbee submit test demo- ./examples/tutorial/15.performance-monitoring.yml ./charts/networking/iperf2/ ./charts/system/ 
 ```
 * **demo-**: demo is the naming prefix, and the `-` indicate an autogenerated postfix (e.g, demo-326)
-* **./examples/tutorial/15.performance-monitoring.yml**: the scenario executes the iperf benchmark and the monitoring stack for observing its execution.
-* **charts/systems**: provides the templates for the telemetry stack.
-* **charts/networking/iperf**: provides the templates for iperf benchmark.
+* [./examples/tutorial/15.performance-monitoring.yml](./examples/tutorial/15.performance-monitoring.yml): the scenario executes the iperf benchmark and the monitoring stack for observing its execution.
+* [./examples/apps/iperf2](./examples/apps/iperf2): provides the application templates used within the scenario.
+* [./charts/system](./charts/system): provides system-wide templates for the telemetry stack, chaos injection, etc.
 
 
 <p align="center">
     <img src="docs/readme.assets/cli-submit.png" width="400">
 </p>
 
 
+
 #### Inspect Submitted Jobs.
 
 To get a list of submitted tests, use:
@@ -259,6 +230,23 @@ kubectl-frisbee report test demo-326 ~/frisbee-reports --pdf  --force
 This will create report on `~/frisbee/reports` directory including the pdf from Grafana.
 
 
+## Use-Cases and Testing Patterns
+
+In declarative testing, a test scenario focuses on what to accomplish rather than on the imperative details of how to manipulate the state of an application under test and verify the final application state against an expected state.
+
+This approach not make tests more *readable, maintainable, and reproducible*, but it also help devops in identifying testing patterns.
+
+Here, you can see some testing patterns we have identified across different application domains.
+
+👉 [Databases](./examples/patterns/databases)
+
+👉 [Federated Learning](./examples/patterns/federated-learning)
+
+👉 [HPC](./examples/patterns/hpc)
+
+👉 [CI](./examples/patterns/ci)
+
+
 ## Features
 
 👉 Workflow templating to store commonly used workflows in the cluster.

charts/databases/tebis/build/tebis.Dockerfile → examples/apps/tebis/build/tebis.Dockerfile

@@ -29,7 +29,7 @@ RUN yum install -y numactl-libs  numactl-devel libibverbs-devel  librdmacm infin
 RUN yum -y install perftest gperf
 
 
-COPY ./CMakeLists.txt /root/CMakeLists.txt
+COPY CMakeLists.txt /root/CMakeLists.txt
 
 # Install Tebis
 RUN git clone --branch master "https://tebis-docker-container:[email protected]/storage/tebis.git" tebis && \

examples/databases/elasticity/install.sh → examples/patterns/databases/elasticity/install.sh

@@ -8,7 +8,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=elasticity
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/databases/cockroachdb ./charts/databases/ycsb)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/cockroachdb ./examples/apps/ycsb)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/databases/network-partition/install.sh → examples/patterns/databases/network-partition/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=network-partition
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/databases/cockroachdb ./charts/databases/ycsb)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/cockroachdb ./examples/apps/ycsb)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/databases/normal-load/install.sh → examples/patterns/databases/normal-load/install.sh

@@ -8,7 +8,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=normal-load
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/databases/cockroachdb ./charts/databases/ycsb)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/cockroachdb ./examples/apps/ycsb)
 export DASHBOARDS=(summary ingleton ycsb)
 
 # Prepare the Reporting folder

examples/databases/sstable-bitrot/install.sh → examples/patterns/databases/sstable-bitrot/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=sstable-bitrot
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/databases/cockroachdb ./charts/databases/ycsb)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/cockroachdb ./examples/apps/ycsb)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/databases/ycsb-sequence/install.sh → examples/patterns/databases/ycsb-sequence/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=ycsb-sequence
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/databases/cockroachdb ./charts/databases/ycsb)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/cockroachdb ./examples/apps/ycsb)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/federated_learning/README.md → examples/patterns/federated-learning/README.md

@@ -3,23 +3,23 @@ This experiment is designed for the evaluation of various ML frameworks on the c
 For this purpose, we use a single client, and we change its backend.
 
 ```shell
-kubectl frisbee submit test fedbed- examples/federated_learning/1.ml-backend.yml  ./charts/system/ charts/federated-learning/fedbed
+kubectl frisbee submit test fedbed- examples/federated-learning/1.ml-backend.yml  ./charts/system/ charts/federated-learning/fedbed
 ```
 
 # Resource Distribution
 This experiment is designed for the evaluation of resource heterogeneity.
 For this purpose, we use multiple clients and assign the total resources to clients according to a  distribution.
 
 ```shell
-kubectl frisbee submit test fedbed- examples/federated_learning/2.resource-distribution.yml  ./charts/system/ charts/federated-learning/fedbed
+kubectl frisbee submit test fedbed- examples/federated-learning/2.resource-distribution.yml  ./charts/system/ charts/federated-learning/fedbed
 ```
 
 # Client Placement
 This experiment is designed for the evaluation of client placement across nodes.
 In this case, we only use the placement primitives, without any kind of resource throttling
 
 ```shell
-kubectl frisbee submit test fedbed- examples/federated_learning/3.client-placement.yml  ./charts/system/ charts/federated-learning/fedbed
+kubectl frisbee submit test fedbed- examples/federated-learning/3.client-placement.yml  ./charts/system/ charts/federated-learning/fedbed
 ```
 
 # Dataset Distribution

examples/federated_learning/crash-on-epoch/install.sh → examples/patterns/federated-learning/crash-on-epoch/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=crash-on-epoch
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/federated-learning/fedbed/)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/fedbed/)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/federated_learning/ml-backend/install.sh → examples/patterns/federated-learning/ml-backend/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=ml-backend
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/federated-learning/fedbed/)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/fedbed/)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/federated_learning/node-placement/install.sh → examples/patterns/federated-learning/node-placement/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=node-placement
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/federated-learning/fedbed/)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/fedbed/)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/federated_learning/parallel-workflows/install.sh → examples/patterns/federated-learning/parallel-workflows/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=parallel-workflows
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/federated-learning/fedbed/)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/fedbed/)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"

examples/federated_learning/resource-distribution/install.sh → examples/patterns/federated-learning/resource-distribution/install.sh

@@ -7,7 +7,7 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 export NAMESPACE=resource-distribution
 export SCENARIO=$(dirname -- "$0")/manifest.yml
 export REPORTS=${HOME}/frisbee-reports/${NAMESPACE}/
-export DEPENDENCIES=(./charts/system/ ./charts/federated-learning/fedbed/)
+export DEPENDENCIES=(./charts/system/ ./examples/apps/fedbed/)
 
 # Prepare the Reporting folder
 mkdir -p "${REPORTS}"