Add benchmark stresscli scripts (#50)

* Add benchmark stresscli scripts

* Profile based multi-run stress tool
* Locust based benchmark script for chatQnA and TGI
* HW/SW Spec and Metrics collection
* Stress result report generation

Signed-off-by: Li Gang <[email protected]>

Co-authored-by: Yingchun Guo <[email protected]>
Co-authored-by: Cathy Zhang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refine stresscli as a lib

Signed-off-by: lvliang-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <[email protected]>
Co-authored-by: Yingchun Guo <[email protected]>
Co-authored-by: Cathy Zhang <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <[email protected]>

evals/benchmark/stresscli/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+*testspec.yaml
+*exceptions.csv
+*failures.csv
+*stats_history.csv
+*stats.csv
+*env/

evals/benchmark/stresscli/README.md

@@ -0,0 +1,132 @@
+# StressCli
+
+This project includes benchmark toolset for AI workloads such as OPEA.
+
+## stresscli.py
+
+`stresscli.py` is a command line tool for dumping test specs and performing load tests.
+
+### Prerequirements
+
+This tool will use `kubectl` to collect Kubernetes cluster information. So you need to make sure `kubectl` is installed and 
+
+This tool uses `locust` by default to do load test. You have to install `locust` to your machine simply by
+```
+pip3 install locust
+```
+For detail information of `locust`, go to [locust website](https://docs.locust.io/en/stable/installation.html).
+
+### Installation
+
+The recommended way to install and run stresscli is in a virtualenv with the latest version of Python 3 (at least Python 3.11). If Python is not installed, you can likely install it using your distribution's
+package manager, or see the [Python Download page](https://www.python.org/downloads/).
+
+```bash
+git clone https://github.com/opea-project/GenAIEval.git
+# go to project root
+cd GenAIEval/evals/benchmark/stresscli
+# create virtual env
+python3 -m venv stresscli_virtualenv
+source stresscli_virtualenv/bin/activate
+# install requirements
+pip install -r requirements.txt
+```
+
+### Usage
+
+```
+$ ./stresscli.py --help
+Usage: stresscli.py [OPTIONS] COMMAND [ARGS]...
+
+  StressCLI - A command line tool for stress testing OPEA workloads.
+
+Options:
+  --kubeconfig PATH  Configuration file to Kubernetes
+  --help             Show this message and exit.
+
+Commands:
+  dump       Dump the test spec
+  load-test  Do load test
+  report     Print the test report
+  validate   Validate against the test spec
+```
+#### Run a test
+
+**Note: Please edit the [run.yaml](./run.yaml) file or create your profile before run the load test.**
+
+```
+./stresscli.py load-test --profile run.yaml
+```
+
+More detail options:
+```
+$ ./stresscli.py load-test --help
+Usage: stresscli.py load-test [OPTIONS]
+
+  Do load test
+
+Options:
+  --profile PATH  Path to profile YAML file
+  --help          Show this message and exit.
+```
+
+#### Generate the test output report
+
+You can generate the report for test cases by:
+```
+$ ./stresscli.py report --folder /home/sdp/test_reports/20240710_004105 --format csv -o data.csv
+```
+
+More detail options:
+```
+$ ./stresscli.py report --help
+Usage: stresscli.py report [OPTIONS]
+
+  Print the test report
+
+Options:
+  --folder PATH      Path to log folder  [required]
+  --format TEXT      Output format, plain_text or csv, default is plain_text
+  --include TEXT     Extract output data from output.log, stats.csv, and
+                     testspec.yaml, default is
+                     output.log|stats.csv|testspec.yaml
+  -o, --output PATH  Save output to file
+  --help             Show this message and exit.
+```
+#### Dump the configuration
+
+You can dump the current testing profile by
+```
+./stresscli.py dump -o <output_file>
+```
+More detail options:
+```
+$ ./stresscli.py dump --help
+Usage: stresscli.py dump [OPTIONS]
+
+  Dump the test spec
+
+Options:
+  -o, --output PATH  YAML file of cluster configuration  [required]
+  --help             Show this message and exit.
+```
+
+#### Validate against the spec
+
+You can validate if the current K8s and workloads deployment comply with the test spec by:
+```
+$ ./stresscli.py validate --file testspec.yaml
+```
+
+More detail options:
+```
+$ ./stresscli.py validate --help
+Usage: stresscli.py validate [OPTIONS]
+
+  Validate against the test spec
+
+Options:
+  --file PATH          Specification YAML file to validate against  [required]
+  --validate_topology  Validate topology in workload specification
+  --help               Show this message and exit.
+```

evals/benchmark/stresscli/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

evals/benchmark/stresscli/commands/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

evals/benchmark/stresscli/commands/dump.py

@@ -0,0 +1,27 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+# stresscli/dump.py
+
+import click
+
+from .utils import dump_k8s_config
+
+
+@click.command()
+# @click.option('--kubeconfig', type=click.Path(), help='Configuration file to Kubernetes')
+@click.option(
+    "-o",
+    "--output",
+    type=click.Path(),
+    default="testspec.yaml",
+    required=True,
+    help="YAML file of cluster configuration",
+)
+@click.pass_context
+#'-o', '--output', type=click.Path(), default='output.yaml', help='YAML file of cluster configuration'
+def dump(ctx, output):
+    """Dump the test spec."""
+    kubeconfig = ctx.parent.params["kubeconfig"]
+    namespace = ctx.parent.params["namespace"]
+    dump_k8s_config(kubeconfig, output, namespace=namespace)