Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add script and article to explain our basic performance test with HtcMock #614

Merged
merged 4 commits into from
Feb 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions .docs/content/0.installation/4.perftests-htcmock.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# ArmoniK.Core performance tests with HtcMock

## Machine setup

An AWS **c5.12xlarge** EC2 instance was used to perform the following tests with Amazon Linux 2 operating system.

Dependencies were installed using the following commands:

```bash
sudo yum install -y git docker yum-utils

# just
mkdir -p ~/bin
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to ~/bin

# terraform
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform

# docker
sudo systemctl enable docker.service
sudo systemctl start docker.service
sudo usermod -a -G docker ec2-user
```

## HtcMock client build

```bash
just buildHtcmockClient
```

or

```bash
docker build -t dockerhubaneo/armonik_core_htcmock_test_client:0.0.0.0-test -f Tests/HtcMock/Client/src/Dockerfile .
```

## Empty tasks and complex dependencies case

ArmoniK.Core infrastructure used:
- Activemq
- Redis
- 1 partition
- HtcMock worker
- 30 compute plane instances

```bash
just worker=mock replicas=30 partitions=1 build-deploy
```

Once deployed, we do some warmup to put ArmoniK in its optimal state:

```bash
docker run --net armonik_network --rm \
-e HtcMock__TotalCalculationTime=00:00:10 \
-e HtcMock__NTasks=300 \
-e HtcMock__SubTasksLevels=4 \
-e HtcMock__EnableFastCompute=true \
-e GrpcClient__Endpoint=http://armonik.control.submitter:1080\
dockerhubaneo/armonik_core_htcmock_test_client:0.0.0.0-test
```

Then we run the actual test:

```bash
docker run --net armonik_network --rm \
-e HtcMock__TotalCalculationTime=00:00:10 \
-e HtcMock__NTasks=10000 \
-e HtcMock__SubTasksLevels=100 \
-e HtcMock__EnableFastCompute=true \
-e GrpcClient__Endpoint=http://armonik.control.submitter:1080 \
dockerhubaneo/armonik_core_htcmock_test_client:0.0.0.0-test
```

We execute this test several times and compute the average tasks throughput.
See this [script](../../../tools/perftest-htcmock.sh) for the exact test process.

## Some throughput results in tasks per seconds

| git rev-parse --short HEAD | git describe --tags | ActiveMQ / Redis |
| -------------------------- | -------------------- | ---------------: |
| 91f62cc2 | 0.20.2-5-g91f62cc2 | 595.0 |
| d436763c | 0.20.2-12-gd436763c | 569.0 |
| f41c9a2c | 0.20.5-33-gf41c9a2c | 575.2 |
| 4ea4db08 | 0.20.5-46-g4ea4db08 | 498.1 |
| 85d90c5b | 0.20.5-91-g85d90c5b | 494.7 |
| 968ca38 | 0.20.5-106-g968ca38e | 491.6 |
| 2bcbe5c2 | 0.20.5-109-g2bcbe5c2 | 494.0 |
49 changes: 49 additions & 0 deletions tools/perftest-htcmock.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash

just tag=0.0.0.0-test buildHtcmockClient

just destroy

just worker=mock replicas=30 partitions=1 build-deploy

SHA=$(git rev-parse --short HEAD)
TAG=$(git describe --tags)
DATE=$(date '+%Y%m%d_%H%M%S')

mkdir -p "logs/$SHA/$DATE"

docker run --net armonik_network --rm \
-e HtcMock__TotalCalculationTime=00:00:10 \
-e HtcMock__NTasks=300 \
-e HtcMock__SubTasksLevels=4 \
-e HtcMock__EnableFastCompute=true \
-e GrpcClient__Endpoint=http://armonik.control.submitter:1080 \
dockerhubaneo/armonik_core_htcmock_test_client:0.0.0.0-test


for i in {0..6}
do
docker run --net armonik_network --rm \
-e HtcMock__TotalCalculationTime=00:00:10 \
-e HtcMock__NTasks=10000 \
-e HtcMock__SubTasksLevels=100 \
-e HtcMock__EnableFastCompute=true \
-e GrpcClient__Endpoint=http://armonik.control.submitter:1080 \
dockerhubaneo/armonik_core_htcmock_test_client:0.0.0.0-test | tee "logs/$SHA/$DATE/mock$i.log"

done

echo
echo
echo ===== Results =====
echo
echo

set -x
grep -h "Throughput for session" "logs/$SHA/$DATE/mock"* | jq -r .sessionThroughput
set +x

AVG=$(grep -h "Throughput for session" "logs/$SHA/$DATE/mock"* | jq -r .sessionThroughput | awk 'BEGIN {sum=0; i=0} {i+=1 ; sum+=$1}; END {printf "%.1f", sum/i}')

echo "| $SHA | $TAG | $AVG |"