add tests for zero downtime scaling with 10 nodes

tests/zero-downtime-scaling/results/1.2.0/1.2.0.md

@@ -0,0 +1,273 @@
+# Results
+
+## Versions
+
+Kubernetes:
+
+```text
+Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8-gke.1067004", GitCommit:"6f460c12ad45abb234c18ec4f0ea335a1203c415", GitTreeState:"clean", BuildDate:"2024-01-04T22:48:32Z", GoVersion:"go1.20.11 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
+```
+
+NGF:
+
+```text
+"version":"edge"
+"commit":"5b9f37c6556388e28d2cd168810b728cca4c0e46"
+"date":"2024-03-19T20:53:54Z"
+```
+
+with NGINX:
+
+```text
+nginx version: nginx/1.25.4
+Built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10)
+OS: Linux 5.15.133+
+```
+
+## Summary
+
+- TODO [will update after all results are collected]
+
+## 10 Node Cluster
+
+### Scale Up Gradually
+
+HTTP wrk output
+
+
+wrk -t2 -c2 -d5m --latency --timeout 2s  http://cafe.example.com/coffee
+Running 5m test @ http://cafe.example.com/coffee
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     1.44ms  802.80us  53.07ms   92.86%
+    Req/Sec   722.83     83.40     0.96k    71.09%
+  Latency Distribution
+     50%    1.31ms
+     75%    1.58ms
+     90%    1.96ms
+     99%    4.31ms
+  431719 requests in 5.00m, 153.98MB read
+  Socket errors: connect 0, read 20, write 0, timeout 0
+Requests/sec:   1438.61
+Transfer/sec:    525.42KB
+
+
+HTTP Graph:
+
+![gradual-up-http.png](10-node/gradual-up-http.png)
+
+
+HTTPS wrk output
+
+
+wrk -t2 -c2 -d5m --latency --timeout 2s  https://cafe.example.com/tea
+Running 5m test @ https://cafe.example.com/tea
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     1.55ms    1.73ms 160.80ms   97.88%
+    Req/Sec   679.18     87.82     0.94k    69.26%
+  Latency Distribution
+     50%    1.38ms
+     75%    1.68ms
+     90%    2.09ms
+     99%    4.51ms
+  405615 requests in 5.00m, 141.96MB read
+  Socket errors: connect 0, read 19, write 0, timeout 0
+Requests/sec:   1351.90
+Transfer/sec:    484.51KB
+
+
+HTTPS Graph:
+
+![gradual-up-https.png](10-node/gradual-up-https.png)
+
+Logs:
+
+- NGF error logs: none
+- NGINX error/warn logs: none
+- NGINX access logs
+  - 842986 [200 Oks]
+  - One 499 `INFO 2024-03-19T21:28:15.266352183Z [resource.labels.containerName: nginx] 10.150.0.27 - - [19/Mar/2024:21:28:15 +0000] "GET /tea HTTP/1.1" 499 0 "-" "-"`
+
+
+### Scale Down Gradually
+
+HTTP wrk output
+
+
+wrk -t2 -c2 -d20m --latency --timeout 2s  http://cafe.example.com/coffee
+Running 20m test @ http://cafe.example.com/coffee
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     8.99ms   71.48ms   1.07s    98.65%
+    Req/Sec   704.32     82.44     0.95k    74.78%
+  Latency Distribution
+     50%    1.35ms
+     75%    1.63ms
+     90%    2.03ms
+     99%  326.29ms
+  1603461 requests in 19.81m, 570.38MB read
+  Socket errors: connect 0, read 21, write 0, timeout 10
+Requests/sec:   1349.02
+Transfer/sec:    491.38KB
+
+
+HTTP Graph
+
+![gradual-down-http.png](10-node/gradual-down-http.png)
+
+
+HTPPS wrk output
+
+
+wrk -t2 -c2 -d20m --latency --timeout 2s  https://cafe.example.com/tea
+Running 20m test @ https://cafe.example.com/tea
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     7.13ms   61.54ms   1.07s    98.98%
+    Req/Sec   661.77     77.50     0.90k    75.68%
+  Latency Distribution
+     50%    1.43ms
+     75%    1.73ms
+     90%    2.13ms
+     99%   85.50ms
+  1500047 requests in 19.75m, 523.58MB read
+  Socket errors: connect 0, read 27, write 0, timeout 12
+Requests/sec:   1265.79
+Transfer/sec:    452.41KB
+
+
+HTTPS Graph
+
+![gradual-down-https.png](10-node/gradual-down-https.png)
+
+Logs:
+
+- NGF error logs: 25
+- NGINX error/warn logs: none
+- NGINX access logs:
+  - 3.06 million 200s
+
+
+### Scale Up Abruptly
+
+HTTP wrk output
+
+
+wrk -t2 -c2 -d2m --latency --timeout 2s  http://cafe.example.com/coffee
+Running 2m test @ http://cafe.example.com/coffee
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     1.52ms    1.00ms  42.25ms   94.80%
+    Req/Sec   693.39    100.35     0.94k    70.54%
+  Latency Distribution
+     50%    1.35ms
+     75%    1.64ms
+     90%    2.08ms
+     99%    5.19ms
+  165726 requests in 2.00m, 58.95MB read
+  Socket errors: connect 0, read 2, write 0, timeout 0
+Requests/sec:   1380.12
+Transfer/sec:    502.71KB
+
+
+HTTP Graph
+
+![abrupt-up-http.png](10-node/abrupt-up-http.png)
+
+HTTPS wrk output
+
+
+wrk -t2 -c2 -d2m --latency --timeout 2s  https://cafe.example.com/tea
+Running 2m test @ https://cafe.example.com/tea
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     1.60ms    0.95ms  31.53ms   93.64%
+    Req/Sec   651.71     93.73     0.88k    69.42%
+  Latency Distribution
+     50%    1.43ms
+     75%    1.74ms
+     90%    2.20ms
+     99%    5.14ms
+  155793 requests in 2.00m, 54.38MB read
+Requests/sec:   1297.29
+Transfer/sec:    463.67KB
+
+
+HTTPS Graph
+
+![abrupt-up-https.png](10-node/abrupt-up-https.png)
+
+Logs:
+
+- NGF error logs - none
+- NGINX error/warn logs - none
+- NGINX access logs
+  - 24289 200s
+
+
+### Scale Down Abruptly
+
+
+HTTP wrk output
+
+
+wrk -t2 -c2 -d2m --latency --timeout 2s  http://cafe.example.com/coffee
+Running 2m test @ http://cafe.example.com/coffee
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     1.60ms    1.00ms  34.67ms   94.04%
+    Req/Sec   653.17    105.34     0.95k    73.63%
+  Latency Distribution
+     50%    1.41ms
+     75%    1.75ms
+     90%    2.23ms
+     99%    5.61ms
+  138169 requests in 2.00m, 49.15MB read
+Requests/sec:   1150.52
+Transfer/sec:    419.08KB
+
+HTTP graph
+
+
+![abrupt-down-http.png](10-node/abrupt-down-http.png)
+
+HTTPS wrk output
+
+
+wrk -t2 -c2 -d2m --latency --timeout 2s  https://cafe.example.com/tea
+Running 2m test @ https://cafe.example.com/tea
+  2 threads and 2 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     1.70ms    1.07ms  41.66ms   94.23%
+    Req/Sec   612.48     95.98     0.96k    76.40%
+  Latency Distribution
+     50%    1.50ms
+     75%    1.86ms
+     90%    2.37ms
+     99%    5.68ms
+  126165 requests in 2.00m, 44.04MB read
+Requests/sec:   1051.24
+Transfer/sec:    375.73KB
+
+HTTPS graph
+
+
+![abrupt-down-http.png](10-node/abrupt-down-http.png)
+
+Logs
+
+- NGINX error/warn logs - none
+- NGF error logs - 25
+- NGINX access logs
+  - 265 K 200s
+  - Four 499
+
+```text
+
+INFO 2024-03-20T03:02:03.477744939Z [resource.labels.containerName: nginx] 10.150.0.26 - - [20/Mar/2024:03:02:03 +0000] "GET /coffee HTTP/1.1" 499 0 "-" "-"
+INFO 2024-03-20T03:02:03.477834719Z [resource.labels.containerName: nginx] 10.150.0.26 - - [20/Mar/2024:03:02:03 +0000] "GET /coffee HTTP/1.1" 499 0 "-" "-"
+10.150.0.27 - - [20/Mar/2024:03:02:06 +0000] "GET /tea HTTP/1.1" 499 0 "-" "-"
+10.150.0.27 - - [20/Mar/2024:03:02:06 +0000] "GET /tea HTTP/1.1" 499 0 "-" "-"
+
+```