Longevity and NFR test results for NGF version 1.5.0 (#2771)

Longevity and NFR test results for NGF v1.5.0

tests/results/dp-perf/1.5.0/1.5.0-oss.md

@@ -0,0 +1,92 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+NGINX Gateway Fabric:
+
+- Commit: 8624530af3c518afd8f7013566a102e8b3497b76
+- Date: 2024-11-11T18:50:09Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.30.5-gke.1443001
+- vCPUs per node: 16
+- RAM per node: 65853972Ki
+- Max pods per node: 110
+- Zone: us-west2-a
+- Instance Type: n2d-standard-16
+
+## Summary:
+
+- Performance seems to have improved. There is improved latency and response times across all routing methods.
+
+
+## Test1: Running latte path based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.03, 999.28
+Duration      [total, attack, wait]             30s, 29.999s, 532.506µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  368.077µs, 659.422µs, 631.038µs, 721.486µs, 756.087µs, 878.907µs, 12.742ms
+Bytes In      [total, mean]                     4800660, 160.02
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           99.93%
+Status Codes  [code:count]                      200:29978  503:22  
+Error Set:
+503 Service Temporarily Unavailable
+```
+
+## Test2: Running coffee header based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.03, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 611.932µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  514.848µs, 666.682µs, 653.935µs, 741.683µs, 777.382µs, 867.041µs, 11.422ms
+Bytes In      [total, mean]                     4830000, 161.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test3: Running coffee query based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.03, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 618.046µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  511.713µs, 672.907µs, 658.846µs, 751.753µs, 786.911µs, 881.607µs, 10.507ms
+Bytes In      [total, mean]                     5070000, 169.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test4: Running tea GET method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.01, 999.99
+Duration      [total, attack, wait]             30s, 30s, 597.097µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  506.955µs, 651.103µs, 638.079µs, 720.439µs, 752.758µs, 828.588µs, 11.282ms
+Bytes In      [total, mean]                     4740000, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test5: Running tea POST method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.02, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 596.477µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  503.899µs, 650.611µs, 639.013µs, 718.258µs, 748.085µs, 827.88µs, 9.075ms
+Bytes In      [total, mean]                     4740000, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```

tests/results/dp-perf/1.5.0/1.5.0-plus.md

@@ -0,0 +1,90 @@
+# Results
+
+## Test environment
+
+NGINX Plus: true
+
+NGINX Gateway Fabric:
+
+- Commit: a0126a6435dd4bd69c1a7f48ee15eecb76c68400
+- Date: 2024-11-12T20:33:03Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.30.5-gke.1443001
+- vCPUs per node: 16
+- RAM per node: 65853972Ki
+- Max pods per node: 110
+- Zone: us-west2-a
+- Instance Type: n2d-standard-16
+
+## Summary:
+
+- Performance seems consistent with previous test run.
+
+## Test1: Running latte path based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.02, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 676.331µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  491.485µs, 689.253µs, 676.054µs, 771.129µs, 806.996µs, 909.616µs, 10.138ms
+Bytes In      [total, mean]                     4800000, 160.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test2: Running coffee header based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.01, 999.99
+Duration      [total, attack, wait]             30s, 30s, 686.479µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  533.29µs, 716.92µs, 703.946µs, 799.238µs, 835.966µs, 942.918µs, 11.356ms
+Bytes In      [total, mean]                     4830000, 161.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test3: Running coffee query based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.01, 999.98
+Duration      [total, attack, wait]             30s, 30s, 682.739µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  549.612µs, 724.458µs, 711.218µs, 810.286µs, 846.648µs, 953.929µs, 9.249ms
+Bytes In      [total, mean]                     5070000, 169.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test4: Running tea GET method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.01, 999.98
+Duration      [total, attack, wait]             30.001s, 30s, 683.465µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  528.936µs, 716.691µs, 698.583µs, 797.784µs, 834.023µs, 930.167µs, 16.219ms
+Bytes In      [total, mean]                     4740000, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test5: Running tea POST method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.01, 999.99
+Duration      [total, attack, wait]             30s, 30s, 719.615µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  545.338µs, 715.216µs, 702.127µs, 799.224µs, 835.977µs, 940.498µs, 11.445ms
+Bytes In      [total, mean]                     4740000, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```

tests/results/longevity/1.5.0/oss.md

@@ -0,0 +1,99 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+NGINX Gateway Fabric:
+
+- Commit: 36f245bcba55935064324ff5803d66110117f7da
+- Date: 2024-11-08T19:20:48Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 2
+- k8s version: v1.30.5-gke.1443001
+- vCPUs per node: 2
+- RAM per node: 4018120Ki
+- Max pods per node: 110
+- Zone: us-west2-a
+- Instance Type: e2-medium
+
+## Traffic
+
+HTTP:
+
+```text
+Running 5760m test @ http://cafe.example.com/coffee
+  2 threads and 100 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency   236.88ms  177.22ms   2.00s    72.93%
+    Req/Sec   232.09    156.40     1.90k    66.16%
+  156451087 requests in 5760.00m, 53.52GB read
+  Socket errors: connect 0, read 350645, write 0, timeout 75472
+Requests/sec:    452.69
+Transfer/sec:    162.39KB
+KB
+```
+
+HTTPS:
+
+```text
+Running 5760m test @ https://cafe.example.com/tea
+  2 threads and 100 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency   223.09ms  138.95ms   2.00s    63.95%
+    Req/Sec   230.23    155.14     1.80k    66.18%
+  155166081 requests in 5760.00m, 52.20GB read
+  Socket errors: connect 0, read 345712, write 0, timeout 176
+Requests/sec:    448.98
+Transfer/sec:    158.37KB
+
+```
+
+
+### Logs
+
+No error logs in nginx-gateway
+
+Error logs in nginx
+
+We could not get non-2xx errors from the cluster but should likely be similar to last release's issues.
+
+### Key Metrics
+
+#### Containers memory
+
+![oss-memory.png](oss-memory.png)
+
+#### NGF Container Memory
+
+![oss-ngf-memory.png](oss-ngf-memory.png)
+
+### Containers CPU
+
+![oss-cpu.png](oss-cpu.png)
+
+### NGINX metrics
+
+![oss-stub-status.png](oss-stub-status.png)
+
+### Reloads
+
+Rate of reloads - successful and errors:
+
+![oss-reloads.png](oss-reloads.png)
+
+Reload spikes correspond to 1 hour periods of backend re-rollouts.
+
+No reloads finished with an error.
+
+Reload time distribution - counts:
+
+![oss-reload-time.png](oss-reload-time.png)
+
+
+## Comparison with previous runs
+
+Graphs look similar to 1.4.0 results.

tests/results/longevity/1.5.0/plus.md

@@ -0,0 +1,91 @@
+# Results
+
+## Test environment
+
+NGINX Plus: true
+
+NGINX Gateway Fabric:
+
+- Commit: 36f245bcba55935064324ff5803d66110117f7da
+- Date: 2024-11-08T19:20:48Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 2
+- k8s version: v1.30.5-gke.1443001
+- vCPUs per node: 2
+- RAM per node: 4018120Ki
+- Max pods per node: 110
+- Zone: us-west2-a
+- Instance Type: e2-medium
+
+## Traffic
+
+HTTP:
+
+```text
+Running 5760m test @ http://cafe.example.com/coffee
+  2 threads and 100 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency   228.08ms  136.20ms   1.92s    63.92%
+    Req/Sec   232.02    153.44     1.71k    66.90%
+  156457702 requests in 5760.00m, 53.53GB read
+  Non-2xx or 3xx responses: 5
+Requests/sec:    452.71
+Transfer/sec:    162.41KB
+```
+
+HTTPS:
+
+```text
+Running 5760m test @ https://cafe.example.com/tea
+  2 threads and 100 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency   229.75ms  136.23ms   1.92s    63.81%
+    Req/Sec   229.91    151.31     1.63k    66.59%
+  155060805 requests in 5760.00m, 52.19GB read
+  Non-2xx or 3xx responses: 3
+Requests/sec:    448.67
+Transfer/sec:    158.33KB
+```
+
+### Logs
+
+No error logs in nginx-gateway
+
+Error logs in nginx
+
+We could not get non-2xx errors from the cluster but should likely be similar to last release's issues.
+
+
+### Key Metrics
+
+#### Containers memory
+
+![plus-memory.png](plus-memory.png)
+
+#### NGF Container Memory
+
+![plus-ngf-memory.png](plus-ngf-memory.png)
+
+### Containers CPU
+
+![plus-cpu.png](plus-cpu.png)
+
+### NGINX Plus metrics
+
+![plus-status.png](plus-status.png)
+
+### Reloads
+
+Rate of reloads - successful and errors:
+
+![plus-reloads.png](plus-reloads.png)
+
+Note: compared to NGINX, we don't have as many reloads here, because NGF uses NGINX Plus API to reconfigure NGINX
+for endpoints changes.
+
+## Comparison with previous runs
+
+Graphs look similar to 1.4.0 results.