QPS drops drastically after processing many requests

[messense]

Using the hello world example from README.md, and benchmark it with oha

1st round

❯ oha -n 10000 http://localhost:5000
Summary:
  Success rate:	0.8308
  Total:	1.2192 secs
  Slowest:	0.1544 secs
  Fastest:	0.0028 secs
  Average:	0.0073 secs
  Requests/sec:	8202.1065

  Total data:	105.47 KiB
  Size/request:	13 B
  Size/sec:	86.51 KiB

Response time histogram:
  0.002 [68]   |
  0.003 [4340] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.005 [2943] |■■■■■■■■■■■■■■■■■■■■■
  0.007 [754]  |■■■■■
  0.009 [149]  |■
  0.010 [4]    |
  0.012 [0]    |
  0.014 [0]    |
  0.016 [0]    |
  0.017 [0]    |
  0.019 [50]   |

Latency distribution:
  10% in 0.0051 secs
  25% in 0.0055 secs
  50% in 0.0062 secs
  75% in 0.0071 secs
  90% in 0.0082 secs
  95% in 0.0089 secs
  99% in 0.0106 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0035 secs, 0.0002 secs, 0.1505 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0002 secs

Status code distribution:
  [200] 8308 responses

Error distribution:
  [1600] connection closed before message completed
  [92] operation was canceled: connection closed

2nd round

❯ oha -n 10000 http://localhost:5000
Summary:
  Success rate:	0.8167
  Total:	7.3648 secs
  Slowest:	0.1063 secs
  Fastest:	0.0006 secs
  Average:	0.0070 secs
  Requests/sec:	1327.3908

  Total data:	101.36 KiB
  Size/request:	13 B
  Size/sec:	13.76 KiB

Response time histogram:
  0.002 [29]   |
  0.004 [47]   |
  0.006 [3783] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.007 [3249] |■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.009 [771]  |■■■■■■
  0.011 [52]   |
  0.013 [5]    |
  0.015 [0]    |
  0.017 [0]    |
  0.018 [0]    |
  0.020 [48]   |

Latency distribution:
  10% in 0.0050 secs
  25% in 0.0055 secs
  50% in 0.0062 secs
  75% in 0.0071 secs
  90% in 0.0081 secs
  95% in 0.0087 secs
  99% in 0.0102 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0034 secs, 0.0002 secs, 0.1020 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0001 secs

Status code distribution:
  [200] 7984 responses

Error distribution:
  [1719] connection closed before message completed
  [73] operation was canceled: connection closed

3rd round

❯ oha -n 10000 http://localhost:5000
Summary:
  Success rate:	0.8108
  Total:	8.9002 secs
  Slowest:	0.0075 secs
  Fastest:	0.0009 secs
  Average:	0.0047 secs
  Requests/sec:	8.3144

  Total data:	780 B
  Size/request:	13 B
  Size/sec:	87 B

Response time histogram:
  0.001 [3]  |■■■■■■■
  0.001 [7]  |■■■■■■■■■■■■■■■■■
  0.002 [4]  |■■■■■■■■■
  0.002 [1]  |■■
  0.003 [10] |■■■■■■■■■■■■■■■■■■■■■■■■
  0.004 [4]  |■■■■■■■■■
  0.004 [2]  |■■■■
  0.005 [2]  |■■■■
  0.005 [6]  |■■■■■■■■■■■■■■
  0.006 [13] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.007 [8]  |■■■■■■■■■■■■■■■■■■■

Latency distribution:
  10% in 0.0019 secs
  25% in 0.0034 secs
  50% in 0.0049 secs
  75% in 0.0065 secs
  90% in 0.0070 secs
  95% in 0.0071 secs
  99% in 0.0075 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0022 secs, 0.0002 secs, 0.0040 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0002 secs

Status code distribution:
  [200] 60 responses

Error distribution:
  [13] connection closed before message completed
  [1] operation was canceled: connection closed

[sansyrox]

@messense , I did not read the title of the issue. I thought that issue was just a metric post. I will have a look at the issue today.

[sansyrox]

@messense , with the latest PR, it can parse 1million requests without dropping any 🥳

➜  ~ oha -n 1000000 http://localhost:5000
Summary:
  Success rate:	1.0000
  Total:	426.8401 secs
  Slowest:	0.3269 secs
  Fastest:	0.0009 secs
  Average:	0.0213 secs
  Requests/sec:	2342.7976

  Total data:	17.17 MiB
  Size/request:	18 B
  Size/sec:	41.18 KiB

Response time histogram:
  0.006 [82867]  |■■■■■■■■■
  0.011 [272171] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.017 [56678]  |■■■■■■
  0.023 [123485] |■■■■■■■■■■■■■■
  0.029 [245641] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.034 [136749] |■■■■■■■■■■■■■■■■
  0.040 [28463]  |■■■
  0.046 [19679]  |■■
  0.052 [17939]  |■■
  0.057 [7251]   |
  0.063 [9077]   |■

Latency distribution:
  10% in 0.0070 secs
  25% in 0.0097 secs
  50% in 0.0229 secs
  75% in 0.0288 secs
  90% in 0.0340 secs
  95% in 0.0423 secs
  99% in 0.0570 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0039 secs, 0.0024 secs, 0.0049 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0006 secs

Status code distribution:
  [200] 1000000 responses