Add simple benchmarks for osrm-* tools #6925
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It was low hanging fruit as we are running these tools for other benchmarks anyway.
Also I noticed that e2e tests I added recently are very "jumpy" :( I will probably address it in some separate PR...
Benchmark Results
plain u32: 1133.8
aliased double: 1167.26
plain double: 1169.79
plain u32: 1138.42
aliased double: 1172.87
plain double: 1179.5
failures: 0
req/s: 6.650req/s
avg: 5.575ms
50%: 3ms
75%: 3ms
95%: 25ms
98%: 49ms
99%: 62ms
min: 1.157ms
max: 96.289ms
failures: 0
req/s: 6.425req/s
avg: 5.557ms
50%: 3ms
75%: 4ms
95%: 19ms
98%: 46ms
99%: 70ms
min: 1.464ms
max: 91.964ms
failures: 0
req/s: 6.034req/s
avg: 4.496ms
50%: 3ms
75%: 4ms
95%: 11ms
98%: 19ms
99%: 53ms
min: 1.013ms
max: 66.555ms
failures: 0
req/s: 6.116req/s
avg: 5.576ms
50%: 3ms
75%: 4ms
95%: 18ms
98%: 52ms
99%: 75ms
min: 1.008ms
max: 87.581ms
failures: 0
req/s: 7.134req/s
avg: 1.420ms
50%: 1ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 2ms
min: 1.005ms
max: 3.117ms
failures: 0
req/s: 7.042req/s
avg: 1.571ms
50%: 2ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 3ms
min: 1.068ms
max: 2.761ms
failures: 0
req/s: 7.100req/s
avg: 1.458ms
50%: 1ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 2ms
min: 1.039ms
max: 6.775ms
failures: 0
req/s: 7.083req/s
avg: 1.524ms
50%: 1ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 2ms
min: 1.033ms
max: 3.204ms
failures: 0
req/s: 6.900req/s
avg: 3.662ms
50%: 4ms
75%: 4ms
95%: 5ms
98%: 6ms
99%: 8ms
min: 1.472ms
max: 9.935ms
failures: 0
req/s: 6.725req/s
avg: 4.276ms
50%: 4ms
75%: 5ms
95%: 6ms
98%: 7ms
99%: 10ms
min: 1.470ms
max: 99.512ms
failures: 0
req/s: 6.200req/s
avg: 4.636ms
50%: 4ms
75%: 5ms
95%: 6ms
98%: 7ms
99%: 10ms
min: 1.373ms
max: 141.264ms
failures: 0
req/s: 6.983req/s
avg: 4.248ms
50%: 4ms
75%: 5ms
95%: 6ms
98%: 7ms
99%: 9ms
min: 1.358ms
max: 97.756ms
failures: 0
req/s: 6.467req/s
avg: 29.454ms
50%: 22ms
75%: 30ms
95%: 110ms
98%: 120ms
99%: 130ms
min: 2.483ms
max: 138.929ms
failures: 0
req/s: 6.675req/s
avg: 35.443ms
50%: 23ms
75%: 34ms
95%: 120ms
98%: 130ms
99%: 140ms
min: 2.451ms
max: 151.331ms
failures: 0
req/s: 6.417req/s
avg: 73.161ms
50%: 79ms
75%: 110ms
95%: 130ms
98%: 140ms
99%: 140ms
min: 4.356ms
max: 165.649ms
failures: 0
req/s: 6.500req/s
avg: 42.225ms
50%: 27ms
75%: 61ms
95%: 130ms
98%: 130ms
99%: 140ms
min: 2.491ms
max: 328.336ms
failures: 0
req/s: 6.150req/s
avg: 13.448ms
50%: 12ms
75%: 17ms
95%: 27ms
98%: 31ms
99%: 32ms
min: 1.994ms
max: 106.068ms
failures: 0
req/s: 5.874req/s
avg: 14.614ms
50%: 13ms
75%: 19ms
95%: 29ms
98%: 32ms
99%: 34ms
min: 1.829ms
max: 36.972ms
failures: 0
req/s: 5.867req/s
avg: 20.310ms
50%: 20ms
75%: 26ms
95%: 34ms
98%: 37ms
99%: 39ms
min: 3.991ms
max: 44.437ms
failures: 0
req/s: 5.850req/s
avg: 14.914ms
50%: 13ms
75%: 19ms
95%: 28ms
98%: 33ms
99%: 36ms
min: 2.381ms
max: 40.495ms
Stringstream: 9.27548ms
Vector: 6.8574ms
Stringstream: 9.44745ms
Vector: 7.01822ms
4.41197ms/req at 82 coordinate
0.0538045ms/coordinate
Radius 5m:
4.39374ms/req at 82 coordinate
0.0535822ms/coordinate
Radius 10m:
15.0527ms/req at 82 coordinate
0.183569ms/coordinate
Radius 15m:
36.7583ms/req at 82 coordinate
0.448272ms/coordinate
Radius 30m:
314.519ms/req at 82 coordinate
3.8356ms/coordinate
4.41337ms/req at 82 coordinate
0.0538215ms/coordinate
Radius 5m:
4.40282ms/req at 82 coordinate
0.053693ms/coordinate
Radius 10m:
15.0342ms/req at 82 coordinate
0.183344ms/coordinate
Radius 15m:
36.9353ms/req at 82 coordinate
0.45043ms/coordinate
Radius 30m:
313.907ms/req at 82 coordinate
3.82813ms/coordinate
2.80752ms/req at 82 coordinate
0.034238ms/coordinate
Radius 5m:
2.74811ms/req at 82 coordinate
0.0335135ms/coordinate
Radius 10m:
10.1372ms/req at 82 coordinate
0.123625ms/coordinate
Radius 15m:
25.9226ms/req at 82 coordinate
0.31613ms/coordinate
Radius 30m:
305.832ms/req at 82 coordinate
3.72966ms/coordinate
2.77088ms/req at 82 coordinate
0.0337912ms/coordinate
Radius 5m:
2.76309ms/req at 82 coordinate
0.0336963ms/coordinate
Radius 10m:
10.1278ms/req at 82 coordinate
0.12351ms/coordinate
Radius 15m:
26.0479ms/req at 82 coordinate
0.317657ms/coordinate
Radius 30m:
302.263ms/req at 82 coordinate
3.68613ms/coordinate
std::vector 11117.9 ms
util::packed_vector 73460.3 ms
slowdown: 6.60738
random read:
std::vector 11015.3 ms
util::packed_vector 29535 ms
slowdown: 2.68128
std::vector 9948.07 ms
util::packed_vector 73306.6 ms
slowdown: 7.36892
random read:
std::vector 8460.91 ms
util::packed_vector 30177.8 ms
slowdown: 3.56673
508.293ms
0.508293ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
350.297ms
0.350297ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
624.345ms
0.624345ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
150.667ms
0.150667ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
97.4145ms
0.0974145ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
131.516ms
0.131516ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
149.903ms
0.149903ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
97.0547ms
0.0970547ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
131.542ms
0.131542ms/req
508.853ms
0.508853ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
349.815ms
0.349815ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
624.741ms
0.624741ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
151.161ms
0.151161ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
96.9316ms
0.0969316ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
132.315ms
0.132315ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
152.045ms
0.152045ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
96.9038ms
0.0969038ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
132.335ms
0.132335ms/req
638.445ms
0.638445ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
439.606ms
0.439606ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
817.78ms
0.81778ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
265.954ms
0.265954ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
162.716ms
0.162716ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
283.928ms
0.283928ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
257.784ms
0.257784ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
161.048ms
0.161048ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
281.935ms
0.281935ms/req
639.416ms
0.639416ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
437.273ms
0.437273ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
810.431ms
0.810431ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
256.097ms
0.256097ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
159.142ms
0.159142ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
280.538ms
0.280538ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
259.231ms
0.259231ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
159.369ms
0.159369ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
282.002ms
0.282002ms/req
202.806ms -> 0.0202806 ms/query
10 results:
237.224ms -> 0.0237224 ms/query
209.887ms -> 0.0209887 ms/query
10 results:
246.082ms -> 0.0246082 ms/query