Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple benchmarks for osrm-* tools #6925

Merged
merged 10 commits into from
Jun 6, 2024
Merged

Conversation

SiarheiFedartsou
Copy link
Member

@SiarheiFedartsou SiarheiFedartsou commented Jun 5, 2024

It was low hanging fruit as we are running these tools for other benchmarks anyway.

Screenshot 2024-06-06 at 14 35 51

Also I noticed that e2e tests I added recently are very "jumpy" :( I will probably address it in some separate PR...

Benchmark Results

Benchmark Base PR
alias aliased u32: 1133.92
plain u32: 1133.8
aliased double: 1167.26
plain double: 1169.79
aliased u32: 1097.71
plain u32: 1138.42
aliased double: 1172.87
plain double: 1179.5
e2e_match_ch requests: 399
failures: 0
req/s: 6.650req/s
avg: 5.575ms
50%: 3ms
75%: 3ms
95%: 25ms
98%: 49ms
99%: 62ms
min: 1.157ms
max: 96.289ms
requests: 385
failures: 0
req/s: 6.425req/s
avg: 5.557ms
50%: 3ms
75%: 4ms
95%: 19ms
98%: 46ms
99%: 70ms
min: 1.464ms
max: 91.964ms
e2e_match_mld requests: 362
failures: 0
req/s: 6.034req/s
avg: 4.496ms
50%: 3ms
75%: 4ms
95%: 11ms
98%: 19ms
99%: 53ms
min: 1.013ms
max: 66.555ms
requests: 367
failures: 0
req/s: 6.116req/s
avg: 5.576ms
50%: 3ms
75%: 4ms
95%: 18ms
98%: 52ms
99%: 75ms
min: 1.008ms
max: 87.581ms
e2e_nearest_ch requests: 428
failures: 0
req/s: 7.134req/s
avg: 1.420ms
50%: 1ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 2ms
min: 1.005ms
max: 3.117ms
requests: 422
failures: 0
req/s: 7.042req/s
avg: 1.571ms
50%: 2ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 3ms
min: 1.068ms
max: 2.761ms
e2e_nearest_mld requests: 426
failures: 0
req/s: 7.100req/s
avg: 1.458ms
50%: 1ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 2ms
min: 1.039ms
max: 6.775ms
requests: 425
failures: 0
req/s: 7.083req/s
avg: 1.524ms
50%: 1ms
75%: 2ms
95%: 2ms
98%: 2ms
99%: 2ms
min: 1.033ms
max: 3.204ms
e2e_route_ch requests: 414
failures: 0
req/s: 6.900req/s
avg: 3.662ms
50%: 4ms
75%: 4ms
95%: 5ms
98%: 6ms
99%: 8ms
min: 1.472ms
max: 9.935ms
requests: 403
failures: 0
req/s: 6.725req/s
avg: 4.276ms
50%: 4ms
75%: 5ms
95%: 6ms
98%: 7ms
99%: 10ms
min: 1.470ms
max: 99.512ms
e2e_route_mld requests: 372
failures: 0
req/s: 6.200req/s
avg: 4.636ms
50%: 4ms
75%: 5ms
95%: 6ms
98%: 7ms
99%: 10ms
min: 1.373ms
max: 141.264ms
requests: 419
failures: 0
req/s: 6.983req/s
avg: 4.248ms
50%: 4ms
75%: 5ms
95%: 6ms
98%: 7ms
99%: 9ms
min: 1.358ms
max: 97.756ms
e2e_table_ch requests: 388
failures: 0
req/s: 6.467req/s
avg: 29.454ms
50%: 22ms
75%: 30ms
95%: 110ms
98%: 120ms
99%: 130ms
min: 2.483ms
max: 138.929ms
requests: 400
failures: 0
req/s: 6.675req/s
avg: 35.443ms
50%: 23ms
75%: 34ms
95%: 120ms
98%: 130ms
99%: 140ms
min: 2.451ms
max: 151.331ms
e2e_table_mld requests: 385
failures: 0
req/s: 6.417req/s
avg: 73.161ms
50%: 79ms
75%: 110ms
95%: 130ms
98%: 140ms
99%: 140ms
min: 4.356ms
max: 165.649ms
requests: 390
failures: 0
req/s: 6.500req/s
avg: 42.225ms
50%: 27ms
75%: 61ms
95%: 130ms
98%: 130ms
99%: 140ms
min: 2.491ms
max: 328.336ms
e2e_trip_ch requests: 369
failures: 0
req/s: 6.150req/s
avg: 13.448ms
50%: 12ms
75%: 17ms
95%: 27ms
98%: 31ms
99%: 32ms
min: 1.994ms
max: 106.068ms
requests: 352
failures: 0
req/s: 5.874req/s
avg: 14.614ms
50%: 13ms
75%: 19ms
95%: 29ms
98%: 32ms
99%: 34ms
min: 1.829ms
max: 36.972ms
e2e_trip_mld requests: 352
failures: 0
req/s: 5.867req/s
avg: 20.310ms
50%: 20ms
75%: 26ms
95%: 34ms
98%: 37ms
99%: 39ms
min: 3.991ms
max: 44.437ms
requests: 351
failures: 0
req/s: 5.850req/s
avg: 14.914ms
50%: 13ms
75%: 19ms
95%: 28ms
98%: 33ms
99%: 36ms
min: 2.381ms
max: 40.495ms
json-render String: 6.56025ms
Stringstream: 9.27548ms
Vector: 6.8574ms
String: 6.66464ms
Stringstream: 9.44745ms
Vector: 7.01822ms
match_ch Default radius:
4.41197ms/req at 82 coordinate
0.0538045ms/coordinate
Radius 5m:
4.39374ms/req at 82 coordinate
0.0535822ms/coordinate
Radius 10m:
15.0527ms/req at 82 coordinate
0.183569ms/coordinate
Radius 15m:
36.7583ms/req at 82 coordinate
0.448272ms/coordinate
Radius 30m:
314.519ms/req at 82 coordinate
3.8356ms/coordinate
Default radius:
4.41337ms/req at 82 coordinate
0.0538215ms/coordinate
Radius 5m:
4.40282ms/req at 82 coordinate
0.053693ms/coordinate
Radius 10m:
15.0342ms/req at 82 coordinate
0.183344ms/coordinate
Radius 15m:
36.9353ms/req at 82 coordinate
0.45043ms/coordinate
Radius 30m:
313.907ms/req at 82 coordinate
3.82813ms/coordinate
match_mld Default radius:
2.80752ms/req at 82 coordinate
0.034238ms/coordinate
Radius 5m:
2.74811ms/req at 82 coordinate
0.0335135ms/coordinate
Radius 10m:
10.1372ms/req at 82 coordinate
0.123625ms/coordinate
Radius 15m:
25.9226ms/req at 82 coordinate
0.31613ms/coordinate
Radius 30m:
305.832ms/req at 82 coordinate
3.72966ms/coordinate
Default radius:
2.77088ms/req at 82 coordinate
0.0337912ms/coordinate
Radius 5m:
2.76309ms/req at 82 coordinate
0.0336963ms/coordinate
Radius 10m:
10.1278ms/req at 82 coordinate
0.12351ms/coordinate
Radius 15m:
26.0479ms/req at 82 coordinate
0.317657ms/coordinate
Radius 30m:
302.263ms/req at 82 coordinate
3.68613ms/coordinate
osrm_contract Time: 93.18s Peak RAM: 185.41MB Time: 94.27s Peak RAM: 185.70MB
osrm_customize Time: 1.31s Peak RAM: 115.05MB Time: 1.30s Peak RAM: 114.91MB
osrm_extract Time: 12.18s Peak RAM: 415.62MB Time: 12.31s Peak RAM: 425.00MB
osrm_partition Time: 2.18s Peak RAM: 148.48MB Time: 2.20s Peak RAM: 148.36MB
packedvector random write:
std::vector 11117.9 ms
util::packed_vector 73460.3 ms
slowdown: 6.60738
random read:
std::vector 11015.3 ms
util::packed_vector 29535 ms
slowdown: 2.68128
random write:
std::vector 9948.07 ms
util::packed_vector 73306.6 ms
slowdown: 7.36892
random read:
std::vector 8460.91 ms
util::packed_vector 30177.8 ms
slowdown: 3.56673
route_ch 1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
508.293ms
0.508293ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
350.297ms
0.350297ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
624.345ms
0.624345ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
150.667ms
0.150667ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
97.4145ms
0.0974145ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
131.516ms
0.131516ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
149.903ms
0.149903ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
97.0547ms
0.0970547ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
131.542ms
0.131542ms/req
1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
508.853ms
0.508853ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
349.815ms
0.349815ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
624.741ms
0.624741ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
151.161ms
0.151161ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
96.9316ms
0.0969316ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
132.315ms
0.132315ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
152.045ms
0.152045ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
96.9038ms
0.0969038ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
132.335ms
0.132335ms/req
route_mld 1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
638.445ms
0.638445ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
439.606ms
0.439606ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
817.78ms
0.81778ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
265.954ms
0.265954ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
162.716ms
0.162716ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
283.928ms
0.283928ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
257.784ms
0.257784ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
161.048ms
0.161048ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
281.935ms
0.281935ms/req
1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
639.416ms
0.639416ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
437.273ms
0.437273ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
810.431ms
0.810431ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
256.097ms
0.256097ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
159.142ms
0.159142ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
280.538ms
0.280538ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
259.231ms
0.259231ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
159.369ms
0.159369ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
282.002ms
0.282002ms/req
rtree 1 result:
202.806ms -> 0.0202806 ms/query
10 results:
237.224ms -> 0.0237224 ms/query
1 result:
209.887ms -> 0.0209887 ms/query
10 results:
246.082ms -> 0.0246082 ms/query

@SiarheiFedartsou SiarheiFedartsou marked this pull request as ready for review June 6, 2024 15:56
@SiarheiFedartsou SiarheiFedartsou merged commit feeed75 into master Jun 6, 2024
21 checks passed
@SiarheiFedartsou SiarheiFedartsou deleted the sf-osrm-tools-bench branch June 6, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants