osrm-prepare issue memory

[tom44000]

Hi there,

I have a problem with preparation on lastest version of OSRM

My config :
process file: latest-europe.pbf (16Go)
Ram : 48 Go
Disk : 220 Go SSD
Swap : 7 Go
LUA version : Lua 5.1.5 Copyright (C) 1994-2012 Lua.org, PUC-Rio

When I launch osrm-prepare, this error occured :

[info] Input file: europe-latest.osrm
[info] Restrictions file: europe-latest.osrm.restrictions
[info] Profile: profile.lua
[info] Threads: 12
[info] Generating edge-expanded graph representation
[info] - 279681 restrictions.
[info] Importing n = 192596056 nodes
[info] - 120343 bollard nodes, 269043 traffic lights
[info] and 202040036 edges
[info] Graph loaded ok and has 202040036 edges
[info] Removing graph geometry while preserving topology
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
[info] removed 156277095 nodes
[info] Geometry successfully removed:
compressed edges: 55963300
compressed geometries: 368452980
longest chain length: 3605
cmpr ratio: 0.151887
avg chain length: 6.58383
[info] new nodes: 36318961, edges 91321368
[info] Node compression ratio: 0.188576
[info] Edge compression ratio: 0.226113
[info] Identifying components of the (compressed) road network
[info] large component [1]=35849024
[info] large component [1590968]=10177
[info] large component [8676785]=12845
[info] SCC run took: 33.8593s
[info] identified: 154229 (compressed) components
[info] identified 19227 (compressed) SCCs of size 1
[info] generating edge-expanded nodes
. 10% . 20% . 30% . 40% . 50% [warn] [exception] std::bad_alloc

With this memory information when crash :

         total       used       free     shared    buffers     cached

Mem: 40168 39315 853 9570 8 11614
-/+ buffers/cache: 27692 12476
Swap: 7597 1236 6361

Thanks in advance for any solutions

[daniel-j-h]

Are you using the latest release or a specific git branch (master, develop, ..)?

std::bad_alloc only gets thrown if an allocation request could not be fulfilled.

Just to make sure, could you run osrm-extract and then osrm-prepare on a smaller dataset (e.g. sweden). If this works, do the following to increase your swap space:

fallocate -l 50G /path/to/swapfile
chmod 600 /path/to/swapfile
mkswap /path/to/swapfile
swapon /path/to/swapfile

Make sure the swapfile is on disk and not in a tmpfs mount. Also, adapt the size (fallocate does not write to the swapfile, it just "reserves" the space, there's no physical writing involved).

And then try the large dataset again.

[tom44000]

Osrm version is the latest commit on master branch
It's ok on smaller dataset..

[daniel-j-h]

Great, then try the swapfile approach. It might be that we are simply running out of memory here for a temporary allocation. That is, even if you may have some memory left, an allocation request might fail in case your memory allocation subsystem fails to find an contiguous area for it.

[tom44000]

Ok, I will testing this solution

Thank a lot Daniel

[TheMarex]

@tom44000 any update here? Is this good to close?

[tom44000]

It's ok for me !! thanks