osrm-extract fails with "double free or corruption" error. Version v5.3.3

[djschilling]

When i use osrm-extract on the europe map downloaded from geofabrik, i get a "double free or corruption" error. Using a smaller map mostly works fine. The issue also sometimes appears using the german map. For extracting i used the car.lua included in the project.

I tested this behaviour with the current master and version v5.3.3.

My setup:
CentOS Linux release 7.2.1511
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio

Full stack trace:

[info] Using script profile.lua
[info] Input file: processed.osm.pbf
[info] Profile: profile.lua
[info] Threads: 16
[info] Parsing in progress..
[STXXL-MSG] STXXL v1.4.1 (prerelease/Release)
[STXXL-ERRMSG] Warning: no config file found.
[STXXL-ERRMSG] Using default disk configuration.
[STXXL-MSG] Disk '/var/tmp/stxxl' is allocated, space: 1000 MiB, I/O implementation: syscall autogrow delete_on_exit queue=0 devid=0
[info] input file generated by 0.43.1
[info] timestamp: n/a
[info] Found no exceptions to turn restrictions
[info] Parsing finished after 29.3122 seconds
[info] Raw input contains 17750609 nodes, 2920983 ways, and 534492 relations
[extractor] Sorting used nodes        ... ok, after 1.18409s
[extractor] Erasing duplicate nodes   ... ok, after 0.941381s
[extractor] Sorting all nodes         ... ok, after 1.92271s
[extractor] Building node id map      ... ok, after 7.62826s
[extractor] setting number of nodes   ... ok
[extractor] Confirming/Writing used nodes     ... ok, after 2.69493s
[info] Processed 17506268 nodes
[extractor] Sorting edges by start    ... ok, after 8.72873s
[extractor] Setting start coords      ... ok, after 17.0263s
[extractor] Sorting edges by target   ... ok, after 9.63335s
[extractor] Computing edge weights    ... ok, after 23.0794s
[extractor] Sorting edges by renumbered start ... ok, after 8.84797s
[extractor] Writing used edges       ... ok, after 6.75602s
[extractor] setting number of edges   ... ok
[info] Processed 18311515 edges
[extractor] Sorting used ways         ... ok, after 0.494343s
[extractor] Sorting 94077 restriction. by from... ok, after 0.008835s
[extractor] Fixing restriction starts ... ok, after 0.316964s
[extractor] Sorting restrictions. by to  ... ok, after 0.012928s
[extractor] Fixing restriction ends   ... ok, after 0.233188s
[info] usable restrictions: 85020
[extractor] writing street name index ... ok, after 0.127942s
[info] Writing turn lane masks...
[info] done (0.000143)
[info] extraction finished after 131.627s
*** Error in `/usr/bin/osrm-extract': double free or corruption (!prev): 0x00000000024cbb90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d023)[0x7f4f348a7023]
/usr/bin/osrm-extract(_ZN5stxxl6vectorIhLj4ENS_9lru_pagerILj8EEELj2097152ENS_2RCEyED2Ev+0x253)[0x4fc723]
/usr/bin/osrm-extract(_ZN4osrm9extractor20ExtractionContainersD1Ev+0xb7)[0x4fe897]
/usr/bin/osrm-extract(_ZN4osrm9extractor9Extractor3runERNS0_20ScriptingEnvironmentE+0x16dd)[0x50fa8d]
/usr/bin/osrm-extract(main+0x47f)[0x4a2eff]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4f3484bb15]
/usr/bin/osrm-extract[0x4a4b11]
======= Memory map: ========
00400000-006ca000 r-xp 00000000 fd:00 4296064025                         /usr/bin/osrm-extract
008ca000-008d2000 r--p 002ca000 fd:00 4296064025                         /usr/bin/osrm-extract
008d2000-008d3000 rw-p 002d2000 fd:00 4296064025                         /usr/bin/osrm-extract
008d3000-008d5000 rw-p 00000000 00:00 0
020bb000-3499d000 rw-p 00000000 00:00 0                                  [heap]
7f4e08000000-7f4e08a3c000 rw-p 00000000 00:00 0
7f4e08a3c000-7f4e0c000000 ---p 00000000 00:00 0
7f4e0c000000-7f4e0c021000 rw-p 00000000 00:00 0
7f4e0c021000-7f4e10000000 ---p 00000000 00:00 0
7f4e10000000-7f4e10a3c000 rw-p 00000000 00:00 0
7f4e10a3c000-7f4e14000000 ---p 00000000 00:00 0
7f4e18000000-7f4e18a3c000 rw-p 00000000 00:00 0
7f4e18a3c000-7f4e1c000000 ---p 00000000 00:00 0
7f4e20000000-7f4e20a3c000 rw-p 00000000 00:00 0
7f4e20a3c000-7f4e24000000 ---p 00000000 00:00 0
7f4e28000000-7f4e28a3c000 rw-p 00000000 00:00 0
7f4e28a3c000-7f4e2c000000 ---p 00000000 00:00 0
7f4e30000000-7f4e30a3c000 rw-p 00000000 00:00 0
7f4e30a3c000-7f4e34000000 ---p 00000000 00:00 0
7f4e38000000-7f4e38a3c000 rw-p 00000000 00:00 0
7f4e38a3c000-7f4e3c000000 ---p 00000000 00:00 0
7f4e40000000-7f4e40a3c000 rw-p 00000000 00:00 0
7f4e40a3c000-7f4e44000000 ---p 00000000 00:00 0
7f4e48000000-7f4e48a3c000 rw-p 00000000 00:00 0
7f4e48a3c000-7f4e4c000000 ---p 00000000 00:00 0
7f4e50000000-7f4e50a3c000 rw-p 00000000 00:00 0
7f4e50a3c000-7f4e54000000 ---p 00000000 00:00 0
7f4e58000000-7f4e58a3c000 rw-p 00000000 00:00 0
7f4e58a3c000-7f4e5c000000 ---p 00000000 00:00 0
7f4e60000000-7f4e60a3c000 rw-p 00000000 00:00 0
7f4e60a3c000-7f4e64000000 ---p 00000000 00:00 0
7f4e68000000-7f4e68a3c000 rw-p 00000000 00:00 0
7f4e68a3c000-7f4e6c000000 ---p 00000000 00:00 0
7f4e6cdf4000-7f4e6cdf5000 ---p 00000000 00:00 0
7f4e6cdf5000-7f4e6d5f5000 rw-p 00000000 00:00 0                          [stack:7761]
7f4e6d5f5000-7f4e6d5f6000 ---p 00000000 00:00 0
7f4e6d5f6000-7f4e6ddf6000 rw-p 00000000 00:00 0                          [stack:7760]
7f4e6ddf6000-7f4e6ddf7000 ---p 00000000 00:00 0
7f4e6ddf7000-7f4e6e5f7000 rw-p 00000000 00:00 0                          [stack:7759]
7f4e6e5f7000-7f4e6e5f8000 ---p 00000000 00:00 0
7f4e6e5f8000-7f4e6edf8000 rw-p 00000000 00:00 0                          [stack:7758]
7f4e6edf8000-7f4e6edf9000 ---p 00000000 00:00 0
7f4e6edf9000-7f4e6f5f9000 rw-p 00000000 00:00 0                          [stack:7757]
7f4e6f5f9000-7f4e6f5fa000 ---p 00000000 00:00 0
7f4e6f5fa000-7f4e6fdfa000 rw-p 00000000 00:00 0                          [stack:7756]
7f4e6fdfa000-7f4e6fdfb000 ---p 00000000 00:00 0
7f4e6fdfb000-7f4e705fb000 rw-p 00000000 00:00 0                          [stack:7755]
7f4e705fb000-7f4e705fc000 ---p 00000000 00:00 0
7f4e705fc000-7f4e70dfc000 rw-p 00000000 00:00 0                          [stack:7754]
7f4e70dfc000-7f4e70dfd000 ---p 00000000 00:00 0
7f4e70dfd000-7f4e715fd000 rw-p 00000000 00:00 0                          [stack:7753]
7f4e715fd000-7f4e715fe000 ---p 00000000 00:00 0
7f4e715fe000-7f4e71dfe000 rw-p 00000000 00:00 0                          [stack:7752]
7f4e7c000000-7f4e7c277000 rw-p 00000000 00:00 0
7f4e7c277000-7f4e80000000 ---p 00000000 00:00 0
7f4e80000000-7f4e80271000 rw-p 00000000 00:00 0
7f4e80271000-7f4e84000000 ---p 00000000 00:00 0
7f4e84000000-7f4e84285000 rw-p 00000000 00:00 0
7f4e84285000-7f4e88000000 ---p 00000000 00:00 0
7f4e88000000-7f4e8827c000 rw-p 00000000 00:00 0
7f4e8827c000-7f4e8c000000 ---p 00000000 00:00 0
7f4e8c000000-7f4e8c278000 rw-p 00000000 00:00 0
7f4e8c278000-7f4e90000000 ---p 00000000 00:00 0
7f4e90000000-7f4e90283000 rw-p 00000000 00:00 0
7f4e90283000-7f4e94000000 ---p 00000000 00:00 0
7f4e94000000-7f4e9427b000 rw-p 00000000 00:00 0
7f4e9427b000-7f4e98000000 ---p 00000000 00:00 0
7f4e98000000-7f4e98264000 rw-p 00000000 00:00 0
7f4e98264000-7f4e9c000000 ---p 00000000 00:00 0
7f4e9c000000-7f4e9c27a000 rw-p 00000000 00:00 0
7f4e9c27a000-7f4ea0000000 ---p 00000000 00:00 0
7f4ea0000000-7f4ea0261000 rw-p 00000000 00:00 0
7f4ea0261000-7f4ea4000000 ---p 00000000 00:00 0
7f4ea4000000-7f4ea4274000 rw-p 00000000 00:00 0
7f4ea4274000-7f4ea8000000 ---p 00000000 00:00 0
7f4ea8000000-7f4ea827d000 rw-p 00000000 00:00 0
7f4ea827d000-7f4eac000000 ---p 00000000 00:00 0
7f4eac000000-7f4eac276000 rw-p 00000000 00:00 0
7f4eac276000-7f4eb0000000 ---p 00000000 00:00 0
7f4eb0000000-7f4eb027b000 rw-p 00000000 00:00 0
7f4eb027b000-7f4eb4000000 ---p 00000000 00:00 0
7f4eb41e0000-7f4eb41e1000 ---p 00000000 00:00 0
7f4eb41e1000-7f4eb49e1000 rw-p 00000000 00:00 0                          [stack:7751]
7f4eb49e1000-7f4eb49e2000 ---p 00000000 00:00 0
7f4eb49e2000-7f4eb51e2000 rw-p 00000000 00:00 0                          [stack:7750]
7f4eb51e2000-7f4eb51e3000 ---p 00000000 00:00 0
7f4eb51e3000-7f4eb55e3000 rw-p 00000000 00:00 0                          [stack:7715]
7f4eb55e3000-7f4eb55e4000 ---p 00000000 00:00 0
7f4eb55e4000-7f4eb59e4000 rw-p 00000000 00:00 0                          [stack:7712]
7f4eb59e4000-7f4eb59e5000 ---p 00000000 00:00 0
7f4eb59e5000-7f4eb5de5000 rw-p 00000000 00:00 0                          [stack:7713]
7f4eb5de5000-7f4eb5de6000 ---p 00000000 00:00 0
7f4eb5de6000-7f4eb61e6000 rw-p 00000000 00:00 0                          [stack:7714]
7f4eb61e6000-7f4eb61e7000 ---p 00000000 00:00 0
7f4eb61e7000-7f4eb65e7000 rw-p 00000000 00:00 0                          [stack:7711]
7f4eb67e8000-7f4eb67e9000 ---p 00000000 00:00 0
7f4eb67e9000-7f4eb6be9000 rw-p 00000000 00:00 0                          [stack:7709]
7f4eb6be9000-7f4eb6bea000 ---p 00000000 00:00 0
7f4eb6bea000-7f4eb6fea000 rw-p 00000000 00:00 0                          [stack:7710]
7f4eb6fea000-7f4eb6feb000 ---p 00000000 00:00 0
7f4eb6feb000-7f4eb73eb000 rw-p 00000000 00:00 0                          [stack:7707]
7f4eb73eb000-7f4eb73ec000 ---p 00000000 00:00 0
7f4eb73ec000-7f4eb77ec000 rw-p 00000000 00:00 0                          [stack:7706]
7f4eb77ec000-7f4eb77ed000 ---p 00000000 00:00 0
7f4eb77ed000-7f4eb7bed000 rw-p 00000000 00:00 0                          [stack:7708]
7f4eb7bed000-7f4eb7bee000 ---p 00000000 00:00 0
7f4eb7bee000-7f4eb7fee000 rw-p 00000000 00:00 0                          [stack:7705]
7f4ecfffa000-7f4edc000000 rw-p 00000000 00:00 0
7f4edc000000-7f4edd269000 rw-p 00000000 00:00 0
7f4edd269000-7f4ee0000000 ---p 00000000 00:00 0
7f4ee0000000-7f4ee0285000 rw-p 00000000 00:00 0
7f4ee0285000-7f4ee4000000 ---p 00000000 00:00 0
7f4ee4000000-7f4ee526a000 rw-p 00000000 00:00 0
7f4ee526a000-7f4ee8000000 ---p 00000000 00:00 0
7f4ee8000000-7f4ee9270000 rw-p 00000000 00:00 0
7f4ee9270000-7f4eec000000 ---p 00000000 00:00 0
7f4eec000000-7f4eecfbd000 rw-p 00000000 00:00 0
7f4eecfbd000-7f4ef0000000 ---p 00000000 00:00 0
7f4ef0000000-7f4ef1241000 rw-p 00000000 00:00 0
7f4ef1241000-7f4ef4000000 ---p 00000000 00:00 0
7f4ef4000000-7f4ef4f86000 rw-p 00000000 00:00 0
7f4ef4f86000-7f4ef8000000 ---p 00000000 00:00 0
7f4ef8000000-7f4ef926f000 rw-p 00000000 00:00 0
7f4ef926f000-7f4efc000000 ---p 00000000 00:00 0
7f4efc000000-7f4efd26c000 rw-p 00000000 00:00 0
7f4efd26c000-7f4f00000000 ---p 00000000 00:00 0
7f4f00000000-7f4f0121d000 rw-p 00000000 00:00 0
7f4f0121d000-7f4f04000000 ---p 00000000 00:00 0
7f4f04000000-7f4f0526d000 rw-p 00000000 00:00 0
7f4f0526d000-7f4f08000000 ---p 00000000 00:00 0
7f4f08000000-7f4f09383000 rw-p 00000000 00:00 0
7f4f09383000-7f4f0c000000 ---p 00000000 00:00 0
7f4f0c000000-7f4f0d26f000 rw-p 00000000 00:00 0
7f4f0d26f000-7f4f10000000 ---p 00000000 00:00 0
7f4f10000000-7f4f1126c000 rw-p 00000000 00:00 0
7f4f1126c000-7f4f14000000 ---p 00000000 00:00 0
7f4f14000000-7f4f15669000 rw-p 00000000 00:00 0
7f4f15669000-7f4f18000000 ---p 00000000 00:00 0
7f4f181eb000-7f4f181ec000 ---p 00000000 00:00 0
7f4f181ec000-7f4f185ec000 rw-p 00000000 00:00 0                          [stack:7704]
7f4f185ec000-7f4f185ed000 ---p 00000000 00:00 0
7f4f185ed000-7f4f189ed000 rw-p 00000000 00:00 0                          [stack:7703]
7f4f189ed000-7f4f189ee000 ---p 00000000 00:00 0
7f4f189ee000-7f4f18dee000 rw-p 00000000 00:00 0                          [stack:7702]
7f4f18dee000-7f4f18def000 ---p 00000000 00:00 0
7f4f18def000-7f4f191ef000 rw-p 00000000 00:00 0                          [stack:7701]
7f4f197fb000-7f4f197fc000 ---p 00000000 00:00 0
7f4f197fc000-7f4f19ffc000 rw-p 00000000 00:00 0                          [stack:7749]
7f4f19ffc000-7f4f19ffd000 ---p 00000000 00:00 0
7f4f19ffd000-7f4f1a7fd000 rw-p 00000000 00:00 0                          [stack:7748]
7f4f1a7fd000-7f4f1a7fe000 ---p 00000000 00:00 0
7f4f1a7fe000-7f4f1affe000 rw-p 00000000 00:00 0                          [stack:7723]
7f4f1affe000-7f4f1afff000 ---p 00000000 00:00 0
7f4f1afff000-7f4f1b7ff000 rw-p 00000000 00:00 0                          [stack:7700]
7f4f1b7ff000-7f4f1b800000 ---p 00000000 00:00 0
7f4f1b800000-7f4f1c000000 rw-p 00000000 00:00 0                          [stack:7699]
7f4f1c000000-7f4f1d94c000 rw-p 00000000 00:00 0
7f4f1d94c000-7f4f20000000 ---p 00000000 00:00 0
7f4f205f8000-7f4f207f9000 rw-p 00000000 00:00 0
7f4f207f9000-7f4f207fa000 ---p 00000000 00:00 0
7f4f207fa000-7f4f20ffa000 rw-p 00000000 00:00 0                          [stack:7698]
7f4f20ffa000-7f4f20ffb000 ---p 00000000 00:00 0
7f4f20ffb000-7f4f217fb000 rw-p 00000000 00:00 0                          [stack:7697]
7f4f217fb000-7f4f217fc000 ---p 00000000 00:00 0
7f4f217fc000-7f4f21ffc000 rw-p 00000000 00:00 0                          [stack:7696]
7f4f21ffc000-7f4f21ffd000 ---p 00000000 00:00 0
7f4f21ffd000-7f4f227fd000 rw-p 00000000 00:00 0                          [stack:7695]
7f4f227fd000-7f4f227fe000 ---p 00000000 00:00 0
7f4f227fe000-7f4f22ffe000 rw-p 00000000 00:00 0                          [stack:7694]
7f4f22ffe000-7f4f22fff000 ---p 00000000 00:00 0
7f4f22fff000-7f4f237ff000 rw-p 00000000 00:00 0                          [stack:7693]
7f4f237ff000-7f4f23800000 ---p 00000000 00:00 0
7f4f23800000-7f4f24000000 rw-p 00000000 00:00 0                          [stack:7692]
7f4f24000000-7f4f25719000 rw-p 00000000 00:00 0
7f4f25719000-7f4f28000000 ---p 00000000 00:00 0
7f4f284a1000-7f4f284a2000 ---p 00000000 00:00 0
7f4f284a2000-7f4f28ca2000 rw-p 00000000 00:00 0                          [stack:7691]
7f4f28ca2000-7f4f28ca3000 ---p 00000000 00:00 0
7f4f28ca3000-7f4f294a3000 rw-p 00000000 00:00 0                          [stack:7690]
7f4f294a3000-7f4f294a4000 ---p 00000000 00:00 0
7f4f294a4000-7f4f29ca4000 rw-p 00000000 00:00 0                          [stack:7689]
7f4f29ca4000-7f4f29ca5000 ---p 00000000 00:00 0
7f4f29ca5000-7f4f2a4a5000 rw-p 00000000 00:00 0                          [stack:7688]
7f4f2a4a5000-7f4f2a4a6000 ---p 00000000 00:00 0
7f4f2a4a6000-7f4f2aca6000 rw-p 00000000 00:00 0                          [stack:7687]
7f4f2afa9000-7f4f2afaa000 ---p 00000000 00:00 0
7f4f2afaa000-7f4f2b7aa000 rw-p 00000000 00:00 0
7f4f2b7aa000-7f4f2b7ab000 ---p 00000000 00:00 0
7f4f2b7ab000-7f4f2c1ab000 rw-p 00000000 00:00 0                          [stack:7747]
7f4f2c1ab000-7f4f326d2000 r--p 00000000 fd:00 4296506938                 /usr/lib/locale/locale-archive
7f4f326d2000-7f4f326d5000 r-xp 00000000 fd:00 8597718765                 /usr/lib64/libdl-2.17.so
7f4f326d5000-7f4f328d4000 ---p 00003000 fd:00 8597718765                 /usr/lib64/libdl-2.17.so
7f4f328d4000-7f4f328d5000 r--p 00002000 fd:00 8597718765                 /usr/lib64/libdl-2.17.so
7f4f328d5000-7f4f328d6000 rw-p 00003000 fd:00 8597718765                 /usr/lib64/libdl-2.17.so
7f4f328d6000-7f4f33ca9000 r-xp 00000000 fd:00 8599390663                 /usr/lib64/libicudata.so.50.1.2
7f4f33ca9000-7f4f33ea8000 ---p 013d3000 fd:00 8599390663                 /usr/lib64/libicudata.so.50.1.2
7f4f33ea8000-7f4f33ea9000 r--p 013d2000 fd:00 8599390663                 /usr/lib64/libicudata.so.50.1.2
7f4f33ea9000-7f4f33eaa000 rw-p 013d3000 fd:00 8599390663                 /usr/lib64/libicudata.so.50.1.2
7f4f33eaa000-7f4f3409a000 r-xp 00000000 fd:00 8599390665                 /usr/lib64/libicui18n.so.50.1.2
7f4f3409a000-7f4f3429a000 ---p 001f0000 fd:00 8599390665                 /usr/lib64/libicui18n.so.50.1.2
7f4f3429a000-7f4f342a6000 r--p 001f0000 fd:00 8599390665                 /usr/lib64/libicui18n.so.50.1.2
7f4f342a6000-7f4f342a8000 rw-p 001fc000 fd:00 8599390665                 /usr/lib64/libicui18n.so.50.1.2
7f4f342a8000-7f4f342a9000 rw-p 00000000 00:00 0
7f4f342a9000-7f4f3440d000 r-xp 00000000 fd:00 8599390683                 /usr/lib64/libicuuc.so.50.1.2
7f4f3440d000-7f4f3460d000 ---p 00164000 fd:00 8599390683                 /usr/lib64/libicuuc.so.50.1.2

[djschilling]

When i run the preprocessing with a debug build i get the same error with this stacktrace:

[info] extraction finished after 5016.5s
*** Error in `osrm-extract': double free or corruption (fasttop): 0x00000000019a7810 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d053)[0x7f213b29e053]
osrm-extract(_ZN9__gnu_cxx13new_allocatorISt10_List_nodeIyEE10deallocateEPS2_m+0xc)[0x8cbb74]
osrm-extract(_ZNSt10_List_baseIySaIyEE11_M_put_nodeEPSt10_List_nodeIyE+0xe)[0x8cbb84]
osrm-extract(_ZNSt10_List_baseIySaIyEE8_M_clearEv+0x1a)[0x8cbba0]
osrm-extract(_ZNSt10_List_baseIySaIyEED1Ev+0x9)[0x8cbbb7]
osrm-extract(_ZNSt4listIySaIyEED1Ev+0x9)[0x8cbbc3]
osrm-extract(_ZN5stxxl9lru_pagerILj8EED1Ev+0x1d)[0x8cbbe3]
osrm-extract(_ZN5stxxl6vectorIjLj4ENS_9lru_pagerILj8EEELj2097152ENS_2RCEyED1Ev+0xc8)[0x8eb6e8]
osrm-extract(_ZN4osrm9extractor20ExtractionContainersD1Ev+0x54)[0x8ec822]
osrm-extract(_ZN4osrm9extractor9Extractor3runERNS0_20ScriptingEnvironmentE+0xb97)[0x8fcd8d]
osrm-extract(main+0x21d)[0x8aa9c1]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f213b242b15]
osrm-extract[0x89fa01]

Running the preprocessing in gdb produces this:

[extractor] Erasing duplicate nodes   ... ok, after 72.4792s
[extractor] Sorting all nodes         ... ok, after 30.2694s
[extractor] Building node id map      ... ok, after 156.191s
[extractor] setting number of nodes   ... ok
[extractor] Confirming/Writing used nodes     ... ok, after 92.6714s
[info] Processed 164609790 nodes
[extractor] Sorting edges by start    ... ok, after 150.524s
[extractor] Setting start coords      ... ok, after 351.095s
[extractor] Sorting edges by target   ... ok, after 160.189s
[extractor] Computing edge weights    ... ok, after 441.904s
[extractor] Sorting edges by renumbered start ... osrm-extract: /usr/include/stxxl/bits/containers/pager.h:90: void stxxl::lru_pager<npages_>::hit(stxxl::lru_pager<npages_>::size_type) [with unsigned int npages_ = 8u; stxxl::lru_pager<npages_>::size_type = long long unsigned int]: Assertion `ipage < size()' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffec668700 (LWP 7114)]
0x00007ffff4f1b5f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install osrm-backend-5.4.0-1.x86_64
(gdb) bt
#0  0x00007ffff4f1b5f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff4f1cce8 in abort () from /lib64/libc.so.6
#2  0x00007ffff4f14566 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4f14612 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000009034b6 in stxxl::lru_pager<8u>::hit (this=this@entry=0x7fffffffb770, ipage=ipage@entry=8) at /usr/include/stxxl/bits/containers/pager.h:90
#5  0x0000000000973b7a in stxxl::vector<unsigned char, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::const_element (this=0x7fffffffb728, offset=...) at /usr/include/stxxl/bits/containers/vector.h:1741
#6  0x0000000000973cf8 in stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u>::operator* (this=this@entry=0x7fffec667720)
    at /usr/include/stxxl/bits/containers/vector.h:640
#7  0x0000000000973df8 in std::__lexicographical_compare<false>::__lc<stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u>, stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u> > (__first1=..., __last1=..., __first2=..., __last2=...) at /usr/include/c++/4.8.2/bits/stl_algobase.h:888
#8  0x0000000000973f4b in std::__lexicographical_compare_aux<stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u>, stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u> > (__first1=..., __last1=..., __first2=..., __last2=...) at /usr/include/c++/4.8.2/bits/stl_algobase.h:927
#9  0x0000000000974018 in std::lexicographical_compare<stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u>, stxxl::const_vector_iterator<unsigned char, stxxl::RC, unsigned long long, long long, 2097152u, stxxl::lru_pager<8u>, 4u> > (__first1=..., __last1=..., __first2=..., __last2=...) at /usr/include/c++/4.8.2/bits/stl_algobase.h:1102
#10 0x0000000000974196 in (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName::operator() (this=this@entry=0x7fffec667af0, lhs=..., rhs=...)
    at /vagrant/recipes/osrm/tmp-build/osrm-backend-branch-feature/edge_weights/src/extractor/extraction_containers.cpp:90
#11 0x000000000098c896 in std::__unguarded_linear_insert<osrm::extractor::InternalExtractorEdge*, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName> (__last=__last@entry=0x7ffeccbb88b0, __comp=...)
    at /usr/include/c++/4.8.2/bits/stl_algo.h:2117
#12 0x000000000098c8e2 in std::__unguarded_insertion_sort<osrm::extractor::InternalExtractorEdge*, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName> (__first=__first@entry=0x7ffecc020590, __last=__last@entry=0x7ffecffffd38,
    __comp=...) at /usr/include/c++/4.8.2/bits/stl_algo.h:2194
#13 0x00000000009ea262 in std::__final_insertion_sort<osrm::extractor::InternalExtractorEdge*, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName> (__first=__first@entry=0x7ffecc020010, __last=__last@entry=0x7ffecffffd38,
    __comp=...) at /usr/include/c++/4.8.2/bits/stl_algo.h:2227
#14 0x00000000009ea2d1 in std::sort<osrm::extractor::InternalExtractorEdge*, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName> (__first=0x7ffecc020010, __last=0x7ffecffffd38, __comp=...)
    at /usr/include/c++/4.8.2/bits/stl_algo.h:5500
#15 0x00000000009ea2f0 in __gnu_parallel::__possibly_stable_sort<false, osrm::extractor::InternalExtractorEdge*, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName>::operator() (this=this@entry=0x7fffec667d7d,
    __begin=<optimized out>, __end=<optimized out>, __comp=...) at /usr/include/c++/4.8.2/parallel/multiway_mergesort.h:264
#16 0x0000000000a055a1 in __gnu_parallel::parallel_sort_mwms_pu<false, true, stxxl::array_of_sequences_iterator<stxxl::typed_block<2097152u, osrm::extractor::InternalExtractorEdge, 0u, void>, osrm::extractor::InternalExtractorEdge, 23831ull>, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName> (__sd=0x7fffffff8fa0, __comp=...) at /usr/include/c++/4.8.2/parallel/multiway_mergesort.h:335
#17 0x0000000000a0581f in __gnu_parallel::parallel_sort_mwms<false, true, stxxl::array_of_sequences_iterator<stxxl::typed_block<2097152u, osrm::extractor::InternalExtractorEdge, 0u, void>, osrm::extractor::InternalExtractorEdge, 23831ull>, (anonymous namespace)::CmpEdgeByInternalSourceTargetAndName> () at /usr/include/c++/4.8.2/parallel/multiway_mergesort.h:461
#18 0x00007ffff54cbde5 in ?? () from /lib64/libgomp.so.1
#19 0x00007ffff69e3dc5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007ffff4fdcced in clone () from /lib64/libc.so.6

It seems like there is a assertion that's failing when running in a debugger. Maybe that can help finding the error.

[danpat]

The error is happening inside STXXL during array sorting. Possibly related:

#2700

In that case, it looks like pre-allocating STXXL space by doing:

echo disk=/tmp/stxxl,50G,syscall > .stxxl

to create an STXXL file and pre-configure 50GB of swap space for it. Try that. We pre-assign 250GB for the OSRM demo server on the OSM planet file, so use that as a guide for sizing.

[djschilling]

I just discovered that we have two stxxl log files in the preprocessing directory.

stxxl.errlog:

[STXXL-ERRMSG] Warning: no config file found.
[STXXL-ERRMSG] Using default disk configuration.

stxxl.log:

[STXXL-MSG] STXXL v1.4.1 (prerelease/Release)
[STXXL-MSG] Disk '/var/tmp/stxxl' is allocated, space: 1000 MiB, I/O implementation: syscall autogrow delete_on_exit queue=0 devid=0

[djschilling]

I preprocessed now again the europe map with 150G pre-allocated space.

The error is the same.

When i use an older OSRM version (4.8.1) it works.

[danpat]

@pongo710 Can you link to the exact map file you're using so we can try to reproduce this?

[sandra-thieme]

@danpat The map file with which we can reliably reproduce this issue is this one: http://maps.contargo.net/maps/2016/unprocessed-osm.pbf

It's a copy from the europe-latest.osm.pbf from download.geofabrik.de from november last year.

[djschilling]

We did some further testing.
If we take the current europe map from geofabrik (http://download.geofabrik.de/europe-latest.osm.pbf) preprocessing works.

So it seems to be an issue with the map we linked. Sadly older maps are not present on geofabrik so we could not test it with older maps.

[danpat]

I've made a copy of the file you liked, it's exposed a few issues that were working on. Haven't reproduced this exact error yet.

[danpat]

Ok, using the latest master (7a52371), on Ubuntu 14.04, I proceed past this step just fine, I don't get the double-free problem.

@rkcpi @pongo710 Can you try again with 7a52371 ? It seems to work OK for me on your older map.

If it still fails, then I'll need to try to reproduce on CentOS - I've currently only tried with Ubuntu 14.04, which has a different version of STXXL and various other libs (I'm linking against STXXL 1.3.1).

[djschilling]

We now tried out the master version with commit d3a6b5a.

It seems to fix some problems but not everything.
Preprocessing the map @rkcpi linked(http://maps.contargo.net/maps/2016/unprocessed-osm.pbf) to you works now.

But we have another map which still produces the double free or corruption error.

You can find the map here: http://maps.contargo.net/maps/2016/processed.osm.pbf.
Its also the europe map, but we deleted a lot of nodes and ways and tags in it which are not necessary for us. The map should be valid, we used it in the previous year.

[danpat]

I set up a CentOS 7.2.1511 Docker container, and installed packages that @pongo710 provided out-of-band. I can reproduce the problem.

Here's the backtrace I get:

[info] usable restrictions: 268204
[extractor] writing street name index ... ok, after 10.4788s
[info] extraction finished after 2130.89s

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff4f63aae in _int_free () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install osrm-backend-5.4.0-1.x86_64
(gdb) bt
#0  0x00007ffff4f63aae in _int_free () from /lib64/libc.so.6
#1  0x00000000008e3542 in __gnu_cxx::new_allocator<std::_List_node<unsigned long long> >::deallocate (this=this@entry=0x7fffffffbdb8, __p=<optimized out>)
    at /usr/include/c++/4.8.2/ext/new_allocator.h:110
#2  0x00000000008e3552 in std::_List_base<unsigned long long, std::allocator<unsigned long long> >::_M_put_node (this=this@entry=0x7fffffffbdb8,
    __p=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_list.h:338
#3  0x00000000008e356e in std::_List_base<unsigned long long, std::allocator<unsigned long long> >::_M_clear (this=0x7fffffffbdb8)
    at /usr/include/c++/4.8.2/bits/list.tcc:79
#4  0x00000000008e3585 in std::_List_base<unsigned long long, std::allocator<unsigned long long> >::~_List_base (this=<optimized out>,
    __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_list.h:378
#5  0x00000000008e3591 in std::list<unsigned long long, std::allocator<unsigned long long> >::~list (this=<optimized out>, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8.2/bits/stl_list.h:438
#6  0x00000000008e35b1 in stxxl::lru_pager<8u>::~lru_pager (this=0x7fffffffbdb8, __in_chrg=<optimized out>)
    at /usr/include/stxxl/bits/containers/pager.h:66
#7  0x0000000000901278 in stxxl::vector<unsigned int, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::~vector (this=0x7fffffffbd70,
    __in_chrg=<optimized out>) at /usr/include/stxxl/bits/containers/vector.h:1515
#8  0x000000000090239a in osrm::extractor::ExtractionContainers::~ExtractionContainers (this=0x7fffffffb910, __in_chrg=<optimized out>)
    at /vagrant/recipes/osrm/tmp-build/osrm-backend-branch-master/include/extractor/extraction_containers.hpp:26
#9  0x00000000008c86f8 in osrm::extractor::Extractor::run (this=this@entry=0x7fffffffe230, scripting_environment=...)
    at /vagrant/recipes/osrm/tmp-build/osrm-backend-branch-master/src/extractor/extractor.cpp:127
#10 0x00000000008b062d in main (argc=4, argv=0x7fffffffe7a8) at /vagrant/recipes/osrm/tmp-build/osrm-backend-branch-master/src/tools/extract.cpp:156

The error is happening during teardown of the ExtractionContainers object - somehow it seems we're triggering a bug in STXXL.

@pongo710 Please try re-building against a different version of STXXL - the one you have may be buggy. STXXL 1.3.1 that's packaged with Ubuntu seems to work just fine, I cannot reproduce the problem when building the same version of OSRM against the Ubuntu supplied libraries.

[djschilling]

I have now a working processing of the map @rkcpi linked above.
The preprocessing uses the current master of osrm and the current master of STXXL. The change to the master of STXXL seems to have fixed it.

I just started a second preprocessing to be sure it's now working.

[djschilling]

Using master version of STXXl did not fix all problems. We now use STXXL Version 1.3.1.

In this configuration every processing we have done is working now.
I am still interested in what actually caused the problem, but we can now work with this setup.

Thanks for all the help.