Understand OSRM Compiled Map Data Files

[wangyoucao577]

Here's overview of OSRM generated files: https://github.com/Project-OSRM/osrm-backend/wiki/Toolchain-file-overview

There could be a sub issue for each file.
This issue can be closed once we have documents for all files or we don't need to understand others anymore.

Deep understanding for the files can help us in debugging. Also, it's the foundation since we hope to have visualization tool to view them: #111

[wangyoucao577]

As discussion in Project-OSRM#5227, all the OSRM generated files are standard tar format. So we can extermine the contents of the files using tar tvf <filename>.osrm.*, or use tar xvf <filename>.osrm.* to extract them.
E.g.

tar -tvf nevada-latest.osrm.nbg_nodes 
-rw-rw-r--  0 0      0           8 Jan  1  1970 osrm_fingerprint.meta
-rw-rw-r--  0 0      0           8 Jan  1  1970 /common/nbn_data/coordinates.meta
-rw-rw-r--  0 0      0     9137888 Jan  1  1970 /common/nbn_data/coordinates
-rw-rw-r--  0 0      0           8 Jan  1  1970 /common/nbn_data/osm_node_ids/number_of_elements.meta
-rw-rw-r--  0 0      0           8 Jan  1  1970 /common/nbn_data/osm_node_ids/packed.meta
-rw-rw-r--  0 0      0     8995392 Jan  1  1970 /common/nbn_data/osm_node_ids/packed

The code include/storage/tar.hpp implements FileReader/FileWriter for all read/write operations based on third party rxi/microtar lib.

[wangyoucao577]

First breakdown issue for .osrm file: #116

[wangyoucao577]

Docs in https://github.com/Telenav/open-source-spec/tree/master/osrm/doc/osrm-toolchain-files

[wangyoucao577]

So far all MLD related files have been breakdown to sub issues as above.

$ ls -lh /osrm-data/
2020-01-22 15:33:27.608 total 477M
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  47M Jan 22 07:32 map.osm.pbf
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  57M Jan 22 07:33 map.osrm
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  64M Jan 22 07:33 map.osrm.cell_metrics
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 536K Jan 22 07:33 map.osrm.cells
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 5.7M Jan 22 07:33 map.osrm.cnbg
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 5.7M Jan 22 07:33 map.osrm.cnbg_to_ebg
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  68K Jan 22 07:33 map.osrm.datasource_names
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  31M Jan 22 07:33 map.osrm.ebg
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 8.3M Jan 22 07:33 map.osrm.ebg_nodes
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 9.0M Jan 22 07:33 map.osrm.edges
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 7.9M Jan 22 07:33 map.osrm.enw
2020-01-22 15:33:27.608 -rwx------ 1 root root  24M Jan 22 07:33 map.osrm.fileIndex
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  28M Jan 22 07:33 map.osrm.geometry
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 4.5M Jan 22 07:33 map.osrm.icd
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 5.0K Jan 22 07:33 map.osrm.maneuver_overrides
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  32M Jan 22 07:33 map.osrm.mldgr
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 964K Jan 22 07:33 map.osrm.names
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  18M Jan 22 07:33 map.osrm.nbg_nodes
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 5.3M Jan 22 07:33 map.osrm.partition
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 6.0K Jan 22 07:33 map.osrm.properties
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 103K Jan 22 07:33 map.osrm.ramIndex
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 4.0K Jan 22 07:33 map.osrm.restrictions
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 4.0K Jan 22 07:32 map.osrm.timestamp
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 4.5K Jan 22 07:33 map.osrm.tld
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 6.5K Jan 22 07:33 map.osrm.tls
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 2.6M Jan 22 07:33 map.osrm.turn_duration_penalties
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root  16M Jan 22 07:33 map.osrm.turn_penalties_index
2020-01-22 15:33:27.608 -rw-r--r-- 1 root root 2.6M Jan 22 07:33 map.osrm.turn_weight_penalties
2020-01-22 15:33:27.608 -rwxr-xr-x 1 root root  78M Jan 22 07:33 wayid2nodeids.csv
2020-01-22 15:33:27.608 -rwxr-xr-x 1 root root  33M Jan 22 07:33 wayid2nodeids.csv.snappy

All MLD route algorithm related files research will be high priority in which case. Others such as CH related, nearest related, guidance related are not important now.