"Could not find node with id" despite node presence in PBF

[knowname]

When running tilemaker v3.0.0 (release binary on Ubuntu 22.04) with the planet on debug profile / config, I get the following error:

tilemaker --config config.json --process process.lua --output planet.mbtiles --store planet.tmp --compact planet-osmium-catted.osm.pbf
[...]
terminate called after throwing an instance of 'std::out_of_range'                                                                                                                         
  what():  Could not find node with id 11498743532                                                                                                                                         
Command terminated by signal 6                                                                                                                                                             

osmium getid confirmed that my input file does contain NodeID 11498743532:

<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="osmium/1.14.0">
  <bounds minlat="-90" minlon="-180" maxlat="90" maxlon="180"/>
  <node id="11498743532" version="1" timestamp="2024-01-08T00:58:56Z" uid="18863578" user="mountaineagle" changeset="146011008" lat="48.711566" lon="13.6249443"/>
</osm>

If I drop --compact, it runs through without a problem. This makes me think, it might have something to do with

tilemaker/src/node_stores.cpp

Lines 93 to 97 in 3af62f6

	LatpLon CompactNodeStore::at(NodeID i) const {
	if(i >= mLatpLons->size())
	throw std::out_of_range("Could not find node with id " + std::to_string(i));
	return mLatpLons->at(i);
	}

.

[cldellow]

Just checking -- is planet-osmium-catted.osm.pbf a PBF that was created by running osmium renumber?

This requirement is noted in the docs here:

tilemaker/docs/RUNNING.md

Lines 50 to 51 in 95644d2

	* `--compact`: Use a smaller, faster data structure for node lookups. __Note__: This requires
	the .pbf to have nodes in sequential order, typically by using `osmium renumber`.

-- but I think we might actually be able to make tilemaker detect and enforce it at runtime if it sees non-sequential numbering when it's reading the PBF

[knowname]

Just checking -- is planet-osmium-catted.osm.pbf a PBF that was created by running osmium renumber?

Yes, sorry for the confusion. The error persists, even if the file is renumbered. I just double checked.

tilemaker --config config-debug.json --process process-debug.lua --output planet.mbtiles --store planet.tmp --compact planet-renumbered.osm.pbf
...
Store size 529G | 1/6 Block 3001/124010 terminate called after throwing an instance of 'std::out_of_range'
  what():  Could not find node with id 8860665610
Command terminated by signal 6

Information on the renumbered PBF file:

osmium fileinfo -e planet-renumbered.osm.pbf 
File:
  Name: planet-renumbered.osm.pbf
  Format: PBF
  Compression: none
  Size: 80383299050
Header:
  Bounding boxes:
    (-180,-90,180,90)
  With history: no
  Options:
    generator=osmium/1.11.1
    osmosis_replication_timestamp=2024-01-08T00:59:59Z
    pbf_dense_nodes=true
    pbf_optional_feature_0=Sort.Type_then_ID
    sorting=Type_then_ID
    timestamp=2024-01-08T00:59:59Z
[======================================================================] 100% 
Data:
  Bounding box: (-180,-90,180,90)
  Timestamps:
    First: 2005-05-21T21:03:22Z
    Last: 2024-01-08T00:59:59Z
  Objects ordered (by type and id): yes
  Multiple versions of same object: no
  CRC32: not calculated (use --crc/-c to enable)
  Number of changesets: 0
  Number of nodes: 8860666091
  Number of ways: 992064323
  Number of relations: 11728203
  Smallest changeset ID: 0
  Smallest node ID: 1
  Smallest way ID: 1
  Smallest relation ID: 1
  Largest changeset ID: 0
  Largest node ID: 8860666091
  Largest way ID: 992064323
  Largest relation ID: 11728203
  All objects have following metadata attributes: version+timestamp+changeset
  Some objects have following metadata attributes: all
  Number of buffers: 14000446 (avg 704 objects per buffer)
  Sum of buffer sizes: 879570319624 (879.57 GB)
  Sum of buffer capacities: 917956198400 (917.956 GB, 96% full)

Information on the (original) planet file I am using is:

osmium fileinfo -e planet-latest.osm.pbf                                                                                                             
File:                                                                                          
  Name: planet-latest.osm.pbf                                                                  
  Format: PBF                                  
  Compression: none                            
  Size: 77937988660                            
Header:                                        
  Bounding boxes:                              
    (-180,-90,180,90)                          
  With history: no                             
  Options:                                     
    generator=planet-dump-ng 1.2.4             
    osmosis_replication_timestamp=2024-01-08T00:59:59Z                                         
    pbf_dense_nodes=true                       
    pbf_optional_feature_0=Has_Metadata                                                        
    pbf_optional_feature_1=Sort.Type_then_ID                                                   
    sorting=Type_then_ID                       
    timestamp=2024-01-08T00:59:59Z             
[======================================================================] 100%                  
Data:                                                                                                                                                                                         
  Bounding box: (-180,-90,180,90)              
  Timestamps:                                  
    First: 2005-05-21T21:03:22Z                
    Last: 2024-01-08T00:59:59Z                 
  Objects ordered (by type and id): yes        
  Multiple versions of same object: no         
  CRC32: not calculated (use --crc/-c to enable)                                               
  Number of changesets: 0                      
  Number of nodes: 8860666091                  
  Number of ways: 992064323                    
  Number of relations: 11728203                
  Smallest changeset ID: 0                     
  Smallest node ID: 1                          
  Smallest way ID: 37                          
  Smallest relation ID: 11                     
  Largest changeset ID: 0                      
  Largest node ID: 11498744706                 
  Largest way ID: 1237933295                   
  Largest relation ID: 16998994                
  All objects have following metadata attributes: version+timestamp+changeset                  
  Some objects have following metadata attributes: all                                         
  Number of buffers: 13472424 (avg 732 objects per buffer)                                     
  Sum of buffer sizes: 879570319624 (879.57 GB)                                                
  Sum of buffer capacities: 883921846272 (883.921 GB, 100% full)                               

[cldellow]

Thanks for the detailed debug data! I think there are two issues here.

1. tilemaker could offer a more helpful error when invoked with --compact on a non-renumbered file. I think this must be the case of the first error, since a node with ID 11,498,743,532 ought not be present in a renumbered file -- the max node ID should be 8,860,666,091

2. The combination of --compact and --shard-stores is broken. (--store implies --shard-stores unless overridden by --fast)

There's actually a comment in the code that explains why the two flags don't work together... but that's not very helpful. :) Instead, the code should not use --shard-stores when --compact is passed. I've opened a PR to improve this.