Open tasks regarding move to osm2pgsql flex backend and new major release

[imagico]

There are a number of things that should be looked up after merging #4978. Since we are going to require deployments to re-import their rendering database it is a good idea to look at a number of things that can only be changed with a re-import of the data. These should best be done before the next release. Others are more long term strategic changes that are facilitated by the flex backend. For those it is more important to thoroughly consider and plan them rather than to rush to have them in this release.

• reviewing the ignore_keys and ignore_key_prefixes list regarding which might have largely vanished from the database in the meantime (so can be removed) and what new metadata tags might have been added to the database in larger numbers that we might want to add there.
• reviewing the roads_info table with the z-order values Use flex backend #4431 had added busway there - which might be a good idea to pre-emptively add in case we decide to render those in some form. bus_guideway might be good to add as well - see highway=bus_guideway layering, tunnels, covered, bridges #3581.
• reviewing the linestring/polygon defaults. Use flex backend #4431 has inverted the logic for waterway here - defaulting to linestring and listing exceptions for certain values to default to polygon. If we want to integrate that change we should take a careful look at what values should default to polygon. Use flex backend #4431 misses waterway=dam here - which is widely used on closed ways representing polygons.
• reviewing the column lists. This is a more impactful part. We discussed removing foot, bicycle and horse in context of the access restrictions rendering work. Because this affects backwards compatibility (see Flex version of the osm2pgsql configuration #4978 (comment)) we should discuss this as a separate issue. This can be - but does not have to be - part of the next release.
• creating a new table representing relations so we can use relation membership in rendering decisions - for example route relations on roads. Use flex backend #4431 provided the ground work for this. And there seems to be consensus on this already.
• developing a consensus strategy on administrative boundary rendering. Background is that we currently render administrative boundaries from the polygons as linestrings representation in the planet_osm_roads table. Use flex backend #4431 demonstrated a method to connect the individual ways ad hoc on a per-tile basis. That is not without issues, in particular discontinuities across tile boundaries and potential performance implications. But is is a method that can be employed ad hoc and that avoids the current white line + comp-op: darken technique (which is somewhat awkward and causes artefacts). It does not solve our long standing issue of low zoom admin boundary rendering (Replace usage of Natural Earth boundaries data #2172) or facilitating reducing the visual noise at the mid to low zoom levels by removing geometric and topological details.
• developing a data preprocessing strategy - already listed in Important todos #4901. Here it concerns in particular the question what kind of tag processing we might want to do during database import. What has been discussed in Introduction of SQL functions to the style #5004 is that style specific tag interpretation should probably not be hardcoded during import to give us and derivative styles flexibility in map design while practically immutable aspects of the OSM data model could be implemented during import - like layernotnull. Also useful might be a way_length column on planet_osm_line in analogy to way_area for polygons. A concrete approach how to do such things needs to be developed and discussed. Also this approach would call for moving the z-order calculation out of the import process. We would need to discuss if we want to do that - and if yes: how to implement that at render time instead.
• discussing the introduction of new geometry tables. Use flex backend #4431 suggested this for admin boundaries and roads. Another idea would be to move all buildings with no tags other than the building tags (that is >500M) out of planet_osm_polygon and have a separate buildings table with very few columns. For any such idea we would need to carefully evaluate the benefit this has under the goals of OSM-Carto.
• evaluating ending use of the planet_osm_roads table - this is connected to the previous item and the administrative boundary strategy.

[joto]

Just a quick note: A reimport is not strictly necessary for the changes in #4978. The database is compatible enough that you can keep updating an old database. That being said, occasional reimports are useful anyway due to database bloat.

And at some point a reimport has to be done, because of other changes in osm2pgsql. Since osm2pgsql 2.0.0 the old middle database format (and some other stuff) is not supported any more. For details see the osm2pgsql release notes and docs.

[imagico]

Yes, i should probably have explained that more explicitly. As per our README we do a new major release based on:

Any change the requires reloading a database, or upgrading software dependencies will trigger a major version change

If we decide to not have any database structure changes beyond #4978 we can mention in our release notes that a database reload is not strictly required. I would assume that most users of the style will do a reload anyway though - hence it is a good opportunity to do some adjustments we'd want to do anyway.

[mboeringa]

• developing a consensus strategy on administrative boundary rendering. Background is that we currently render administrative boundaries from the polygons as linestrings representation in the planet_osm_roads table. Use flex backend #4431 demonstrated a method to connect the individual ways ad hoc on a per-tile basis. That is not without issues, in particular discontinuities across tile boundaries and potential performance implications. But is is a method that can be employed ad hoc and that avoids the current white line + comp-op: darken technique (which is somewhat awkward and causes artefacts).

I am not to familiar with the entire openstreetmap-carto stack, but are you referring to the ST_LineMerge usage in project.mml? That usage is a specific choice in project.mml, and has little to do with osm2pgsql, if I understand the stack right.

The openstreetmap-carto.lua file input to osm2pgsql's flex processing does not include such logic for tile based processing, or merging of the individual boundary lines. It just imports them "as-is", so as individual line elements / way objects.

In my own postprocessing, I merge all lines not on tile boundaries, but for an entire section between two bordering administrative boundaries. I subsequently generalize them depending on scale, and subdivide if they become to big.

[imagico]

I am not to familiar with the entire openstreetmap-carto stack, but are you referring to the ST_LineMerge usage in project.mml? That usage is a specific choice in project.mml, and has little to do with osm2pgsql, if I understand the stack right.

The connection to osm2pgsql is that the current approach to administrative boundary rendering uses the polygon outlines as linestring representation in planet_osm_roads. If we want to take a renewed look at the database structure and what to change there, the administrative boundaries are one of the things that is going to be affected.

The different principal options are essentially known (though it would not hurt if someone would summarize them when starting a strategy discussion). And just to make this absolutely clear again: #2172 is not about line simplification, it is about topological simplification.