Switch to Lua transforms and change database schema #2533

pnorman · 2017-01-03T07:18:51Z

Not yet ready for merge

What this does

This PR accomplishes four main tasks

Lua transforms

It switches to Lua transforms, which allows pre-processing of tags like layer. This allows SQL like

ORDER BY COALESCE(layer,0), way_area DESC

Instead of the longer

ORDER BY CASE WHEN layer~E'^-?\\d+$' AND length(layer)<10 THEN layer::integer ELSE 0 END, way_area DESC

There are also more significant changes for z_order, but that can be done post-merge

Disjoint areas

It uses -G, so disjoint areas are imported as PostGIS MULTIPOLYGONs instead of multiple rows as POLYGONs. These are often found on administrative boundary relations where islands are disjoint.

Tag columns

It uses --hstore, so tags that don't have a column end up in a special tags column. This allows for more flexability, and the number of columns has been significantly reduced

Multipolygon handling

The multipolygon handling is in the transforms, so instead of the C transforms, we are responsible for defining the handling.

For the purposes of this PR and the transforms, a multipolygon is either new-style or old-style. A new-style multipolygon has a tag other than type=multipolygon on the relation, and tags on its members are ignored. An old-style multipolygon is one with only a type=multipolygon and all the member ways have the same tags, or a member may be untagged. For technical reasons, deleted tags (e.g. source=*) are ignored.

There is also implicitly the definition of an invalid multipolygon, which is a type=multipolygon relation which doesn't meet the above criteria.

Some properties of MPs as defined like this are

Any new-style MP will never change to an old-style MP by only changing tags on its member ways, nor will it become invalid. Sudden changes to interpretation were possible with the C transform logic, where adding building=yes to a small object would set the building column to yes for a large object generated by a relation.
Any old-style MP will never change to an new-style MP by only changing tags on its member ways, but may become invalid if the tags become conflicting
type=multipolygon relations are always areas
Processing mulitpolygons does not require any knowledge of tagging semantics like is required for area formation from polygons. For technical reasons, deleted keys are ignored, which does involve tagging semantics.

Area handling

We can now determine if a way corresponds to an area feature or a linear feature however we want, instead of just based on tag keys. We want to avoid the interpretation of a correctly mapped feature depending on if a way is closed or not.

Open issues

Add comments to openstreetmap-carto.style pointing to INSTALL.md
Add something on using hstore to the code style guide (tags->'foo' instead of tags -> 'foo', etc) (Add SQL style notes on hstore #2552)
The interim need to backport PRs against both master and 3.x needs documenting in CONTRIBUTING.md
Define more precisely how long we will maintain 3.x and 4.x compatibility
Better document MPs somewhere

Pre-merge changes

The changelog and README.md version information need changing

A new git branch for 3.x.x work needs creating

Post-merge changes

Mark database layout milestone as finished

Fixes #101, #1362, #1504

Preview

I have a demo at https://lua.osm-carto.paulnorman.ca/. It is not consuming OSM updates. Before reporting a bug please

See if it's covered already in the discussion
See if it's an expected change
Check the OSM tagging so you can report what it is, if relevant
Check if the OSM data has recently changed

@pnorman

**This is a pull-request against the lua branch** This PR proposes a database-reload. It changes the documentation on the use of osm2pgsql, and adds a .lua file for preprocessing. This database import aims to be backwards compatible with older versions of the style. This resolves (at least part of) #1504. #Hstore Adding the hstore option to osm2pgsql allows the database to use all keys. This PR proposes using hstore for new keys, and using traditional columns for old keys. This allows us to stay compatible with old versions of the style. According to [@pnorman's benchmarks](http://www.paulnorman.ca/blog/2014/03/osm2pgsql-and-hstore/), using hstore without dropping columns would lead to a 9% size increase and a 0.38% speed decrease. Given the benefits of having all columns available, I would consider this acceptable. # Make polygon/linestring decision based on value This is based on the following unmerged PR to osm2pgsql: osm2pgsql-dev/osm2pgsql#346 It makes the polygon/linestring decision based on the tag value, in addition to the tag key. In particular, highway=services and junction=yes are now treated as polygon, while leisure=track, man_made=embankment, man_made=breakwater, man_made=groyne, natural=cliff, natural=tree_row, historic=citywalls, waterway=derelict_canal, waterway=ditch, waterway=drain, waterway=river, waterway=stream, waterway=wadi, waterway=weir, power=line, and power=minor_line are now treated as linestring. This resolves #1362, resolves #137, resolves #268, and resolves #892. # Rendering order The new .lua file creates a osmcarto_z_order value in the database, which stores the rendering order we use. This will allow us to simplify the road queries. # Recommend -G flag for multipolygons This resolves #59. # Import ele on buildings This resolves #101. # Delete more meta-tags from imports The .lua file drops some more meta-information from imports that is of no need for rendering. This is based on the following unmerged PR to osm2pgsql: * osm2pgsql-dev/osm2pgsql#532

…ua file

…ua_changes

* Convert layer to integer and use it in queries * Use layer instead of z_order in queries where possible

… into lua_changes

…p-carto into lua_changes

Conflicts: openstreetmap-carto.style project.mml

nebulon42 · 2017-04-14T06:20:28Z

Regarding old-style MPs I would be much in favour of that. I'm sure that the cleanup will progress to a point where this is no problem anymore.

matthijsmelissen · 2017-04-14T08:10:18Z

That's great news. I would also be strongly in favour of dropping support for old-style multipolygons.

Remove support for old-style multipolygons

Tag v3.2.0

pnorman · 2017-04-20T21:54:28Z

This is now ready. Before merging we want to

clean out the PR backlog,
do another full planet demo,
release 3.3.0, and
impose a feature freeze for cartographic changes after 3.3.0 until 4.0.0 is released

I'd also like to move religion to hstore, but if that doesn't get done it won't hold up the release.

pnorman · 2017-05-05T08:00:41Z

I squeezed some more disk space out of one of my servers, so it looks like I can do a test over the weekend, then we can proceed.

pnorman · 2017-05-08T22:43:16Z

https://lua.osm-carto.paulnorman.ca is running with the latest code, I'm looking to tag 3.3.0 and freeze in a couple days.

imagico · 2017-05-08T23:22:20Z

Nice. Comparison is somewhat difficult since the style version is different in the standard map at the moment and the low zoom tiles are also very old there right now.

Am i right to assume that this does not yet use libosmium for geometry building (osm2pgsql-dev/osm2pgsql#684)?

matthijsmelissen · 2017-05-08T23:30:49Z

Looks good!

I had no time to analyse this yet, but here is a difference:
https://lua.osm-carto.paulnorman.ca/#18/49.58117/6.12971

pnorman · 2017-05-08T23:46:10Z

Am i right to assume that this does not yet use libosmium for geometry building

No, it's using released osm2pgsql.

pnorman · 2017-05-09T08:03:05Z

I had no time to analyse this yet, but here is a difference:
https://lua.osm-carto.paulnorman.ca/#18/49.58117/6.12971

Object in question is http://www.openstreetmap.org/way/77686091 with leisure=track, so this is an expected difference, and, as it happens, a tagging error.

matthijsmelissen · 2017-05-09T09:05:31Z

Object in question is http://www.openstreetmap.org/way/77686091 with leisure=track, so this is an expected difference, and, as it happens, a tagging error.

We might want to add a rendering rule though. We could render it equal to racetracks for now, so there is at least something.

imagico · 2017-05-09T09:27:02Z

Yes, we should render leisure=track on lines but no, not like racetracks. The latter is for motor sports, the former for running sports, very different things we also render very differently.

Ideally i think leisure=track on lines should be the same as on areas, i.e. line with thin casing. And preferably in a way that combines with areas, i.e. render casings below and fills above areas.

imagico · 2017-05-09T13:41:48Z

Ok, i inspected the map a bit more and it seems to me most of the differences that are not either due to the style version difference or due to data changes are

the vanishing of independently labeled multipolygon components and the resulting duplicate labels which is mostly a very positive effect (see Recommend -G flag for multipolygons #59).
the different interpretation of closed ways with leisure=track as @math1985 already spotted.
the lack of transfer of member way tags to multipolygon relations due to the removal of support for old style MPs. Like for example here:

http://www.openstreetmap.org/way/100827142

This water area tag is now interpreted while previously it was ignored because it was redundant with the relation tag so the islands in the river vanish. I expected this effect to be much more widespread, especially for water areas since in the past it had been common practice to simply add waterway=riverbank to all outer ways of a a river polygon and if a water area tag was added to the relation it was not removed from the ways. But it is less frequent than i thought although still pretty visible in some areas, in particular India and China. Since this particular kind of faulty double tagging is not covered by OSMI and Osmose and this is kind of connected to the elimination of old style multipolygons because it only matters if you ignore old style tagging it might be useful to think about including these in the multipolygon cleanup effort - cc @joto.

pnorman · 2017-05-09T18:46:06Z

I guess the question is, do we want to hold up 3.3.0/4.0.0 to make any of these changes?

imagico · 2017-05-09T20:22:37Z

I don't think so. If anyone wants to add rendering of leisure=track as lines that would be most welcome but it does not really matter much if this happens before or after the release. It is not that anything that has been rendered correctly before is rendered incorrectly with this change and would be rendered exactly right if we add new support for this. We drop support for rendering closed ways with leisure=track and without area=yes as areas and we would in the future like to add support for rendering them as lines.

And the faulty double tagging can be fixed as easily after the change than before - in fact if the change is rolled out the motivation for fixing is probably higher since it would fix a visible error while right now it would be an invisible change.

matthijsmelissen · 2017-05-09T20:38:40Z

I guess the question is, do we want to hold up 3.3.0/4.0.0 to make any of these changes?

As far as I'm concerned, no.

pnorman · 2017-05-09T21:46:28Z

I'll look at tagging 3.3.0 tonight, which will also involve a feature freeze until 4.0.0.

sommerluk · 2017-05-10T14:50:22Z

A big “thanks” for making this working!

matthijsmelissen and others added 30 commits May 17, 2016 01:03

Add highway=rest_area to list of tags treated as polygon

e0aa01d

Merge duplicate code to determine if object is polygon

1ec1cfc

Explicitly make variables in .lua script local

02edb0a

Whitespace in .lua file

e5660e6

Move filtering out objects with 0 tags to beginning of function in .l…

d524487

…ua file

Code simplification

d276c90

Typo

1b82cb6

Small performance optimization

343d1c9

Small code optimization

8d0489e

Small code optimization

d34bebc

Combine boundary relation checks

21d9b80

Move columns that are not used in WHERE statements to osm2pgsql

118c5cf

Merge remote-tracking branch 'pnorman/combine_boundary_checks' into l…

6798824

…ua_changes

Convet layer to integer and user layer instead of z_order

ae3f38e

* Convert layer to integer and use it in queries * Use layer instead of z_order in queries where possible

Clean up and speed up filter_tags_generic

04956b4

Use luadoc strings

d0b1f53

Only handle area=yes

73496d8

Refactor polygon special case lists for simpler code

a43cb7a

Name polygon function isarea

659840c

Use local vars for polygon stuff

10bff06

Search through possible polygon tags

5175195

Refactor lua z_order and roads code

be2eb57

Add testing framework

4499350

Merge branch 'isarea' of https://github.com/pnorman/openstreetmap-carto…

3621bd7

… into lua_changes

Merge branch 'lua_z_order' of https://github.com/pnorman/openstreetma…

dc36874

…p-carto into lua_changes

Use spaces, not tabs

f2030f2

Regenerate .mml

2ec88f3

Merge remote-tracking branch 'math1985/lua_changes' into lua

b5e4dbd

Conflicts: openstreetmap-carto.style project.mml

Use 4sp for Lua indents

7216ac7

pnorman added 2 commits April 16, 2017 13:02

Remove support for old-style multipolygons

cb23299

Merge pull request #2610 from pnorman/lua/no_old_style_mps

799aebf

Remove support for old-style multipolygons

pnorman self-assigned this Apr 17, 2017

pnorman added 3 commits April 17, 2017 01:31

Document length of committment for 3.x and 4.x compatibility

77cc9c4

Document what happens with PRs and 3.x

b6b155b

Merge tag 'v3.2.0' into lua/master

a002f47

Tag v3.2.0

nebulon42 mentioned this pull request Apr 18, 2017

Render disused:railway=rail like railway=disused #2030

Closed

pnorman mentioned this pull request Apr 19, 2017

Drop old-style multipolygons ClearTables/ClearTables#111

Closed

nebulon42 mentioned this pull request Apr 30, 2017

Add rendering for golf=* #661

Closed

pnorman merged commit a002f47 into master May 10, 2017

joto mentioned this pull request May 10, 2017

Ways with same tags as mp relation osmlab/fixing-polygons-in-osm#36

Open

imagico mentioned this pull request May 11, 2017

Rendering leisure=track on lines #2632

Closed

imagico mentioned this pull request Apr 23, 2022

Render polygonal breakwaters as polygons #4537

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to Lua transforms and change database schema #2533

Switch to Lua transforms and change database schema #2533

pnorman commented Jan 3, 2017 •

edited

Loading

nebulon42 commented Apr 14, 2017

matthijsmelissen commented Apr 14, 2017

pnorman commented Apr 20, 2017 •

edited

Loading

pnorman commented May 5, 2017

pnorman commented May 8, 2017 •

edited

Loading

imagico commented May 8, 2017

matthijsmelissen commented May 8, 2017

pnorman commented May 8, 2017

pnorman commented May 9, 2017

matthijsmelissen commented May 9, 2017 •

edited

Loading

imagico commented May 9, 2017

imagico commented May 9, 2017

pnorman commented May 9, 2017

imagico commented May 9, 2017

matthijsmelissen commented May 9, 2017

pnorman commented May 9, 2017

sommerluk commented May 10, 2017

Switch to Lua transforms and change database schema #2533

Switch to Lua transforms and change database schema #2533

Conversation

pnorman commented Jan 3, 2017 • edited Loading

What this does

Lua transforms

Disjoint areas

Tag columns

Multipolygon handling

Area handling

Open issues

Pre-merge changes

Post-merge changes

Preview

nebulon42 commented Apr 14, 2017

matthijsmelissen commented Apr 14, 2017

pnorman commented Apr 20, 2017 • edited Loading

pnorman commented May 5, 2017

pnorman commented May 8, 2017 • edited Loading

imagico commented May 8, 2017

matthijsmelissen commented May 8, 2017

pnorman commented May 8, 2017

pnorman commented May 9, 2017

matthijsmelissen commented May 9, 2017 • edited Loading

imagico commented May 9, 2017

imagico commented May 9, 2017

pnorman commented May 9, 2017

imagico commented May 9, 2017

matthijsmelissen commented May 9, 2017

pnorman commented May 9, 2017

sommerluk commented May 10, 2017

pnorman commented Jan 3, 2017 •

edited

Loading

pnorman commented Apr 20, 2017 •

edited

Loading

pnorman commented May 8, 2017 •

edited

Loading

matthijsmelissen commented May 9, 2017 •

edited

Loading