Skip to content

Commit

Permalink
Add support for data generalization
Browse files Browse the repository at this point in the history
This large commit adds a framework to generalize OSM data using
different strategies and implements several strategies. The source is
all in the new `src/gen` directory. It includes a new command
`osm2pgsql-gen` to access this functionality. Future versions might
integrate it with the `osm2pgsql` command, but because this is all
experimental it is kept separate for now. Call `osm2pgsql-gen` with
`-h` to get usage help.

See also https://osm2pgsql.org/generalization/ and the chapter on
Generalization in the manual:
https://osm2pgsql.org/doc/manual.html#generalization

For the raster support this adds two new library dependency: CImg and
potrace. Both have been around for a long time and are readily available
on all systems.

The following strategies work on a tile-by-tile basis and operate on
polygons:

The "vector-union" strategy buffers and unionizes polygons using vector
operations.

The "raster-union" strategy does a similar thing but does it in raster
space which is much faster. First the polygons are rendered into a
raster, an open/close operation is called (which basically does the same
thing as the buffering in vector space) and finally the resulting raster
is vectorized again.

The "builtup" strategy is intended to derive a layer of builtup areas
from landuse=residential/industrial etc. as well as building cover and
dense road networks.

The following strategies always work on all data (not tile-based):

The "discrete-isolation" strategy rates places based on some importance
metric to get a more even distribution of places on the map.

The new "rivers" strategy finds important rivers, this is still very
much work in progress.
  • Loading branch information
joto committed Mar 7, 2023
1 parent 344e9d8 commit 899c56a
Show file tree
Hide file tree
Showing 34 changed files with 3,361 additions and 5 deletions.
2 changes: 2 additions & 0 deletions .github/actions/ubuntu-prerequisites/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,12 @@ runs:
- name: Install software
run: |
sudo apt-get install -yq --no-install-suggests --no-install-recommends \
cimg-dev \
libboost-filesystem-dev \
libboost-system-dev \
libbz2-dev \
libexpat1-dev \
libpotrace-dev \
libpq-dev \
libproj-dev \
pandoc \
Expand Down
3 changes: 1 addition & 2 deletions .github/actions/win-install/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@ runs:

steps:
- name: Install packages
run: vcpkg install bzip2:x64-windows expat:x64-windows zlib:x64-windows proj4:x64-windows boost-geometry:x64-windows boost-system:x64-windows boost-filesystem:x64-windows boost-property-tree:x64-windows lua:x64-windows libpq:x64-windows
run: vcpkg install cimg:x64-windows bzip2:x64-windows expat:x64-windows zlib:x64-windows proj4:x64-windows boost-geometry:x64-windows boost-system:x64-windows boost-filesystem:x64-windows boost-property-tree:x64-windows lua:x64-windows libpq:x64-windows
shell: bash

- name: Install psycopg2 and beahve
run: python -m pip install psycopg2 behave
shell: bash
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:

- name: Install prerequisites
run: |
brew install lua boost postgis pandoc
brew install lua boost postgis pandoc cimg potrace
pip3 install psycopg2 behave
pg_ctl -D /usr/local/var/postgres init
pg_ctl -D /usr/local/var/postgres start
Expand Down Expand Up @@ -45,6 +45,7 @@ jobs:
env:
CC: gcc-10
CXX: g++-10
EXTRA_FLAGS: -Wno-unused-but-set-parameter # workaround for GCC bug
LUA_VERSION: 5.3
LUAJIT_OPTION: ON
POSTGRESQL_VERSION: 9.6
Expand Down Expand Up @@ -79,6 +80,7 @@ jobs:
env:
CC: gcc-10
CXX: g++-10
EXTRA_FLAGS: -Wno-unused-but-set-parameter # workaround for GCC bug
LUA_VERSION: 5.3
LUAJIT_OPTION: OFF
POSTGRESQL_VERSION: 10
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/test-install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,14 @@ jobs:
run: |
sudo apt-get purge -yq postgresql*
sudo apt-get install -yq --no-install-suggests --no-install-recommends \
cimg-dev \
libboost-filesystem-dev \
libboost-system-dev \
libbz2-dev \
libexpat1-dev \
liblua${LUA_VERSION}-dev \
libluajit-5.1-dev \
libpotrace-dev \
libpq-dev \
libproj-dev \
lua${LUA_VERSION} \
Expand Down
34 changes: 33 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,11 @@ include_directories(SYSTEM ${PostgreSQL_INCLUDE_DIRS})

find_package(Threads)

find_path(POTRACE_INCLUDE_DIR potracelib.h)
find_library(POTRACE_LIBRARY NAMES potrace)

find_path(CIMG_INCLUDE_DIR CImg.h)

############### Libraries are found now ########################

set(LIBS ${Boost_LIBRARIES} ${PostgreSQL_LIBRARY} ${OSMIUM_LIBRARIES})
Expand Down Expand Up @@ -276,6 +281,33 @@ add_subdirectory(src)
add_executable(osm2pgsql src/osm2pgsql.cpp)
target_link_libraries(osm2pgsql osm2pgsql_lib ${LIBS})

if (${POTRACE_LIBRARY} STREQUAL "POTRACE_LIBRARY-NOTFOUND" OR ${CIMG_INCLUDE_DIR} STREQUAL "CIMG_INCLUDE_DIR-NOTFOUND")
message(STATUS "Did not find cimg and/or potrace library. Not building osm2pgsql-gen.")
else()
if (WITH_LUA)
message(STATUS "Found cimg and potrace library. Building osm2pgsql-gen.")
set(BUILD_GEN 1)
include_directories(SYSTEM ${CIMG_INCLUDE_DIR})
include_directories(SYSTEM ${POTRACE_INCLUDE_DIR})
add_executable(osm2pgsql-gen src/gen/osm2pgsql-gen.cpp
src/gen/canvas.cpp
src/gen/gen-base.cpp
src/gen/gen-create.cpp
src/gen/gen-discrete-isolation.cpp
src/gen/gen-rivers.cpp
src/gen/gen-tile-builtup.cpp
src/gen/gen-tile-raster.cpp
src/gen/gen-tile-vector.cpp
src/gen/gen-tile.cpp
src/gen/params.cpp
src/gen/raster.cpp
src/gen/tracer.cpp)
target_link_libraries(osm2pgsql-gen osm2pgsql_lib ${LIBS} ${POTRACE_LIBRARY})
else()
message(STATUS "No Lua. Not building osm2pgsql-gen.")
endif()
endif()

#############################################################
# Optional "clang-tidy" target
#############################################################
Expand All @@ -287,7 +319,7 @@ find_program(CLANG_TIDY
if (CLANG_TIDY)
message(STATUS "Looking for clang-tidy - found ${CLANG_TIDY}")

file(GLOB CT_CHECK_FILES src/*.cpp tests/*cpp)
file(GLOB CT_CHECK_FILES src/*.cpp src/*/*.cpp tests/*cpp)

add_custom_target(clang-tidy
${CLANG_TIDY}
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ Required libraries are
* [zlib](https://www.zlib.net/)
* [Boost libraries](https://www.boost.org/), including geometry, system and
filesystem
* [CImg](https://cimg.eu/) (Optional, see README-gen.md)
* [potrace](https://potrace.sourceforge.net/) (Optional, see README-gen.md)
* [PostgreSQL](https://www.postgresql.org/) client libraries
* [Lua](https://www.lua.org/) (Optional, used for Lua tag transforms
and the flex output)
Expand Down Expand Up @@ -80,14 +82,15 @@ On a Debian or Ubuntu system, this can be done with:

```sh
sudo apt-get install make cmake g++ libboost-dev libboost-system-dev \
libboost-filesystem-dev libexpat1-dev zlib1g-dev \
libboost-filesystem-dev libexpat1-dev zlib1g-dev libpotrace-dev cimg-dev \
libbz2-dev libpq-dev libproj-dev lua5.3 liblua5.3-dev pandoc
```

On a Fedora system, use

```sh
sudo dnf install cmake make gcc-c++ boost-devel expat-devel zlib-devel \
potrace-devel cimg-devel \
bzip2-devel postgresql-devel proj-devel proj-epsg lua-devel pandoc
```

Expand Down
11 changes: 11 additions & 0 deletions docs/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,17 @@ if(PANDOC)
VERBATIM)

list(APPEND MANPAGE_TARGETS osm2pgsql.1)

if(BUILD_GEN)
add_custom_command(OUTPUT osm2pgsql-gen.1
COMMAND ${PANDOC} ${PANDOC_MAN_OPTIONS} -o osm2pgsql-gen.1
${CMAKE_CURRENT_SOURCE_DIR}/osm2pgsql-gen.md
DEPENDS osm2pgsql-gen.md manpage.template
COMMENT "Building manpage osm2pgsql-gen.1"
VERBATIM)
list(APPEND MANPAGE_TARGETS osm2pgsql-gen.1)
endif()

else()
message(STATUS "Looking for pandoc - not found")
message(STATUS " osm2pgsql manual page can not be built")
Expand Down
87 changes: 87 additions & 0 deletions docs/osm2pgsql-gen.1
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
.TH "OSM2PGSQL" "1" "1.8.1" "" ""
.SH NAME
.PP
osm2pgsql-gen - Generalize OpenStreetMap data - EXPERIMENTAL!
.SH SYNOPSIS
.PP
\f[B]osm2pgsql-gen\f[R] [\f[I]OPTIONS\f[R]]\&...
.SH DESCRIPTION
.PP
THIS PROGRAM IS EXPERIMENTAL AND MIGHT CHANGE WITHOUT NOTICE!
.PP
\f[B]osm2pgsql-gen\f[R] reads data imported by \f[B]osm2pgsql\f[R] from
the database, performs various generalization steps specified by a Lua
config file and writes the data back to the database.
It is used in conjunction with and after \f[B]osm2pgsql\f[R] and reads
the same config file.
.PP
This man page can only cover some of the basics and describe the command
line options.
See the Generalization chapter in the osm2pgsql
Manual (https://osm2pgsql.org/doc/manual.html#generalization) for more
information.
.SH OPTIONS
.PP
This program follows the usual GNU command line syntax, with long
options starting with two dashes (\f[C]--\f[R]).
Mandatory arguments to long options are mandatory for short options too.
.SH MAIN OPTIONS
.TP
-a, --append
Run in append mode.
.TP
-c, --create
Run in create mode.
This is the default if \f[B]-a, --append\f[R] is not specified.
.TP
-S, --style=FILE
The Lua config file.
Same as for \f[B]osm2pgsql\f[R].
.TP
-j, -jobs=NUM
Specifies the number of parallel threads used for certain operations.
Setting this to the number of available CPU cores is a reasonable
starting point.
.SH HELP/VERSION OPTIONS
.TP
-h, --help
Print help.
.TP
-V, --version
Print osm2pgsql version.
.SH LOGGING OPTIONS
.TP
--log-level=LEVEL
Set log level (`debug', `info' (default), `warn', or `error').
.TP
--log-sql
Enable logging of SQL commands for debugging.
.SH DATABASE OPTIONS
.TP
-d, --database=NAME
The name of the PostgreSQL database to connect to.
If this parameter contains an \f[C]=\f[R] sign or starts with a valid
URI prefix (\f[C]postgresql://\f[R] or \f[C]postgres://\f[R]), it is
treated as a conninfo string.
See the PostgreSQL manual for details.
.TP
-U, --username=NAME
Postgresql user name.
.TP
-W, --password
Force password prompt.
.TP
-H, --host=HOSTNAME
Database server hostname or unix domain socket location.
.TP
-P, --port=PORT
Database server port.
.SH SEE ALSO
.IP \[bu] 2
osm2pgsql website (https://osm2pgsql.org)
.IP \[bu] 2
osm2pgsql manual (https://osm2pgsql.org/doc/manual.html)
.IP \[bu] 2
\f[B]postgres\f[R](1)
.IP \[bu] 2
\f[B]osm2pgsql\f[R](1)
88 changes: 88 additions & 0 deletions docs/osm2pgsql-gen.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# NAME

osm2pgsql-gen - Generalize OpenStreetMap data - EXPERIMENTAL!

# SYNOPSIS

**osm2pgsql-gen** \[*OPTIONS*\]...

# DESCRIPTION

THIS PROGRAM IS EXPERIMENTAL AND MIGHT CHANGE WITHOUT NOTICE!

**osm2pgsql-gen** reads data imported by **osm2pgsql** from the database,
performs various generalization steps specified by a Lua config file and
writes the data back to the database. It is used in conjunction with and
after **osm2pgsql** and reads the same config file.

This man page can only cover some of the basics and describe the command line
options. See the [Generalization chapter in the osm2pgsql
Manual](https://osm2pgsql.org/doc/manual.html#generalization) for more
information.

# OPTIONS

This program follows the usual GNU command line syntax, with long options
starting with two dashes (`--`). Mandatory arguments to long options are
mandatory for short options too.

# MAIN OPTIONS

-a, \--append
: Run in append mode.

-c, \--create
: Run in create mode. This is the default if **-a, \--append** is not
specified.

-S, \--style=FILE
: The Lua config file. Same as for **osm2pgsql**.

-j, \-jobs=NUM
: Specifies the number of parallel threads used for certain operations.
Setting this to the number of available CPU cores is a reasonable starting
point.

# HELP/VERSION OPTIONS

-h, \--help
: Print help.

-V, \--version
: Print osm2pgsql version.

# LOGGING OPTIONS

\--log-level=LEVEL
: Set log level ('debug', 'info' (default), 'warn', or 'error').

\--log-sql
: Enable logging of SQL commands for debugging.

# DATABASE OPTIONS

-d, \--database=NAME
: The name of the PostgreSQL database to connect to. If this parameter
contains an `=` sign or starts with a valid URI prefix (`postgresql://` or
`postgres://`), it is treated as a conninfo string. See the PostgreSQL
manual for details.

-U, \--username=NAME
: Postgresql user name.

-W, \--password
: Force password prompt.

-H, \--host=HOSTNAME
: Database server hostname or unix domain socket location.

-P, \--port=PORT
: Database server port.

# SEE ALSO

* [osm2pgsql website](https://osm2pgsql.org)
* [osm2pgsql manual](https://osm2pgsql.org/doc/manual.html)
* **postgres**(1)
* **osm2pgsql**(1)

Loading

0 comments on commit 899c56a

Please sign in to comment.