Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rendrer implementation #1

Closed
malcolmh opened this issue Apr 24, 2019 · 20 comments
Closed

Rendrer implementation #1

malcolmh opened this issue Apr 24, 2019 · 20 comments

Comments

@malcolmh
Copy link
Member

This is an issue opened by stevo01 on openseseamap github and moved here.

Hallo Malcolm,
I had a closer look into your scripts today and now some questions for realization of a "t1.openseamap.org" clone:

are the world.osm and next.osm files an osm extract of the whole world?
what are the recommenced hardware requirements (number of CPU cores, RAM and disk space) and whats the estimated runtime for rendering of all tiles?
Getworld seems to produce the next.osm file. Can you give me a hint how to install and configure "./Overpass/osm3s_query" which is used by this script.

Best Regards

Steffen

@malcolmh
Copy link
Member Author

next.osm is the new seamark extract that will be compared with world.osm to determine any changes. After the diff is performed next.osm is moved to worl.osm.

This renderer is implemented on a Mac Mini - the new model with a 4-core CPU, 32GB memory & 500GB SSD. I run both the renderer & the Overpass server on this computer.

Information on setting up an Overpass server can be found here: https://dev.overpass-api.de/

@stevo01
Copy link

stevo01 commented Apr 24, 2019

Thanks for fast response. In the meantime I found docker container for setup of overpass api (https://github.com/aAXEe/docker-overpass-api/commits/master) and webserver for tile delivery (https://github.com/aAXEe/tiles-docker). Just the render process needs to be pushed into next docker container to get a simple plug and play solution :-)

@stevo01
Copy link

stevo01 commented May 15, 2019

Hello malcolm,
I am still working on setup of sea mark render server. Here a list with commands, that I used to setup the environment:

clone projects and build applications

mkdir /workspace
cd /workspace
git clone https://github.com/OpenNauticalChart/renderer.git
cd renderer
cd jsearch
ant

cd /workspace
git clone https://github.com/openstreetmap/josm-plugins.git
cd josm-plugins/seachart/jrender
ant

prepare everything for first call of jsearch.jar

cd /workspace/renderer/work
mkdir tmp
cp planet-seamarks.osm next.osm
touch world.osm
diff world.osm next.osm | grep id= | grep -v "<tag" > diffs

call jsearch.jar

java -jar ./../jsearch/jsearch.jar ./

note: the jsearch application seems to produce the expected osm files in tmp directory.
After around 1 day it is still working on zoomlevel 9 and command du -sh shows 405M
Question: can you give me estimation how much time the processing needs?

Today I tried to use jrender.jar first time. The call of application shows no error but I got no png file in tiles directory. Please have look to my logfile which shows the observed behavior.

Here a log from my tests:

$ cat tmp/98-186-9.osm
<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.6' upload='false' generator='Jrender'>
<bounds minlat='43.45291889' minlon='-111.26953125' maxlat='44.21370991' maxlon='-110.21484375'/>
  <node id="3497124656" lat="43.8921386" lon="-110.6553738" version="2" timestamp="2015-05-24T05:47:20Z" changeset="31411931" uid="128186" user="malcolmh">
    <tag k="seamark:buoy_special_purpose:colour" v="yellow"/>
    <tag k="seamark:buoy_special_purpose:shape" v="conical"/>
    <tag k="seamark:type" v="buoy_special_purpose"/>
  </node>
  <node id="3497124657" lat="43.8925208" lon="-110.6544291" version="2" timestamp="2016-04-09T14:38:48Z" changeset="38430045" uid="128186" user="malcolmh">
    <tag k="seamark:buoy_special_purpose:colour" v="orange;white"/>
    <tag k="seamark:buoy_special_purpose:colour_pattern" v="horizontal"/>
    <tag k="seamark:buoy_special_purpose:shape" v="can"/>
    <tag k="seamark:type" v="buoy_special_purpose"/>
  </node>
</osm>

$ java -jar jrender.jar 
Usage: java -jar jrender.jar <osm source directory> <tile directory> <zoom> <xtile> <ytile>
$ java -jar jrender.jar tmp/ tiles/ 9 98 186

$ ls tiles/ -al
total 8
drwxr-xr-x  2 stevo stevo 4096 May 15 11:51 .
drwxr-xr-x 10 stevo stevo 4096 May 15 11:51 ..

@malcolmh
Copy link
Member Author

malcolmh commented May 16, 2019 via email

@stevo01
Copy link

stevo01 commented May 16, 2019

Do not use this repo. It is two years out-of-date and contains incomplete and broken code. Instead, us the SVN repo (https://svn.openstreetmap.org/applications/editors/josm/plugins/)

ok. I just followed info from subversion server "NOTICE: Please note that this OpenStreetMap svn repo is deprecated and many items have now moved to repositories under the github.com/OpenStreetMap project.". After a lock into svn history I switched back to svn version.

@stevo01
Copy link

stevo01 commented May 16, 2019

Do not run jrender until jsearch is finished. It may open files, but they may not be complete until jsearch closes all its output files.

ok. In the meantime I made copy of the tmp dir and started render application on this copy (just for test).
The Result: I got first png files in tile directory 👍

@stevo01
Copy link

stevo01 commented May 16, 2019

Do not try to diff the entire planet with an empty file. This will result in a huge “diffs” that will take weeks to process. Instead: cp next.osm world.osm nano world.osm diff … etc

I tried to follow you instruction but the tmp dir is empty after call of jearch application. Yould you have another look into the logfile please?

bash-4.4# ls osm -l
total 629228
-rw-r--r--    1 1000     1000     321867132 May 11 14:56 world.osm
-rw-r--r--    1 root     root     322459678 May 16 08:42 xapi.osm
bash-4.4# ls tmp/ -l
total 0
bash-4.4# cp osm/xapi.osm next.osm
bash-4.4# cp next.osm world.osm
bash-4.4# vi world.osm 
note: I changed the version from node with id =13391742 from 8 to 7. This node seems to be a  lighthouse.
bash-4.4# time diff world.osm next.osm | grep id= | grep -v "<tag" > diffs
real	0m10.442s
user	0m9.456s
sys	0m0.972s
bash-4.4# cat diffs 
   <node id="13198991" lat="59.0920040" lon="11.2526896" version="7" timestamp="2017-08-24T06:51:30Z" changeset="51395976" uid="156646" user="archie"/>
   <node id="13198992" lat="59.0963239" lon="11.2512210" version="7" timestamp="2018-03-20T20:28:39Z" changeset="57364854" uid="5853411" user="j1scad"/>
   <node id="13198993" lat="59.0964541" lon="11.2514471" version="8" timestamp="2017-04-03T22:37:04Z" changeset="47426912" uid="586394" user="Kristoffer_hh"/>
-  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="7" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
+  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
bash-4.4# time java -jar jsearch.jar ./

real	0m30.663s
user	1m7.052s
sys	0m1.616s

bash-4.4# ls -l tmp/
total 0

@stevo01
Copy link

stevo01 commented May 16, 2019

Do not try to diff the entire planet with an empty file. This will result in a huge “diffs” that will take weeks to process. Instead: cp next.osm world.osm nano world.osm diff … etc

And how can I start the sea mark render process from scratch?

@stevo01
Copy link

stevo01 commented May 16, 2019

just for information: I commited initial version of dockercontainer on github today. You can find it here:
https://github.com/stevo01/SeaMarkRenderer

@malcolmh
Copy link
Member Author

malcolmh commented May 16, 2019 via email

@malcolmh
Copy link
Member Author

malcolmh commented May 16, 2019 via email

@stevo01
Copy link

stevo01 commented May 17, 2019

On 16 May 2019, at 15:04, stevo01 @.***> wrote: I tried to follow you instruction but the tmp dir is empty after call of jearch application. Yould you have another look into the logfile please?
The problem seems to be with diff. You should get: <

(That second line should start with ">" & not be quoted text) I tried the command: 'diff world.osm next.osm | grep id= | grep -v "<tag" > diffs’ on both macOS & debian linux. Both gave the result above.

Hello Malcolm,
the docker container is based on alpine and is using busybox implementation of diff which seems to generate the output in "unified diff" format instead the expected "normal" format. I added the diffutils to container and now the diff producing data in "normal format". A call of jsearch is producing *.osm files in tmp directory now.

Best Regards

Steffen

stevo01 added a commit to stevo01/SeaMarkRenderer that referenced this issue May 17, 2019
@stevo01
Copy link

stevo01 commented May 17, 2019

Hello Malcolm,
first of all, thanks for spending your time to support me. Unfortunatally my investigation stucks again.
I used the renderer to generate the tiles after the jrender application generates the osm files in temp directory. Unfortunatally I got no tiles. Please have additional look into attached log file and give me feedback.

bash-4.4# cp osm/xapi.osm next.osm
bash-4.4# cp osm/world.osm .
bash-4.4# vi world.osm (note: I changed the version from node with id =13391742 from 8 to 7. This node seems to be a  lighthouse.)
bash-4.4# ls tmp/
bash-4.4# ls tiles/
bash-4.4# cat diffs 
<   <node id="13391742" lat="64.7298413" lon="-13.9856468" version="7" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
>   <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
bash-4.4# jsearch.sh 
+ cd /workspace
+ diff world.osm next.osm
+ grep id=
+ grep -v '<tag'
+ java -jar jsearch.jar ./
+ mv next.osm world.osm

bash-4.4# ls tmp/ -l
total 156
-rw-r--r--    1 root     root          8231 May 17 08:43 1887-1072-12.osm
-rw-r--r--    1 root     root          7622 May 17 08:43 1887-1073-12.osm
-rw-r--r--    1 root     root          7622 May 17 08:43 1887-1074-12.osm
-rw-r--r--    1 root     root          8231 May 17 08:43 1888-1072-12.osm
-rw-r--r--    1 root     root          4136 May 17 08:43 1888-1073-12.osm
-rw-r--r--    1 root     root          4136 May 17 08:43 1888-1074-12.osm
-rw-r--r--    1 root     root         13755 May 17 08:43 1889-1072-12.osm
-rw-r--r--    1 root     root          4136 May 17 08:43 1889-1073-12.osm
-rw-r--r--    1 root     root          4136 May 17 08:43 1889-1074-12.osm
-rw-r--r--    1 root     root         23416 May 17 08:43 236-134-9.osm
-rw-r--r--    1 root     root         23416 May 17 08:43 472-268-10.osm
-rw-r--r--    1 root     root         19573 May 17 08:43 944-536-11.osm

bash-4.4# grep -rnw 'tmp/' -e '13391742'
tmp/1887-1074-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/472-268-10.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1888-1072-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1888-1074-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1887-1073-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/236-134-9.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1889-1072-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1887-1072-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/944-536-11.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1889-1073-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1889-1074-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
tmp/1888-1073-12.osm:4:  <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">

bash-4.4# cat /usr/local/bin/render.sh 
#!/bin/bash

#   tmp/   - input directory, application expect to find osm
#            file in this folder
#   tiles/ - output directory, application stores generated
#            tile files in this folder

# loop over all osm files in directory tmp
for file in $(ls tmp | grep "\.osm"); do
  tx=$(echo $file | cut -f 1 -d'-')
  ty=$(echo $file | cut -f 2 -d'-')
  z=$(echo $file | cut -f 3 -d'-')
  z=$(echo $z | cut -f 1 -d'.')
  echo "$(date) rendering $z $tx $ty" >> log.txt

  # start render application
  java -jar jrender.jar tmp/ tiles/ $z $tx $ty 2>/dev/null

  # remove the processed osm file
  rm tmp/$file
done

echo "$(date) rendering process is done" >> log.txt

bash-4.4# render.sh 
bash-4.4# ls tmp/ -l
total 0
bash-4.4# ls tiles/ -l
total 0
bash-4.4# cat log.txt 
Fri May 17 08:42:44 UTC 2019 New rendering queued
Fri May 17 08:43:37 UTC 2019 rendering ready
Fri May 17 08:56:27 UTC 2019 rendering 12 1887 1072
Fri May 17 08:56:28 UTC 2019 rendering 12 1887 1073
Fri May 17 08:56:28 UTC 2019 rendering 12 1887 1074
Fri May 17 08:56:28 UTC 2019 rendering 12 1888 1072
Fri May 17 08:56:28 UTC 2019 rendering 12 1888 1073
Fri May 17 08:56:28 UTC 2019 rendering 12 1888 1074
Fri May 17 08:56:28 UTC 2019 rendering 12 1889 1072
Fri May 17 08:56:28 UTC 2019 rendering 12 1889 1073
Fri May 17 08:56:28 UTC 2019 rendering 12 1889 1074
Fri May 17 08:56:28 UTC 2019 rendering 9 236 134
Fri May 17 08:56:28 UTC 2019 rendering 10 472 268
Fri May 17 08:56:28 UTC 2019 rendering 11 944 536
Fri May 17 08:56:28 UTC 2019 rendering process is done

bash-4.4# ls -l
total 315076
-rw-r--r--    1 root     root           314 May 17 08:42 diffs
drwxr-xr-x  109 root     root          4096 May 17 05:32 josm-plugins
lrwxrwxrwx    1 root     root            28 May 17 07:07 jrenderer.jar -> seachart/jrender/jrender.jar
lrwxrwxrwx    1 root     root            39 May 17 05:32 jsearch.jar -> /workspace/renderer/jsearch/jsearch.jar
-rw-r--r--    1 root     root           767 May 17 08:56 log.txt
drwxr-xr-x    2 1000     1000          4096 May 17 08:39 osm
drwxr-xr-x    2 1000     1000          4096 May 16 15:22 query
drwxr-xr-x    5 root     root          4096 May 17 05:31 renderer
drwxr-xr-x   16 root     root          4096 May 17 07:06 seachart
drwxr-xr-x    2 root     root          4096 May 17 05:24 tiles
drwxr-xr-x    2 root     root        139264 May 17 08:56 tmp
-rw-r--r--    1 root     root     322459678 May 17 08:40 world.osm

note: I used jrenderer from subversion


bash-4.4# cd seachart/
bash-4.4# svn info
Path: .
Working Copy Root Path: /workspace/seachart
URL: https://svn.openstreetmap.org/applications/editors/josm/plugins/seachart
Relative URL: ^/applications/editors/josm/plugins/seachart
Repository Root: https://svn.openstreetmap.org
Repository UUID: b9d5c4c9-76e1-0310-9c85-f3177eceb1e4
Revision: 35000
Node Kind: directory
Schedule: normal
Last Changed Author: malcolmh
Last Changed Rev: 34993
Last Changed Date: 2019-04-30 20:36:31 +0000 (Tue, 30 Apr 2019)

bash-4.4# svn status

@malcolmh
Copy link
Member Author

I ran the same & got tiles OK. Please try this:

cd seachart/jrender/
ant clean; ant

re-run the test and please do "ls tiles" at the end. If all goes well you should see this:

imac:test mherring$ ls tiles/
14 15 16 17 18
imac:test mherring$ ls tiles/14
7554 7555 7556
imac:test mherring$ ll tiles/14/7555
total 24
-rw-r--r--+ 1 mherring staff 4.6K 17 May 10:37 4292.png
-rw-r--r--+ 1 mherring staff 3.3K 17 May 10:37 4293.png

@stevo01
Copy link

stevo01 commented May 17, 2019

Thanks for fast feedback. I tested the render app on my linux workstation (outside docker) and it worked like expected. Thats a little bit strange right? I think the test setup allows me to find the problem with docker version now.

@stevo01
Copy link

stevo01 commented May 17, 2019

After update of docker container, your applications is working without modifications. The container is now based on debian/stretch instead alphine linux and this solves the trouble with call of renderer.jar without further investigations. I tried to add useful commends to container and shell scripts...

@stevo01
Copy link

stevo01 commented May 23, 2019

On 16 May 2019, at 15:06, stevo01 @.***> wrote: And how can i start the sea mark render process from scratch?
The starting point is the existing tile directories. From there, my rendering setup tracks changes. It would not be practical to create this from zero as your would be trying to generate 10 years of rendering history. The tiles currently occupy about 15GB. If you could provide me with the necessary permissions, I could rsync the tiles to your system. (Send me a PM if you want to do this) Alternatively, simply start from zero and build from the future changes. I have observed that the most densely mapped areas get edited most often, so this would quickly build your tile library for those areas. If a single node is edited, my renderer will re-render an area of 3x3 zoom12 tiles around that node, so the trick of decrementing a node version will force that to happen.

Hi Malcolm,

I observed that jsearch is running in a single thread during the setup of the dockercontaine. I am thinking now about enhanced setup which allows to generate all tiles from stretch in short time I mean days instead of weeks. From my point of view it should be possible to separate the *.osm file into different regions and start the jsearch process for each of this regions in parallel. This allows the usage of multiple processor cores and should speed-up the processing.
Before I start with such type of tests: Is it even possible to use the jsearch application in this way?

Best Regards

Steffen

@malcolmh
Copy link
Member Author

It is certainly possible to run multiple parallel instances of jsearch. Also, by adding bounding box arguments to the OverPassQL string, the world data can be split up into different regions, one for each instance of jsearch. Each instance of next.osm would have to be in a separate directory, together with a null world.osm and each instance of jsearch would be given the path to one of those directories.

This may still take a very long time! It would be better to start with a pre-loaded tile directory. My offer of a one-time rsync to your system still stands.

@stevo01
Copy link

stevo01 commented May 23, 2019

It is certainly possible to run multiple parallel instances of jsearch. Also, by adding bounding box arguments to the OverPassQL string, the world data can be split up into different regions, one for each instance of jsearch. Each instance of next.osm would have to be in a separate directory, together with a null world.osm and each instance of jsearch would be given the path to one of those directories.

ok. I plane to perform some benchmarks and check the runtimes.

This may still take a very long time! It would be better to start with a pre-loaded tile directory. My offer of a one-time rsync to your system still stands.

Thanks for offer. I got access to tiles and world.osm file which are served by open sea map server. I used rsync to sync this files to my workstation and used the docker container for processing a update some days ago. I think it worked like expected. One open point is a "desaster recovery" strategy or in other words: How to setup a server if no backup is available or in the case that we need to render all tiles after update of jrenderer.

by the way: I created mbtile files from this extract and deployed them to ftp server for evaluation of opencpn (Version 5.0 supports this feature).
You can find this files here: https://ftp5.gwdg.de/pub/misc/openstreetmap/openseamap/charts/mbtiles/test/

@malcolmh
Copy link
Member Author

If you already have the tiles & the world.osm file from OpenSeaMap tileserver, then you do not need to do any special initialization. Just start running with a single instance of jsearch using new data from your OverPass system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants