-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rendrer implementation #1
Comments
next.osm is the new seamark extract that will be compared with world.osm to determine any changes. After the diff is performed next.osm is moved to worl.osm. This renderer is implemented on a Mac Mini - the new model with a 4-core CPU, 32GB memory & 500GB SSD. I run both the renderer & the Overpass server on this computer. Information on setting up an Overpass server can be found here: https://dev.overpass-api.de/ |
Thanks for fast response. In the meantime I found docker container for setup of overpass api (https://github.com/aAXEe/docker-overpass-api/commits/master) and webserver for tile delivery (https://github.com/aAXEe/tiles-docker). Just the render process needs to be pushed into next docker container to get a simple plug and play solution :-) |
Hello malcolm, clone projects and build applications
prepare everything for first call of jsearch.jar
call jsearch.jar
note: the jsearch application seems to produce the expected osm files in tmp directory. Today I tried to use jrender.jar first time. The call of application shows no error but I got no png file in tiles directory. Please have look to my logfile which shows the observed behavior. Here a log from my tests:
|
On 15 May 2019, at 11:23, stevo01 ***@***.***> wrote:
clone projects and build applications
mkdir /workspace
cd /workspace
git clone https://github.com/OpenNauticalChart/renderer.git
cd renderer
cd jsearch
ant
cd /workspace
git clone https://github.com/openstreetmap/josm-plugins.git
cd josm-plugins/seachart/jrender
ant
Do not use this repo. It is two years out-of-date and contains incomplete and broken code. Instead, us the SVN repo (https://svn.openstreetmap.org/applications/editors/josm/plugins/)
prepare everything for first call of jsearch.jar
cd /workspace/renderer/work
mkdir tmp
cp planet-seamarks.osm next.osm
touch world.osm
diff world.osm next.osm | grep id= | grep -v "<tag" > diffs
Do not try to diff the entire planet with an empty file. This will result in a huge “diffs” that will take weeks to process. Instead:
cp next.osm world.osm
nano world.osm
<change just the first node version value>
diff … etc
That will create just a small “diffs” which will be processed very quickly for test purposes.
call jsearch.jar
java -jar ./../jsearch/jsearch.jar ./
note: the jsearch application seems to produce the expected osm files in tmp directory.
After around 1 day it is still working on zoomlevel 9 and command du -sh shows 405M
Tody I tyed to use jrender.jar in parallel, The call of application shows no errror but I got no png file in tiles directory.
Do not run jrender until jsearch is finished. It may open files, but they may not be complete until jsearch closes all its output files.
Good luck!
Malcolm
|
ok. I just followed info from subversion server "NOTICE: Please note that this OpenStreetMap svn repo is deprecated and many items have now moved to repositories under the github.com/OpenStreetMap project.". After a lock into svn history I switched back to svn version. |
ok. In the meantime I made copy of the tmp dir and started render application on this copy (just for test). |
I tried to follow you instruction but the tmp dir is empty after call of jearch application. Yould you have another look into the logfile please?
|
And how can I start the sea mark render process from scratch? |
just for information: I commited initial version of dockercontainer on github today. You can find it here: |
On 16 May 2019, at 15:04, stevo01 ***@***.***> wrote:
I tried to follow you instruction but the tmp dir is empty after call of jearch application. Yould you have another look into the logfile please?
The problem seems to be with diff. You should get:
< <node id="13391742" lat="64.7298413" lon="-13.9856468" version="8" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
<node id="13391742" lat="64.7298413" lon="-13.9856468" version="7" timestamp="2018-11-08T02:32:58Z" changeset="64279059" uid="1772368" user="pizzaiolo">
(That second line should start with ">" & not be quoted text)
I tried the command: 'diff world.osm next.osm | grep id= | grep -v "<tag" > diffs’ on both macOS & debian linux. Both gave the result above.
|
On 16 May 2019, at 15:06, stevo01 ***@***.***> wrote:
And how can i start the sea mark render process from scratch?
The starting point is the existing tile directories. From there, my rendering setup tracks changes. It would not be practical to create this from zero as your would be trying to generate 10 years of rendering history. The tiles currently occupy about 15GB. If you could provide me with the necessary permissions, I could rsync the tiles to your system. (Send me a PM if you want to do this)
Alternatively, simply start from zero and build from the future changes. I have observed that the most densely mapped areas get edited most often, so this would quickly build your tile library for those areas. If a single node is edited, my renderer will re-render an area of 3x3 zoom12 tiles around that node, so the trick of decrementing a node version will force that to happen.
|
Hello Malcolm, Best Regards Steffen |
Hello Malcolm,
note: I used jrenderer from subversion
|
I ran the same & got tiles OK. Please try this: cd seachart/jrender/ re-run the test and please do "ls tiles" at the end. If all goes well you should see this: imac:test mherring$ ls tiles/ |
Thanks for fast feedback. I tested the render app on my linux workstation (outside docker) and it worked like expected. Thats a little bit strange right? I think the test setup allows me to find the problem with docker version now. |
After update of docker container, your applications is working without modifications. The container is now based on debian/stretch instead alphine linux and this solves the trouble with call of renderer.jar without further investigations. I tried to add useful commends to container and shell scripts... |
Hi Malcolm, I observed that jsearch is running in a single thread during the setup of the dockercontaine. I am thinking now about enhanced setup which allows to generate all tiles from stretch in short time I mean days instead of weeks. From my point of view it should be possible to separate the *.osm file into different regions and start the jsearch process for each of this regions in parallel. This allows the usage of multiple processor cores and should speed-up the processing. Best Regards Steffen |
It is certainly possible to run multiple parallel instances of jsearch. Also, by adding bounding box arguments to the OverPassQL string, the world data can be split up into different regions, one for each instance of jsearch. Each instance of next.osm would have to be in a separate directory, together with a null world.osm and each instance of jsearch would be given the path to one of those directories. This may still take a very long time! It would be better to start with a pre-loaded tile directory. My offer of a one-time rsync to your system still stands. |
ok. I plane to perform some benchmarks and check the runtimes.
Thanks for offer. I got access to tiles and world.osm file which are served by open sea map server. I used rsync to sync this files to my workstation and used the docker container for processing a update some days ago. I think it worked like expected. One open point is a "desaster recovery" strategy or in other words: How to setup a server if no backup is available or in the case that we need to render all tiles after update of jrenderer. by the way: I created mbtile files from this extract and deployed them to ftp server for evaluation of opencpn (Version 5.0 supports this feature). |
If you already have the tiles & the world.osm file from OpenSeaMap tileserver, then you do not need to do any special initialization. Just start running with a single instance of jsearch using new data from your OverPass system. |
This is an issue opened by stevo01 on openseseamap github and moved here.
Hallo Malcolm,
I had a closer look into your scripts today and now some questions for realization of a "t1.openseamap.org" clone:
Best Regards
Steffen
The text was updated successfully, but these errors were encountered: