How can I render a tile from a .pbf or .osm file (OSM) ?

[ststeiger]

I need to render the tiles of the OSM map for Switzerland for offline use on demand in IIS.
I'm using the pbf for monaco, because it's a lot smaller, for testing.

I figured simple code to do that with MapsUI looks appx. somthing along the lines of this

            Mapsui.Map map = CreateMap();

            var mr = new Mapsui.Rendering.Skia.MapRenderer();
            
            using (System.IO.MemoryStream ms = mr.RenderToBitmapStream(viewport, map.Layers, map.BackColor))
            {
                byte[] bytes = ms.ToArray();
                System.IO.File.WriteAllBytes("GuessFileFormat.png", bytes);
            }

However, I cannot figure out how to load the contents of monaco-latest.osm.pbf into a MemoryProvider.
There is a sample that produces GeoJSON, and a sample that reads a pbf.

I get a NetTopologySuite.Features.FeatureCollection and I somehow need IEnumerable<Mapsui.Providers.IFeature> for the memory provider.

This is the test-code i have:
https://github.com/ststeiger/OsmTilePrerenderer/blob/master/OsmTilePrerenderer/Program.cs

using OsmSharp;
using OsmSharp.Streams;
using OsmSharp.Geo; // for ToFeatureSource 

using NetTopologySuite.Features;
using NetTopologySuite.Geometries;

using System.Linq;



namespace OsmTilePrerenderer
{


    class Program
    {



        // https://github.com/AliFlux/VectorTileRenderer
        static void Main(string[] args)
        {
            ReadGeometryStream();
            System.Console.WriteLine("Hello World!");


            var viewport = new Mapsui.Viewport
            {
                Center = new Mapsui.Geometries.Point(0, 0),
                Width = 600,
                Height = 400,
                Resolution = 63000
            };


            Mapsui.Map map = CreateMap();

            var mr = new Mapsui.Rendering.Skia.MapRenderer();
            using (System.IO.MemoryStream ms = mr.RenderToBitmapStream(viewport, map.Layers, map.BackColor))
            {
                byte[] bytes = ms.ToArray();
                System.IO.File.WriteAllBytes("GuessFileFormat.png", bytes);
            }


        }



        private static Mapsui.Layers.MemoryLayer CreatePointLayer()
        {
            var random = new System.Random();
            var features = new Mapsui.Providers.Features();
            for (var i = 0; i < 100; i++)
            {
                var feature = new Mapsui.Providers.Feature
                {
                    Geometry = new Mapsui.Geometries.Point(random.Next(100000, 5000000), random.Next(100000, 5000000))
                };
                features.Add(feature);
            }


            var provider = new Mapsui.Providers.MemoryProvider(features);

            return new Mapsui.Layers.MemoryLayer { DataSource = provider };
        }



        // D:\Stefan.Steiger\Documents\Visual Studio 2017\Projects\Mapsui\Tests\Mapsui.Rendering.Xaml.Tests\MapRendererTests.cs

        public static void foo()
        {
            OsmStreamSource source = null;

            var merger = new OsmSharp.Streams.Filters.OsmStreamFilterMerge();
            merger.RegisterSource(source);
        }


        public static Mapsui.Map CreateMap()
        {
            var map = new Mapsui.Map
            {
                BackColor = Mapsui.Styles.Color.Transparent,
                Home = n => n.NavigateTo(new Mapsui.Geometries.Point(0, 0), 63000)
            };

            // OsmStreamSource source = null;
            // OsmSharp.Geo.Streams.IFeatureStreamSource features = null;
            // System.Collections.Generic.List<IFeature> features = null; ;
            FeatureCollection features = new FeatureCollection();

            Mapsui.Providers.IProvider source = null; // new Mapsui.Providers.MemoryProvider(features);

            Mapsui.Layers.ILayer layer = new Mapsui.Layers.MemoryLayer
            {
                Style = null,
                DataSource = source,
                Name = "Line"
            };

            map.Layers.Add(layer);
            return map;
        }


        static void ReadXmlStream()
        {
            string fileName = @"D:\username\Documents\Visual Studio 2017\Projects\OsmTilePrerenderer\OsmTilePrerenderer\Data\monaco-latest.osm.pbf";

            using (System.IO.FileStream fileStream = System.IO.File.OpenRead(fileName))
            {
                var target = new XmlOsmStreamTarget(fileStream);

                // var filtered = target.FilterSpatial(polygon, true);
                // target.RegisterSource(filtered);

                target.Pull();
            }
            
        }

        static void ReadGeometryStream()
        {
            // let's show you what's going on.
            OsmSharp.Logging.Logger.LogAction = (origin, level, message, parameters) =>
            {
                System.Console.WriteLine(string.Format("[{0}] {1} - {2}", origin, level, message));
            };

            // Download.ToFile("http://files.itinero.tech/data/OSM/planet/europe/luxembourg-latest.osm.pbf", "luxembourg-latest.osm.pbf").Wait();

            using (System.IO.FileStream fileStream = System.IO.File.OpenRead(@"D:\username\Documents\Visual Studio 2017\Projects\OsmTilePrerenderer\OsmTilePrerenderer\Data\monaco-latest.osm.pbf"))
            {
                // create source stream.
                OsmStreamSource source = new PBFOsmStreamSource(fileStream);

                // show progress.
                OsmStreamSource progress = source.ShowProgress();

                // filter all powerlines and keep all nodes.
                System.Collections.Generic.IEnumerable<OsmGeo> filtered = 
                    from osmGeo in progress
                    where osmGeo.Type == OsmSharp.OsmGeoType.Node ||
                            (osmGeo.Type == OsmSharp.OsmGeoType.Way && osmGeo.Tags != null && osmGeo.Tags.Contains("power", "line"))
                    select osmGeo;

                // convert to a feature stream.
                // WARNING: nodes that are partof powerlines will be kept in-memory.
                //          it's important to filter only the objects you need **before** 
                //          you convert to a feature stream otherwise all objects will 
                //          be kept in-memory.
                OsmSharp.Geo.Streams.IFeatureStreamSource features = filtered.ToFeatureSource();

                // filter out only linestrings.
                System.Collections.Generic.IEnumerable<IFeature> lineStrings = from feature in features
                                  where feature.Geometry is LineString
                                  select feature;

                // build feature collection.
                FeatureCollection featureCollection = new FeatureCollection();
                foreach (IFeature feature in lineStrings)
                {
                    featureCollection.Add(feature);
                }


                // convert to geojson.
                string json = ToJson(featureCollection);



                // var st = new Mapsui.Providers.MemoryProvider(json);
                System.IO.File.WriteAllText("output.geojson", json);
            }
        }
        

        private static string ToJson(FeatureCollection featureCollection)
        {
            Newtonsoft.Json.JsonSerializer jsonSerializer = NetTopologySuite.IO.GeoJsonSerializer.Create();
            System.IO.TextWriter jsonStream = new System.IO.StringWriter();
            jsonSerializer.Serialize(jsonStream, featureCollection);
            string json = jsonStream.ToInvariantString();
            return json;
        }


    }


}

[charlenni]

You didn‘t say, what you want to do with the data. Draw only the lines? Draw a rendered map?

[ststeiger]

@charlenni: I know that this are only the lines (as i said: along the lines of this).
Yes, the map.
Draw the map each tile at a time.
There's some old code for that in Itinero, but it uses OsmSharp.UI, and that in turn relies on OsmSharp.Osm, which does not work with the current version of OsmSharp.
It seems it has either been forked and diverged, or discarded. Or both.

[xivk]

OsmSharp.UI has been discontinued because building a rendering library for OSM data was a bit too much for the project to handle let alone maintain. I would be better fit for mapsui.

Feel free to reuse anything you can still find there but I wouldn't recommend using it in anything but experiments.

The current OsmSharp just focuses on processing and reading/writing OSM data.

[ststeiger]

@xivk: So it has been discontinued in OsmSharp, but with no replacement ins Mapsui.
Thanks for the info.
Now this is really messed up.
I'd have taken libmapnick in the first place, but there is no support for libmapnick binaries for mapnick 3 or libmapnick 3 on windows.
Looks like there is no other choice but to pre-render that 53GB on Linux and deploy the result to a windows machine in a separate application.

[charlenni]

Ok, than I understand you correct. You want to use vector offline maps and render them on the fly. Mapsui couldn‘t do this. It sounds easy, but it isn‘t. You need first a theme style, which determin, how things are rendered. This is the easy part. The difficult part is to render the labels.

Perhaps you could use MBTiles for offline usage. They can contain tiles as PNG. You could download them from different sites.

[pauldendulk]

@xivk Do you have an idea what should be done to support this in Mapsui? Which libraries te use?

[ststeiger]

@pauldendulk: In all C# rendering projects, I either see SkiaSharp or WPF.
For pure .NET, you could use SixLabors.ImageSharp.
SixLabors also has font-rendering capabilities, though I don't know how well they support right-to-left fonts, etc.
It would probably work for western europe & the americas.
But even if it worked, whether it would be fast enough is the next question.

[xivk]

In OsmSharp.UI I built everything myself on top of whatever rendering capability was there on Android/iOS. I think when using SkiaSharp it is doable to get it fast enough.

There is also the issue of data management, how to manage the offline map data, but I guess nowadays that can be solved with vector tiles. On top of that there is the styling language, a good idea could be to just reuse something that already exists, this has the added advantage that users can just reuse existing styles or use other tools to design the style.

Building something like this can be a lot of work and then maintaining it is also very difficult. There are so many things to consider when rendering a base map from vector data.

[charlenni]

@ststeiger I assume, that Paul talks about libraries to draw vector map data, not a library to use for drawing. There Mapsui uses Skia, one of the fastes library, which could be used on many different platforms (Windows, Android, iOS, Mac and so on).

@xivk I made such a thing for MapboxGL style files and Mapbox vector data (see here). I transfered them from the native vector data to Mapsui features with style from the MapboxGL style file. This features than are added to a normal layer. Got it up to 3 fps. Problem of MapboxGL style is, that the style could change with the zoom factor. Not only with integers, but with float zoom level too. So much time is wasted with getting the right style for the feature (checking for the filters with tags from feature). And I stopped, when I get to the labels. As I mentioned, this isn't an easy job.

Now I do a redesign of this part. I think, the Mapsui feature approach isn't fast enough. So I use the SKPicture from Skia and create the tile recording the drawing commands. There are some advantages:

• Like Mapsui features, SKPicture is a vector image. So it draws always with very good quality. But it could be transfered to a PNG or JPG very easy.
• Coordinates from the vector data (here integers between 0 and 4095) could be used directly. So no translation into Mapsui coordinates and, when drawing, back again takes place.
• Filters are only calculated once when the feature is loaded.
• Constant values for paintig (color, stroke width and so on) are precalculated when loading the tile.
• Zoom dependent values for painting are calculated in the moment, when the picture is drawn to a canvas. This could be made, because Skia knows the SKDrawable object, which determins its style in the moment of drawing.

It should be designed in a way, that it could not only handle MapboxGL styles and Mapbox vector data, but others too, like Mapsforge and its style file format.

For the labels there are some articles in literature. There is even a implementation in C#: MapSurfer.NET (see here), but it isn't open source.

So, it should be possible to visualize vector map data in C#, but there isn't any library up to now.

[xivk]

I remember in OsmSharp.UI that I calculated everything that had to with the styling once per 'feature' and the styling was not done every frame but only once when the data was loaded or zoom level changed and crossed a boundary where the style also changes.

[xivk]

Also in OsmSharp.UI I was alone in working on most of this, mapsui seems to have a good core of contributors, perhaps it will be easier to maintain things.

[charlenni]

I remember in OsmSharp.UI that I calculated everything that had to with the styling once per 'feature' and the styling was not done every frame but only once when the data was loaded or zoom level changed and crossed a boundary where the style also changes.

Yes, thats the correct way. But be carefull, sometimes one feature has different styles. So you have to add the feature twice or more, because the order, in which the features with special styles are drawn, is important.

Also in OsmSharp.UI I was alone in working on most of this, mapsui seems to have a good core of contributors, perhaps it will be easier to maintain things.

No, I assume not. Most come and want a special function or want to mention, that their app don't work as expected. Than they want help. Even if they have full access to the complete source code. I assume, that this is the case in most small projects as OsmSharp or Mapsui.

[ststeiger]

@xivk: For offline storage, either Firebird or SQLite. On the web, there is WebSQL or IndexedDB.

[pauldendulk]

Yes, I would like to know what you would use to read the .pbf or .osm. Translating that to Mapsui.Geometries with a default style might be relatively easy. Getting proper styles and good performance will be hard.

[ststeiger]

@pauldendulk: OsmSharp and OsmSharp.IO.Binary are pretty much the only thing in C# for that.
BruTile for MbTiles.
There are examples in how to do that in the source code.

I don't think parsing CSS, and putting that style-info in a struct/class and using that during rendering would be that much of a problem.
I think the hardes thing would be the labels.
Right language for the label (multilanguage support), getting the right data, right size, right background-color for a certain foreground color, readable, color-blindness considerations, left-to-right/right-to-left/rtl-downside-up, rotate270, rotateX, edge-cases, esotheric fonts, features, unicode etc.

Microsoft's GDI renderer would certainly be an epic fail in the readability department when rotating fonts by x old-degrees.

[xivk]

Translating OSM data into renderable data is non-trivial when doing for example country boundaries or coastlines but for most data using OsmSharp should be sufficient. OsmSharp.IO.Binary is a custom format osm.bin, reading/writing osm.pbf is supported in OsmSharp itself.

The osm.bin format supports read/writing incomplete OSM objects without changeset, user information or with negative IDs, it's not recommended for general use, just use osm.pbf.

[pauldendulk]

@ststeiger How do you need BruTile if you translate to vector? Store it as vector tiles?

[ststeiger]

@xivk: Yes, I know - i understand some data is also scattered in shapefiles there for some reason that i don't know...

@pauldendulk: Don't need BruTile for rendering/translating vector data. Just to read the MbTiles file format if you don't want to render vector data and instead use a pre-rendered tile. To get the right tile for lat/long or X/Y/Z.

Never mind, I think I get the hint with vector tiles now.
It's best to generate an mbtiles file with vector tile data, or for non-commercial use take the one from openmaptiles.
Then use leaflet with mapbox-gl-leaflet plugin.
That way, I don't have to render anything in .NET, and can get the file size down to appx. 400 MB.
Much better than 53 GB.
Bonus points since then, I can use dynamic/custom styles.
/Outsourced to JavaScript and WebGL - no .NET needed for rendering.
Now I just need to write a server component to serve vector tile data.

[pauldendulk]

I am glad you found a solution.

If we ever what to render sm.pbf's we should not render then directly to canvas but instead translate them to vectors and styling and render it in the regular way. There may be performance issues with a such a solution but we should fix those issues to make Mapsui more performant in general.

[ststeiger]

@pauldendulk:
May I propose - render them to svg (or XAML) - this is vectors.
Then you can just add a css stylesheet - and can put whatever the hell you want into it - problem with styling solved as well.
If you need raster graphics, then it can be rendered with any program/library capable of rendering svgs, such as libwkhtml2X (formerly wkhtml2image).
Alternatively, with a WPF/WinForms/iOS/Android svg-display-control, or an embedded svg-capable-web-browser-control.

[pauldendulk]

Mapsui uses SkiaSharp which has itself support to render svg. However, it will still need to translate the svg to native skia features. So for performance we will probably keep a cached skia native feature, like SKPicture.