-
Notifications
You must be signed in to change notification settings - Fork 819
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Roadmap / strategic discussion #1975
Comments
Some more questinons:
To start off the discussion, some potential points of attention (some of which might overlap):
|
I'm going to split my comments up, since they'll be on a few different issues and I don't want to lose partial posts (again). Compilation performance#1947 and #1944 helped here, but we're still about 10 seconds. We used to be under 2.5 and were still one of the slower stylesheets to compile. I recently compiled osm-carto, Stamen Toner, CartoDB basemaps, Humanitarian, OSM Bright, and the unnamed Wikimedia OSM Bright 2 derived style. Nothing was close to us in compilation time, even with the Humanitarian bug that prevented compilation on carto 0.15.0 (hotosm/HDM-CartoCSS#277). This graph is a based on a blog post but updated to include current changes and expanded where we were a long time between releases, using This increases the delay between saving a style change and seeing its effects, and at high zooms working with an extract the style recompilation is the biggest part of the delay. One of the big advantages to switching to a Tilemill/Kosmtik workflow is you could make changes and see them right away. We've lost that, and at 15-20 seconds we were actually slower than saving a change to Mapnik XML, restarting renderd, and hitting reload in the browser. |
Rendering performanceNo one has done any rendering performance profiling in over a year. I don't know if we're IO bound, PostgreSQL CPU bound, or Mapnik CPU bound. I don't know which layers have slow SQL statements, and there are even tools for analyzing this. I have no idea if we're slower than two years ago, given the same OSM data, and the same extends to a per-layer level. |
From a design perspective i think the following broader issues are the most important:
More specific issues of importance are IMO:
|
We need to consider this very carefully. I like the idea that the database is rendering-agnostic: you could in principle use one rendering database for different stylesheets. How well this works in practice, and if anybody uses this in production, I'm not sure about.
+1 |
I'm happy to see the routine improvements to colours, icons etc continue - they are the main thing we need to do. I still think there's a few big areas of work there, around changing some more colours to fit in the points/lines/polygons hierarchy (for example, sports centre polygons are more saturated than most POIs, and almost all lines). My main zoom levels of concern are 8-12. They are high enough to contain significant numbers of distinct features (unlike e.g. z4) and low enough that they are often indistinct in what they are showing (and use tiny text too). They are also low enough that, for me at least, they are a complete pain to design since re-rendering them takes forever on my laptop. My approach to these zooms is often to look at them and think "is this the best map possible of this area? If not, what would it look like if it was better?" In saying this, my personal goals for the next 12 months are around technology, not really cartography:
We've started with the first two of these, but they need to be completed. The third is more open for discussion and has impacts beyond the stylesheet, but I believe it's the right thing to aim for. |
This is the basis of my work on the ArcGIS Renderer during the past two years: the ability to arbitrarily swap styles, only adding new keys to an existing database when needed to fulfill specific render rules requiring certain keys not yet available. Swapping entire styles and viewing the latest results isn't a one-second operation though, as it requires to reload layer files and, as said, potentially to extract keys from ESRI's implementation of an "hstore-like" key-value storage, which can be a rather lengthy operation (minutes to potentially hours or days depending on the size of the OSM extract and the number of keys to add). It is not "dynamic" like hstore is: you can't run a SQL statement against the key-value storage in ESRI's implementation, you need to extract the key and add a new field to the table containing the "hstore-like" column (1). I have fully automated this key extraction process though, by writing a routine that evaluates SQL statements of render rules, and automatically adds the required keys. This works like a charm, albeit limited to basic "WHERE" clause style statements like "X IN ('1','2'), as this is a limitation of standard ArcGIS layer file's Definition Queries (there are Query Layers in ArcGIS that don't have this limitation and can contain arbitrary SQL, but I don't use them, and they have other limitations, like not being editable in ArcGIS). That said, simply changing basic symbology, labelling, or a SQL Definition Query of an ArcGIS layer file of already "rendered" data, can be done instantaneously in ArcGIS if it doesn't require new keys (and with rendering in my concept I don't mean creating raster tiles but creating ArcGIS vector layers and the underlying needed database structure to support them, from this basis, you can export to a number of useful formats, e.g. Adobe PDF or TIFF, or setup a webservice). (1) *** EDIT *** : Although this may be considered, or is, a limitation compared to hstore, there is also a positive side: SQL statements can remain much more "standard", due to operating against ordinary database fields, and thus cross-database compatible. Since ArcGIS supports multiple databases, this is desirable feature. |
HStoreThere's nothing published about moving a style to hstore and making schema choices based on data. We know the german .style file reduces performance, and that using hstore should increase performance in the right conditions. Because I feel the main pressure for hstore is just to increase the number of features we render, I have not had it at the top of my priority list. |
Andy does, see comments about However, using a non-render specific database and pre-processed data are orthogonal issues. Right now we use a non-specific database, but use preprocessed data in the form of shapefiles for coastlines and Antarctica ice. pre-processed data has two problems
Right now most of my time is going into a database schema which is very much not osm2pgsql with C tag transforms |
It's much more than this to me: it also means better granularity, which helps making rendering more sane, as not all objects are equal regarding to rendering. Two examples: we could show domestic and private airports later than international, once we have the scheme for it (#1143; using way_area for it would be just intermediate solution) and make the memorial plaques less obtrusive than currently by pushing them later and rendering with smaller and more accurate icon (#1605 (comment); in my city there are so many of them, that I'm reluctant to map them until I'm sure they won't eclipse too many things). |
Agree. |
RoadsI can see some tweaks coming to road styles, but I'd like like to see improvements to the technical side. We have 3k lines MSS and 7 layers: tunnels, casing, fill, bridges, ref, roads name, path name + low zoom stuff. Four of these layers are giant 120 line 5.5kb queries, so there's a lot of complexity there too. Mapbox Streets has 700 lines and 4 layers, the new WM style has two layers and 150 lines. I think the WM style gets layering, bridges, and tunnels horribly wrong, but I haven't seen problems with MB Streets. Are we handling complicated cases they are ignoring? Are we doing something wrong? Are their vector tiles limiting what kind of street styling people do? Maybe there's some cascading that we could do to cut back on the MSS. The WM style has an interesting comment // Roads have a "is" field that specifies what it is: 'road', 'bridge',
// or 'tunnel'. Each road segment will only belong to one of them. The
// 'bridge' sublayer makes use of Mapnik's group-by rendering mode;
// attachments in this layer will be grouped by layer for appropriate
// rendering of multi-level overpasses. I'm not sure that I trust that, since it doesn't seem to be accurately reflecting what's going on, but could we do this, reducing the number of layers? omniscale/magnacarto#7 (comment) for an explanation of group-by. |
// Roads have a "is" field that specifies what it is: 'road', 'bridge', Where is that "is" field coming from? It isn't a regular key, like "bridge" or "tunnel", so must be some pre-processed field.
Well, whether you trust it, or whether it is an accurate description of the process, the // "Each road segment will only belong to one of them. statement isn't always right in reality. There are rare but real road segments tagged as both bridge and tunnel, and they aren't necessarily wrong, a famous example is the https://en.wikipedia.org/wiki/Conwy_Railway_Bridge near Conwy, see also this comment by me in another issue: Render covered=yes on bridges too if they are under a building Of course, whether these rare cases need rendering support is another matter..., and @math1985 closed the old issue due to its rarity, so that question is probably actually answered. |
I suspect our code handles various layering issues that Mapbox Streets are ignoring. The most problematic of these, I suspect, are highway areas, like pedestrian areas. We render highway area fill in between casing and fill of layer 0-highways. As bridges are rendered above layer-0 fills and tunnels are rendered below layer 0-casings, we cannot really merge bridges and tunnels without breaking highway area rendering. Is there somewhere a demo of Mapbox Streets on the web? I'm interested to see how they are handling highway areas. |
|
Our road style styling is quite complex - from quick looking at https://www.mapbox.com/mapbox.js/example/v1.0.0/ - they have one colour (white) for all roads, footways and cycleways. Some highway values are displayed in the same style (or nearly the same) - for example there is no distinction between cycleway and footway, between highway=pedestrian and highway=construction. All displayed railway=* values seem to be rendered in the same style, the only difference is zoom level where features appear. It is may not be the biggest reason for this difference in code length but it certainly contributes. |
I've been trying to decide how handle bridge+tunnel objects in another project, and came to the conclusion I shouldn't.
I see that MBS vector tiles have lines, polygons and points in the in the roads layer. Would this help? cc @ajashton, in case he has any insights into how MBS is handling this |
Yes, it would make things a lot easier. It would make the query more complicated, but it would allow to reduce the number of queries (and thus the amount of duplication). |
2015-11-29 21:09 GMT+01:00 Paul Norman [email protected]:
+1, I'd give priority to bridge. Tunnel objects are really interesting in |
I wasn't suggesting to give them special priority, just to show there are "those exceptions...", which is always good to know. Actually, the current rendering for the Conwy Railway Bridge, isn't bad, the labelling though must take care of the "covered" part of the story: |
Some more comments from my side:
Of course, also non-maintainers are more than welcome to contribute to this discussion. |
I agree. E.g. some complexity of the SQL could be reduced by using a custom database schema. I also think that not having hstore available holds us back from making proper cartographic decisions, which I see as major problem. Unfortunately there is nothing I could contribute to the transition to hstore currently. |
Any examples how hstore could help us (other than more granular objects)? |
Examples are rendering communication masts only, improvements for airports, replacing deprecated tags with new ones (ford, emergency phone), render very small memorials on z19, and more. Regarding more granular objects hstore does not mean that each category has to be distinguished by e.g. an icon, it could also be used to drop rendering for some sub-category. |
I wouldn't like to add a lot of additional (and duplicated) code to this project to support this. I think if we want to do this, we'd need support for this in either Mapnik or Carto, or perhaps in an additional preprocessor. The same holds for latitude-based size decisions. |
In the spirit of solving problems at the right place this might be relevant here too: mapbox/carto#424. |
Regarding
in combination with what has been said in mapbox/carto#424 (comment):
It seems this creates a kind of problem. The tools employed here (mainly Carto, Mapnik, Osm2pgsql and PostGIS) are not necessarily used exactly for the purpose they are primarily intended for and even if they originally were this might have changed over time like in case of CartoCSS. So it cannot generally be expected for these projects to adapt to the requirements of this style. Of course these are all open source tools that can be adapted to individual needs by anyone but i for example would have no interest in working on Carto to add new language features or to add support for new Mapnik features since it is IMO a dead end because:
Beyond that it is imperative for a high quality style that people can contribute with improvements without having to delve into the inner workings of the programs used for making any non-trivial changes. The fairly dynamic development this style has experienced in the last years is largely because it was fairly easy to productively contribute in a lot of fields. If this was my own project i would probably start looking into re-evaluating the choice of tools used which would probably make sense to do in connection with considerations for vector tiles. This will however likely create two possibilities both not really attractive:
|
Except closed and created swapped. |
Fixed now. |
This is an interesting statistic but very difficult to read without knowing the reason issues are closed. The distribution of reasons why issues are closed has shifted significantly over time and the patterns in that have a strong influence back on reporting activities. This may not always be obvious to long time contributors and maintainers here but casual contributors typically react strongly to how issues they report are dealt with. So overall neither the dealing of developers/maintainers with open issues nor the reporting patterns are constant over the time looked at. What would be rather interesting to look at in terms of development efficiency is how changes made fare in terms of problems solved vs. problems created. This relates strongly to what i mentioned in #2291 (comment). This is difficult to do because you cannot base it exclusively on issues actually reported, you would need to consider unreported problems as well. This is what in the end decides on if this style improves quality wise or not. These unreported problems are critical to identify the unknown unknowns so to speak - the changes necessary to fix problems you are not even aware or do not understand. For improving and maintaining quality of this style it is therefore essential to minimize the unknown unknowns and to maximize completeness of the problems reported and documented and awareness of them. IMO strategies for handling issues would therefore need to concentrate on that goal. |
All the statistics lie this way or another. Quantity is (mostly) easy to get, quality is not - and one of the reasons is that it's very subjective matter. For me this style keeps getting better at reasonable costs, so the overall quality is constantly rising from my point of view, but for anybody else it might be otherwise. It always strikes me that you tend to solve unknown problems while avoiding big and visible ones. This is not working for me. Big visions are great at the phase of designing the prototype, but after that the details matter a lot more. Then redesign or competing solution might happen at some point, but in the world of complex systems there are a lot of gotchas which should be addressed to make things better. I want to have a great map, not a great design of the map, even if I like conceptualism. Ignoring details and concentrating on documentation instead of the code and community will hit the map quality in my opinion. |
Reopening this ticket, since it looks like I'm able to express my visions at last and some people might be interested in changing documentation to reflect different state of the style. The first of my articles on the subject is about dealing with a lot of data in big cities and how it relates to rendering outdoor (I hope I will write more of them): |
Some of my thoughts:
|
Disagree, it is semantically different, we had discussed that before. |
@polarbearing You seem to confuse you private opinion with output of discussion (which still wasn't established in #3342) |
I would like to see some hard data showing it some day, but my gut feeling is that some big tag transitions is simply impossible without OSM Carto adopting it. I agree that this is pretending and it has bad consequences.
That's also what I think. It would be good to have some easy to modify vector styles to show flexibility and richness, but it's not yet clear to me when it might happen, and even then raster map will still be needed at least for some time.
I plan to propose and test landcover/natural on low zoom soon. However that's an easy part, adding rivers on low zoom (requires special tagging) or changing border colors seems to be much harder.
I think that the name of project is misleading today - "open" is still true, but "map" just shows typical usage and "street" is also another typical usage (routing). OpenGeoData would be closer to reality (and this domain is still used by OSM!). But I have nothing against tuning road colors. There are more people claiming that they are interested, but just no one tries it. Only @imagico fork made it lately in some limited way: http://blog.imagico.de/more-new-colors/ |
Here is my second article about map designing principles - this time it's about the size: |
I think this is the right place to discuss problems from #2436 (comment). All the questions from #2436 (comment) are also what I'd like to ask, but I would also like to know:
What do you consider original vision? I don't know the original vision before OSM Carto, but I think that original vision was just to port XML into Carto CSS avoiding adding anything. |
Interesting comment on wiki from iD developer @bhousel, which will definitely have some impact on OSM ecosystem, since it's highly popular editor:
https://lists.openstreetmap.org/pipermail/tagging/2019-January/041906.html |
frankly I am not astonished he got some of his edits reverted, he has a record of insisting on unilateral decisions, e.g. introducing formerly unused and undocumented tags via iD presets even where competing tags were already established.
The bigger problem are things that don’t find general consensus but happen to slip through without someone noticing.
|
@dieterdreist, kind of like the car: and motorcycle: tags that being pushed hard by Rtfm on the wiki and with retagging despite opposition to both of them. I noticed you gave him feedback a while ago his car: proposal that he ignored. I partly agree with bhousel about the wiki being worthless since Rtfm hasn't been stopped in his tracks since then. I know some things will slip through the tracks, but I feel like the wiki has to be very accurate when it comes to what tag is prefered and the tags definitions, or else its pretty much worthless. Especially when it comes to the really important tags. Someone definently shouldnt be able to insert their own tag into the pages of established ones as if it over rides them without someone intervining. or else whats the point in the wiki and tag status? Anyway, at least its being disscussed I guess. I imagine it would probably be hard for projects like this and iD editor to function without an agreed on and clear longterm tagging foundation in place. Which probably takes the wiki being consistant and reliable. So hopefully it will get worked out. |
sent from a phone
On 8. Jan 2019, at 15:37, Adamant36 ***@***.***> wrote:
I partly agree with bhousel about the wiki being worthless since Rtfm hasn't been stopped in his tracks since then.
Rtfm‘s edits and proceedings are currently discussed and eventually reverted. bhousel has a record of acting similarly with regard to (not) discussing things with the community and (not) accepting community decisions, just that he has more power because he controls the iD presets.
|
Let's please stay on topic,, which bad-mouthing maintainers of other projects is not. The Wiki has never been a source of truth for us. We've used it to greater and lesser degrees depending on how useful it is, but it's frequently not. |
I would like to start talking about the planned direction of this style again, mainly regarding cartography and goals. However, I'm not sure if it's best to do it in this issue, where most of the comments are from Nov 2015 to Nov 2016, and much of the discussion was around limitations in the current tools (Mapnik, Carto, Osm2pgsql and PostGIS). What I would like to discuss:
|
Feel free to open a new issue, but I think it is important to stated the goals (definition of done) in the opening message. For example: 'this ticket should lead to a list of agreed on cartographic goals for 2019' or 'this ticket should lead to a modification in CARTOGRAPHY.txt'. |
A new issue would be best - a lot of the comments here are no longer relevant. |
With the road update rolled out, a large part of the style sheet has now been modified since the stylesheet development has been taken up by the current tram. Roads, buildings, landuse colours, and icons have all been changed to a greater or lesser extent.
I think that makes it time again to have a look at the bigger picture.
What should development concentrate on over the next year or so? What are the issues that need improving most at this point?
This discussion should lead to an update of the roadmap in README.md.
CC @gravitystorm @pnorman @matkoniecz.
The text was updated successfully, but these errors were encountered: