Importing Berlin

[jvolker]

I've used ./importer --oneshot=/ with the binary version and this (extracted) OSM file: http://download.geofabrik.de/europe/germany/berlin-latest.osm.bz2

The importer crashes with Unknown turn restriction no_entry:

- Running convert_osm on /Users/jerry/Downloads/berlin-latest.osm
Read /Users/jerry/Downloads/berlin-latest.osm (1,040)... 162.1816s
OSM doc has 5522394 nodes, 818090 ways, 14293 relations
processing OSM nodes (5,522,394)... 4.4821s
processing OSM ways (818,090)... 6.3400s
Relation 1653527 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 2107105 has unhandled member role inner, ignoring it
thread 'main' panicked at 'Unknown turn restriction no_entry', map_model/src/raw.rs:373:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
dropping Timer while doing progress processing OSM relations, due to panic?

[dabreegster]

The OSM importing code is extremely strict about stuff I haven't seen before, to prevent regressions when converting regularly used maps. This one is easy to fix (edit map_model/src/raw.rs:373) if you've built from source. You're bound to hit a few more issues though. I'll download the Berlin map and fix whatever I encounter.

[dabreegster]

The entire Berlin area is huge. Could we start with a smaller area? Not neighborhood small, maybe the general city center area, just not 1.1GB. :) I use geojson.io or geoman.io to draw polygon boundaries.

[jvolker]

Wow, thanks for doing this.

Sorry. I didn't know how much this file covered.

Could we start with a smaller area?

Of course, how about this for the core city centre:

"coordinates": [
          [
            [
              13.361434936523436,
              52.49532344352079
            ],
            [
              13.432674407958984,
              52.49532344352079
            ],
            [
              13.432674407958984,
              52.53199183474197
            ],
            [
              13.361434936523436,
              52.53199183474197
            ],
            [
              13.361434936523436,
              52.49532344352079
            ]
          ]
        ]

and this for a larger part of the city which is surrounded by the circle train:

"coordinates": [
          [
            [
              13.271141052246094,
              52.45935663683681
            ],
            [
              13.47644805908203,
              52.45935663683681
            ],
            [
              13.47644805908203,
              52.561116680853836
            ],
            [
              13.271141052246094,
              52.561116680853836
            ],
            [
              13.271141052246094,
              52.45935663683681
            ]
          ]
        ]

I can try to create those filter files.

[jvolker]

Filter files: berlin-polys.zip

I've tried another time with berlin-centre.poly but am still getting the same error.

[dabreegster]

Thanks! I got the large one working. You can put this file in data/system/maps: https://www.dropbox.com/s/4uia1pk9aj1d8gd/berlin.bin?dl=0
My GPU half-melted during loading and my mouse was briefly unresponsive, but in the end, only took 20s to load. :)

I'll commit a fix in a moment that deals with almost all of the issues. There are a few intersections where seemingly different combinations of roads wind up with exactly the same turn geometry, which breaks some assertions. I want to figure out why that's happening instead of relaxing that code permanently.

[dabreegster]

Interesting: I think Berlin is the first place I've run across with OSM parking tags filled out somewhat. It's been quite painful mapping them in Seattle. Odd that the coverage in Berlin is so spotty; IIRC from a prior visit, there's definitely more than just these areas

[jvolker]

Wow! That's amazing. Thanks a lot for your quick help.

Unfortunately, abstreet is crashing when trying to load the Berlin map. output.txt contains:

Loading map ../data/system/maps/berlin.bin

Reading ../data/system/maps/berlin.bin: 0/149 MB... 0.0005s

../data/system/maps/berlin.bin is missing or corrupt. Check https://github.com/dabreegster/abstreet/blob/master/docs/dev.md and file an issue if you have trouble.

invalid value: integer `512`, expected variant index 0 <= i < 8

Interesting: I think Berlin is the first place I've run across with OSM parking tags filled out somewhat.

That's great!

[dabreegster]

I found what I think is a problem with Bayernring in OSM:

• https://www.openstreetmap.org/way/722013503
• https://www.openstreetmap.org/way/722013169
  The first one is an overlapping road of the second. I think the second one needs to be split at the point where the road attributes are different. Are you part of the OSM community in Berlin? If not, I can attempt a fix or raise the issue on the appropriate slack.

The crash is because the binary map format has changed since Sunday. That's the price of rapid iteration. :( If you can build the game crate from source, it'll handle the new format. If not, https://github.com/dabreegster/abstreet/suites/866601914/artifacts/10031908 is a Mac binary from last night that should work

[jvolker]

The new Mac binary is working with the Berlin map. Fantastic! Thanks so much.

Are you part of the OSM community in Berlin?

No sorry, I'm not. But I'm part of CityLAB Berlin. I should introduce myself properly in the next few days. And maybe my colleagues can answer some of your questions on the Berlin data. Let me know if you have any more questions.

Thanks so much! 🙏 That was the quickest support.

[jvolker]

About the parking data, my colleague @Lisa-Stubert mentioned an official data set from 2014 that contains this data in some format. It might be useful to fill in the gaps on OSM.

FIS-Broker > Straßenbefahrung > Parkfläche > WFS address

The data set contains polygons. @Lisa-Stubert mentioned we might need lines instead.

[dabreegster]

The direct link to WFS address seemed broken, and I couldn't find Parkfläche on the first page. Is there a different direct link?

Seattle has a perhaps similar dataset, blockface. I'm matching that to the nearest side of each road here. We could do something similar for this dataset.

Ideally the data winds up in OSM, so anybody can use it, and it can be maintained more easily. OSM has strict rules around importing data automatically. Regardless of an import, theres a tool in A/B Street to update parking tags in OSM a little more conveniently than other editors.

[jvolker]

The direct link to WFS address seemed broken, and I couldn't find Parkfläche on the first page. Is there a different direct link?

I'm not familiar with the workflow yet, but a former colleague has written this guide on how to use that data source: https://lab.technologiestiftung-berlin.de/projects/fisbroker-to-qgis/en/

Apparently you can also load it via command line (with GDAL installed): ogr2ogr -f gpkg Parkflächen.gpkg WFS:"https://fbinter.stadt-berlin.de/fb/wfs/data/senstadt/s_Parkflaeche"

I'm going to get back to this next week.

[Lisa-Stubert]

https://fbinter.stadt-berlin.de/fb/index.jsp?loginkey=zoomStart&mapId=k_StraDa@senstadt&bbox=383640,5815972,386377,5817593 this is the direct link to the data set of the "Straßenbefahrung". It contains all possible areas and objects of road traffic for Berlin. The dark grey areas are the parking areas. They can only be downloaded via the WFS service @jvolker posted.

[jvolker]

I've just tried to load the berlin map in A/B street compiled from source on Windows and received this error:

switch map...
Wrote ../data/player/camera_state/montlake.json
load map...
Loading map ../data/system/maps/berlin.bin
Reading ../data/system/maps/berlin.bin: 0/149 MB... 0.0001smemory allocation of 14934484003840414750 bytes failederror: process didn't exit successfully: `C:\Users\PC\Desktop\AB-Street\abstreet\target\release\game.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)

Do you have any idea what could cause this? Thanks.

[dabreegster]

The binary format of the map changed yesterday.

I can regenerate the Berlin map and give you a new dropbox link in the next ~30 mins

[dabreegster]

https://www.dropbox.com/s/2l2muvha0dubuvp/berlin.bin?dl=0

The simulation crashes 3m58s in. I can investigate soon; I suspect this is the same as #143 (comment) -- there are some bus stops mis-tagged in OSM. I'll improve the error message to find these better, or filter out the invalid ones.

[dabreegster]

Fixed Berlin map: https://www.dropbox.com/s/2l2muvha0dubuvp/berlin.bin?dl=0

The possible bus route issues:

Route Bus M76: U Walther-Schreiber-Platz <=> S Lichtenrade has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 43238974, i1: OriginalIntersection { osm_node_id: 27433695 }, i2: OriginalIntersection { osm_node_id: 27434791 } }
Route Bus 277: S+U Hermannstraße <=> Marienfelde, Stadtrandsiedlung has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 391673769, i1: OriginalIntersection { osm_node_id: 3726792172 }, i2: OriginalIntersection { osm_node_id: 3708789255 } }
Route Bus N50: U Tierpark <=> Buchholz-West/Hugenottenplatz has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 626027608, i1: OriginalIntersection { osm_node_id: 5910363710 }, i2: OriginalIntersection { osm_node_id: 2519032893 } }
Route Bus X54: S+U Pankow <=> U Hellersdorf has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 4783453, i1: OriginalIntersection { osm_node_id: 30625193 }, i2: OriginalIntersection { osm_node_id: 2101949795 } }
Route Bus N5: S+U Alexanderplatz <=> Hellersdorf, Riesaer Straße has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 139877314, i1: OriginalIntersection { osm_node_id: 270718978 }, i2: OriginalIntersection { osm_node_id: 7312391564 } }
Route Buslinie M29 has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 451544125, i1: OriginalIntersection { osm_node_id: 294494421 }, i2: OriginalIntersection { osm_node_id: 845075600 } }

These stops are just skipped for now. Not high priority to figure out what's wrong here.

[jvolker]

Thanks. It's working again.

[dabreegster]

Just checking, are you blocked on me for anything besides importing and matching parking data? (which I'm not likely to get to anytime super soon)

[jvolker]

Thanks for asking @dabreegster. It seems the most important thing is to get trip data to get a more realistic simulation. We are currently looking into data sources and are going to post them here.

[tori-d]

Hey Dustin!

So we've done some brainstorming and come up with some additional data sources that might be useful for the project. Maybe you can take a look and give us an indication which data sources you see as more or less useful, or what's still missing?

The biggest thing we don't have right now is any sort of traffic pattern model like the SoundCast model. It's unclear to me to what extent this data exists in Berlin at all, and to the extent it does, it's not available as open data and we currently lack the contacts to rustle up that data. Obviously we'll keep exploring that, because it's something we're very interested in, but I don't know how soon we would have something concrete to offer there, if that even happens at all.

So what do we have?

• Public transit data (GTFS): https://daten.berlin.de/datensaetze/vbb-fahrplandaten-gtfs (available as a regularly-updated upload; there is also an API, but it requires first contacting the local transit operator to get access)
• Car traffic speed data: https://movement.uber.com/explore/berlin/speeds/query?dt%5Btpb%5D=ALL_DAY&dt%5Bwd;%5D=1,2,3,4,5,6,7&dt%5Bdr%5D%5Bsd%5D=2020-03-01&dt%5Bdr%5D%5Bed%5D=2020-03-31&ff=&lang=en-US (this is a dataset made available by Uber; shows average speeds over various road segments in both directions – could be useful as a proxy for traffic flows?)
• Traffic volume data: https://fbinter.stadt-berlin.de/fb/index.jsp?loginkey=zoomStart&mapId=wmsk_07_01verkmeng2014@senstadt&bbox=388185,5818529,395105,5822530 (This is a very general dataset, available via WFS, showing estimated number of vehicles driving over major streets in a 24 hour period – this data is based on traffic counts originally gathered in 2014, so it's pretty outdated)

What might we have?

I looked at the dataset containing information on "blockfaces" to see what it contained (wasn't familiar with this term myself). Seems like it includes data on things like zones (e.g. loading and unloading zones, no parking zones, etc.) as well as metered parking/parking fees. Not sure what out of the blockfaces dataset is most relevant for A/B Street – if it's just about knowing where there is streetside parking, then the aforementioned "Straßenbefahrung" dataset is probably our best bet there. If you also want information on no-parking zones etc., I think we can possibly also get those out of the Straßenbefahrung dataset, but I'm not certain. We definitely won't be to get get good data on where parking is metered (and under what circumstances); there isn't currently a comprehensive city-wide dataset for that.

We may be able to rustle up some data on off-street parking – @Lisa-Stubert wanted to check out another large geospatial dataset we have to see if parking lots and parking garages are in it (although if they are, my guess is we won't have data on capacities, but maybe I'll be proven wrong).

Let me know what questions you have. If there's an acute need for data that has not yet been addressed, also let me know about that and we can go sleuthing...

[dabreegster]

Thanks for doing this research!

• The GTFS upload will be useful eventually. The transit modelling needs lots of work first; currently just a single bus is created per route, and it runs all day. No need for API access; the schedule information is used when building the map as a one-time process.
• Speed and traffic volume could be helpful. I've seen research papers before talking about how to use volume data to infer a set of start/end points that would cause the measured volume counts. I have no familiarity with the algorithms for doing this; it'd be future work to implement them or find an existing tool that does it.
• Straßenbefahrung: It sounds like this is the equivalent of the Seattle blockface data indeed. We could use it just to infer the existence of space along the shoulder of the road for street parking. The data in Seattle wasn't adequate; it says "no parking restrictions" on narrow roads that physically had no space for parking. So I use it as a first pass, manually look at areas of the map with odd road geometry, then fix upstream in OpenStreetMap using a tool.
• There's no modelling of metered parking, resident-only restrictions, 2 hours max, etc yet. No immediate plans to figure this out either.
• Offstreet parking with garages/lots could be helpful. We can estimate capacity from the geometry. Some lots are already in OSM: https://www.openstreetmap.org/way/669540789

I think the biggest thing that would help generate better trip data would be census. In the US, there are census tracts like this that chop up the area into pretty small regions, then assign a bunch of demographics to each tract -- mainly a population count. Then there's the American Commuter Survey that has questions like "how many cars owned per household" and "how many trips > 5 miles taken per day." The sample size is probably really small, but it's better than nothing, and I think the survey results are correlated with census tracts. So from this, the really simple trip generation model would take the number of people inside a tract, randomly distribute them to houses in that area (using OSM's residential vs commercial building tagging), guess a workplace/school/common destination for them using stats on how far people in that area travel, and so on.

I see https://en.wikipedia.org/wiki/Demographics_of_Berlin, and drilling down, https://en.wikipedia.org/wiki/Tiergarten,_Berlin is a small area with a population of ~12.5k. So I think we could use this! I haven't looked for where the wiki article pulls data; if there's a more consolidated format that ideally has the GPS coordinates of each neighborhood with this population count, that'd be a huge step forward.

[tori-d]

With respect to the Census Tracts / finding a corollary in Berlin:

So there are various administrative units that Berlin as a city is broken up into. A coworker actually did a nice breakdown of them (in English) here: https://lab.technologiestiftung-berlin.de/projects/spatial-units/en/. The example you found, Tiergarten, is an example of an Ortsteil/locality. Of the top of my head, I'm not actually sure what the smallest possible unit is (there are a few different systems for breaking up the city that are used in different concepts, so it's not necessarily straightforward to arrange all of the units from smallest to biggest. I guess what we want is the smallest possible unit for which we also have demographic data?

I think the smallest unit for which we would have overall population data is a Planungsraum/Planning Area. The population data is available here: https://daten.berlin.de/datensaetze/einwohnerinnen-und-einwohner-berlin-lor-planungsr%C3%A4umen-am-31122018

We also have the geometry of the Planungsräume available in various geospatial formats: https://data.technologiestiftung-berlin.de/dataset/lor_planungsgraeume/en (I think this file is slightly outdated though, we'll have to update it first).

I don't know if we'll be able to find additional census-like data for that level of specificity, however. Within Berlin itself I haven't seen any sort of survey data around mobility habits.

@Lisa-Stubert found this dataset, which probably has what we want? https://daten.clearingstelle-verkehr.de/279/ It includes a regional level, where an area is broken up into a grid of squares that are at least 500x500m and have at least 500 residents in them. The data, as far as I can tell, would include information on the average trip length from a given cell, whether the people living there own cars, etc. Unfortunately, it is not available as open data and I suspect it is only available for a fee. Moreover, the data seems like it is likely to be fairly complex, so even if we were to acquire it, I think it might be beyond our resources at least to integrate it into the program.

I can keep poking around to see if I find anything else at the Berlin level, but at least right now, we definitely have population data for sub-units of Berlin.

[matkoniecz]

Maybe 2> log.txt (stdeer capture) will work?

[jvolker]

Maybe 2> log.txt (stdeer capture) will work?

No, nothing, unfortunately.

[matkoniecz]

This is a function we've kind of made up out of vague intuitions, and tuned for Kraków

"tuned" is overstatement, it was more of "AB Street is no longer crashing and there are some pedestrians, cars and cyclists" and "problem of bicyclist traffic jams, with bicycle traffic over 100 times reality, is reduced to 5 times reality" (it is not overstatement, bicycle traffic in one iteration was so large that it caused nearly instant city-wide gridlock).

Feel free to change numbers or tweak it based solely on Berlin, do not worry about Krakow (though if you want you may test it also there).

[dabreegster]

Ah, maybe running a separate shell script on Windows launches a new terminal. Try:

RUST_BACKTRACE=1 cargo run --release --manifest-path importer/Cargo.toml -- --raw --map --city=berlin

I'm not sure about Windows syntax, but you could also try . import.sh --raw --map --city=berlin. The leading . means run all of the commands in a file, but ignore the first line that says what shell to execute with.

Also I noticed you're doing --city=berlin_center; it should be --city=berlin. The difference isn't interesting when a city only has one map, but we'll probably split into multiple maps at some point, like Seattle. The city name is berlin and the only map there is called berlin_center, currently.

[jvolker]

Thanks, the following has worked:

set RUST_BACKTRACE=1 && cargo run --release --manifest-path importer/Cargo.toml -- --raw --map --city=berlin

But I'm now getting the following errors:

   Compiling importer v0.1.0 (C:\Users\PC\Desktop\AB-Street\abstreet\importer)
warning: unreachable expression
  --> importer\src\main.rs:96:13
   |
91 | /             panic!(
92 | |                 "Can't do --scenario or --scenario_everyone without the scenarios feature \
93 | |                  compiled in"
94 | |             );
   | |______________- any code following this expression is unreachable
95 |               // Nonsense to make the type-checker work
96 |               (Some(true), Some(true))
   |               ^^^^^^^^^^^^^^^^^^^^^^^^ unreachable expression
   |
   = note: `#[warn(unreachable_code)]` on by default

warning: unused variable: `maybe_popdat`
  --> importer\src\main.rs:80:10
   |
80 |     let (maybe_popdat, maybe_huge_map) = if job.scenario || job.scenario_everyone {
   |          ^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_popdat`
   |
   = note: `#[warn(unused_variables)]` on by default

warning: unused variable: `maybe_huge_map`
  --> importer\src\main.rs:80:24
   |
80 |     let (maybe_popdat, maybe_huge_map) = if job.scenario || job.scenario_everyone {
   |                        ^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_huge_map`
warning: unused variable: `maybe_map`
   --> importer\src\main.rs:112:13
    |
112 |         let mut maybe_map = if job.raw_to_map {
    |             ^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_map`

warning: variable does not need to be mutable
   --> importer\src\main.rs:112:13
    |
112 |         let mut maybe_map = if job.raw_to_map {
    |             ----^^^^^^^^^
    |             |
    |             help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

warning: function is never used: `adjust_private_parking`
   --> importer\src\seattle.rs:116:8
    |
116 | pub fn adjust_private_parking(map: &mut Map, scenario: &Scenario) {
    |        ^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: 6 warnings emitted

    Finished release [optimized] target(s) in 6m 27s
     Running `target\release\importer.exe --raw --map --city=berlin`
- Working on all berlin maps
import map data...
- Missing data/input/berlin/osm/berlin-latest.osm.pbf, so downloading http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf
- Running "curl" "--fail" "-L" "-o" "tmp_output" "http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 57.2M  100 57.2M    0     0  11.4M      0  0:00:05  0:00:05 --:--:-- 11.1M
- Running "mv" "tmp_output" "data/input/berlin/osm/berlin-latest.osm.pbf"
thread 'main' panicked at 'Failed to run "mv" "tmp_output" "data/input/berlin/osm/berlin-latest.osm.pbf": Os { code: 2, kind: NotFound, message: "Das System kann die angegebene Datei nicht finden." }', importer\src\utils.rs:132:13
stack backtrace:
   0: backtrace::backtrace::trace_unsynchronized
             at C:\Users\VssAdministrator\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\backtrace-0.3.46\src\backtrace\mod.rs:66
   1: std::sys_common::backtrace::_print_fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:78
   2: std::sys_common::backtrace::_print::{{impl}}::fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:59
   3: core::fmt::write
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libcore\fmt\mod.rs:1076
   4: std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\io\mod.rs:1537
   5: std::sys_common::backtrace::_print
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:62
   6: std::sys_common::backtrace::print
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:49
   7: std::panicking::default_hook::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:198
   8: std::panicking::default_hook
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:218
   9: std::panicking::rust_panic_with_hook
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:486
  10: std::panicking::begin_panic_handler
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:388
  11: std::panicking::begin_panic_fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:342
  12: importer::utils::osmconvert
  13: importer::utils::download
  14: importer::berlin::osm_to_raw
  15: alloc::collections::btree::search::search_tree
  16: csv::reader::Reader<R>::deserialize
  17: std::rt::lang_start_internal::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\rt.rs:52
  18: std::panicking::try::do_call
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:297
  19: std::panicking::try
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:274
  20: std::panic::catch_unwind
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panic.rs:394
  21: std::rt::lang_start_internal
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\rt.rs:51
  22: main
  23: invoke_main
             at d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  24: __scrt_common_main_seh
             at d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  25: BaseThreadInitThunk
  26: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
import map data took 5.4766s

- import map data took 5.4766s

- import map data took 5.4766s
error: process didn't exit successfully: `target\release\importer.exe --raw --map --city=berlin` (exit code: 101)

[dabreegster]

You can ignore the warnings. It tried to execute this command: mv tmp_output data/input/berlin/osm/berlin-latest.osm.pbf. Looks like Windows calls it move, not mv.

I'll switch importer to use system calls instead of relying on Linux/Mac commands when possible. Not going to get to that for a few hours. As a temporary workaround, you could edit importer/src/utils.rs, look for all Command::new, and if you get an error about one of them, change it appropriately. I think mv becomes move, cp -> copy, and rm -> del (possibly messing with the -rfv flags).

[dabreegster]

Number of people in houses around here down from 360k to 180k, after adjusting for the planning areas that only partly overlapped the map. Does this seem generally more right to you?

[tori-d]

A couple additional data sources to throw in there:

• I found this study published in 2018 that I think actually gets at this census-level data you mentioned before, @dabreegster. Basically one of the universities here did a survey of 40,000 Berliners in which they were asked about their mobility habits (e.g. modal split, types of trips, etc.). They have detailed reports for each of Berlin's 12 districts/Bezirke. However, there is currently no open data available, just PDFs. We could try to reach out to this university however and see if they would be interested in sharing their data, perhaps they would find our project interesting: https://www.berlin.de/sen/uvk/verkehr/verkehrsdaten/zahlen-und-fakten/mobilitaet-in-staedten-srv-2018/

• For bikes, we do have open data from the automatic counting stations the city has. The table is available here: https://www.berlin.de/sen/uvk/verkehr/verkehrsplanung/radverkehr/weitere-radinfrastruktur/zaehlstellen-und-fahrradbarometer/#dauer (if you scroll there is an Excel file). It shows the hourly counts for each station over several years. Using that, we could at least get pretty accurate counts of how many bikers are going past a given point in the city on a given day / at a given time (there are 17 stations across the city). We did an analysis with the data up to 2017 a couple years ago, so we do have some already have some cleaned up versions of this data that theoretically could be used as a basis? Not sure what data specifically would be most helpful to have here though (e.g.: should the hourly data be aggregated to a certain level for each location?)

[dabreegster]

The travel surveys look really promising! Right now, our approach knows roughly how many people reside in each building, but we don't know how far they travel, or how. If we could put together a table of (origin district, destination district, percentage of bike trips, pct of driving, ...), then we could pretty easily do something more realistic. I can't understand the PDFs too easily, but if they have this quadratic table of trips between every pair of districts, 144 things isn't unreasonable to manually copy from a PDF.

Bike counts may be helpful, but I don't know how to turn the data into individual trips. This is a sample paper describing how to do it. It's way beyond my current ability to understand.

The more I read about https://activitysim.github.io/activitysim/index.html, the more I'm convinced this could be the more trustworthy way forward. Seattle's Soundcast model makes use of this. (Edit: just kidding, they use DaySim.) Glancing through the amount of detail in the different parts of their model, there's no hope of reproducing this level of rigor. I think A/B Street ought to require trip data as input. We can do extremely simple approximations that just look at OpenStreetMap data and maybe some extra census counts, but I don't know if it's worth trying to recreate what ActivitySim has done.

[dabreegster]

An update: there's renewed interest in getting a good travel demand model for Berlin, from the folks creating CargoBikeCity. Since July, a few more ideas/options for generating demand data have arisen:

• @behrisch clued us in to https://github.com/matsim-scenarios/matsim-berlin
• https://github.com/DLR-VF/TAPAS was released
• https://github.com/lcodeca/SUMOActivityGen
• We renewed our efforts to generate a simple model using only OSM data and population counts: Activity modelling #424

[tordans]

@dabreegster great to see this long thread about Berlin. We have a dedicated OSM UserGroup about "Traffic evolution" in Berlin. More at https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende. Some things we work on:

• We are looking into extending the tagging for bike lanes with new, safety related tagging and where required separate ways to represent bike-lane-only turns (around trees and such) – examples at https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Radwege#Tagging-Beispiele. This is still experimentell, however.
• Same for pedestrian routing; this experimental area has full separately mapped sidewalks https://www.openstreetmap.org/node/4794919616#map=17/52.47639/13.42334
• We are looking at improving the pedestrian crossing mapping and tagging – this is in an early stage https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Fu%C3%9Fwege#Tagging-Beispiele
• We mapped all Neukölln with the parking:lane Schema. This allows reports like https://www.openstreetmap.org/user/Supaplex030/diary/393423.

I would love to see the map with lanes and such running to start checking the OSM data and adjust it where needed. What is your recommendation how to get this running locally (Mac) in this UseCase?

[matkoniecz]

What is your recommendation how to get this running locally (Mac) in this UseCase?

Where you are stuck? At running ABStreet at all or at running specifically Berlin?

[dabreegster]

Hi @tordans! I'm constantly amazed by the level of detail in OSM being added, and the new proposals on the Verkehrswende wiki page look exciting. I'm quite interested in improving the accuracy of the map model in A/B Street from this data, but some of it is going to be quite difficult.

One big problem is that A/B Street doesn't yet handle sidewalks and cyclepaths mapped separately from the "main" road way. a-b-street/osm2streets#59 and a-b-street/osm2streets#81 have details. In short, it's hard for a few reasons:

1. The simulation needs to relate sidewalks with the rest of the road to handle "modality transfers", like walking from a building to the road, then starting to bike or entering a parked car. When the sidewalk isn't explicitly associated with the rest of the road, inferring these connections is hard.

2. Rendering and inferring good geometry for separate ways is hard. Part of the reason is due to widths -- since I haven't seen width explicitly tagged in most places I've looked, all lanes have fixed widths. Separate paths tend to be very close to the road, and because the inferred width is wrong or the GPS is slightly out of alignment, the generated roads tend to overlap.

3. The traffic simulation has trouble when there are lots of short "roads" broken up by extra "intersections." Separate crosswalks aggravate this problem even more.

That being said, you could enable using more of the OSM sidewalk data by using config like this.

I'd love to work on fixing these problems, but I tend to over-commit how much I can work on. If anybody from your group is interested in helping with development, that would be ideal. I can mentor if Rust isn't a familiar language, but some programming experience is necessary to make progress.

One good first step could be to extend the area of Berlin in A/B Street. We just have the one shape right now. If Neukölln is a useful area to focus on, for example, then we could chop up the area into more pieces. This is best done by locals familiar with the city to choose meaningful boundaries. https://dabreegster.github.io/abstreet/howto/new_city.html#including-the-city-to-ab-street-more-permanently has more info -- geojson.io is a useful tool for drawing the polygons.

I spotted this from y'all's Telegram group:

Once we have a polygon covering this area and are regularly importing, I can look into specific problems like this.

What is your recommendation how to get this running locally (Mac) in this UseCase?

https://dabreegster.github.io/abstreet/dev/index.html has instructions for compiling locally -- is that what you meant?

[SupaplexOSM]

Hey Dustin,

I am also happy to have discovered this thread (I'm another member of the "Verkehrswende" group). Neukölln could be an perfect playground for testing detailed street information. As you've already noticed, we have parking lanes mapped all over the place here. In addition, we are currently starting to systematically collect other data that could be interesting for traffic modelling: For example, road widths (a discussion for this purpose resulted in the OSM tagging "width:carriageway" a few months ago, see OSM-Wiki). Most roads here have no markings, so width data is most relevant for estimating traffic flow. Do you already take width data into account in any way?

As mentioned by @tordans we have also started to record separate sidewalks. "sidewalk=separate" on a highway is an indicator for this, which you can evaluate even when analysing the street lines without having to generate relationships to separate geometries. We need to improve this a bit, because an indication like "sidewalk:both=separate" or "sidewalk:left=separate" would be clear to show on which sides of the road there is a sidewalk. Detailed attributes for the sidewalks (like width – if mapped at all) are lost with separate mapping, but not the information that there is a sidewalk at all. We do it similarly for cycle tracks.

There are other details like curbs, curb extensions or certain forms of crossings that we are working on, but the two points mentioned above (street width and the street vs. pavement problem) are probably the most interesting for you.

I'd be happy if our well-mapped neighbourhood is enriching your developments and perhaps encourages other mappers to map in more detail. Applications like A/B Street and the analyses that are possible with them show what potential can be found in open data! I'm not a programmer, unfortunately, so I can't help with A/B Street (but I am happy to contribute on local or tagging issues).

P.S. If you're interested: I'm currently working on a parking analysis (with QGIS) for this Neukölln district, which I'll finish during the next weeks: see some first impressions here.

[matkoniecz]

sidewalk=separate" on a highway is an indicator for this, which you can evaluate even when analysing the street lines without having to generate relationships to separate geometries.

This is implemented already, though maybe it can be tweaked a bit. AB is smart enough to handle it specially on major oneway roads (dual carriageways).

[dabreegster]

Do you already take width data into account in any way?

Not really, except for narrow=yes. https://github.com/dabreegster/abstreet/blob/master/map_model/src/make/initial/lane_specs.rs is the one place that transforms OSM tags into A/B Street's lane model. Maybe more interesting than the (disorganized) code are the tests at the bottom. You've probably noticed that regular driving lanes and bike lanes both have the same width in A/B Street. I would very much like to have more reasonable defaults, and incorporate the exact lane and road widths if they're tagged. But it might make editing lanes in the game, one of the main actions the player can take, more complicated. Do we have to figure out if the road has enough width to support a lane type transformation? Do we need to shift the geometry of the lane slightly? Does the UI for changing lanes need to get more detailed -- maybe closer to streetmix.net? This is the sort of thing I'd love to tackle when I get time or find somebody interested in designing+implementing this.

we have also started to record separate sidewalks

Things like sidewalk:left=separate would be a massive improvement over sidewalk=separate, in my opinion, because it would let me defer the complex problem of handling separate sidewalks at least for a little while. Although extremely unwieldy to work with, a relation would be even better -- something with 2 ways as members, for example -- role="main_road", role="sidewalk". I can imagine some algorithms to effectively infer this relation automatically by "tracing" the separated sidewalks and figuring out where they have a crossing over the road, but also there are lots of edge cases.

I still think the best next step is to get more boundary polygons for areas around Berlin in.

[dabreegster]

@SupaplexOSM, I just found your micromap thanks to @tordans in the OSMUS Slack. This is frankly the coolest thing I have seen in a long time. It's an unprecedented level of detail, yet immediately clear how to read -- the design and color choices are awesome.

I'm still reading through the blog posts and code, so apologies if some of these questions are answered there.

1. Do you have a method for calculating the width of a road from the mapped curbs? Does QGIS have a tool like this that can be applied automatically?
2. When you know the total width of a road, do you have heuristics for dividing up that width amongst some number of lanes with different purposes?

Since last December, A/B Street has made progress representing roads and lanes with modifiable width: https://a-b-street.github.io/docs/project/retrospective/index.html#road-editor

We've also made some progress logically and geometrically consolidating complex junctions. I just published a long article about how this works, https://a-b-street.github.io/docs/tech/map/geometry/index.html.

Finally, I'm curious how your parking space analysis is being received by the community. Are you using this to argue that there's too much public space being used to store vehicles? Or is this project more aimed at representing data in OSM instead of other sources?

[SupaplexOSM]

Oh, I'm glad you like the map! Unfortunately I'm not using such methods and I don't know if they would be possible... I am actually rather a non-expert in scripting etc. (and at the beginning of the project also in GIS) and therefore often have to look for simple solutions... There are two elements in the map so far that are not 100% generated from OSM:

1. One are the carriageway areas, which I have as separate shapes. They are a "hand-made" mixture of OpenData from the city of Berlin with (partly outdated) kerb information as well as our more up-to-date, often better, but not complete area-covering OSM data (barrier=kerb and residential landuse areas, which are often traditionally mapped to the kerb in Berlin). Theoretically, you could replace this data completely with "area:highway" OSM data, but we don't use them here yet (except in some intersection areas, see below).

2. Secondly, the line geometries of the parking lanes/parked cars are manually post-processed so that they fit exactly to the kerbs. Sorry, that I can't provide a magic receipe for that ;) With snapping, this can be automated well in QGIS, but to make it perfect and exclude "artefacts", I invested a few hours once to clean the data manually.

We now increasingly map accurate road width information in our area ("width:carriageway") and from various attributes (lanes, highway classification, oneway, number and location of parking and cycle lanes etc.) this information can also be interpolated for roads without any width attribute. I use this partly in the Python script for QGIS to generate the "basic version" of the parking lanes. But if you would render them on this basis, then unfortunately no such exact representation of the parking lanes would be possible as it is in the map - most parking lanes/cars would be a bit misaligned... (This could also be changed by mapping according to the "kerbside" scheme proposed by the makers of Curb LR: https://medium.com/sharedstreets/openstreetmap-and-curb-regulations-7812ee582a33 - but this is not yet used in OSM and the "parking:lane" scheme is so well established that it would probably not stand a chance).

Last week, the road markings and turn lanes have been added to the rendering of the micro map - these can be derived more or less well from the OSM data if lane attributes are mapped consistently (you know that, of cause). To improve the rendering, I have added "placement" tags in some places. In addition, intersection areas with "area:highway" + "junction=yes" are taken into account to "cut off" the markings in intersection areas. In return, I'm rendering "footway=crossing" ways there (or derive them from crossing nodes) to get more realistic crossing representation.

So a lot of this rendering only works because the area here is mostly mapped in a very detailed way... And unfortunately, the methodological requirements mentioned above mean that this micro-mapping map style cannot be easily extended to other areas or that it would require a generic lane model such as that of A/B Street (which can certainly come very close to reality if road attributes are mapped well). The generation of your data/the A/B Street lane model probably works completely differently and far more professionally, and therefore in a generic way (but as I said, I'm more an amateur and initially the map was only intended to visualise the parking data in our area...).

Regarding the second question: The topic of "public parking space" is currently being discussed very actively at all political and social levels in Berlin. However, there is a lack of empirical data on how much parking space there actually is - even more a lack of open data. We discovered that this data can be generated very precisely on the basis of OSM and the parking space analysis was therefore initially a little "tech demo" of how something like this can work on the basis of geodata. The response was great, some local initiatives use the data for political argumentation and planning and even two engineering companies, which do such data collection professionally, had questions, as they apparently do not yet work in such a "modern" way ;)

Lastly: Regarding A/B Street developments, I am always really impressed... "This is frankly the coolest thing I have seen in a long time" actually expresses pretty exactly what I feel when I see it ;) By the way, I love the Streetmix-like Road Editor approach - I've been dreaming of something like that for a very long time! It's also nice to see that the cycleway:separation scheme is taken into account ;)

[dabreegster]

(Sorry for the slow response)

I'm more an amateur and initially the map was only intended to visualise the parking data in our area

No need to be humble here -- even if you feel like a GIS beginner and this work would be hard to replicate elsewhere, it's still a fantastic map. I consider it a gold standard to build towards in more areas. I'm slowly puzzling through some ideas how to generalize the approach with less effort and in places without carriageway area data.

But actually, I was inspired to dig through Seattle's open data portal again, and found that there is pavement edge data (links here):

I think I've come across a few other examples of this in the US before. If the license is usable and we could devise some heuristic for chopping up the white negative space there into distinct road and junction polygons, then possibly we could propose importing these as area:highway polygons in some cities, with lots of human editing. And once we build a few demos of things using this kind of data, maybe mappers will be inspired to add more coverage and maintain it.

Another idea: integrate A/B Street's intersection geometry algorithm as a plugin to ID and JOSM. Then somebody can click a road center-line tagged in OSM, invoke the plugin, and auto-generate polygons for the road and junction. The result will be pretty bad in complex situations, but decent for many cases. The plugin could have a slider to adjust things like road width to match it to satellite easily. This could be another way to seed an area with more detailed area polygons.

I invested a few hours once to clean the data manually.

A lesson that I constantly forget is that automating GIS data is never a complete solution; cleaning things up manually is always necessary. I don't how many hours I've cumulatively spent trying to tune geometry algorithms to work well in a pretty small chunk of a city that I'm interested in. At some point, it feels like it'd be much more effective to instead develop simple plugins to more easily map highway areas in ID/JOSM. If the UI is easy and responsive, then drawing more detail against satellite -- in cases where tree cover doesn't make it impossible -- seems more plausible.

However, there is a lack of empirical data on how much parking space there actually is - even more a lack of open data.

I've seen similar debates in US cities. I'm curious, have you attempted to measure parking spaces associated with private buildings? Not just big commercial parking garages (though I don't remember seeing many of those in Berlin...), but the few spots tucked away behind an apartment building. Quantifying the number of spots is pretty hard, since people can get creative with packing vehicles into the space. But... somehow at least mapping the physical space would be interesting. But of course, much tougher than public street parking.

[matkoniecz]

the few spots tucked away behind an apartment building

behind seems relatively easy but time consuming with decent aerial imagery (as in case of such areas getting turned into parkings they tend to eliminate any trees). And street parking mapping is also really, really time consuming anyway.

Underground parkings are much trickier.

[SupaplexOSM]

The "private" parking spaces are also included in the evaluation to a (probably) very large extent. Some can be seen from the public space ("behind the fence"), others only become "visible" on aerial images. In Berlin, we are lucky that the city provides very good aerial images as OpenData every year, so that you can look under almost every tree using the images from different years ;) "Informal" parking spaces are not included in the analysis (as far as they are distinguishable from the official ones from remote).
And yeah, the underground parkings are a bit more complicated - partly their footprints are visible in our OpenData, but sometimes not. In some cases you can roughly derive their footprints from aerial imagery (according to the structures above the underground parking), in others you can just estimate ("somewhere below this building/behind this entrance"), or sometimes I was able to go in and count how many vehicles park there, got the numbers from the owners, from websites or plans. (Because this data is mostly not verifiable on the ground for the public, I didn't include them to OSM – only the entrances.)
In the end, with all the accuracy, one should keep in mind that the margin of error is rather small compared to the high total number of parked cars. So if some parking spaces are missing behind a building or the capacity of an underground garage is wrongly calculated, it doesn't make that much difference.