Create a new RouteInfo service, and use it to auto-fill the name of a route #154

dabreegster · 2023-05-09T10:55:29Z

This is a big step towards #69, #137, and all other ideas where we start to understand routes drawn in ATIP not just as geospatial line-strings, but as deeply rich semantic structures. First, the demo. After drawing a route in the v1 mode, you can press a new Autofill button, and generate a name for the route based on street names:

screencast.mp4

Technical design decisions

We want to understand and visualize many things about the route someone draws and things nearby:

speed limits along the sections on carriageways
crossings
existing lane configuration, to help with "will-it-fit" checks, determining if part of a route shares a bus lane, etc
what buildings and other amenities exist along the route
possible turning conflicts at big junctions

I'm proposing two design decisions to do that kind of thing:

Don't attempt to do this analysis in JS or TS, or on top of "raw" OpenStreetMap data. We could try to express some of this as Overpass queries and post-process data coming back. I think that's doable for some of the simpler questions we have, but will quickly break down in complexity.
Use A/B Street's map model as a starter. Years of development and polish have gone into the map importer there, handling various problems in OSM. The map model API has been re-shaped over time to handle traffic simulations, walkshed analyses, low-traffic neighbourhood analysis, etc. I don't think it'll do everything we need in its current form (crossings as they're tagged in OSM are a big gap right now), but I'd push for investing in more improvements there, rather than starting something from scratch.

As an alternative to this approach, we could consider doing all of the route analysis "offline" -- read in the ATIP GeoJSON files, write R or Python scripts to calculate some of the things listed above. This would limit interactivity in ATIP -- maybe someone could later load the results of that and match things up, but the method started in this PR gives instant results. The results of the analysis will require us to develop custom UIs to display things like turn conflicts, and I think the Svelte components are the right place to do that frontend work. We can treat the Rust layer as a backend and keep the two pieces coupled.

Future-proofing

Because initially deserializing the A/B Street map file is slow, running using web workers is immediately desirable, because it happens off the main thread. I found https://github.com/GoogleChromeLabs/comlink, which makes dealing with web workers very easy. This gives us another advantage as well -- the Svelte layer has to deal with async queries upfront. This means we could just as easily make HTTP requests to some new remote API somewhere to answer the same questions. So if we wanted, we could implement the RouteInfo service with a Python server running somewhere, and use whatever backend methods/data we want.

File management

Edit: acteng/atip-data-prep#7 imports all of the maps for the UK. They're already in S3, aside from 4 areas with bugs.

dabreegster

I'm working on generating the map data files everywhere, will followup soon.

Two UX questions immediately:

Should we have a second progress bar somewhere to show the second file loading?
Where should this auto-fill button go? It's just in the old v1 form for now. It's only relevant for routes, so maybe also in v2.

dabreegster · 2023-05-09T10:57:26Z

.github/workflows/web.yml

      - name: Build main branch
        run: |
          npm ci
+          # TODO Remove the directory check after this is present everywhere


After merging this PR, I'll rebase all other branches right now, so we can always run the wasm-pack step and simplify things

dabreegster · 2023-05-09T10:57:50Z

.github/workflows/web.yml

@@ -31,6 +36,8 @@ jobs:
          for branch in $(git branch -r | grep -vE "origin/(main|gh-pages)" | sed 's/origin\///'); do
            echo "Building $branch"
            git checkout $branch
+            # TODO Handle failure here


Same as above. We can just include npm run wasm-release before the npm run build below

dabreegster · 2023-05-09T10:58:24Z

.gitignore

@@ -6,3 +6,5 @@ node_modules/
 /playwright-report/
 /playwright/.cache/
 dist
+
+assets/bristol.bin


TODO: Remove this. I've manually produced and copied this file over. Working on stuff on the https://github.com/acteng/abstreet-to-atip/ side to generate files everywhere

dabreegster · 2023-05-09T10:58:46Z

README.md

+- `npm run fmt` to auto-format code
+- `npm run check` to see TypeScript errors
+
+If you're using Firefox locally to develop and get "import declarations may


It works fine in Chrome. See comments in App.svelte for details

dabreegster · 2023-05-09T11:00:02Z

route_info/src/lib.rs

+
+        web_sys::console::log_1(&format!("Got {} bytes, deserializing", map_bytes.len()).into());
+
+        let map: Map = bincode::deserialize(map_bytes).map_err(err_to_js)?;


So after we pass in a binary map file (produced in abstreet-to-atip and soon hosted in S3), we deserialize it. Then we also build a quadtree to snap to an intersection, since all the queries will make use of that

dabreegster · 2023-05-09T11:01:04Z

route_info/src/lib.rs

+    pub fn name_for_route(&self, raw_line_string: JsValue) -> Result<String, JsValue> {
+        let feature: geojson::Feature = serde_wasm_bindgen::from_value(raw_line_string)?;
+        // TODO Duplicate non-adjacent points; make a more permissive version of PolyLine.
+        //let polyline = PolyLine::from_geojson(&feature, Some(self.map.get_gps_bounds())).map_err(err_to_js)?;


A/B Street's geometry library has methods to much more easily parse GeoJSON, but they're too strict about duplicate points. We could improve this later to simplify code here.

dabreegster · 2023-05-09T11:02:03Z

route_info/src/lib.rs

+            return Err(err_to_js("couldn't get line-string"));
+        };
+
+        // Find the closest intersections to the route endpoints


The route snapper just gives raw GPS coordinates. Here we do a super simple version of map matching to find the closest intersection in the street network to each of the route's raw endpoints.

dabreegster · 2023-05-09T11:03:18Z

route_info/src/lib.rs

+            .closest_intersection
+            .closest_pt(pt2, threshold)
+            .ok_or_else(|| err_to_js("no intersection close to last pt"))?;
+        let i1_name = self.map.get_i(i1).name(lang, &self.map);


Then over in A/B Street, there's an existing method to print a nice name for an intersection, based on road names: https://github.com/a-b-street/abstreet/blob/dc74f7de5455b122c5636b34eb0d83b80116a20b/map_model/src/objects/intersection.rs#L229

(This can even use alternate languages, when a street is tagged with multiple names!)

dabreegster · 2023-05-09T11:03:54Z

src/App.svelte

+    //
+    // Note this should work fine in older browsers when doing 'npm run build'.
+    // It's only a problem during local dev mode.
+    interface WorkerConstructor {


This is a small amount of setup / TS boilerplate to make Comlink wrap our class

dabreegster · 2023-05-09T11:04:29Z

src/App.svelte

@@ -3,6 +3,10 @@
  import { onMount } from "svelte";
  import authoritiesUrl from "../assets/authorities.geojson?url";
  import type { Schema } from "./types";
+  import bristolUrl from "../assets/bristol.bin?url";
+  import * as Comlink from "comlink";
+  import workerWrapper from "./worker?worker";


The ?worker syntax is Vite build magic... which only works in some browsers when doing npm run dev. See comments below

This is magic to me ✨

acteng/atip#154

…p#154

Sparrow0hawk · 2023-05-09T16:32:05Z

Just glancing at this and will try give it a proper look through tomorrow. Quick question, is there a reason route_info has to live in this repo? Or could it live in its own repo like route-snapper and we just npm install it?

dabreegster · 2023-05-09T16:37:22Z

Or could it live in its own repo like route-snapper and we just npm install it?

No hard limit, but in the short-term, I think it's more convenient to keep it here. The local development workflow is more of a hassle when you have to edit a few local repos in sync, merge PRs in sync, etc. I've hit that annoyance already with route-snapper; https://github.com/dabreegster/route_snapper/blob/main/CHANGES.md is the amusing result. I feel like we're in the exploratory phase right now, and the API of the route info service will change constantly. I expect to write new Svelte components tightly coupled to the data it returns (like to display speed limit broken down along a route).

So I'd propose keeping it here for now, and splitting it out into a standalone repo once the dust has settled a bit.

robinlovelace-ate · 2023-05-09T21:15:49Z

Regarding your proposals

Don't attempt to do this analysis in JS or TS, or on top of "raw" OpenStreetMap data. We could try to express some of this as Overpass queries and post-process data coming back. I think that's doable for some of the simpler questions we have, but will quickly break down in complexity.

Agreed. Seeing the routes as part of the network along which people travel makes sense and is the most future proof option. I see it as hard to build and cannot imagine building such functionality, hence had not thought of it, but fully approve of the thinking here if it's actually doable, which I believe is.

Use A/B Street's map model as a starter. Years of development and polish have gone into the map importer there, handling various problems in OSM. The map model API has been re-shaped over time to handle traffic simulations, walkshed analyses, low-traffic neighbourhood analysis, etc. I don't think it'll do everything we need in its current form (crossings as they're tagged in OSM are a big gap right now), but I'd push for investing in more improvements there, rather than starting something from scratch.

Sounds reasonable. I've seen A/B Street in action and must agree: it's rock solid. Questions:

Isn't A/B Street's map model itself based on osm2streets now? That seems like a smaller, easier to update, and more agile codebase to build on.
Isn't A/B Street's map model overkill for some things? A design goal for ATIP is simplicity and A/B Street's map model is quite complex I thought. As an example, all A/B Street routes are represented in 2D while ATIP routes are 1D. I think it makes sense to move towards 2D but not sure at what rate.
Ease of maintenance and contribution?

I imagine you've thought of all these already just raising them for conversation. Overall: love the ambition and it make sense to build on what is good and works well.

robinlovelace-ate · 2023-05-09T21:24:46Z

As an alternative to this approach, we could consider doing all of the route analysis "offline" -- read in the ATIP GeoJSON files, write R or Python scripts to calculate some of the things listed above. This would limit interactivity in ATIP -- maybe someone could later load the results of that and match things up, but the method started in this PR gives instant results. The results of the analysis will require us to develop custom UIs to display things like turn conflicts, and I think the Svelte components are the right place to do that frontend work. We can treat the Rust layer as a backend and keep the two pieces coupled.

If the business logic is written in Rust wouldn't that allow us to get best of both worlds, using the Rust code in the browser via Wasm and in the backend via a CLI like cargo install --git https://github.com/dabreegster/odjitter ?

robinlovelace-ate · 2023-05-09T21:32:21Z

Two UX questions immediately:

Should we have a second progress bar somewhere to show the second file loading?

No strong feelings. Having 2 progress bars is not common and may look a bit complex but that's just a random thought without having seen it. As a sketch, where would the 2nd progress bar go compared with this?

Where should this auto-fill button go? It's just in the old v1 form for now. It's only relevant for routes, so maybe also in v2.

In the left hand panel just below each question would make sense to me on first glance.

robinlovelace-ate

I'm not qualified to comment on the technical details here. This looks like some hardcore modern TypeScript and Svelte code, combined with a nice bit of tightly integrated high-performance yet concise Rust of good measure. Based on the video and thinking expressed in the PR comments, which greatly helps understand the intention, it's a thumbs-up from me pending a few questions in my previous comments.

dabreegster · 2023-05-10T05:43:50Z

Thanks for asking great, critical questions! This is a longer-term decision starting in this PR, so very nice to get some discussion before diving farther in.

Isn't A/B Street's map model itself based on osm2streets now? That seems like a smaller, easier to update, and more agile codebase to build on.

Yes. For Alex's benefit, some context: about a year ago, we started splitting https://github.com/a-b-street/osm2streets out from the rest of A/B Street. That's made it easier to spin out smaller projects that just need to understand the street network part of OSM data. It's also a chance to more rapidly iterate on that code without having to immediately keep all A/B Street apps up-to-date. The decision to split has paid off very well (unlike the one here of ATIP and the route snapper). I'm trying to slowly move more and more of A/B Street logic into osm2streets, taking the chance to tackle it with fresh perspective and rewrite -- things like turns, representing edits to the network, pathfinding.

I started basing off A/B Street map model, which is a layer after osm2streets, partly based on convenience/rapid prototyping speed, and partly because the map model layer also has buildings snapped to roads. That could be helpful if we want to show amenities along a route.

But...

Ease of maintenance and contribution?

I think this is a strong argument. The osms2streets codebase is comparatively much cleaner, better documented, better tested, and less burdened with older assumptions. Things like zebra and unmarked crossings are a total mess in the map model layer, with US-specific assumptions baked pretty deeply in and crossing data tagged in OSM ignored. We'll likely want to revamp that relatively soon to understand the pedestrian experience. osm2streets is a much better place to do that.

And for buildings, there are also some old hacks (like losing any association between an OSM relation of a university campus and the 100 individual buildings in it) that'd be helpful to improve for ATIP.

Additionally, storing files with just the osm2streets StreetNetwork will be smaller. Map model files have more stuff crammed in, and we don't need it all.

So, I propose:

merge this PR as a first, already working step
immediately followup by cutting over to osm2streets instead, and moving+rewriting building, crossing, turn, etc logic there as needed

Isn't A/B Street's map model overkill for some things? A design goal for ATIP is simplicity and A/B Street's map model is quite complex I thought. As an example, all A/B Street routes are represented in 2D while ATIP routes are 1D.

osm2streets and A/B Street are the same here. Roads and routes are "either" 1D or 2D. Through the API, you can access just 1D road center-lines and glue them together into a longer route. Or you can use the road's estimated width and see more of a 2D perspective. We won't make use of that yet.

If the business logic is written in Rust wouldn't that allow us to get best of both worlds

+1, also true. I would summarize it like this:

Rust: run as an offline CLI tool, directly in the browser, or as a remote API server
anything else*: offline CLI tool or as a remote API server (some other languges are getting WASM compatibility, but probably Rust has the best support so far?)

Having 2 progress bars is not common and may look a bit complex

Agreed, and once the tool is loaded, there's not even an immediately actionable thing for the user to do with it. So actually, how about by the autofill button nested in a route's form? If the RouteInfo API isn't ready to use yet, that button can be disabled with a tooltip saying it's loaded, or we just put the progress bar there. Most people won't even see it, unless they open a route before it finishes loading.

robinlovelace-ate · 2023-05-10T09:37:55Z

So, I propose:

merge this PR as a first, already working step

immediately followup by cutting over to osm2streets instead, and moving+rewriting building, crossing, turn, etc logic there as needed

👍 to that.

robinlovelace-ate · 2023-05-10T09:39:46Z

Through the API, you can access just 1D road center-lines and glue them together into a longer route. Or you can use the road's estimated width and see more of a 2D perspective. We won't make use of that yet.

Very cool, I didn't realise that. The fact that a multi-lane road can simultaneously be represented as a 1D linestring and as a complex polygon is amazing. Great that we'll benefit from the ability to almost instantly switch between the two representations. Big 👍

robinlovelace-ate · 2023-05-10T09:41:41Z

Agreed, and once the tool is loaded, there's not even an immediately actionable thing for the user to do with it. So actually, how about by the autofill button nested in a route's form?

Sounds reasonable, let's see how it looks...

models. Add a button to v1 to calculate a name for a route automagically.

workflow, and document it

…though it's not present in every branch yet. This'll cause errors until the other branches have been rebased

dabreegster · 2023-05-10T17:56:25Z

Just noting we'll see Github actions failures until this is merged and the other pending branches are rebased. I tried making the wasm-pack step be conditional on whether the route_info directory exists (and thus we're in a branch where this new code exists and needs building), but when switching between all branches in the build script, it leaves behind some files there and falsely tries to build. Seems simplest to me to push past this and make sure everything's fine after merging, instead of making the script more complicated for a short time.

acteng/atip#154

…p#154

dabreegster · 2023-05-11T12:40:47Z

Since we discussed the high-level approach yesterday, going to merge this now and continue:

fix up the CI by rebasing all the other branches
showing a progress bar nested under routes
cutting over to osm2streets
gzipping for smaller files

Happy for async feedback as always!

dabreegster commented May 9, 2023

View reviewed changes

dabreegster requested review from Sparrow0hawk and robinlovelace-ate May 9, 2023 11:06

dabreegster added a commit to acteng/atip-data-prep that referenced this pull request May 9, 2023

Import A/B Street map data for ATIP route info. See

2f18b8d

acteng/atip#154

dabreegster mentioned this pull request May 9, 2023

Import A/B Street map data for ATIP route info. See acteng/atip-data-prep#7

Merged

dabreegster added a commit to acteng/atip-data-prep that referenced this pull request May 9, 2023

fixup! Import A/B Street map data for ATIP route info. See acteng/ati…

31d32a9

…p#154

dabreegster force-pushed the map_model branch from b4a82f5 to af9cc18 Compare May 9, 2023 16:09

robinlovelace-ate approved these changes May 9, 2023

View reviewed changes

dabreegster added 5 commits May 10, 2023 18:51

Scaffold a new WASM library to augment ATIP data from A/B Street map

5e8b82b

models. Add a button to v1 to calculate a name for a route automagically.

Use comlink to run the A/B Street helper asynchronously in a web worker

e518a09

Rename the A/B Street helper to RouteInfo, include it in the Github

eb37a3d

workflow, and document it

Use real URLs, instead of a hardcoded local file

0c0c100

Assume we're building the WASM library in the Github workflows, even …

789e73d

…though it's not present in every branch yet. This'll cause errors until the other branches have been rebased

dabreegster force-pushed the map_model branch from af9cc18 to 789e73d Compare May 10, 2023 17:53

dabreegster added a commit to acteng/atip-data-prep that referenced this pull request May 11, 2023

Import A/B Street map data for ATIP route info. See

429fc0e

acteng/atip#154

dabreegster added a commit to acteng/atip-data-prep that referenced this pull request May 11, 2023

fixup! Import A/B Street map data for ATIP route info. See acteng/ati…

0975991

…p#154

dabreegster merged commit 50103e2 into main May 11, 2023

dabreegster deleted the map_model branch May 11, 2023 12:41

dabreegster mentioned this pull request May 11, 2023

Switch route_info to using osm2streets directly, using a simple CLI t… acteng/atip-data-prep#10

Merged

This was referenced May 11, 2023

Use osm2streets for route info files, not A/B Street map models #161

Merged

New speed limit layers #169

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a new RouteInfo service, and use it to auto-fill the name of a route #154

Create a new RouteInfo service, and use it to auto-fill the name of a route #154

dabreegster commented May 9, 2023 •

edited

Loading

dabreegster left a comment

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

dabreegster May 9, 2023

robinlovelace-ate May 9, 2023

Sparrow0hawk commented May 9, 2023

dabreegster commented May 9, 2023

robinlovelace-ate commented May 9, 2023 •

edited

Loading

robinlovelace-ate commented May 9, 2023

robinlovelace-ate commented May 9, 2023

robinlovelace-ate left a comment

dabreegster commented May 10, 2023 •

edited

Loading

robinlovelace-ate commented May 10, 2023

robinlovelace-ate commented May 10, 2023

robinlovelace-ate commented May 10, 2023

dabreegster commented May 10, 2023

dabreegster commented May 11, 2023


		web_sys::console::log_1(&format!("Got {} bytes, deserializing", map_bytes.len()).into());

		let map: Map = bincode::deserialize(map_bytes).map_err(err_to_js)?;

Create a new RouteInfo service, and use it to auto-fill the name of a route #154

Create a new RouteInfo service, and use it to auto-fill the name of a route #154

Conversation

dabreegster commented May 9, 2023 • edited Loading

Technical design decisions

Future-proofing

File management

dabreegster left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sparrow0hawk commented May 9, 2023

dabreegster commented May 9, 2023

robinlovelace-ate commented May 9, 2023 • edited Loading

robinlovelace-ate commented May 9, 2023

robinlovelace-ate commented May 9, 2023

robinlovelace-ate left a comment

Choose a reason for hiding this comment

dabreegster commented May 10, 2023 • edited Loading

robinlovelace-ate commented May 10, 2023

robinlovelace-ate commented May 10, 2023

robinlovelace-ate commented May 10, 2023

dabreegster commented May 10, 2023

dabreegster commented May 11, 2023

dabreegster commented May 9, 2023 •

edited

Loading

robinlovelace-ate commented May 9, 2023 •

edited

Loading

dabreegster commented May 10, 2023 •

edited

Loading