Implement OpenStreetMap data source

[johan12345]

as discussed in #97, this adds support for OpenStreetMap as a data source for charging stations.

TODO:

• implement filters
  • basic filters: free parking/charging, open 24/7, min power, connectors, number of connectors
  • filter by operator or network (requires a ReferenceData implementation which saves the list of possible choices)
• make sure that we don't consume too much RAM when importing the data (currently, they are first fully loaded into memory and only then stored in the database - instead we could do this in chunks)
• Add better progress indicator while initially importing the data?
  • phone UI
  • Android Auto / AAOS
• Implement a mechanism to manually or automatically (e.g. once per week) update the data
• Try to make real-time data work (should work for chargers where OSM has data about connectors and powers)
• Ask Chargeprice whether they can add support for OSM (currently their API only supports OCM and GE - adding OSM support would probably be quite a bit of work for them, especially as OSM data may include different spellings of the same charger network/operator)
• improve representation of missing values in the UI (e.g. currently when the charging power is missing, "null" is shown - and if no details about connectors are available, the screen is pretty much empty)
  • phone UI
  • Android Auto / AAOS
• Move to production webserver
• Update description of OSM in the data source selection screen
• Implement opening hours parsing (for this we need to replace the current EVMap OpeningHours structure with the more flexible OSM format, see https://github.com/simonpoole/OpeningHoursParser)
• Check if we can speed up marker clustering by letting the Spatialite DB do the calculations
• Check out if we can get any more details from the OSM data, such as
  • the address?
  • links to photos (not sure if this is being used for chargers so far?)
  • EVSEID (may be useful for better realtime data)
  • Charger website
• Add EVMap to https://github.com/taginfo/taginfo-projects

[dbrgn]

make sure that we don't consume too much RAM when importing the data (currently, they are first fully loaded into memory and only then stored in the database - instead we could do this in chunks)

That might be something that I could contribute, I do quite a lot of network programming and this seems quite an isolated and easily testable thing.

Implement a mechanism to manually or automatically (e.g. once per week) update the data

I think initially a manual mechanism would be enough. Later, the Android JobScheduler API could be used for scheduling (e.g. once a week).

Using .setRequiredNetworkType(int) we could even run the job only when an unmetered network (e.g. home WiFi) is used.

Check out if we can get any more details from the OSM data, such as

• the address?

If addr:* fields are set, that should be fairly easy to extract. Otherwise, a reverse geocoder would need to be used. Android offers such an API (IIRC through Google Play, so might be tricky on F-Droid), but the quality isn't always good, and I'm not sure if it's worth it.

links to photos (not sure if this is being used for chargers far?)

I've never seen this used for chargers, but why not 🙂

EVSEID (may be useful for better realtime data)

Yeah, that's a great idea, since it allows uniquely identifying the charger.

[johan12345]

That might be something that I could contribute, I do quite a lot of network programming and this seems quite an isolated and easily testable thing.

cool! Currently quite a lot of the JSON parsing into an in-memory list of OSMChargingStations is abstracted away by Retrofit and Moshi. I think Moshi does have a streaming API, so we can keep using that, but we might need to drop Retrofit for this and use the underlying OkHttpClient directly.

I think initially a manual mechanism would be enough. Later, the Android JobScheduler API could be used for scheduling (e.g. once a week).

Using .setRequiredNetworkType(int) we could even run the job only when an unmetered network (e.g. home WiFi) is used.

Yep, makes sense. As it is set up now, it will already do the download on demand when the app is started and no chargers are stored in the DB (or too outdated, currently the hard limit is set to 300 days). So you could even already trigger a manual update by simply clearing the cache in the settings and then going back to the map. But intuitively an extra button to do it manually might still be useful.

We already have a scheduled job to delete outdated entries from the DB, a very similar one with the extra network type constraint could be used to re-download the OSM chargers.

If addr:* fields are set, that should be fairly easy to extract.

Yeah, let's check how many chargers have that field set to see whether it's worth implementing support for it (or, implement it anyway to encourage people to include it? 😅). We could also decide to e.g. show coordinates in the address field in case the address is not available - just so we have at least some kind of useful information in the UI at that position.

Yeah, that's a great idea, since it allows uniquely identifying the charger.

Yes - it's not a one-to-one mapping though, as one EVSEID doesn't correspond to a whole charging site, but either to a single plug, or sometimes a combination of plugs (e.g. when a charger has both Type 2 and a household plug, but only one of them can be used at a time).
Apparently OSM stores the EVSEIDs simply as a semicolon-separated list, so we would also have store them as a list in our ChargeLocation data structure (without mapping them to a specific plug and power) - on the other hand, when obtaining the realtime availability data, I do try to match the EVSEIDs to the plugs and powers, if possible.

[johan12345]

@dbrgn

If addr:* fields are set, that should be fairly easy to extract. Otherwise, a reverse geocoder would need to be used.

There seems to be another way to get at least the country & city directly from the Overpass API, see the two examples here:

• https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_API_by_Example#Adding_geocoding_details_to_POIs
• https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_API_by_Example#Adding_Georeference_details_to_village_nodes_(since_0.7.54)

This would actually be pretty useful - e.g., we use the country to decide which of the real-time data sources should be queried for a charger.

However, the wiki page mentions, at least for the second example:

Performance hint: don't run this kind of query on 1000 or 10000+ objects on the overpass-api.de instance, as it consumes too much resources.

I wonder why this is not mentioned for the first example, even though both examples do pretty similar things... Our query for all charging stations worldwide is very large, so not sure if this is feasible.

[dbrgn]

@johan12345 Hmm, that's interesting! I assume this information is extracted by using the geographical relation (it's "inside" a certain administrative boundary). However, that does indeed sound expensive to calculate for a lot of points. And by looking at the syntax (without knowing in detail how Overpass works), I think foreach.posts will result in an "n+1" query. So if we get back 20'000 charging points, that would be 20'001 queries against the database.

Additionally, I wonder how much value there is in knowing the city in which a chargepoint is. I think if there is no address associated with a charger, then the distance (or the marker on a map) would be more useful from a user point of view.

[dbrgn]

Regarding contributions: I did mean to take a look, but my calendar is totally stuffed until at least mid-October... So don't count on me, sorry. (I'm happy to test and give feedback though!)

[johan12345]

This weekend I finished some more important steps, such as adding a progress indicator for the download, making the necessary adjustments to the Android Auto/AAOS app, and setting up a production-ready webserver for the OSM data.

Now the main point that is still missing to finally get this merged (and which should not be too much work, as already mentioned above) is:

Implement a mechanism to manually or automatically (e.g. once per week) update the data

the remaining points could be handled later independently of this PR.

The data source selection screen will look like this:

I think the description is fine for now. Later, when it has been tested more thoroughly and things like opening hours have been implemented, we could change it to something like

Worldwide, with varying quality - many charging stations included, but often with less details. Community-maintained.

[dbrgn]

Very cool! Some initial feedback:

• Download started immediately after choosing OSM as provider. This should probably be done after confirmation (with a hint about the approximate size, for people withotu flatrate). Maybe initial download could be a wizard step?
• I have a 10Gbps internet uplink, so even though my phone is connected via WiFi, speed should be very fast. However, download took a few minutes. Is there some rate limit on the filehost? Or is there live processing of the downloaded data that slows things down a bit?
• When zooming/scrolling on the map while the download is in progress, the "Lade herunter..." text in the top bar flickers a bit. I think it briefly falls back to a default text. Nothing bad, but maybe it could be easily fixed?
• When viewing a large map area, loading chargers can be really slow! It took maybe 1-2 minutes on my Pixel 7a. Might be because I'm using the debug build, but I suspect some database index is missing? I also managed to trigger some ANRs when scrolling around on the map.
• I'm not sure how important automatic updates are, I think at least initially a manual update mechanism/activity (maybe with a "data is old, please update" hint) would be enough.

[johan12345]

Download started immediately after choosing OSM as provider. This should probably be done after confirmation (with a hint about the approximate size, for people withotu flatrate). Maybe initial download could be a wizard step?

Gzipped the file is still around 5 MB, so not that large for today's standards... But yeah, due to the significant time it takes to import, it might still make sense to have a confirmation dialog.

I have a 10Gbps internet uplink, so even though my phone is connected via WiFi, speed should be very fast. However, download took a few minutes. Is there some rate limit on the filehost? Or is there live processing of the downloaded data that slows things down a bit?

Exactly, the data is being parsed (from JSON into ChargeLocation objects) and then inserted into the SQLite database on the fly to avoid having to keep all chargers in memory at once. Currently this is done in chunks of 100. Probably it could be improved a bit with parallelization, i.e. one thread/coroutine parses the response and inserts items into a buffer (of limited size), while another thread/coroutine takes elements from the buffer and inserts them into the DB.

When zooming/scrolling on the map while the download is in progress, the "Lade herunter..." text in the top bar flickers a bit. I think it briefly falls back to a default text. Nothing bad, but maybe it could be easily fixed?

Yes, I'll look into that

When viewing a large map area, loading chargers can be really slow! It took maybe 1-2 minutes on my Pixel 7a. Might be because I'm using the debug build, but I suspect some database index is missing? I also managed to trigger some ANRs when scrolling around on the map.

There should at least be a spatial index, which is probably the most important when viewing a large area without many filters applied. I think (but haven't benchmarked yet) that the main problem here is that we first load all chargers in the visible map area into memory, and only then apply calculations for grouping them into clusters.
With some clever queries, we could probably let the DB do the clustering instead, so that the ORM would have much less work to do (however it seems to be harder to do this in Spatialite than, e.g., PostGIS...).
Or at least we could try to initially only load the fields that we actually need for clustering (i.e. location) and not all the charger details. Maybe that already speeds it up...

I'm not sure how important automatic updates are, I think at least initially a manual update mechanism/activity (maybe with a "data is old, please update" hint) would be enough.

Manual updating, in principle, is already possible using the "clear cache" button in settings (just maybe not that obvious to the user). 😉

[Altonss]

• EVSEID (may be useful for better realtime data)

The EVSE ID can be obtained thanks to this key in OSM: https://wiki.openstreetmap.org/wiki/Key:ref:EU:EVSE

[danieldegroot2]

Ask Chargeprice whether they can add support for OSM (currently their API only supports OCM and GE - adding OSM support would probably be quite a bit of work for them, especially as OSM data may include different spellings of the same charger network/operator)

For notable brands/networks/operators OSM uses Name Suggestion Index (NSI). This data shows up in some of the editors contributors use to improve OSM. It is open to issues/PRs.

[PinguDEV-original]

Uhm, hows the status here? Would absolutely love to see this feature be released!

[johan12345]

Hi! Sorry, I have been very busy with other things lately and didn't have much time to work on EVMap except for necessary maintenance and minor improvements. Hopefully that will improve a bit later this summer.

The main open points are still the ones in the TODO list above, but some (like parsing opening hours, Chargeprice support, and making use of EVSEIDs) can be tackled later in separate PRs.

What I still would like to implement before merging this is moving the task of marker clustering into the DB query instead of doing it with in-memory ChargeLocation objects to reduce the amount of work for the ORM when browsing the map at low zoom levels. For the other data sources, we currently use server-side clustering at low zoom levels, but that's not possible for the OSM data source as it is always loaded from the local DB.

I added a PoC SQL query for this in 107ec10, but it doesn't respect additional filters yet and is not yet used by the higher abstraction layers (ChargeLocationRepository). So that would be the next step.

[PinguDEV-original]

Is there documentation to build this? I would be absolutely happy if I could test it (even rn)

[johan12345]

sure, there is some documentation at https://github.com/ev-map/EVMap/?tab=readme-ov-file#development-setup

[PinguDEV-original]

Sorry to bother you again, but it now says I have to update my system?

[PinguDEV-original]

Ooops, I just had to use a virt automotive, then it works. Right now, it looks really good, would absolutly love to see this merged!

[johan12345]

Sorry to bother you again, but it now says I have to update my system?

You just have to choose the correct build variant - for a phone the ...normal ones and for an automotive emulator the ...automotive ones :)

[Altonss]

Just tried out this build, a few things I noticed:

• During the big initial data download, the app feels really slow. I don't know how this could be improved, but for now it is not great;
• Browsing the map feels a little bit slow (even after filtering out to see only HPCs);
• Realtime data is missing at several locations, this might be due to the poor data quality.

Otherwise it looks very promising, congrats :)