OSM Planet update

[ivanbranco]

Hi, I noticed at this link: https://osm2rdf.cs.uni-freiburg.de/ that the timestamp for the OSM Planet is 2024-04-19T04:22:33.000Z
Yet at https://qlever.cs.uni-freiburg.de/osm-planet, in "Index Information", the timestamp is 21.03.202
Is the second date not updated or is correct as it is? Also what's the update rate of the OSM Planet, weekly I guess?

[1ec5]

In #1322, the OHM Planet dataset was recently configured to update daily based on a planet endpoint. In case the developers aren’t aware, OSM has a similar endpoint and RSS feed that always returns the latest weekly planet.

[hannahbast]

@1ec5 Thank you for the reminder. We are aware of the weekly updates for OSM Planet and our RDF dump on https://osm2rdf.cs.uni-freiburg.de is also updated weekly, as you can see on that page. We just haven't set up an automatic update of https://qlever.cs.uni-freiburg.de/osm-planet yet, and I forgot to update it manually. I will update it now and the new version should be ready by tonight.

[ivanbranco]

I don't understand something. I run this query: https://qlever.cs.uni-freiburg.de/osm-planet/qLMDVV which should return me trees with circumference=0
the Index Information says version 02.05.2024, but it returns me elements in which the circumference=0 tag was deleted in 23.04.2024, such as: https://www.openstreetmap.org/node/9050663567/history

It returns me 222 elements while they should be only around 92

[hannahbast]

@1ec5 @ivanbranco @Danysan1 @nyurik https://qlever.cs.uni-freiburg.de/osm-planet has now been updated to the latest PBF dump from https://planet.openstreetmap.org/pbf . We will soon automate this, like we already did for OHM Planet. But for now, we still want to do it manually because the GeoSPARQL part is currently in heavy development. Please ping us again on this thread, whenever we miss a new PBF dump. That being said, I have two questions:

1. We would be happy to update daily. Is there a simple way to get a daily dump of the full PBF data?

2. I wonder which of the following kinds of queries are how important (an estimated percentage for each category would be nice):

A. Queries involving only the tags, but not the geometries
B. Queries involving tags and geometries, but without visualization of the results set
C. Queries with visualization of the result set

[1ec5]

We would be happy to update daily. Is there a simple way to get a daily dump of the full PBF data?

As far as I can tell, based on this documentation, OSM only releases a full planet file once a week; otherwise, you’re expected to ingest minutely updates. But apparently there are a few daily mirrors that presumably collate these minutely updates into daily planets. I have no idea about their reliability.

I wonder which of the following kinds of queries are how important (an estimated percentage for each category would be nice):

I’ve only begun to explore QLever’s potential, so it’s hard for me to say for sure how I’d use it long-term. But I’ve been quite enamored with the geometry capabilities. Even with the existing rudimentary ogc:sfContains and ogc:sfIntersects support, QLever starts to bridge the gap between Sophox (strong on charting and data item integration, weak on geometry) and Overpass (strong on geometry and topology, weak on text processing).

Admittedly, I’ve seldom opened Petrimaps when running personal queries on QLever, but whenever I share my results with others, I’m pretty sure that’s what they look at first. For perspective, over 90% of the queries I perform in Overpass turbo make use of the Map tab; less than 10% are raw queries that use the Data tab. But if Overpass had even basic text processing support like SUBSTR(), then I probably would use the Data tab over a third of the time.

My experience may not be representative of the others in this discussion, let alone in the broader OSM community. The OSM community rarely discusses analysis tools as a kind of data consumer, on par with renderers, routers, and geocoders, so I’m less certain of what users expect.

[Danysan1]

On question 1 I agree with the previous answer, the only thing I'd like to add is that if I understand correctly "daily" in the mirrors on OSM wiki means "daily mirroring from the source" so they serve weekly, not daily. If you look inside them you will find only weekly dumps, for example on https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/planet/2024/ the last dump is currently 2024-04-22 . I believe the only choice for daily updates is keeping a DB with the planet and updating it regularly with minutely updates.

On question 2 my experience is more or less aligned with the previous answer, use case C is the most frequent followed by B and A.

[ivanbranco]

Please ping us again on this thread, whenever we miss a new PBF dump.

Seems like a new planet dump is available, could be possible to manually update it on QLever?

[hannahbast]

@ivanbranco The index version is currently the version of the TTL file, and not of the original PBF file. The TTL file from 02.05.2024 was built from the planet-latest.osm.pbf from 26.04.2024. However, I just realize by looking at https://planet.openstreetmap.org/pbf that the planet-latest.osm.pbf with date stamp 26.04.2024 is actually a snapshot of the dataset from 22.04.2024. That explains your observation but it also means that it takes OSM a whopping four days to provide a new planet-latest.osm.pbf. Three comments:

1. We are in the process of building an index for the latest planet-latest.osm.pbf, which is dated 03.05.2024 and is a snapshot of the dataset from 29.04.2024. I should be done by tonight.
2. We will make sure that, starting from the next version, the QLever UI mentions the date of the snapshot of the OSM data because that is the most interesting of the three dates (snapshot date, date of the PBF file, date of the TTL file).
3. It would be really great to get snapshots of the OSM data faster and any help would be appreciated for that. Once we have a snapshot, we can load the new dataset into QLever in about a day (the only reason it currently takes a bit longer is because we are currently working on new GeoSPARQL features and the code is still experimental).

[hannahbast]

https://qlever.cs.uni-freiburg.de/osm-planet has now been updated to the latest planet-latest.pbf.osm, which is from 03.05.2024. There are now triples for all of the following GeoSPARQL relations:

ogc:sfContains , ogc:sfCovers, ogc:sfIntersects, ogc:sfEquals, ogc:sfTouches, ogc:sfCrosses, ogc:sfOverlaps

There is also osmmeta:timestamp, which yields the time of the last change for each object according to the PBF.

[ivanbranco]

However, I just realize by looking at https://planet.openstreetmap.org/pbf that the planet-latest.osm.pbf with date stamp 26.04.2024 is actually a snapshot of the dataset from 22.04.2024

Ah, this clarifies everything! I just noticed that you added "(which is a snapshot of the OSM dataset from dd.mm.yyyy)" which resolves any potential confusion. Thanks for your work!

[hannahbast]

https://qlever.cs.uni-freiburg.de/osm-planet has now been updated to the most recent planet-240506.osm.pbf. And there are no more trees with circumference 0 :-) See https://qlever.cs.uni-freiburg.de/osm-planet/qLMDVV

[hannahbast]

https://qlever.cs.uni-freiburg.de/osm-planet has now been updated to the most recent planet-240513.osm.pbf

[1ec5]

osm-planet is still on planet-241111.osm.pbf from over a month ago, even though a TTL extract from yesterday is available on the osm2rdf site.

[hannahbast]

@1ec5 Thanks for the reminder, I am just building a QLever index for the latest version and it will be finished today, as a Christmas present :-)

[1ec5]

Thank you, merry Christmas!