You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The osrm-extract task takes a while, particularly on large datasets. When used in unreliable computing environments (i.e. market-priced cloud providers), it's not uncommon for long-running tasks to get interrupted.
osrm-extract can be modified to save its progress.
This line represents approximately the halfway processing point for osrm-extract.
Through an serendipitous accident of apathy/not-enough-time, osrm-extract could re-start processing at L245 if it happens to get killed after that for some reason.
I don't think it makes sense to create checkpoints in the middle of a tool. This complicates the control flow, and breaks with expected behavior (basically the cache invalidation problem).
If we need a checkpoint we should split the tool into two parts with defined control flows.
In this specific example we could split it apart easily. But I suspect rewriting the initial OSM extraction code to be in-memory would probably make it unnecessary to do so, since we could exploit parallelism for all operations, rather then be IO bound. See #3767 for an actionable task for reducing the memory usage enough to make this feasible.
The
osrm-extract
task takes a while, particularly on large datasets. When used in unreliable computing environments (i.e. market-priced cloud providers), it's not uncommon for long-running tasks to get interrupted.osrm-extract
can be modified to save its progress.osrm-backend/src/extractor/extractor.cpp
Line 245 in c48fc58
This line represents approximately the halfway processing point for
osrm-extract
.Through an serendipitous accident of apathy/not-enough-time,
osrm-extract
could re-start processing at L245 if it happens to get killed after that for some reason./cc @miccolis
The text was updated successfully, but these errors were encountered: