speeding up batch processing during tests?

[emiltin]

a (wild?) idea for speeding up initial test runs -
keep osrm-extract and osrm-prepare running as background processes, so they don't have to be lauched and terminated over and over. it seems allocating stxxl space and such takes significant time.

[TheMarex]

What could work, is to allow multiple files as parameters. Should not be too hard to add.
I think background processes are generally not such a good idea for data processing tools.

[emiltin]

mulitple files would work for batch processsing, but probably not for testing, since you want to test each file after they're processed, not process everything first and then test?

[DennisOSRM]

I could make sense to use the exact workflow during testing that is used during normal operations, i.e. not to introduce a separate workflow.

[emiltin]

i agree that we should test realistic work flow.
another idea would be to combine extract and prepare into a single binary, which could either extract, prepare or do both in a single command. this could simplify the typical workflow, while also potentially reduce the allocation/deallocation required, since you do everything in a single step.

[DennisOSRM]

Don't think this is a good solution either.

[emiltin]

:-(

[DennisOSRM]

Here's a bit of background on the tool chain: The binaries were meant to separate the basic preprocessing steps from the data set. E.g. it was meant to make integration of different data sources fairly easy. It also introduces a common intermediate file format that allows to take the extracted data and use it in different tools. There are users of OSRM doing either right now.

[emiltin]

Makes sense, although the preprocessing tool could still do each step separate, but also allow you to do both in a single step, and optionally skip the intermediate files?

A downside related to testing would be that combining extract and compare would mean the tools less chance to cache intermediate test files based on what tool changed.

[kaligrafy]

Is there anyway to keep the osrm-prepare and osrm-extract opened, like osrm-routed waiting for the next request? I need to generate lots of .osrm files from ruby, but after a while, I get a fork: resource limited because it seems the processes are not killed successfuly in ruby, so it needs to reopen osrm-prepare and osrm-extract each time.

[emiltin]

you cannot keep osrm-prepare and osrm-extract open and process more files.

but the error you get sounds like you're not closing them properly. the cucumber tests that come with osrm opens and closes osrm-prepare and osrm-extract open hundreds of times quickly, without problems. see these files for how the cucumber ruby files lanches the osrm bins:
https://github.com/Project-OSRM/osrm-backend/blob/develop/features/support/run.rb
https://github.com/Project-OSRM/osrm-backend/blob/develop/features/support/launch.rb

[kaligrafy]

Well, I just verified and it is indeed the osrm-routed that is causing problem… All my Processes are killed after use in my code, which is weird.

I’ll take a look at cucumber tests to fix it. Thanks!

you cannot keep osrm-prepare and osrm-extract open and process more files.

but the error you get sounds like you're not closing them properly. the cucumber tests that come with osrm opens and closes osrm-prepare and osrm-extract open hundreds of times quickly, without problems. see this file for how the cucumber ruby files lanches the osrm bins https://github.com/Project-OSRM/osrm-backend/blob/develop/features/support/launch.rb

—
Reply to this email directly or view it on GitHub.

[kaligrafy]

I think this was missing from my code:

Process.detach(@@pid) # avoid zombie processes

you cannot keep osrm-prepare and osrm-extract open and process more files.

but the error you get sounds like you're not closing them properly. the cucumber tests that come with osrm opens and closes osrm-prepare and osrm-extract open hundreds of times quickly, without problems. see this file for how the cucumber ruby files lanches the osrm bins https://github.com/Project-OSRM/osrm-backend/blob/develop/features/support/launch.rb

—
Reply to this email directly or view it on GitHub.

[kaligrafy]

It works now with detach :-)
Thanks!

I am building a genetic algorithm that creates transit network using a OD matrix (taken from regional travel surveys). It looks very promising. The input parameters are static stops and od matrix for theses stops. Than, the algorithm creates random routes connecting these stops and evolve by minimizing operating costs and total users travel time. OSRM is used to create networks and calculate travel time for each users.

[kaligrafy]

After 8300 networks tested, I get this error from osrm-routed:
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_implboost::exception_detail::error_info_injector<boost::system::system_error >: remote_endpoint: Invalid argument

What could cause this error?

[emiltin]

sounds like this could be a bug in osrm. @DennisOSRM?

[DennisOSRM]

I have to guess here, but I have seen similar errors when the machine running osrm-routed had network issues.

[kaligrafy]

I tested the osm/osrm files separately from the algorithm and all works well, so it must be a network issue or something similar to #247

[DennisOSRM]

#247 is most probably unrelated.