Memory leak

[ZhiWeiCui]

Hi:

Thank you so much for providing an amazing package.

Use this package according to the method you said. When the route method is executed multiple times, it is found that the memory keeps increasing. I don't know if you have encountered this situation.

Looking forward to your reply!

[daniel-j-h]

You need to call the _destruct functions to release memory, see

• libosrmc/libosrmc/osrmc.h

  Lines 6 to 111 in 38e6af2

  	/*
  	* The following provides a high-level interface overview for libosrmc.
  	*
  	*
  	* Workflow
  	* ========
  	*
  	* The library provides access to the following services: Route, Table, Nearest, Match.
  	* These services are hidden behind the osrmc_osrm_t type which you have to create first.
  	* This can be done by constructing it from osrmc_config_t setting the .osrm extract path.
  	* Note: in the following error handling is omitted for brevity. See section Error Handling.
  	*
  	* Example:
  	*
  	* osrmc_config_t config = osrmc_config_construct(path, &error);
  	* osrmc_osrm_t osrm = osrmc_osrm_construct(config, &error);
  	*
  	* For querying a service you have to first create a service-specific parameters object.
  	* Constructing an object follows the naming convention osrmc_service_params_construct.
  	* Destructing an object follows the naming convention osrmc_service_params_destruct.
  	*
  	* You then add coordinates and service-specific setting to the parameters.
  	*
  	* Adding coordinates works across all parameters using osrmc_params_add_coordinate and the
  	* more specific osrmc_params_add_coordinate_with allowing for more control over snapping.
  	* Both functions work on the structural base type osrmc_params_t.
  	*
  	* Example:
  	*
  	* osrmc_route_params_t params = osrmc_route_params_construct(&error);
  	* osrmc_params_add_coordinate((osrmc_params_t)params, longitude, latitude, &error);
  	*
  	* There are service-specific functions for fine-tuning responses and behavior.
  	*
  	* Example:
  	*
  	* osrmc_route_params_t params = osrmc_route_params_construct(&error);
  	* osrmc_route_params_add_alternatives(params, 1);
  	*
  	* Finally, you query the service passing the parameters object and extract specific results from it.
  	*
  	* Example:
  	*
  	* response = osrmc_route(osrm, params, &error);
  	* distance = osrmc_route_response_distance(response, &error);
  	*
  	*
  	* Types
  	* =====
  	*
  	* The library provides opaque types hiding the implementation.
  	* Constructing an object of such a type follows the osrmc_type_construct naming convention.
  	* Destructing an object of such a type follows the osrmc_type_destruct naming convention.
  	*
  	* Example:
  	*
  	* config = osrmc_config_construct(path, &error);
  	* osrmc_config_destruct(config);
  	*
  	*
  	* Error handling
  	* ==============
  	*
  	* Error handling is done by passing an osrmc_error_t out parameter to functions that may fail.
  	* Only on failure: the library fills the error object from which you can get a error message via osrmc_error_message.
  	* Only on failure: you take over ownership and have to destruct the error object via osrmc_error_destruct.
  	*
  	* Example:
  	*
  	* osrmc_error_t error = NULL;
  	* params = osrmc_route_params_construct(&error);
  	* if (error) {
  	* fprintf(stderr, "Error: %s\n", osrmc_error_message(error));
  	* osrmc_error_destruct(error);
  	* return EXIT_FAILURE;
  	* }
  	*
  	*
  	* Responses and Callbacks
  	* =======================
  	*
  	* The library provides both functions returning response objects from services as well as functions taking callbacks.
  	*
  	* Response object types follow the osrmc_service_response_t naming convention.
  	* You take over ownership and have to destruct the response object via osrmc_service_response_destruct.
  	* The library provides functions for extracting data from the response objects.
  	*
  	* Example:
  	*
  	* response = osrmc_route(osrm, params, longitude, latitude, &error);
  	* distance = osrmc_route_response_distance(response, &error);
  	* osrmc_route_response_destruct(response);
  	*
  	* For functions taking callbacks, the library expects you to provide handlers, for example for waypoints.
  	* The library handles response ownership and cleanup internally, that is all you have to do is check for failure.
  	* All handlers and functions invoking handlers take user-provided void* for transparently passing through client data.
  	*
  	* Example:
  	*
  	* void my_waypoint_handler(void* data, const char* name, float longitude, float latitude) {
  	* printf("Longitude: %f, Latitude: %f\n", longitude, latitude);
  	* }
  	*
  	* osrmc_route_with(osrm, params, my_waypoint_handler, NULL, &error);
  	*
  	*/

• libosrmc/libosrmc/osrmc.h

  Lines 95 to 97 in 38e6af2

  	* response = osrmc_route(osrm, params, longitude, latitude, &error);
  	* distance = osrmc_route_response_distance(response, &error);
  	* osrmc_route_response_destruct(response);

[ZhiWeiCui]

Hi:
Thank you very much for your reply!
I used the example from GeographicaGS/libosrmc by python3. I saw this method in osrmcpy.py, but I'm not sure.

libosrmc/bindings/osrmcpy.py

Lines 153 to 157 in 38e6af2

	def __del__(_):
	if _.osrm:
	lib.osrmc_osrm_destruct(_.osrm)
	if _.config:
	lib.osrmc_config_destruct(_.config)

My test code is as follows. When I put router = OSRM(OSRM_DATASET.encode('utf-8'), contraction=True) in the for loop, the memory is stable. I don't want to instantiate OSRM in every loop based on performance. Would you like to ask if I have other options ?

from pyosrm import PyOSRM, Status
from dpsolver.Util import randomly_location
import subprocess, os, re

from osrmcpy import OSRM, Coordinate


def convert_size(size):
    if size <1024:
        return size
    elif (size >= 1024) and (size < (1024 * 1024)):
        return "%.2f KB"%(size/1024)
    elif (size >= (1024*1024)) and (size < (1024*1024*1024)):
        return "%.2f MB"%(size/(1024*1024))
    else:
        return "%.2f GB"%(size/(1024*1024*1024))


def process_info():
    pid = os.getpid()
    res = subprocess.getstatusoutput('ps aux|grep ' + str(pid))[1].split('\n')[0]

    p = re.compile(r'\s+')
    l = p.split(res)
    info = {'user': l[0],
            'pid': l[1],
            'cpu': l[2],
            'mem': l[3],
            'vsa': convert_size(int(l[4])*1024),
            'rss': convert_size(int(l[5])*1024),
            'start_time': l[6]}
    return info

# location
location_start = randomly_location(10000)
location_end = randomly_location(10000)

# osrmcpy
DATA_DIR = '/mnt/d/Data/osrm/bicycle/ch'
OSRM_DATASET = os.path.join(DATA_DIR, 'china-latest.osrm')
router = OSRM(OSRM_DATASET.encode('utf-8'), contraction=True)

# pyosrm
# router = PyOSRM(use_shared_memory=True, algorithm='CH')

for start, end in zip(location_start.values(), location_end.values()):
    result = router.route([start[::-1], end[::-1]])
    print(process_info())

[daniel-j-h]

I don't know about the geographica fork, looks like they implement some more of the api surface. Maybe ask there. The shared memory flag could be the issue here, after all these hears I don't recall anymore off the top of my head how it works but if it's using mmap under the hood there's a high chance your kernel dynamically walks the page-sized chunks and loads them into memory; try without.

…

On June 4, 2021 2:03:32 AM UTC, ZhiWei Cui ***@***.***> wrote: Hi: Thank you very much for your reply! I used the example from [GeographicaGS/libosrmc](https://github.com/GeographicaGS/libosrmc) by python3. I saw this method in [osrmcpy.py](https://github.com/daniel-j-h/libosrmc/blob/master/bindings/osrmcpy.py), but I'm not sure. https://github.com/daniel-j-h/libosrmc/blob/38e6af24c0389854eea8065235e626b608ff8b32/bindings/osrmcpy.py#L153-L157 My test code is as follows. When I put _router = OSRM(OSRM_DATASET.encode('utf-8'), contraction=True)_ in the for loop, the memory is stable. I don't want to instantiate OSRM in every loop based on performance. Would you like to ask if I have other options ? from pyosrm import PyOSRM, Status from dpsolver.Util import randomly_location import subprocess, os, re from osrmcpy import OSRM, Coordinate def convert_size(size): if size <1024: return size elif (size >= 1024) and (size < (1024 * 1024)): return "%.2f KB"%(size/1024) elif (size >= (1024*1024)) and (size < (1024*1024*1024)): return "%.2f MB"%(size/(1024*1024)) else: return "%.2f GB"%(size/(1024*1024*1024)) def process_info(): pid = os.getpid() res = subprocess.getstatusoutput('ps aux|grep ' + str(pid))[1].split('\n')[0] p = re.compile(r'\s+') l = p.split(res) info = {'user': l[0], 'pid': l[1], 'cpu': l[2], 'mem': l[3], 'vsa': convert_size(int(l[4])*1024), 'rss': convert_size(int(l[5])*1024), 'start_time': l[6]} return info # location location_start = randomly_location(10000) location_end = randomly_location(10000) # osrmcpy DATA_DIR = '/mnt/d/Data/osrm/bicycle/ch' OSRM_DATASET = os.path.join(DATA_DIR, 'china-latest.osrm') router = OSRM(OSRM_DATASET.encode('utf-8'), contraction=True) # pyosrm # router = PyOSRM(use_shared_memory=True, algorithm='CH') for start, end in zip(location_start.values(), location_end.values()): result = router.route([start[::-1], end[::-1]]) # result = router.route([start[::-1], end[::-1]]) print(process_info())

[ZhiWeiCui]

Thank you again for your suggestion, I will consider it carefully !