Add support for disabling feature datasets

This change adds support for disabling datasets, such that specific files are not loaded into memory when running OSRM. This enables users to not pay the memory cost for features they do not intend to use. Initially, there are two options: - ROUTE_GEOMETRY, for disabling overview, steps, annotations and waypoints. - ROUTE_STEPS, for disabling steps only. Attempts to query features for which the datasets are disabled will lead to a DisabledDatasetException being returned.
Project-OSRM · Aug 1, 2023 · d34f259 · d34f259
1 parent 522d0f0
commit d34f259
Show file tree

Hide file tree

Showing 34 changed files with 999 additions and 198 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,7 @@
   - Changes from 5.27.1
     - Features
       - ADDED: Add support for a default_radius flag. [#6575](https://github.com/Project-OSRM/osrm-backend/pull/6575)
+      - ADDED: Add support for disabling feature datasets. [#6666](https://github.com/Project-OSRM/osrm-backend/pull/6666)
     - Build:
       - ADDED: Add CI job which builds OSRM with gcc 12. [#6455](https://github.com/Project-OSRM/osrm-backend/pull/6455)
       - CHANGED: Upgrade to clang-tidy 15. [#6439](https://github.com/Project-OSRM/osrm-backend/pull/6439)

diff --git a/docs/http.md b/docs/http.md
@@ -88,10 +88,11 @@ Every response object has a `code` property containing one of the strings below
 | `InvalidService`  | Service name is invalid.                                                         |
 | `InvalidVersion`  | Version is not found.                                                            |
 | `InvalidOptions`  | Options are invalid.                                                             |
-| `InvalidQuery`    | The query string is syntactically malformed.                                    |
+| `InvalidQuery`    | The query string is syntactically malformed.                                     |
 | `InvalidValue`    | The successfully parsed query parameters are invalid.                            |
-| `NoSegment`       | One of the supplied input coordinates could not snap to the street segment.          |
+| `NoSegment`       | One of the supplied input coordinates could not snap to the street segment.      |
 | `TooBig`          | The request size violates one of the service-specific request size restrictions. |
+| `DisabledDataset` | The request tried to access a disabled dataset.                                  |
 
 - `message` is a **optional** human-readable error message. All other status types are service-dependent.
 - In case of an error the HTTP status code will be `400`. Otherwise, the HTTP status code will be `200` and `code` will be `Ok`.
@@ -130,7 +131,7 @@ In addition to the [general options](#general-options) the following options are
 |number      |`integer >= 1` (default `1`)  |Number of nearest segments that should be returned. |
 
 As `waypoints` is a single thing, returned by that service, using it with the option `skip_waypoints` set to `true` is quite useless, but still
-possible. In that case, only the `code` field will be returned. 
+possible. In that case, only the `code` field will be returned.
 
 **Response**
 
@@ -953,11 +954,11 @@ The object is used to describe the waypoint on a route.
 
 ## Flatbuffers format
 
-The default response format is `json`, but OSRM supports binary [`flatbuffers`](https://google.github.io/flatbuffers/) format, which 
+The default response format is `json`, but OSRM supports binary [`flatbuffers`](https://google.github.io/flatbuffers/) format, which
 is much faster in serialization/deserialization, comparing to `json`.
 
 The format itself is described in message descriptors, located at `include/engine/api/flatbuffers` directory. Those descriptors could
-be compiled to provide protocol parsers in Go/Javascript/Typescript/Java/Dart/C#/Python/Lobster/Lua/Rust/PHP/Kotlin. Precompiled 
+be compiled to provide protocol parsers in Go/Javascript/Typescript/Java/Dart/C#/Python/Lobster/Lua/Rust/PHP/Kotlin. Precompiled
 protocol parser for C++ is supplied with OSRM.
 
 `Flatbuffers` format provides exactly the same data, as `json` format with a slightly different layout, which was optimized to minimize
@@ -971,7 +972,7 @@ Root object is the only object, available from a 'raw' `flatbuffers` buffer. It
 
 **Properties**
 
-- `error`: `bool` Marks response as erroneous. An erroneous response should include the `code` fieldset, all the other fields may not be present. 
+- `error`: `bool` Marks response as erroneous. An erroneous response should include the `code` fieldset, all the other fields may not be present.
 - `code`: `Error` Error description object, only present, when `error` is `true`
 - `waypoints`: `[Waypoint]` Array of `Waypoint` objects. Should present for every service call, unless `skip_waypoints` is set to `true`. Table service will put `sources` array here.
 - `routes`: `[RouteObject]` Array of `RouteObject` objects. May be empty or absent. Should present for Route/Trip/Match services call.
@@ -983,21 +984,21 @@ Contains error information.
 
 **Properties**
 
-- `code`: `string` Error code 
+- `code`: `string` Error code
 - `message`: `string` Detailed error message
 
 ### Waypoint object
 
 Almost the same as `json` Waypoint object. The following properties differ:
 
-- `location`: `Position` Same as `json` location field, but different format. 
+- `location`: `Position` Same as `json` location field, but different format.
 - `nodes`: `Uint64Pair` Same as `json` nodes field, but different format.
 
 ### RouteObject object
 
 Almost the same as `json` Route object. The following properties differ:
 
-- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats. 
+- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
 - `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format.
 - `legs`: `[Leg]` Array of `Leg` objects.
 
@@ -1012,7 +1013,7 @@ Almost the same as `json` Leg object. The following properties differ:
 
 Almost the same as `json` Step object. The following properties differ:
 
-- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats. 
+- `polyline`: `string` Same as `json` geometry.polyline or geometry.polyline6 fields. One field for both formats.
 - `coordinates`: `[Position]` Same as `json` geometry.coordinates field, but different format.
 - `maneuver`: `StepManeuver` Same as `json` maneuver field, but different format.
 
@@ -1035,7 +1036,7 @@ Almost the same as `json` Step object. The following properties differ:
 | `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction)                                                                                                                                                                                                                                  |
 | `ExitRotary`     | Describes the maneuver exiting a rotary (large named roundabout)                                                                                                                                                                                                                                                             |
 
-- `driving_side`: `bool` Ttrue stands for the left side driving. 
+- `driving_side`: `bool` Ttrue stands for the left side driving.
 - `intersections`: `[Intersection]` Same as `json` intersections field, but different format.
 
 ### Intersection object
@@ -1049,7 +1050,7 @@ Almost the same as `json` Intersection object. The following properties differ:
 
 Almost the same as `json` Lane object. The following properties differ:
 
-- `indications`: `Turn` Array of `Turn` enum values. 
+- `indications`: `Turn` Array of `Turn` enum values.
 
 | `value`                | Description                                                                                                               |
 |------------------------|---------------------------------------------------------------------------------------------------------------------------|
@@ -1089,16 +1090,16 @@ Almost the same as `json` StepManeuver object. The following properties differ:
 | `ExitRoundabout` | Describes a maneuver exiting a roundabout (usually preceded by a `roundabout` instruction) |
 | `ExitRotary`    | Describes the maneuver exiting a rotary (large named roundabout) |
 
-- `modifier`: `Turn` Maneuver turn (enum) 
+- `modifier`: `Turn` Maneuver turn (enum)
 
 ### Annotation object
 
-Exactly the same as `json` annotation object. 
+Exactly the same as `json` annotation object.
 
 
 ### Position object
 
-A point on Earth. 
+A point on Earth.
 
 ***Properties***
 - `longitute`: `float` Point's longitude

diff --git a/docs/nodejs/api.md b/docs/nodejs/api.md
@@ -31,6 +31,7 @@ var osrm = new OSRM('network.osrm');
                Old behaviour: Path to a file on disk to store the memory using mmap.  Current behaviour: setting this value is the same as setting `mmap_memory: true`.
     -   `options.mmap_memory` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)?** Map on-disk files to virtual memory addresses (mmap), rather than loading into RAM.
     -   `options.path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** The path to the `.osrm` files. This is mutually exclusive with setting {options.shared_memory} to true.
+    -   `options.disable_feature_dataset` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)?** Disables a feature dataset from being loaded into memory if not needed. Options: `ROUTE_STEPS`, `ROUTE_GEOMETRY`.
     -   `options.max_locations_trip` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in trip query (default: unlimited).
     -   `options.max_locations_viaroute` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in viaroute query (default: unlimited).
     -   `options.max_locations_distance_table` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)?** Max. locations supported in distance table query (default: unlimited).

diff --git a/features/lib/osrm_loader.js b/features/lib/osrm_loader.js
@@ -68,8 +68,9 @@ class OSRMDirectLoader extends OSRMBaseLoader {
         super(scope);
     }
 
-    load (inputFile, callback) {
-        this.inputFile = inputFile;
+    load (ctx, callback) {
+        this.inputFile = ctx.inputFile;
+        this.loaderArgs = ctx.loaderArgs;
         this.shutdown(() => {
             this.launch(callback);
         });
@@ -78,7 +79,7 @@ class OSRMDirectLoader extends OSRMBaseLoader {
     osrmUp (callback) {
         if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
 
-        const command_arguments = util.format('%s -p %d -i %s -a %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM);
+        const command_arguments = util.format('%s -p %d -i %s -a %s %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM, this.loaderArgs);
         this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => {
             if (err && err.signal !== 'SIGINT') {
                 this.child = null;
@@ -101,8 +102,9 @@ class OSRMmmapLoader extends OSRMBaseLoader {
         super(scope);
     }
 
-    load (inputFile, callback) {
-        this.inputFile = inputFile;
+    load (ctx, callback) {
+        this.inputFile = ctx.inputFile;
+        this.loaderArgs = ctx.loaderArgs;
         this.shutdown(() => {
             this.launch(callback);
         });
@@ -111,7 +113,7 @@ class OSRMmmapLoader extends OSRMBaseLoader {
     osrmUp (callback) {
         if (this.osrmIsRunning()) return callback(new Error("osrm-routed already running!"));
 
-        const command_arguments = util.format('%s -p %d -i %s -a %s --mmap', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM);
+        const command_arguments = util.format('%s -p %d -i %s -a %s --mmap %s', this.inputFile, this.scope.OSRM_PORT, this.scope.OSRM_IP, this.scope.ROUTING_ALGORITHM, this.loaderArgs);
         this.child = this.scope.runBin('osrm-routed', command_arguments, this.scope.environment, (err) => {
             if (err && err.signal !== 'SIGINT') {
                 this.child = null;
@@ -134,8 +136,9 @@ class OSRMDatastoreLoader extends OSRMBaseLoader {
         super(scope);
     }
 
-    load (inputFile, callback) {
-        this.inputFile = inputFile;
+    load (ctx, callback) {
+        this.inputFile = ctx.inputFile;
+        this.loaderArgs = ctx.loaderArgs;
 
         this.loadData((err) => {
             if (err) return callback(err);
@@ -148,7 +151,7 @@ class OSRMDatastoreLoader extends OSRMBaseLoader {
     }
 
     loadData (callback) {
-        const command_arguments = util.format('--dataset-name=%s %s', this.scope.DATASET_NAME, this.inputFile);
+        const command_arguments = util.format('--dataset-name=%s %s %s', this.scope.DATASET_NAME, this.inputFile, this.loaderArgs);
         this.scope.runBin('osrm-datastore', command_arguments, this.scope.environment, (err) => {
             if (err) return callback(new Error('*** osrm-datastore exited with ' + err.code + ': ' + err));
             callback();

diff --git a/features/options/data/disabled_dataset.feature b/features/options/data/disabled_dataset.feature
@@ -0,0 +1,141 @@
+@routing @disable-feature-dataset
+Feature: disable-feature-dataset command line options
+    Background:
+        Given the profile "testbot"
+        And the node map
+            """
+            0
+            a b c
+            """
+        And the ways
+            | nodes |
+            | ab    |
+            | bc    |
+
+    Scenario: disable-feature-dataset - geometry disabled error
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
+
+        # The default values
+        And the query options
+          | overview       | simplified |
+          | annotations    | false      |
+          | steps          | false      |
+          | skip_waypoints | false      |
+
+        When I route I should get
+            | from | to | code              |
+            | a    | c  | DisabledDataset  |
+
+        When I plan a trip I should get
+            | waypoints | code             |
+            | a,b,c     | DisabledDataset |
+
+        When I match I should get
+          | trace | code             |
+          | abc   | DisabledDataset |
+
+    Scenario: disable-feature-dataset - geometry disabled error table
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
+
+        When I request nearest I should get
+            | in | code              |
+            | 0  | DisabledDataset  |
+
+        When I request a travel time matrix with these waypoints I should get the response code
+            | waypoints  | code             |
+            | a,b,c      | DisabledDataset |
+
+
+    Scenario: disable-feature-dataset - geometry disabled success
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
+
+        # No geometry values returned
+        And the query options
+          | overview       | false |
+          | annotations    | false |
+          | steps          | false |
+          | skip_waypoints | true  |
+
+        When I route I should get
+            | from | to | code     |
+            | a    | c  | Ok       |
+
+        When I plan a trip I should get
+            | waypoints |  code |
+            | a,b,c     |  Ok   |
+
+        When I match I should get
+          | trace | code |
+          | abc   | Ok   |
+
+    Scenario: disable-feature-dataset - geometry disabled error table
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_GEOMETRY"
+
+        And the query options
+          | skip_waypoints | true  |
+
+        # You would never do this, but just to prove the point.
+        When I request nearest I should get
+            | in | code              |
+            | 0  | Ok                |
+
+        When I request a travel time matrix with these waypoints I should get the response code
+            | waypoints  | code  |
+            | a,b,c      | Ok    |
+
+
+    Scenario: disable-feature-dataset - steps disabled error
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
+
+        # Default + annotations, steps
+	And the query options
+          | overview       | simplified |
+          | annotations    | true       |
+          | steps          | true       |
+
+        When I route I should get
+            | from | to | code               |
+            | a    | c  | DisabledDataset   |
+
+        When I plan a trip I should get
+            | waypoints |  code            |
+            | a,b,c     | DisabledDataset |
+
+        When I match I should get
+          | trace | code             |
+          | abc   | DisabledDataset |
+
+
+    Scenario: disable-feature-dataset - geometry disabled error table
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
+
+        When I request nearest I should get
+            | in | code |
+            | 0  | Ok   |
+
+        When I request a travel time matrix with these waypoints I should get the response code
+            | waypoints  | code  |
+            | a,b,c      | Ok    |
+
+
+    Scenario: disable-feature-dataset - steps disabled success
+        Given the data load extra arguments "--disable-feature-dataset ROUTE_STEPS"
+
+        # Default + steps
+        And the query options
+          | overview       | simplified |
+          | annotations    | true       |
+          | steps          | false      |
+
+        When I route I should get
+            | from | to | code     |
+            | a    | c  | Ok       |
+
+        When I plan a trip I should get
+            | waypoints |  code |
+            | a,b,c     |  Ok   |
+
+        When I match I should get
+          | trace | code |
+          | abc   | Ok   |
+
diff --git a/features/step_definitions/data.js b/features/step_definitions/data.js
@@ -33,6 +33,11 @@ module.exports = function () {
         callback();
     });
 
+    this.Given(/^the data load extra arguments "(.*?)"$/, (args, callback) => {
+        this.loaderArgs = this.expandOptions(args);
+        callback();
+    });
+
     this.Given(/^a grid size of ([0-9.]+) meters$/, (meters, callback) => {
         this.setGridSize(meters);
         callback();