Grammar support for Russian way names #102
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| ## Grammar support | ||
|
|
||
| Many languages - all Slavic (Russian, Ukrainian, Polish, Bulgarian, etc), Finnic (Finnish, Estonian) and others - have [grammatical case feature](https://en.wikipedia.org/wiki/Grammatical_case) that could be supported in OSRM Text Instructions too. | ||
| Originally street names are being inserted into instructions as they're in OSM map - in [nominative case](https://en.wikipedia.org/wiki/Nominative_case). | ||
| To be grammatically correct, street names should be changed according to target language rules and instruction context before insertion. | ||
|
|
||
| Actually grammatical case applying is not the simple and obvious task due to real-life languages complexity. | ||
| It even looks so hard so, for example, all known native Russian navigation systems don't speak street names in their pronounceable route instructions at all. | ||
|
|
||
| But fortunately street names have restricted lexicon and naming rules and so this task could be relatively easily solved for this particular case. | ||
|
|
||
| ### Implementation details | ||
|
|
||
| The quite universal and simplier solution is the changing street names with the prepared set of regular expressions grouped by required grammatical case. | ||
| The required grammatical case should be specified right in instruction's substitution variables: | ||
|
|
||
| - `{way_name}` and `{rotary_name}` variables in translated instructions should be appended with required grammar case name after colon: `{way_name:accusative}` for example | ||
| - [languages/grammar](languages/grammar/) folder should contain language-specific JSON file with regular expressions for specified grammar case: | ||
| ```json | ||
| { | ||
| "v5": { | ||
| "accusative": [ | ||
| ["^ (\\S+)ая-(\\S+)ая [Уу]лица ", " $1ую-$2ую улицу "], | ||
| ["^ (\\S+)ая [Уу]лица ", " $1ую улицу "], | ||
| ... | ||
| ``` | ||
| - All such JSON files should be registered in common [languages.js](languages.js) | ||
| - Instruction text formatter ([index.js](index.js) in this module) should: | ||
| - check `{way_name}` and `{rotary_name}` variables for optional grammar case after colon: `{way_name:accusative}` | ||
|
There was a problem hiding this comment. I like this syntax – it leaves open the possibility of additional transformations like capitalization, which could replace the "capitalizeFirstLetter" meta property currently set on some localizations. |
||
| - find appropriate regular expressions block for target language and specified grammar case | ||
| - call standard [string replace with regular expression](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace) for each expression in block passing result from previous call to the next; the first call should enclose original street name with whitespaces to make parsing words in names a bit simplier. | ||
| - Strings replacement with regular expression is available in almost all other programming language and so this should not be the problem for other code used OSRM Text Instructions' data only. | ||
| - If there is no regular expression matched source name (that's for names from foreign country for example), original name is returned without changes. This is also expected behavior of standard [string replace with regular expression](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace). And the same behavior is expected in case of missing grammar JSON file or grammar case inside it. | ||
|
|
||
| ### Example | ||
|
|
||
| Russian _"Большая Монетная улица"_ street from St Petersburg (_Big Monetary Street_ in rough translation) after processing with [Russian grammar rules](languages/grammar/ru.json) will look in following instructions as: | ||
| - _"Turn left onto `{way_name}`"_ => `ru`:_"Поверните налево на `{way_name:accusative}`"_ => _"Поверните налево на Большую Монетную улицу"_ | ||
| - _"Continue onto `{way_name}`"_ => `ru`:_"Продолжите движение по `{way_name:dative}`"_ => _"Продолжите движение по Большой Монетной улице"_ | ||
| - _"Make a U-turn onto `{way_name}` at the end of the road"_ => `ru`:_"Развернитесь в конце `{way_name:genitive}`"_ => _"Развернитесь в конце Большой Монетной улицы"_ | ||
| - _"Make a U-turn onto `{way_name}`"_ => `ru`:_"Развернитесь на `{way_name:prepositional}`"_ => _"Развернитесь на Большой Монетной улице"_ | ||
|
|
||
| ### Design goals | ||
|
|
||
| - __Cross platform__ - uses the same data-driven approach as OSRM Text Instructions | ||
| - __Test suite__ - has [prepared test](test/grammar_tests.js) to check available expressions automatically and has easily extendable language-specific names testing pattern | ||
| - __Customization__ - could be easily extended for other languages with adding new regular expressions blocks into [grammar support](languages/grammar/) folder and modifying `{way_name}` and other variables in translated instructions only with necessary grammatical case labels | ||
|
|
||
| ### Notes | ||
|
|
||
| - Russian regular expressions are based on [Garmin Russian TTS voices update](https://github.com/yuryleb/garmin-russian-tts-voices) project; see [file with regular expressions to apply to source text before pronouncing by TTS](https://github.com/yuryleb/garmin-russian-tts-voices/blob/master/src/Pycckuu__Milena%202.10/RULESET.TXT). | ||
|
There was a problem hiding this comment. Thank you for dedicating this rule set to the public domain! |
||
| - There is another grammar-supporting module - [jquery.i18n](https://github.com/wikimedia/jquery.i18n) - but unfortunately it has very poor implementation in part of grammatical case applying and is supposed to work with single words only. | ||
| - Actually it would be great to get street names also in target language not from default OSM `name` only - there are several multi-lingual countries supporting several `name:<lang>` names for streets. But this the subject to address to [OSRM engine](https://github.com/Project-OSRM/osrm-backend) first. | ||
|
There was a problem hiding this comment. This is a good idea – please file an issue in osrm-backend. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| var languages = require('./languages'); | ||
| var instructions = languages.instructions; | ||
| var grammars = languages.grammars; | ||
|
|
||
| module.exports = function(version, _options) { | ||
| var opts = {}; | ||
|
|
@@ -104,7 +105,6 @@ module.exports = function(version, _options) { | |
| switch (type) { | ||
| case 'use lane': | ||
| laneInstruction = instructions[language][version].constants.lanes[this.laneConfig(step)]; | ||
| if (!laneInstruction) { | ||
| // If the lane combination is not found, default to continue straight | ||
| instructionObject = instructions[language][version]['use lane'].no_lanes; | ||
|
|
@@ -136,8 +136,9 @@ module.exports = function(version, _options) { | |
| // if both are the same we assume that there used to be an empty name, with the ref being filled in for it | ||
| // we only need to retain the ref then | ||
| name = ''; | ||
| } else { | ||
| name = name.replace(' (' + step.ref + ')', ''); | ||
| } | ||
|
|
||
| // In attempt to avoid using the highway name of a way, | ||
| // check and see if the step has a class which should signal | ||
|
|
@@ -199,10 +200,36 @@ module.exports = function(version, _options) { | |
|
|
||
| return this.tokenize(language, instruction, replaceTokens); | ||
| }, | ||
| grammarize: function(language, name, grammar) { | ||
| // Process way/rotary name with applying grammar rules if any | ||
| if (name && grammar && grammars && grammars[language] && grammars[language][version]) { | ||
| var rules = grammars[language][version][grammar]; | ||
| if (rules) { | ||
| // Pass original name to rules' regular expressions enclosed with spaces for simplier parsing | ||
| var n = ' ' + name + ' '; | ||
|
There was a problem hiding this comment. It's currently possible for clients to manipulate a step's name before compilation. For example, the Mapbox Directions API currently inserts SSML markup around numbers within the name so that speech synthesizers pronounce them more casually. I think that could potentially interfere with this feature. That would be a good argument for implementing #52, so that clients can reliably manipulate the name after grammaticalization. There was a problem hiding this comment. No problem, if clients will not damage There was a problem hiding this comment. But actually it's no problem, even if Mapbox will catch and replace There was a problem hiding this comment. I’m referring to a JavaScript port of mapbox/mapbox-navigation-ios#552 for the Directions API’s (not yet documented) I’ll be the first to admit it’s a hack, but I wanted to point this out because clients may be inclined to do similar things to the value of /cc @allierowan |
||
| rules.forEach(function(rule) { | ||
| var re = new RegExp(rule[0]); | ||
|
There was a problem hiding this comment. Should these rules be case insensitive? There was a problem hiding this comment. Actually no (and no for But for general case it looks useful to add some |
||
| n = n.replace(re, rule[1]); | ||
| }); | ||
|
|
||
| return n.trim(); | ||
| } | ||
| } | ||
|
|
||
| return name; | ||
| }, | ||
| tokenize: function(language, instruction, tokens) { | ||
| // Keep this function context to use in inline function below (no arrow functions in ES4) | ||
| var that = this; | ||
| var output = instruction.replace(/\{(\w+):?(\w+)?\}/g, function(token, tag, grammar) { | ||
| var name = tokens[tag]; | ||
| if (typeof name !== 'undefined') { | ||
| return that.grammarize(language, name, grammar); | ||
| } | ||
|
|
||
| // Return unknown token unchanged | ||
| return token; | ||
| }) | ||
| .replace(/ {2}/g, ' '); // remove excess spaces | ||
|
|
||
| if (instructions[language].meta.capitalizeFirstLetter) { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are the various cases always regular and easy to determine from the nominative case? I noticed a few features have been tagged
name:dativein OpenStreetMap – would it be desirable or necessary for OSRM to pass such tags along to OSRM Text Instructions?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMHO it's not deal of OSM - it's impossible to store all case variants in all languages for each name. Perhaps CLDR will be the better place (if will 😉 )