Serbian Cyrillic to Serbian Latin transliteration (and possibly vice versa?)

[Susexe]

Problem

Currently, Serbian translations in the "ingredients.txt" file are a mess. There's no standardized way of translating them to Serbian; it's a mix of Cyrillic and Latin words under the "sr" language code across different entries. There are also inconsistent language codes: "sr-la" and "sr-el" are both used for Serbian Latin, although those strings are commented. Similarly, "sr" and "sr-ec" are both used for Serbian Cyrillic.

Proposed solution

My suggestion is to only enter Serbian Cyrillic translations of ingredients, additives, etc. under the "sr" language code and create an automatic Serbian Cyrillic-to-Latin conversion. Why Cyrillic to Latin, you may ask? Well, Serbian Latin contains so-called digraphs, which make the transliteration prone to mistakes. Serbian Cyrillic has none of them: NJ is Њ, LJ is Љ, and colloqually DJ (Đ) is Ђ, which guarantees Cyrillic to Latin conversion is 100% correct. The opposite is problematic, though there are some tools that include exception dictionaries (the majority of English words would probably still be transliterated). The tool could potentially be tested for product search too, as the majority of products are in Latin and cannot be found when Cyrillic is entered in the search.

[teolemon]

We have some custom mappings in the Crowdin config we need to sort out, to see if they are helping or harming:
sr-CS: sr_CS
sr: sr_RS
https://github.com/openfoodfacts/openfoodfacts-server/blob/main/crowdin.yml

[alexgarel]

@Susexe do you think we can do a single script to transform the current taxonomies for that and then add a check on taxonomy change to avoid going back to ascii ?

[Susexe]

"CS" was the former country code for the state union of Serbia and Montenegro until it was dissolved in 2006 and Serbia and Montenegro became separate countries. Some time ago, Microsoft deprecated the codes "sr-Cyrl-CS" and "sr-Latn-CS" in favor of "sr-Cyrl-RS" and "sr-Latn-RS". I'm not sure what Crowdin uses internally, but if it's "sr-CS", then it's misleading. If you ask me, "sr" for Cyrillic (since it's the official script in Serbia) and "sr-Latn" for Latin are the way to go.

I wanted to manually update the taxonomies anyway since they're inconsistent, and my preferred script would be Cyrillic for the reasons I mentioned above.

[Susexe]

I've just realized MediaWiki is deprecating sr-ec and sr-el codes, which is great news. Google uses sr and sr-Latn.

[alexgarel]

@Susexe, yes, we sadly don't handle language variants in taxonomies, yet !

I think it's fair to consider Cyrillic here as there are more speakers writing this way.

So feel free to edit taxonomies to move entries to Cyrillic. (see https://wiki.openfoodfacts.org/Global_taxonomies and https://wiki.openfoodfacts.org/Taxonomy_Maintenance)