wrong default mapping of some Romanian diacritics

[latrau]

Environment

Debian Linux

• Tesseract Version: tesseract 4.00.00alpha

• Platform: Linux 4.15.0 SMP PREEMPT 2018 x86_64 GNU/Linux

Current Behavior:

using the ron option (Romanian):

romanian diacritics șȘțȚ are mapped into the wrong Unicode codes, namely:
Ș -> Ş=U+015E
ș -> ş=U+015F
Ț -> Ţ=U+0162
ț -> ţ=U+0163

Expected Behavior:

Ș -> Ș=U+0218
ș -> ș=U+0219
Ț -> Ț=U+021A
ț -> ț=U+021B

Suggested Fix:

edit the map accordingly;

[zdenop]

Where is input image or something that would demonstrate problem?

[latrau]

the Romanian typographical convention is that the diacritics s and t are with a comma below not with cedilla (as specified also in UNICODE Latin ext A and B).

best would be that any diacritical s or t in the -ron (Romanian) option should be mapped into Latin ext B code above; meaning that in the tesseract's ron unicharset there should be no trace of [15e ] [15f ] [162 ] or [163 ], only [218 ]-[21a ].

e.g.

the wrong mapping is everywhere once the -ron option is selected...

let me quote UNICODE 10 (chap.07) on this:

The Unicode Standard provides unambiguous representations for all of the forms, for
example, U+0219 ș latin small letter s with comma below versus U+015F ş latin
small letter s with cedilla. In modern usage, the preferred representation of Roma-
nian text is with U+0219 ș latin small letter s with comma below, while Turkish data
is represented with U+015F ş latin small letter s with cedilla.

same goes for ȘțȚ.

so option -ron means șțȚȘ [U+0218-A] with no ambiguity and should nowhere involve şŞŢţ [U+015e-f][U+0162-3].

[amitdo]

This issue is not caused by Tesseract itself. It should be moved to another repo (not sure which one).

[stweil]

I think langdata_lstm is a good one and transfer the issue.

[stweil]

@latrau, so each of the wrong characters should be replaced? Do you want to send a pull request which fixes ron.training_text, maybe also ron.singles_text and ron.wordlist?

[stweil]

@latrau, was cedilla used in historic Romanian texts? If yes, it might be a good idea to keep both forms (with cedilla for the historic characters and with comma for the modern ones).