[WIP] additionnal lib/cleanup for French language to improve quality of inputs

[CapitainFlam]

better french input for better french output ?

Hi !
⚠️ Firsts commits in GitHub, first pull request ever in a repo, so please neither shoot or shout at me 😅 ⚠️

This commit comes from different discussion, that I propose you to have a look to understand the context.

• Rejet des abbréviations commonvoice-fr#21 (...This issue should be partially solved with this pull)
• setup basic FR preprocessing CorporaCreator#87 (...This is inspirational commit that I took some ideas from)

and it is somehow connected to :

• Corrections diverses sur Corpora Creator commonvoice-fr#20
• https://commonvoice.mozilla.org/fr/criteria
• https://discourse.mozilla.org/t/rfc-criteres-dacceptation-pour-la-validation-denregistrements/73063/12
• https://discourse.mozilla.org/t/proportion-de-noms-propres-a-consonnance-etrangere/72887/26
• Remove sentences containing foreign-language proper nouns / last names common-voice#3786
• [BUG] Words and dates are missing in french written sentences common-voice#3792
• Scrape sentences from wikidata cv-sentence-extractor#92

TL;DR:
I'm trying to improve input quality for french sentence collector, to avoid
A/ garbage IN - garbage OUT, and
B/ avoid to remove in the end (in the CorporaCreator) the garbage that went through collecting, recording, review... to finally being dropped and not being included in the final release for training batchs. Better to remove it as soon as sentence collector.

What have you done ? 😱

To do so, I duplicated a EN to FR file in sentence collector, and modified it according to a previous job made by Nicolas Panel for CorporaCreator (and sadly not commited).

According to discussion (links here... Did you follow it from the list above ?!), it's recommanded that I create a WIP pull request, to allow everyone to comment, throw tomatos and/or additionnal commits to it.

footnote : it can be hard (...it IS hard !!!) to understand REGEX (REGular EXpressions). Do not hesitate to catch up with https://regex101.com/ to understand and test it.

[drzraf]

You could use a dedicated function like
https://stackoverflow.com/a/9083076 in combination with passing a callback to replace() (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace)

• You could drop the test for the s. suffix.
• It would be reusable for the non-ordinal case you handled below

[CapitainFlam]

well,

First,

I have my regex to work with :
-- kudos to gregory-r in this post in stackoverflow, licence CC BY-SA

function deromanize(roman) {
  var r = 0;
  // regular expressions to check if valid Roman Number.
  if (!/^M*(?:D?C{0,3}|C[MD])(?:L?X{0,3}|X[CL])(?:V?I{0,3}|I[XV])$/.test(roman))
    throw new Error('Invalid Roman Numeral.');

  roman.replace(/[MDLV]|C[MD]?|X[CL]?|I[XV]?/g, function(i) {
    r += {M:1000, CM:900, D:500, CD:400, C:100, XC:90, L:50, XL:40, X:10, IX:9, V:5, IV:4, I:1}[i]; 
  });

  return r;
}

So, I'll put it here to not forget it, because :

Second,

I don't have a clue of how to implement a function in JS ! (I know Pascal and VB ✌️ , but never learned JS 😭... I have to jump in the rabbit hole 🕳️ 🐇 (it is this way) )

Edit: My feeling about that ? "Why JS? because!" xkcd explined it well.

...To however follow, either the white rabbit, or crumbs, including future me, well, this link
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace#specifying_a_function_as_the_replacement
could be a good start to look at.

/me : jumps in the 🕳️

[MichaelKohler]

before you jump into that rabbit hole, you might want to check out my other comment regarding the roman numerals, as I don't think we need this at all. I'll also have a look at your other issue and add the documentation to the README for better understanding

[CapitainFlam]

Indeed, I don't understand that, collecting sentences, we can NOT "correct and clean" before "validation".

no problem, I'll wait for the issue #636 😸

[drzraf]

Maybe worth using a library like this one or this one (although these are not actually handle ordinal but be useful for plain numbers)

That may help other languages too.

[CapitainFlam]

I actually don't know how to import external library, I think it's a bit out of my league right now.

[drzraf]

If the last character was a ? or a ! I don't think it should be removed. It serves for the intonation purpose and isn't part of the acronym.

[CapitainFlam]

Est ce que le commit aef4284 répond au problème ?

EDIT : sorry, replied in FR. Translation : does the commit answer the issue ?

[drzraf]

Not sure about this one. This could very well be a date. Il est né le 25/12 qui ne devrait pas être remplacé de la sorte et prend priorité sur la regexp suivante (date format mm/yy)

[CapitainFlam]

oh shoot ! you're right !
To solve it, we shall test numbers, and see "if it's in date [1..31]/[1..12] (dd/mm) or [1..12]/[00..99] or [1..12]/[1700..2030] (mm/yy(yy)?) range, it shall be a date stamp, otherwise it's a fraction ?!"
Wow! It escalated quickly!
Sometimes, only context will show us that "mettre 1/4 de litre de lait" will go to "mettre un avril de litre de lait"... But it shall be more readable than "mettre de litre de lait".
On the other hand, if we manage poolry date (not checking mm/yy but only mm/yyyy), it will go like : "il est né en 12/88." to "il est né en douze sur quatre-vingt huit." instead of "il est née en décembre quatre-vingt huit."... Again, it's more readable than ""il est né en."

IMHO, [1..31]/[1..12] (dd/mm) or [1..12]/[1700..2030] (mm/yyyy) range shall work fine. The reste will go through 'fraction replacement', and will always be better that 'full removal'.

[CapitainFlam]

Let's first wait for issue #636 resolution, 'coz I don't want to work for nothing :-/

[drzraf]

Given the impact of creating blank in the middle of a sentence after the replacement of a date, I'm a proponent of either:

• omitting these sentence altogether
• making deep replacement (spelling date, month, year) which may or may not be practicable.

[drzraf]

­I suggested substituting "–" (long hyphen) with the common ("-") counterpart and testing against it.

[CapitainFlam]

commit 18fdf37

[drzraf]

Aren't we missing something about the french apostrophe (´)? I don't know how the model would benefit (or not) from this situation. I'm pretty sure text processing would do the uniformisation (before TTS for example) so I believe we should make it too.

[CapitainFlam]

commit 1a0cff5

[CapitainFlam]

BTW, sources files like https://github.com/common-voice/commonvoice-fr/blob/master/CommonVoice-Data/data/debats-assemblee-nationale/20130718093000000.txt are using french apostrophe ?!
(it only shows in "ui-monospace" code font, not the usual "Segoe UI" font.

line 2 : J’appelle maintenant, dans le texte de la commission, les articles du projet de loi.
if I change it myself : 
line 2 : J'appelle maintenant, dans le texte de la commission, les articles du projet de loi.

[CapitainFlam]

so... commit 8d7d973

[drzraf]

Side note: Acronyms are a strange case : They are actually spelled as distincts letters (except for NUPES).

But we don't want "STT" to process R.S.A. as erre et ça but we currently have nothing to train it that way.

Should we expect a specific future dictionary or just not strip some of them in the first place (but remove the dots)?

[MichaelKohler]

⚠️ Firsts commits in GitHub, first pull request ever in a repo, so please neither shoot or shout at me 😅 ⚠️

No worries, you're doing fine :) Thanks for your contributions!

I have a few comments, but generally I think it's a good thing to update these files.

[MichaelKohler]

I think we can remove all of the roman numerals, as they are not allowed from the beginning (see the abbreviation pattern in the validation file).

[MichaelKohler]

These can be removed, as numbers are not allowed anyway in the Sentence Collector.

[MichaelKohler]

These can be removed, as abbreviations are not allowed in the validation file.

[CapitainFlam]

Well, I think I understand wrongly how it works. Let's solve this #636 issue first 😸

[MichaelKohler]

Numbers are not allowed by the validation file.

[CapitainFlam]

Shouldn't we keep it for "future-proof rules" and/or "bulk load rules if you don't have yours" ?
Let's discuss it here : https://discourse.mozilla.org/t/sentence-collector-cleanup-before-export-vs-cleanup-on-upload/105411/15

[MichaelKohler]

Numbers are not allowed by the validation file.

[MichaelKohler]

Also note https://discourse.mozilla.org/t/sentence-collector-cleanup-before-export-vs-cleanup-on-upload/105411, though that probably doesn't matter too much for this here :)

[drzraf]

Any hope to get this in, in one shape or another?

[MichaelKohler]

The Sentence Collector has now been integrated into the Common Voice platform. Therefore I'm archiving this project here. The validation files now live here and I'm sure it would still benefit from the validation rules being added there: https://github.com/common-voice/common-voice/tree/main/server/src/core/sentences/validation. Unfortunately moving this PR over there is way harder than manually recreating it. Would you mind creating a new PR for this? Sorry for the troubles.