Can add additional parameters in calculating scores like rouge?

[hookSSi]

🚀 Feature

Add tokenizer, normalizer parameters in calculating scores like rouge

Motivation

I wish I could consider another language when calculating scores like rouge

Because of _normalize_and_tokenize_text, I can only use english.

If only this is supported as a parameter, it will be good to be able to use other languages easily.

Pitch

rouge_score(preds, target, tokenizer, normalizer)

def _normalize_and_tokenize_text(text: str, tokenizer:lambda, normalizer:lambda, stemmer: Optional[Any] = None) -> Sequence[str]:
    """Rouge score should be calculated only over lowercased words and digits. Optionally, Porter stemmer can be
    used to strip word suffixes to improve matching. The text normalization follows the implemantion from `Rouge
    score_Text Normalizition`_
    Args:
        text:
            An input sentence.
        stemmer:
            Porter stemmer instance to strip word suffixes to improve matching.
    """
    # Replace any non-alpha-numeric characters with spaces.
    if normalizer is None:
        text = re.sub(r"[^a-z0-9]+", " ", text.lower())
    text = normalizer(text)

    tokens = re.split(r"\s+", text)
    if stemmer:
        # Only stem words more than 3 characters long.
        tokens = [stemmer.stem(x) if len(x) > 3 else x for x in tokens]

    # One final check to drop any empty or invalid tokens.
    if tokenizer is None:
        tokens = [x for x in tokens if (isinstance(x, str) and re.match(r"^[a-z0-9]+$", x))]
    tokens = tokenizer(text)

    return tokens

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

cc: @stancld opinion?

[stancld]

Hi @hookSSi, I think it's a good idea. The sacrebleu implementation of BLEUScore also allows to use of different tokenization techniques depending on the target language, so feel free to open PR. Looking forward :]