Create per.py

[ssh-meister]

Script for calculation Punctuation Error Rate and related rates (correct rate, deletions rate, etc.)
@karpnv @ekmb @vsl9 @KunalDhawan

[jubick1337]

And we have a package for ASR-related metrics.

Why not move the metric there?

[karpnv]

move it from nemo/collections/common/metrics/per.py to nemo/collections/asr/metrics/

[ssh-meister]

@jubick1337 @karpnv because this metric is applicable for ASR and NLP PC models both (there is noticed in the paper that PER allows to compare prediction accuracy between PC models in general)

[itzsimpl]

move it from nemo/collections/common/metrics/per.py to nemo/collections/asr/metrics/

Note that coneptually this implementation of PER will give "useful" results only for NLP, where the input is the text that is to be punctuated and is immutable (i.e. the reference is this same text punctuated by hand; a clear ground truth to compare against).

For the ASR case, eg. with hybrid_pc models, the input is audio, and the reference is the punctuated transcription (typically obtained by hand). If recognition is flawless then PER will give the correct information (the quality of the punctuation subtask), if recognition is not flawless punctuation might be affected by WER, and PER might be affected as well (eg. a word that is typically followed by a comma gets deleted). In certain occasions the punctuation of a imperfect transcription might still be flawless from the standpoint of punctuation, but since the metric is computed agains the original reference punctuated transcription, the PER score will be affected as well. There is no simple solution to this except to manually re-punctuate the transcription obtained by the ASR task and use that as reference, which might not be viable, but is something to keep in mind when reporting or. using the PER metric.

[jubick1337]

Could you please add tests for PER here?

[jubick1337]

minor format changes, otherwise looks good to me

[ssh-meister]

Could you please add tests for PER here?

@jubick1337, I've added tests, could you please review them again?

[karpnv]

Lets move rate_punctuation.py to
https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text_eval.py by adding use_per=False option.

[ssh-meister]

Lets move rate_punctuation.py to https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text_eval.py by adding use_per=False option.

This type of modification will result in a reduction of certain features. For instance, in the speech_to_text_eval.py script, there is currently no provision for computing the supported metrics (WER/CER) separately for each sample and then saving them to the manifest. This capability could be essential when considering the use of punctuation rates (correct rate / deletions rate / etc.) as thresholds for further sample filtering (e.g. SDE).

It may be advisable to consider adding a feature to speech_to_text_eval.py to calculate only the total PER value on a dataset, but in addition to the existing rate_punctuation.py functionality, rather than replacing it.