Add preprocessing scripts for the librilight datasets

[HarryHe11]

✨ Description

This update introduces preprocessing scripts for the Libri-Light datasets, enhancing their usability and compatibility with our processing workflows.

🚧 Related Issues

No related issues.

👨‍💻 Changes Proposed

• Implemented the main workflow preprocessors/librilight.py for preprocessing Libri-Light datasets.
• Developed a utils/cut_by_vad.py script to segment audio files using multiprocessing (Step 1: Segmentation).
• Created an utils/mfa_prepare.py script to convert audio files to 16kHz and 16-bit PCM, and to filter out longer audio files (Steps 2 & 3: Filter and Preprocess).
• Added utils/whisper_transcription.py for audio transcriptions using distilled-whisper – a more efficient variant of Whisper, and included text preprocessing functions for these transcriptions (Steps 4 & 5: Transcription & Text-Preprocess).
• Integrated an MFA alignment function specifically tailored for Libri-Light in preprocessors/librilight.py(Step 6: Alignment).
• Enabled data splitting into train/dev/eval sets, along with statistics calculation and metadata construction for Libri-Light in preprocessors/librilight.py (Steps 7-9).
• Provided support for different subsets of Libri-Light, including "tiny", "small", "medium", and "large".

🧑‍🤝‍🧑 Who Can Review?

• @lmxue
• @RMSnow
• @HeCheng0625

🛠 TODO

• Test on Libri-Light-tiny (custom split from Libri-Light-small).
• Test on Libri-Light-small.
• Test on Libri-Light-medium.
• Test on Libri-Light-large.

✅ Checklist

• Code has been reviewed
• Code complies with the project's code standards and best practices
• Code has passed all tests
• Code does not affect the normal use of existing features
• Code has been commented properly
• Documentation has been updated (if applicable)
• Demo/checkpoint has been attached (if applicable)

[HarryHe11]

In this comment, I provide screenshots from testing the implemented scripts on Libri-Light-tiny (a custom split from Libri-Light-small).

The Running Process

preprocessors/librilight.py

Part 1 of 5:

Part 2 of 5:

Part 3 of 5:

Part 4 of 5:

Part 5 of 5:

The Outcome:

Processed Data:

MetaData:

[lmxue]

Thanks for your efforts. Please check out the comments.

[HarryHe11]

Thanks for your efforts. Please check out the comments

Thank you so much for reading my PR; I have addressed your concerns, and please see my most recent commits.

[lmxue]

LGTM.

[lmxue]

LGTM.
P.S. This PR has been tested on Libri-Light-tiny. However, three other subdatasets need to be tested as listed in TODO. You may need to test them when the dataset is ready.

[HarryHe11]

LGTM.

P.S. This PR has been tested on Libri-Light-tiny. However, three other subdatasets need to be tested as listed in TODO. You may need to test them when the dataset is ready.

sure, I test them then.