adding whisper large peft+int8 training example

[pacman100]

What does this PR do?

1. Fixes [Whisper] fine-tune Whisper with int-8 (scripts + example notebook) #87 - adding whisper large peft+int8 training example notebook.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[younesbelkada]

This looks amazing! 🔥 Looks excellent to me!

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

sayakpaul commented on 2023-02-16T10:25:22Z
----------------------------------------------------------------

Add a small introduction section and club the code cells before datasets import under a section called "Initial setup".

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

sayakpaul commented on 2023-02-16T10:25:23Z
----------------------------------------------------------------

Better to define openai/whisper-large-v2 in a variable like model_ckpt and then reuse it throughout. Reduces the cognitive load.

Also, for models that include multiple modalities like this one, we usually maintain a standalone processor instead of separate feature extractors and tokenizers. I think we should use WhisperProcessor (https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperProcessor) here, no? An example of using the processor of a multimodal model is available here: https://huggingface.co/docs/transformers/main/tasks/image_captioning

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

sayakpaul commented on 2023-02-16T10:25:25Z
----------------------------------------------------------------

Line #7.    class DataCollatorSpeechSeq2SeqWithPadding:

Maybe this could later go into transformers like the other data collators we have.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

sayakpaul commented on 2023-02-16T10:25:27Z
----------------------------------------------------------------

🔥

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

sayakpaul commented on 2023-02-16T10:25:28Z
----------------------------------------------------------------

Maybe add a sentence drawing the reader's attention to the fact that we're ONLY training 1% of the total model params.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

sayakpaul commented on 2023-02-16T10:25:29Z
----------------------------------------------------------------

Put Seq2SeqTrainingArguments in Seq2SeqTrainingArguments? 

[sayakpaul]

तुम्ही खूप चांगले काम केले! 🔥

Maybe just format the code with something like jupyter-black so that the code reads more beautiful?