Add Musicgen

[sanchit-gandhi]

What does this PR do?

Adds the musicgen model by fairseq to transformers

This model is made of three components:

1. T5Encoder (which import as AutoModelForTextEncoding)
2. MusicgenDecoder (which we copy as much as possible from modeling_bart.py)
3. Encodec (which we import as AutoModel)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ArthurZucker]

I think it's very nicely handled! As we talked offline, cool that everyting stays in the modeling code instead of having logit processors!

[sanchit-gandhi]

Would be great to hear your thoughts on the design here @patrickvonplaten (adding the tests otherwise now)

TODO:

• convert m/l checkpoints
• handle padded tokens from encodec (in delay pattern mask, then again when we decode)
• fast tests
• integration tests
• add method for unconditional generation (no need to use processor to get input ids)
• finish docs / docstrings

[patrickvonplaten]

clean!

[patrickvonplaten]

I think the modeling design is nice. The changes to the main generate method are too model-specific IMO and I also don't think it's a good idea to create a dependency via super().generate(...).

I'd propose two things:

• 1.) Change the forward method to accept [batch_size x num_codevectors, seq_length] instead of [batch_size, num_codevectors, seq_length]. I think if we explain it nicely in the docs, there is no real disadvantage to moving num_codevectors to the batch dimension right away.
• 2.) I'd just directly call sample and greedy_search here: https://github.com/huggingface/transformers/pull/24109/files#r1232442770

[patrickvonplaten]

Interesting!

[sanchit-gandhi]

Refactored the .generate method as discussed @patrickvonplaten - would be great to hear your thoughts on this before committing to it for the final design! Will update the docstrings as required once we've settled on the design.

Overall, my thoughts are that this approach to explicitly calling greedy_search and sample works. It's not super compact, but is definitely easier to understand than the super().generate call that we had before. It will allow us to build on top with other generation methods (e.g. assisted generation) in the future, so I think it's the way to go

[patrickvonplaten]

clean!

[sanchit-gandhi]

FYI @sgugger this is the only new functionality since your review (the rest was just getting the doc tests to pass by fixing bugs in the docstrings)

As discussed offline, we want a method in the processor to strip any padding from our generated audio values. This method does exactly that, and is tested for with a new processor test.

The API looks as follows:

from transformers import AutoProcessor, MusicgenForConditionalGeneration
from datasets import load_dataset

processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")

dataset = load_dataset("sanchit-gandhi/gtzan", split="train", streaming=True)
sample = next(iter(dataset))["audio"]

inputs = processor(
    audio=sample["audio"],
    sampling_rate=sample["sampling_rate"],
    text="80s blues track with groovy saxophone",
    padding=True,
    return_tensors="pt",
)

audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)

# NEW: post-process to remove padding from the batched audio
audio_values = processor.decode_audio(audio_values, padding_mask=inputs.padding_mask)

[sgugger]

Can we do decode(text=...) or decode(audio=...) instead? This would be more in-line with how processor deal with multimodal inputs/outputs and we have something similar in call and pad already.

[sanchit-gandhi]

Resolved in c0235d3 - we now decode the audios if audio values are passed, or decode token ids with the tokenizer otherwise

I had originally used the .pad function in the feature extractor, but this actually proved to be more complicated than a hand-written pad. This is because we want to pack the padding mask to max length with the non-padding token:

1. The padding mask is constructed based on the input ids, padding left based on the length of the inputs
2. We then generate our input ids to a new max length, generating new tokens on the right always
3. The newly generated ids are all valid, so the padding mask should not pad these out, hence we pad with the non-padding token on the right side

This means that to use the .pad method, we first have to flip the attributes of the feature extractor, perform the padding, then flip them back again

1. Flip the padding token
2. Ensure the padding side is set to "right"
3. Do the padding
4. Flip the padding token back again
5. Ensure the padding side is set to the original

=> overall this was more complicated than writing two lines of padding