Cogvideox-5B Model adapter change

[zRzRzRzRzRzRzR]

What does this PR do?

This is the draft of PR CogVideoX-5B (CogVideoX-Pro), including:

1. using get_3d_rotary_pos_embed
2. convert_transformer need to remove some parm
3. using with CogVideoXAttnProcessor2_0 with image_rotary_emb

This is still a draft and need to do more adaptation for run

@a-r-r-o-w

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

thanks!
I left some comments, but code changes look good to me to merge if we are in a hurry (can do some follow-up refractory)

[yiyixuxu]

let's refactor with get_1d_rotary_pos_embeds (

diffusers/src/diffusers/models/embeddings.py

Line 441 in dc07fc2

def get_1d_rotary_pos_embed(

) (in a follow-up PR )

[yiyixuxu]

ok but I'm wondering why upcasting will cause not able to generate videos for cogvideox
maybe we can test out not downcast rotary embedding instead (all our other pipelines keep it in float32)

[a-r-r-o-w]

I mainly added this to numerically match the output of apply_rotary_emb with original implementation. It seems that even if we compute in float32, the results are great and the small diff does not impact quality, now that other bugs have been resolved. We can undo this change

[a-r-r-o-w]

@yiyixuxu @zRzRzRzRzRzRzR Would it be okay to remove the following limit in the follow-up PR?

diffusers/src/diffusers/pipelines/cogvideo/pipeline_cogvideox.py

Lines 527 to 530 in dc07fc2

	if num_frames > 49:
	raise ValueError(
	"The number of frames must be less than 49 for now due to static positional embeddings. This will be updated in the future to remove this limitation."
	)

I tested the 5B-model with generations of 57, 65, and 73 frames and they all turn out good - maybe the RoPE embeddings help the model generalize better. For the 2B-model, the outputs are bad for the above values probably due to limitations of normal PE. We could add a recommendation in docs mentioning that 49 frames and below is the good setting for 2B.

In the refactor, I'd also like to create the normal positional embeddings in the pipeline instead of the transformer, similar to rope embeds, because it does not make sense to create them for the 5B model (currently they are created and saved to the module with a call to register_buffer regardless of whether 2B or 5B).

[yiyixuxu]

@a-r-r-o-w

if we're going to undo the change about upcast in apply_rotary_embedding I think it is better to not downcast freqs_cos and freqs_sin here too
i.e. change to freqs_cos = freqs_cos.to(device=device) here

otherwise I think the rotary embedding is created in float32, downcasted to bfloat16 here, and then upcast to float32 when applying - this cannot be good even if it does not have a noticeable impact

can you test it out?

[a-r-r-o-w]

Ah sorry, I overlooked this. Testing

[a-r-r-o-w]

Also just noticed that apply_rotary_emb moves the embeddings to the correct device, so we can just remove these 2 lines entirely