fix: CogVideox train dataset _preprocess_data crop video

[glide-the]

• Removed int8 to float32 conversion (* 2.0 - 1.0) from train_transforms as it caused image overexposure.

• Added _resize_for_rectangle_crop function to enable video cropping functionality. The cropping mode can be configured via video_reshape_mode, supporting options: ['center', 'random', 'none'].

@sayakpaul

[sayakpaul]

Cc: @a-r-r-o-w

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

I am not sure why we are making the tensors to range [0, 1], instead of [-1, 1]. In the original codebase, we convert to [-1, 1] as well here if I understand correctly, yes?

[glide-the]

I am not sure why we are making the tensors to range [0, 1], instead of [-1, 1]. In the original codebase, we convert to [-1, 1] as well here if I understand correctly, yes?

You're right, it should be in the [-1, 1] range. In fact, this is for matrix calculations during fine-tuning, and the [-1, 1] range is easier for computation. I forgot that this step is handled in the latten2img process, so the image is in the [0, 1] range, while the latent space is in the [-1, 1] range.

I've already verified that the cause of the training result showing a blank screen is that I input a 960x720 image into the dataset, and it was compressed to a 460x720 image for training directly.

[glide-the]

https://github.com/THUDM/CogVideo/blob/111756a6a68a8df375ef9c31f9f325818699dfaa/sat/data_video.py#L437
The number 127.5 may experience precision loss during division operations.

encode : images / 255.0 * 2.0 - 1.0
decode: (images / 2 + 0.5).clamp(0, 1)

encode : (frames - 127.5) / 127.5
decode: (images / 2 + 0.5).clamp(0, 1)

[a-r-r-o-w]

Thanks, I have one question. Apart from that, this looks great! Could you also run make style?

cc @yiyixuxu if we want the ipynb notebook here or not

[a-r-r-o-w]

Thanks for the improvements! Have you verified training on the new settings? I think it would be good to default --video_reshape_mode to maintain compatibility with existing *.sh scripts for launching training that others might have locally setup.

Could you host the ipynb notebook on https://gist.github.com/ instead of here, and link to it instead? We try to limit notebooks, pngs/mp4s/gifs and other files in the repo otherwise it can soon compound to a bulky clone.

I think make style is also needed to quality tests

[glide-the]

Thanks for the improvements! Have you verified training on the new settings? I think it would be good to default --video_reshape_mode to maintain compatibility with existing *.sh scripts for launching training that others might have locally setup.

Could you host the ipynb notebook on https://gist.github.com/ instead of here, and link to it instead? We try to limit notebooks, pngs/mp4s/gifs and other files in the repo otherwise it can soon compound to a bulky clone.

I think make style is also needed to quality tests

move video_fix_rgb_float_and_crop.ipynb to https://gist.github.com/glide-the/7658dbfd5f555be0a1a687a4139dba40