Skip to content

Latest commit

 

History

History
155 lines (109 loc) · 5.87 KB

data_process.md

File metadata and controls

155 lines (109 loc) · 5.87 KB
Raw

💃 SMPL & Rendering

Try Champ with your dance videos! It may take time to setup the environment, follow the instruction step by step🐢, report issue when necessary.

Notice that it has been tested only on Linux. Windows user may encounter some environment issues for pyrender.

Install dependencies

  1. Install 4D-Humans

    git clone https://github.com/shubham-goel/4D-Humans.git
conda create --name 4D-humans python=3.10
conda activate 4D-humans
pip install -e 4D-Humans

    or you can install via pip by a simple command

    pip install git+https://github.com/shubham-goel/4D-Humans

  2. Install detectron2

    gcc and g++ 12 is necessary to build detectron2

    conda install -c conda-forge gcc=12 gxx=12

    Then

    git clone https://github.com/facebookresearch/detectron2

pip install -e detectron2

    or you can install via pip by a simple command

    pip install git+https://github.com/facebookresearch/detectron2

  3. Install Blender

    You can download Blender 3.x version for your operation system from this url https://download.blender.org/release/Blender3.6.

Download models

  1. DWPose for controlnet

    First, you need to download our Pose model dw-ll_ucoco_384.onnx (baidu, google) and Det model yolox_l.onnx (baidu, google), then put them into ${PROJECT_ROOT}/annotator/ckpts/.

  2. HMR2 checkpoints

    python -m scripts.pretrained_models.download --hmr2

  3. Detectron2 model

    python -m scripts.pretrained_models.download --detectron2

  4. SMPL model

    Please download the SMPL model from the official site https://smpl.is.tue.mpg.de/download.php. Then move the .pkl model to 4D-Humans/data:

    mkdir -p 4D-Humans/data/
mv basicModel_neutral_lbs_10_207_0_v1.0.0.pkl 4D-Humans/data/

Produce motion data

  1. Prepare video

    Prepare a "dancing" video, and use ffmpeg to split it into frame images:

    mkdir -p driving_videos/Video_1/images
ffmpeg -i your_video_file.mp4 -c:v png driving_videos/Video_1/images/%04d.png

  2. Fit SMPL

    Make sure you have splitted the video into frames and organized the image files as below:

    |-- driving_videos
    |-- your_video_1
        |-- images
            |-- 0000.png
                ...
            |-- 0020.png
                ...
    |-- your_video_2
        |-- images
            |-- 0000.png
                ...
    ...

|-- reference_imgs
    |-- images
        |-- your_ref_img_A.png
        |-- your_ref_img_B.png
                ...

    Then run script below to fit SMPL on reference images and driving videos:

    python -m scripts.data_processors.smpl.generate_smpls --reference_imgs_folder reference_imgs --driving_video_path driving_videos/your_video_1 --device YOUR_GPU_ID

    Once finished, you can check reference_imgs/visualized_imgs to see the overlay results. To better fit some extreme figures, you may also append --figure_scale to manually change the figure(or shape) of predicted SMPL, from -10(extreme fat) to 10(extreme slim).

  3. Smooth SMPL

    blender --background --python scripts/data_processors/smpl/smooth_smpls.py --smpls_group_path driving_videos/your_video_1/smpl_results/smpls_group.npz --smoothed_result_path driving_videos/your_video_1/smpl_results/smpls_group.npz

    Ignore the warning message like unknown argument printed by Blender. There is also a user-friendlty CEB Blender Add-on to help you visualize it.

  4. Transfer SMPL

    python -m scripts.data_processors.smpl.smpl_transfer --reference_path reference_imgs/smpl_results/your_ref_img_A.npy --driving_path driving_videos/your_video_1 --output_folder transferd_result --figure_transfer --view_transfer

    Append --figure_transfer when you want the result matches the reference SMPL's figure, and --view_transfer to transform the driving SMPL onto reference image's camera space.

  5. Render SMPL via Blender

    blender scripts/data_processors/smpl/blend/smpl_rendering.blend --background --python scripts/data_processors/smpl/render_condition_maps.py --driving_path transferd_result/smpl_results --reference_path reference_imgs/images/your_ref_img_A.png

    This will rendering in CPU on default. Append --device YOUR_GPU_ID to select a GPU for rendering. It will skip the exsiting rendered frames under the transferd_result. Keep it in mind when you want to overwrite with new rendering results. Ignore the warning message like unknown argument printed by Blender.

  6. Render DWPose Clone DWPose

    DWPose is required by scripts/data_processors/dwpose/generate_dwpose.py. You need clone this repo to the specific directory DWPose by command below:

    git clone https://github.com/IDEA-Research/DWPose.git DWPose
conda activate champ

    Then

    python -m scripts.data_processors.dwpose.generate_dwpose --input transferd_result/normal --output transferd_result/dwpose

Now, the transferd_result is prepared to be used in Champ🥳!