DreamPhysics

DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians from Video Diffusion Priors. A demo for optimizing physical parameters of dynamic 3D Gaussians via the distillation of video diffusion prior.

demo.mp4

Installation

Since we use original gaussian-splatting as a submodule, please clone this repo and then clone gaussian-splatting as follows:

git clone https://github.com/tyhuang0428/DreamPhysics
cd DreamPhysics
git clone https://github.com/graphdeco-inria/gaussian-splatting

The implementation is mainly based on PhysGaussian and threestudio. Therefore, the required packages from these two repos should be included by the following commands:

conda create -n DreamPhysics python=3.9
conda activate DreamPhysics

pip install -r requirements.txt
pip install -e gaussian-splatting/submodules/diff-gaussian-rasterization/
pip install -e gaussian-splatting/submodules/simple-knn/

Quick Start

We support using text-to-video (ModelScope) and image-to-video (Stable Video Diffusion) diffusion models to guide the optimization of physical parameters. You can refer to the following command:

# text-to-video
python ms_simulation.py --model_path <path to gs model> --prompt <input text prompt> --output_path <path to output folder> --physics_config <path to physics-related config file> --guidance_config <path to video-diffusion config file>

# image-to-video
python svd_simulation.py --model_path <path to gs model> --prompt <input image prompt> --output_path <path to output folder> --physics_config <path to physics-related config file> --guidance_config <path to video-diffusion config file>

You can download prepared models and corresponding config files, and then have a quick try:

python ms_simulation.py --model_path ./model/ficus_whitebg-trained/ --prompt "a ficus swaying in the wind" --output_path ./output_ms --physics_config ./config/physics/ficus_config.json

python svd_simulation.py --model_path ./model/ball/ --prompt ./model/ball/input.png --output_path ./output_svd --physics_config ./config/physics/ball_config.json

We will keep increasing the scale of our 3D assets. You can also load your own 3D Gaussian pre-trained models to this pipeline. For the setting details of physical configs, you can refer to PhysGaussian. Note that, to optimize Young's modulus (E), we rescale the value of E (reduced by 1e7 times, e.g., 1.0 represents 1e7). The common Young's modulus is between 1e4 and 1e7, so make sure that you set an appropriate value.

Limitation

• Only support collision and rotation of soft-body objects.
• Only support the optimization of Young's modulus (E).
• The simulation is unstable as the physical property changes.

TODO

• Add more types of physical animation.
• Implement the optimization of other physical properties.
• Increase the scale of available 3D assets.

Since it is a simple demo for optimizing physical parameters of dynamic 3D Gaussians, we are willing to discuss and improve the implementation.

Acknowledgement

This repo is built based on several open-sourced projects:

• 3D generation project threestudio and its extension for Animate124.
• WARP-based MPM solver warp-mpm and physics-based 3D Gaussian method PhysGaussian.

We also use LGM to generate 3D assets with 3D GS representation.

Citation

@article{huang2024dreamphysics,
  title={DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with Video Diffusion Priors},
  author={Huang, Tianyu and Zeng, Yihan and Li, Hui and Zuo, Wangmeng and Lau, Rynson WH},
  journal={arXiv preprint arXiv:2406.01476},
  year={2024}
}