In recent years, there has been rapid development in 3D generation models, opening up new possibilities for applications such as simulating the dynamic movements of 3D objects and customizing their behaviors. However, current 3D generative models tend to focus only on surface features such as color and shape, neglecting the inherent physical properties that govern the behavior of objects in the real world. To accurately simulate physics-aligned dynamics, it is essential to predict the physical properties of materials and incorporate them into the behavior prediction process. Nonetheless, predicting the diverse materials of real-world objects is still challenging due to the complex nature of their physical attributes. In this paper, we propose Physics3D, a novel method for learning various physical properties of 3D objects through a video diffusion model. Our approach involves designing a highly generalizable physical simulation system based on a viscoelastic material model, which enables us to simulate a wide range of materials with high-fidelity capabilities. Moreover, we distill the physical priors from a video diffusion model that contains more understanding of realistic object materials.
Given an object represented as 3D Gaussians, We first simulate it using the Material Point Method (MPM). The simulation comprises two distinct components: an elastoplastic part and a viscoelastic part. These components operate independently to calculate the individual stresses within the object and are combined to determine the overall stress. After the simulation, a series of Gaussians with varying orientations are generated, reflecting the dynamic evolution of the scene. Then, we render these Gaussians from a fixed viewpoint to produce a sequence of video frames. Finally, we utilize a pretrained video diffusion model with Score Distillation Sampling (SDS) strategy to iteratively optimize physical parameters.
We show the visual results of our approach across a wide range of materials, including elastic entities, plastic metals, and fibre materials. Physics3D achieves high-fidelity and realistic performance in a wide range of materials.
Visual comparison between our synthesized videos and baseline methods (PhysDreamer, DreamGaussian4D and PhysGaussian). We show that Physics3D is able to generate realistic scene movement while maintaining good motion consistency.
| Real Capture | Ours | PhysDreamer | PhysGaussian | DreamGaussian4D | |
|---|---|---|---|---|---|
| hat | |||||
| alocasia | |||||
| carnation | |||||
| telephone | |||||
We compare results w/ and w/o viscoelastic part. We observe that with both elastoplastic and viscoelastic parts can fit the real-world object better.
@article{liu2024physics3d,
title={Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion},
author={Liu, Fangfu and Wang, Hanyang and Yao, Shunyu and Zhang, Shengjun and Zhou, Jie and Duan, Yueqi},
journal={arXiv preprint arXiv:2406.04338},
year={2024}
}