We present a novel approach for synthesizing 3D talking heads with controllable emotion, enhancing lip synchronization and rendering quality. To address multi-view consistency and emotional expressiveness issues, we propose a ‘Speech-to-Geometry-to-Appearance’ mapping framework trained on the EmoTalk3D dataset, enabling controllable emotion, wide-range view rendering, and fine facial details.
ECCV 2024    Project Page    Code

We propose STAG4D, a novel framework for high-quality 4D generation, integrating pre-trained diffusion models with dynamic 3D Gaussian splatting. Our method outperforms prior 4D generation works in rendering quality, spatial-temporal consistency, and generation robustness, setting a new state-of-the-art for 4D generation from diverse inputs, including text, image, and video.
ECCV 2024    Project Page    Code

We introduce a method for animating human images, using the SMPL 3D human parametric model within a latent diffusion framework to improve shape alignment and motion guidance. By incorporating various maps and skeleton-based guidance, we enrich the model with detailed 3D shape and pose attributes, fusing them via a multi-layer motion fusion module with self-attention mechanisms.
ECCV 2024    Project Page    Code

We present a novel differentiable point-based rendering framework for material and lighting decomposition from multi-view images, enabling editing, ray-tracing, and real-time relighting of the 3D point cloud. Our framework showcases the potential to revolutionize the mesh-based graphics pipeline with a relightable, traceable, and editable rendering pipeline solely based on point cloud.
ECCV 2024    Project Page   Code

Author's picture

Hao Zhu

NJU-3DV Lab, Nanjing University
E-mail: zh@nju.edu.cn


Assistant Professor, PhD Advisor


Nanjing, China