We introduce a method for animating human images, using the SMPL 3D human parametric model within a latent diffusion framework to improve shape alignment and motion guidance. By incorporating various maps and skeleton-based guidance, we enrich the model with detailed 3D shape and pose attributes, fusing them via a multi-layer motion fusion module with self-attention mechanisms.
arXiv 2024 Paper    Project Page

We propose a fully differentiable framework named neural ambient illumination (NeAI), which incorporates Neural Radiance Fields (NeRF) as a physically-based lighting model to handle complex lighting. Our method integrates physically based rendering into NeRF, utilizing roughness-adaptive specular lobe encoding and precise decomposition via the pre-convoluted background.
AAAI 2024 Paper    Project Page

We propose VividTalk, a two-stage generic framework that supports generating high-visual quality talking head videos with all the above properties. Extensive experiments show that the proposed VividTalk can generate high-visual quality talking head videos with lip-sync and realistic enhanced by a large margin.
arXiv 2023 Paper    Project Page

We present a novel differentiable point-based rendering framework for material and lighting decomposition from multi-view images, enabling editing, ray-tracing, and real-time relighting of the 3D point cloud. Our framework showcases the potential to revolutionize the mesh-based graphics pipeline with a relightable, traceable, and editable rendering pipeline solely based on point cloud.
arXiv 2023 Paper    Project Page   Code

We present a large-scale detailed 3D face dataset, FaceScape, and the corresponding benchmark to evaluate single-view facial 3D reconstruction. By training on FaceScape data, a novel algorithm is proposed to predict elaborate riggable 3D face models from a single image input.
TPAMI 2023 Paper    Code & Dataset

We propose a novel method, AvatarBooth, for generating high-quality 3D avatars from text prompts or photos. Unlike previous approaches that are based on only text descriptions, our method enables the users to customize the generated avatars according to casually captured photos of the face or full body.
arXiv 2023 Paper    Project Page

We present LoD-NeuS, an efficient neural representation for high-frequency geometry detail recovery and anti-aliased novel view rendering. A multi-scale tri-plane-based scene representation is introduced to capture the LoD of the signed distance function (SDF) and the space radiance.
SIGGRAPH Aisa Conf. 2023 Paper

We propose to synthesize high-quality 3D face models from natural language descriptions including both concrete and abstract descriptions. Describe3D dataset is established with large-scale 3D faces and fine-grained text descriptions for text-to-3D face generation task.
CVPR 2023 Paper    Project Page   Code

We propose a novel approach for the single-view 3D face reconstruction task in a non-parametric scheme. Our method gets rid of the heavy dependence on the statistic model and, therefore, its limitations and achieves state-of-the-art performance by learning from our created pseudo 2D&3D datasets.
AAAI 2023 Paper (Oral)    Project Page    Code

We propose a parametric model that maps free-view images into a vector space of coded facial shape, expression and appearance using a neural radiance field, namely Morphable Facial NeRF. MoFaNeRF can be used to synthesize free-view images by fitting to a single image or generating from a random code. The synthesized face is morphable that can be rigged to any expressions and be edited to any shapes or appearances.
ECCV 2022 Paper    Project Page    Code