NeRF-Diffusion for 3D-Consistent Face Generation and Editing

Héctor Laria, Héctor Laria, Kai Wang, Joost van de Weijer, Joost van de Weijer, Bogdan Raducanu, Bogdan Raducanu, Yaxing Wang

2024

Abstract

Generating high-fidelity 3D-aware images without 3D supervision is a valuable capability in various applications. Current methods based on NeRF features, SDF information, or triplane features have limited variation after training. To address this, we propose a novel approach that combines pretrained models for shape and content generation. Our method leverages a pretrained Neural Radiance Field as a shape prior and a diffusion model for content generation. By conditioning the diffusion model with 3D features, we enhance its ability to generate novel views with 3D awareness. We introduce a consistency token shared between the NeRF module and the diffusion model to maintain 3D consistency during sampling. Moreover, our framework allows for text editing of 3D-aware image generation, enabling users to modify the style over 3D views while preserving semantic content. Our contributions include incorporating 3D awareness into a text-to-image model, addressing identity consistency in 3D view synthesis, and enabling text editing of 3D-aware image generation. We provide detailed explanations, including the shape prior based on the NeRF model and the content generation process using the diffusion model. We also discuss challenges such as shape consistency and sampling saturation. Experimental results demonstrate the effectiveness and visual quality of our approach.

Paper Citation

in Harvard Style

Laria H., Wang K., van de Weijer J., Raducanu B. and Wang Y. (2024). NeRF-Diffusion for 3D-Consistent Face Generation and Editing. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 587-594. DOI: 10.5220/0012381500003660

in Bibtex Style

@conference{visapp24,
author={Héctor Laria and Kai Wang and Joost van de Weijer and Bogdan Raducanu and Yaxing Wang},
title={NeRF-Diffusion for 3D-Consistent Face Generation and Editing},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
year={2024},
pages={587-594},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012381500003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP
TI - NeRF-Diffusion for 3D-Consistent Face Generation and Editing
SN - 978-989-758-679-8
AU - Laria H.
AU - Wang K.
AU - van de Weijer J.
AU - Raducanu B.
AU - Wang Y.
PY - 2024
SP - 587
EP - 594
DO - 10.5220/0012381500003660
PB - SciTePress