Skull-to-Face: Anatomy-Guided 3D Facial Reconstruction and Editing
Abstract
Deducing the 3D face from a skull is an essential but challenging task in forensic science and archaeology. Existing methods for automated facial reconstruction yield inaccurate results, suffering from the non-determinative nature of the problem that a skull with a sparse set of tissue depth cannot fully determine the skinned face. Additionally, their texture-less results require further post-processing stages to achieve a photo-realistic appearance. This paper proposes an end-to-end 3D face reconstruction and exploration tool, providing textured 3D faces for reference. With the help of state-of-the-art text-to-image diffusion models and image-based facial reconstruction techniques, we generate an initial reference 3D face, whose biological profile aligns with the given skull. We then adapt these initial faces to meet the statistical expectations of extruded anatomical landmarks on the skull through an optimization process. The joint statistical distribution of tissue depths is learned on a small set of anatomical landmarks on the skull. To support further adjustment, we propose an efficient face adaptation tool to assist users in tuning tissue depths, either globally or at local regions, while observing plausible visual feedback. Experiments conducted on a real skull-face dataset demonstrated the effectiveness of our proposed pipeline in terms of reconstruction accuracy, diversity, and stability.
An overview of the proposed Skull-to-Face pipeline
Real-world Facial Reconstruction from Skull
Facial reconstruction demo on historical figure: Robert the Bruce, was King of Scots from 1306 to his death in 1329. 3D scan of the skull, portrait image and bust are downloaded from the Internet ( skull, portrait, bust). See demo video 4 for more details.
Demo Videos
Demo video 1 shows the end-to-end pipeline of the facial reconstruction from a skull.
Demo video 2 shows animations of 3D facial editing, globally and in local regions.
Demo video 3 shows the comparison between our method and existing methods.
Demo video 4 shows a real-world facial reconstruction example, which uses the skull of Robert the Bruce, King of Scots from 1306 to 1329.
Comparisons with SOTA methods
Qualitative comparisons with existing methods. (a) is the input skull. (b-e) show the results of HF-GGR [Jia et al, C&G'22], Pix2Pix [Isola et al, CVPR'17], CycleGAN [Zhu et al, ICCV'17], and CGAN-CFG [Zhang et al, ACMMM'22]. (f) show our reconstructed face with texture and geometry. (g) is the CT-scanned GT.
BibTeX
@article{liang2024skulltoface,
title={Skull-to-Face: Anatomy-Guided 3D Facial Reconstruction and Editing},
author={Yongqing Liang and Congyi Zhang and Junli Zhao and Wenping Wang and Xin Li},
year={2024},
eprint={2403.16207},
archivePrefix={arXiv},
primaryClass={cs.CV}
}