Pose-aware 3D talking face synthesis using geometry-guided audio-vertices attention

Li, Bo, Wei, Xiaolin, Liu, Bin, He, Zhifen, Cao, Junjie and Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680 2024. Pose-aware 3D talking face synthesis using geometry-guided audio-vertices attention. IEEE Transactions on Visualization and Computer Graphics 10.1109/TVCG.2024.3371064

Preview

PDF - Accepted Post-Print Version Download (20MB) | Preview

Abstract

Most of the existing 3D talking face synthesis methods suffer from the lack of detailed facial expressions and realistic head poses, resulting in unsatisfactory experiences for users. In this paper, we propose a novel pose-aware 3D talking face synthesis method with a novel geometry-guided audio-vertices attention. To capture more detailed expression, such as the subtle nuances of mouth shape and eye movement, we propose to build hierarchical audio features including a global attribute feature and a series of vertex-wise local latent movement features. Then, in order to fully exploit the topology of facial models, we further propose a novel geometry-guided audio-vertices attention module to predict the displacement of each vertex by using vertex connectivity relations to take full advantage of the corresponding hierarchical audio features. Finally, to accomplish pose-aware animation, we expand the existing database with an additional pose attribute, and a novel pose estimation module is proposed by paying attention to the whole head model. Numerical experiments demonstrate the effectiveness of the proposed method on realistic expression and head movements against state-of-the-art methods.

Item Type:	Article
Date Type:	Published Online
Status:	In Press
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1077-2626
Date of First Compliant Deposit:	21 March 2024
Date of Acceptance:	20 February 2024
Last Modified:	27 Mar 2024 09:48
URI:	https://orca.cardiff.ac.uk/id/eprint/167426

Actions (repository staff only)

Edit Item