NEURAL HEAD REENACTMENT FOR NOVEL DOMAINS
Abstract
Despite recent advancements in deep learning-based head reenactment, they still have a limited capacity to produce high-quality head images for non-human domains (e.g., animation and artistic portrait), which bear significant potentials in virtual reality and meta-verse applications. This is mainly due to the head structure and texture gap between human and novel domain heads; in practice a model trained with human heads does not work well for non-human heads. In this research, we propose to improve the performance in the arbitrary domains via joint training of real and animation datasets and a domain-conditional texture generator. In particular, our research aims to construct a novel pose representation transformation to achieve the shared motion space of two disjoint domains, where a human head motion can be easily transferred to an arbitrary head. In addition, such a pose representation and a domain label are fed to the texture generator to synthesize the high-quality head reenactment outputs given a driving motion. After training, the proposed method is able to process an arbitrary head from novel domains, thus giving a chance to be applied to various fields such as gaming avatar and VTuber.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Jan 04, 2023
- Source ID
- FA23862214024
Entities
People
- Jaegul Choo
Organizations
- Air Force Office of Scientific Research
- KAIST
- United States Air Force