**Introduction:**
Deep learning allows to synthesize photorealistic images of human faces with novel identities. Most modern methods lack explicit control over appearance, head pose, face shape, and facial expressions. This yields in inconsistent results when animating these faces. In this project, we aim to fix this and generate controllable 3D facial avatars.
Please check out [2] for slides.
**Context:**
We build on top of previous work based on neural rendering of faces, called VariTex [1]. VariTex can generate faces for unseen identities and animate the face interior consistently. For the face exterior, VariTex produces per-frame plausible results, but not yet consistent animations.
To generate images of complete human heads, VariTex proposes an additive decoder that generates per-frame plausible exterior regions, such as hair. The goal of this project is to replace this additive decoder with a temporally consistent variant to achieve smooth animations for exteriors regions, also including upper bodies.
(Optional) Possible further extensions could include semantic controls - eye gaze, eye blinks, lighting, emotions - or animations from other modalities - audio, text, and reference videos.
**Prerequisites**
_Required_: Deep learning basics; hands-on project experience (preferrably PyTorch); communication skills.
_Preferred_: Basic knowledge of 3D vision; you have heard of 3D morphable models, implicit functions, neural radiance fields; willingness to aim for a publication.
**References:**
[1] https://arxiv.org/abs/2104.05988
[2] https://docs.google.com/presentation/d/1pi1gCwWpV_yFeblNR_CIPqnAaUp8Amw0fp46AoSgIsY/edit?usp=sharing
**Introduction:** Deep learning allows to synthesize photorealistic images of human faces with novel identities. Most modern methods lack explicit control over appearance, head pose, face shape, and facial expressions. This yields in inconsistent results when animating these faces. In this project, we aim to fix this and generate controllable 3D facial avatars.
Please check out [2] for slides.
**Context:** We build on top of previous work based on neural rendering of faces, called VariTex [1]. VariTex can generate faces for unseen identities and animate the face interior consistently. For the face exterior, VariTex produces per-frame plausible results, but not yet consistent animations.
To generate images of complete human heads, VariTex proposes an additive decoder that generates per-frame plausible exterior regions, such as hair. The goal of this project is to replace this additive decoder with a temporally consistent variant to achieve smooth animations for exteriors regions, also including upper bodies.
(Optional) Possible further extensions could include semantic controls - eye gaze, eye blinks, lighting, emotions - or animations from other modalities - audio, text, and reference videos.
**Prerequisites**
_Required_: Deep learning basics; hands-on project experience (preferrably PyTorch); communication skills.
_Preferred_: Basic knowledge of 3D vision; you have heard of 3D morphable models, implicit functions, neural radiance fields; willingness to aim for a publication.