Skip to main content

Video system lets you control another human's expressions in real-time

Video system lets you control another human's expressions in real-time

Share this story

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

Researchers have created a video-modifying program that transfers the expressions of a subject onto live video of another subject's face in real time. The video warrants cliché: it has to be seen to be believed.

The project is a collaboration between researches from University of Erlangen-Nuremberg, Max-Planck-Institude for Informatics, and Stanford University. Footage of the project was been released in September with a paper titled "Real-time Expression Transfer for Facial Reenactment." Here's how the abstract describes what makes their system unique:

The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video.

For those who'd like to know the nitty-gritty, the paper is worth a read, and written in a refreshingly accessible fashion. The video speculates about use-cases, like a translator applying their expressions and voice over their subject, or someone in casual clothes projecting onto a source in formalwear. But what I see is an opportunity for Disney to using a blank-faced Robert Downey Jr. wax model as a target model to produce Iron Man movies for the rest of time.

How exactly would Disney do this? I'm glad you asked.