Mixed reality videos — where a video camera and a green screen are used to show what a person using VR is experiencing — aren’t particularly new. But a recent collaboration between Google’s Daydream and Research teams has come up with a new spin on mixed reality by making the VR headsets transparent so you can actually “see” the user’s face.
Of course, Google hasn’t actually invented a way to magically X-ray through a VR headset. Rather, using a combination of a 3D scanned model of a person’s face, along with an HTC Vive headset modified with eye tracking technology allows for a digital recreation of the person’s face to be substituted in place. To prevent a disconcerting uncanny valley effect, a translucent version of the headset is left in place, to create some mental distance for a viewer.
The Google Research blog dives deeper into how the system works. Each 3D scan typically takes less than a minute and creates a database of eye gazes and blinks, which the software then maps to the data from the eye tracking system in the headset. Additionally, the software color corrects the digitally created face portion to match the lighting of the rest of the user’s face.
For now, the transparent headset effect remains a cool research project, with no formal release information for either developers or consumers. But Google is apparently “exploring how to bring this technology to select creators in the future,” so it’s possible we could see the tech show up somewhere down the line.