Facebook showed off its latest ideas for social interaction in virtual reality today, demoing more complex avatars at Oculus Connect, the annual developer conference for the VR company. The avatars could smile, laugh, or express anger and shock. They had arms attached to their hands and personalized clothing. All this was new and more robust virtual representation than anything we've seen from the company so far.
We've known for a while that Mark Zuckerberg's long-term goal for virtual reality was to transform his social network from a place where you post text, photos, and video to one where people can hang out in virtual space and feel as if they are conversing face to face. He elaborated on this dream a few months ago at F8, and today the company showed off a few new pieces of software that built on these early demos.
Toybox is the basic social sandbox for VR. You and another person can build with blocks, play ping pong, or team up to take out a targets in a virtual shooting gallery. But your avatar in that program is an anonymous blue head and a pair of floating blue hands. There is nothing to distinguish between any two people in the space.
In the more advanced demo at F8 Facebook showed off a social interaction where people can customize their avatars, adding details like facial hair and glasses. Until today there was no tech with the Oculus Rift that can track your face, so it remains a frozen mask, devoid of smiles, frowns, or rage.
A blog post from Facebook explained how these more advanced avatars worked. "While we can't yet track and reproduce the precise movement of torso and limbs, we can algorithmically approximate arm and body position based on the tracked position of head and hands. We chose to represent avatar arms this way because we found that having a more realistic physical form amplifies the feeling of connection," wrote Michael Booth, the product manager for Social VR.
For your face Facebook used a few tricks. "Lip motion creates a visual indicator that someone is talking, helping conversation flow more naturally. We enabled basic mouth movement triggered by your voice," wrote Booth. The system combines signals from your voice, head, and hand position to recognize and render a few basic moods. "Our avatars can display emotions and expressions like "smiling," "confused," "skeptical," or "listening." This makes it easier for people to communicate how they're feeling.
Along with VR avatars, the social experience now connects with Facebook Messenger. Zuckerberg called his wife, Priscilla Chan, who appeared on a live video call that sat at the table alongside the cartoon version of Zuckerberg and his team. Mixing cartoonish but three-dimensional avatars with live video rendered in 2D results in a fairly awkward scene for now, but it did allow for the world first Mixed Reality selfie.