Skip to main content

Google’s secret VR weapon is Project Tango

Google’s secret VR weapon is Project Tango


Putting the pieces together

Share this story

The next few months are going to be awash with virtual reality news. In a couple weeks, we should learn a lot more about the Playstation VR. Then, nearly four years after we first tried the Oculus Rift, it’s finally going to be officially released on March 28th. Not long thereafter, HTC’s Vive will also finally be released to the public. These three headsets represent some of the most exciting technology anybody’s tried in the last few years. They’re powerful, immersive, and may finally have games and apps that are more than just tech demos. And if that’s not enough VR for you, Samsung’s Galaxy S7 is also getting released next month. It's the latest phone to work with the company’s Gear VR headset accessory.

So get ready for an ocean of VR news, and the next month is just going to be the start. Because after that wave of VR news, another one is going to crash: Google will almost surely make VR the center of its annual developer conference Google IO, which is happening on May 18th. The company is widely expected to move beyond its early experiments with Google Cardboard and into the next phase of VR. The big question is whether it’s going to look and feel more like those expensive, high-end headsets or more like Samsung’s less powerful but more portable mobile efforts.

A wave of VR is coming

I think it’s going to be exactly in the middle of those things, with capabilities that haven’t been possible on a phone-based device — only headsets attached to an expensive gaming PC or console via a wire. It will likely still use a phone and still technically be less powerful than those other headsets, but nevertheless signal that Google is in this game in a real way. And the reason I believe all that is that last year, just before Google IO, I got the chance to try out some demonstrations of exactly the kind of system I’m describing here. It’s called Project Tango, and I am betting that it’s going to be at the heart of Google’s next big VR push.

In one of the craziest demos I’ve ever tried, I strapped an Android tablet to my face in a room and started walking around in VR, untethered to a computer. Without the aid of cameras, the tablet automatically mapped the room I was in and let me explore an ethereal space with a white tree and floating heads. But the crazy bit was that Google’s employees, including Tango lead Johnny Lee, were also in that same space. I could see their heads in VR and they could see me, and their position in VR was exactly the same position in real life. I reached out to one of those virtual floating heads and my real hand tapped Lee’s real shoulder. It was wild.

Before I get into Tango more deeply, let’s review some of the recent things that we’ve heard about Google’s VR push. First and most importantly, Google has created an entire VR division within the company, as Mark Bergen reported at Recode in January. It’s led by Clay Bavor, whose title is VP of Virtual Reality. Bavor had been running Google’s apps division, but he also oversaw Google Cardboard and some other VR experiments within the company.

Last year, I talked to him about both Cardboard and another project called Jump, an open source VR camera-rig design that works with Google’s cloud computing platform to create fully steroscopic 360-degree video. It remains one of the most immersive things I’ve ever done with mobile VR. Now Bavor, whom Bergen described as a "precocious and well-liked exec inside Google," has made it his whole job to work on VR and has a team behind him to do it.

The next piece of Google’s VR puzzle is a piece published earlier this month by the Financial Times. It reported that later this year Google will release something similar to the Gear VR, but it will "feature better sensors, lenses, and a more solid plastic casing." Google is also expected to bake VR capabilities right into the next version of Android, presumably so it can take full advantage of the phone’s computing power to enable better (and less-nauseating, a problem with some mobile VR) experiences.

Mobile VR has a problem

That all sounds great, but the problem is that untethered, mobile VR has a problem. The Gear VR and Google Cardboard can detect how your head is moving with a gyroscope and compass, but those sensors aren’t very precise. More importantly, they don't detect your head’s position in space. Doing that second part is what enables headsets like the Rift and the Vive to feel fully immersive, letting you duck and dodge and have your real movements precisely mapped to the virtual world.

Those high-end headsets solve that problem with an external camera or laser array you mount in your room. They track tiny dots on your headset and controllers to track their positions in space. It’s called "outside-in" tracking and thus far it’s the best and most accurate way to ensure that what you do in reality matches what you do in virtual reality. But those cameras represent extra hardware you need to set up and configure and mean that you can only use the associated headset in the room where they are.

It would be far better to have the headset itself handle positional tracking, to do it "inside-out." You wouldn’t need the extra hardware and you could use it in any room, not just in the VR cave where you’ve set it up. So far, we haven’t seen any consumer-grade hardware that can do it. Five months ago, Oculus founder Palmer Luckey said on Reddit that "VR-grade inside-out tracking is not currently workable on mobile devices."

That’s not totally true, though. The Project Tango demo I described above only worked because it had inside-out tracking. Google’s experiment with augmented and virtual reality lets a device track its own position in space using an array of specialized cameras and sensors. It’s able to map a room and the objects in it, and more importantly knows its own position relative to those things — all without external cameras or preset mapping data. It might not be up to Luckey’s standards for "VR-grade inside-out tracking," but it’s nevertheless damn compelling.

My guess is that the "better sensors" that the Financial Times is referring to are precisely the sensors that enable Project Tango to work. So far, Google hasn’t shown the public a lot of Tango’s VR capabilities. Most of what we’ve seen has been focused on augmented reality and letting you do indoor mapping and navigation — the company just did it again this past week at Mobile World Congress in Barcelona. But Tango’s VR capabilities are perhaps even more exciting than indoor mapping.

Lee told me that Google’s ambition for Tango is to make those sensors just as essential to a smartphone as GPS and a compass are now. "Nowadays, you wouldn’t consider buying a phone without GPS," Lee said last year. "We hope to see Tango kind of reach the same level of adoption."

So far, only a tiny handful of prototype phones — including some from Lenovo and Qualcomm — support Tango directly. Waiting for a plurality of Android phones to adopt those sensors before its VR efforts can take off seems like a very bad strategy for Google. Building them into a separate headset and then building the capability to talk to them directly into the next version of Android, however, seems like a great strategy. Building that headset and also building a standalone headset that doesn’t require a phone but has the same capabilities also seems like a good idea, and that’s exactly what the Wall Street Journal reported Google is doing (though it may not be ready this year — and I doubt that a new VR-enabled Nexus is in the cards for May).

Yes, this is all speculation (and yes, Google declined to comment on this story), but the pieces are all there. Strategically, Google can’t make a big splash with VR if it’s not differentiated from the Gear VR, which has been on the market for awhile and has a growing library of apps and games to support it. It also needs to compete with the Rift, Vive, and PSVR, and having the same positional tracking as those headsets without the need to tether to a computer would be a big deal.

Google already has all the technology it needs for a surprisingly good VR product

I could be wrong (or premature) with this prediction, but I think the stars are aligning for this kind of system to be revealed at Google IO in May — it’s a developer conference, and Google will need those developers to learn how to use it all.

Last year, I asked Bavor why Cardboard (and VR in general) seemed to be perennially stuck in the zone of experimentation. When was Google going to get serious about VR? "Cardboard is just the tip of the iceberg of our VR investments," he told me, arguing that it was "well beyond an experiment." He then added that there are "many things going on behind the curtains."

If Google is smart, one of those things behind those curtains is a VR headset with Project Tango sensors and a new version of Android that directly supports them. And I think that Google is very smart.