Skip to main content

Here's how you build an augmented reality game for HoloLens

Here's how you build an augmented reality game for HoloLens


Now you're thinking with holograms

Share this story

Programming a hologram sounds like something that should be done with some kind of special cybergloves on a computer the size of a ‘60s IBM mainframe. But at Build 2015, Microsoft has been quietly taking developers through the "Holographic Academy," a 90-minute training session that teaches them the basics of building projects for its HoloLens augmented reality headset. I’m not a developer, but Microsoft let me and some other journalists go through it as well — and it turns out that basic hologram creation is, if not exactly straightforward, at least pretty understandable.

Technical details aside, the biggest difference between HoloLens and any other platform is the amount of information it collects. Microsoft is secretive about what exactly is in the device, but among other things, it can accept voice commands, read very simple finger gestures, and scan rooms well enough to build a detailed, fairly accurate depth map. This is all fed into the Unity game engine, which announced augmented reality support earlier this week. Microsoft is pushing the idea of truly universal apps — which include HoloLens editions — at Build, but at the training session we created a HoloLens-specific Unity project, exported it through Microsoft’s Visual Studio, and loaded it directly onto the device via Micro USB.

Building a simple game for HoloLens feels a lot like just building a gameThe game wasn’t the most impressive HoloLens demo around, compared to more complex experiences like Holo Studio or the Mars Rover. It was a little floating island of notepaper and origami, with two paper balls floating above it. The individual pieces, like art, music, and specific scripts, had been created already — some specifically for this project, some as part of a more general "HoloToolkit." Our job was to bring a partially assembled Unity project to completion, creating a series of increasingly complex behaviors.

There wasn't any real programming in the Holographic Academy, but it was easy to at least look through the code, and putting the game together felt like building any beginner's tutorial in Unity. Instead of placing a virtual camera, you place a marker that represents the user's head. Instead of assigning behaviors when someone moves or clicks the mouse, you assign them when someone looks at an object and makes an "air tap." And so on.

Unity Screenshot

Here's Unity's workspace with a non-augmented reality game. Our project was a lot simpler.

Voice and motion control aren’t new to anyone who’s played a Kinect game, and gaze-tracking — the headset’s ability to see where its wearer is looking — is a central component of VR. But being able to "see" the real-world changes is everything. The HoloLens origami collection was just a group of disembodied objects floating in space, but by the time we were done, you could tap to place the notebook on a table, say "move, ball" (or another phrase of your choice) to drop the two paper balls, then watch them roll off the table, onto the floor, and around each other. As long as you've created objects that will respond to basic physics, the HoloLens sensors can do the rest.

We still don't know how to let players share an experienceThere are more advanced options that we didn't get a chance to try, like an occlusion feature that would supposedly make the ball realistically disappear if it rolled behind, say, a couch leg. And there are obvious scenarios we didn't get to explore. Microsoft has touted HoloLens as a way to get people interacting with technology together, instead of looking at their phones or squeezing in front of a TV. But I haven't yet seen a demo where people share holographic space, and I'm not sure how developers would align objects so multiple people could see them. There are no external markers or scanning devices, and it's not clear if HoloLenses can sense each other. I also asked if the headset's sensors could detect things besides depth — like real-world color — and got a noncommittal response.

Being able to program something on HoloLens doesn't mean you know anything about its inner workings. Microsoft won't say much about the technology that turns those simple rendered objects into startlingly realistic projections, and the software development kit just acts as an intermediary. It also doesn't mean you can design a good HoloLens game. There are probably going to be a lot of holographic Rubik's Cubes and puzzle boxes in the early days of the platform, except that you'll only be able to move them by staring and tapping. But just understanding the basics of building for such an unfamiliar platform makes it seem more real — and, despite all its problems, more viable.