Inside Microsoft's HoloLens

Opening up the HoloLens is a startling reminder that this is a self-contained holographic computer. Unlike the Oculus Rift or HTC Vive VR headsets, it’s completely untethered and doesn’t need to communicate with a PC or phone to project holograms into the environment around you. It's a Windows 10 PC that sits on your head. That’s impressive in itself, but it wasn’t always that way. Microsoft has spent years refining the headset and associated hardware so you don’t need a giant PC to use it. The first time I used HoloLens in January last year, I had to hang a miniature PC around my neck. It has come a long way since those early units.

The magic of HoloLens is a combination of see-through holographic lenses, an array of sensors, and the processing required to understand your gestures, gaze, voice, and the environment around you. It’s tempting to compare it to the work Google did with its Glass project, but the HoloLens is trying something different and it feels more immersive as a result. This headset understands the objects in your living room, and it projects holograms into your eyes that don’t jerk around. I played a Conker game and the squirrel literally climbed up a coffee table and ran across pillows. It really felt, at times, like Conker was in the room with me.

An important part of the HoloLens is the depth sensing camera that sits above the visor at the front of the headset. It’s essentially a miniature version of the Kinect camera that only needs about a tenth of the power, and it looks like a regular camera from the outside. "It is engineered and designed by Microsoft," explains Microsoft’s Ori Amiga, who has been working on the headset for years. Microsoft doesn’t usually engineer and manufacture hardware chips and parts, so this is relatively unusual for the company. As you’re gesturing around with your fingers, the depth camera senses exactly where your hand is, and whether you’re using your left or right hand.

The depth sensing camera works in tandem with two "environment understanding" cameras on either side of the HoloLens. These also look like regular cameras from the outside of the headset, but they’re used to capture the world around you and help HoloLens understand where your walls and coffee table are. There’s also a 2-megapixel video camera at the front so you can record exactly what you see with the holograms included. Microsoft calls this mixed reality capture, and it’s a fun way to record what you see with the holograms that are being projected into your living room. During my HoloLens hands-on, I found myself activating the feature a lot (using Cortana, of course). I can imagine a lot of people are going to have fun sharing clips to Facebook and Twitter.

So the cameras help HoloLens see, but the real magic is how holograms are projected into your vision. The HoloLens optics system is composed of see-through holographic lenses (Microsoft refers to them as "waveguides") that use an optical projection system to beam holograms into your eyes. These look like regular glasses when you pull them out of the visor, and it’s easy to imagine a future where just these glasses are the entirety of HoloLens. While traditional smartphones have displays measured in pixels, Microsoft is counting light points to boost HoloLens’ resolution and the number of light points per radian for density. Essentially, more radiants and light points mean holograms are brighter and richer. These bright and rich holograms are projected and bounced around as particles of light (photons) through the waveguides and into your eyes. I’ve used the HoloLens three times now, and I’m convinced this latest version has sharper text rendering. Amiga couldn’t comment on exactly what Microsoft has improved over the past year, but a combination of hardware and software tweaks have definitely made holograms crisper.

The holographic lenses are also the source of some HoloLens complaints. The device has a limited field of view, and it’s an optical limitation that won’t change much, if at all, by the time this ships to actual consumers. Microsoft is using a transparent display, and holograms need to be projected into your eyes. It’s still a challenge for the HoloLens and other head-mounted augmented reality devices to cover your entire eye as you move your pupil around. It could take years before this field of view greatly improves, but this is just the first generation.

Perhaps the most impressive part of the HoloLens is its motherboard. Unlike a powerful gaming PC or even your average laptop, it’s tiny and uniquely shaped for the HoloLens. I was surprised at just how small it is. "I’m not sure you’ve ever seen a motherboard that isn’t rectangular or square, but it’s not easy," explains Amiga. Microsoft has packed everything into this motherboard. The CPU (32bit Intel-based), GPU, flash memory, RAM, Wi-Fi chip, and Bluetooth chip are all on this tiny little board. There’s also a new chip on this HoloLens motherboard: the holographic processing unit (HPU).

"We believe that it’s time for the world to get the third processor, so you’ll have a CPU, GPU, and a HPU," says Amiga. The HPU does a lot of the heavy lifting here, so that the CPU and GPU are able to just launch apps and display the holograms. Microsoft custom designed the HPU and it’s really meant to take all of the data from the cameras and sensors and process it in real-time so you can use gestures accurately.

All of these sensors need battery power, and Microsoft has packed a bunch into the rear of the headset. There are three separate batteries on each side of the HoloLens, but Microsoft isn’t revealing their exact capacity yet. They look like little battery packs you might use inside a toy, and they’re small enough to fit in the sides of the HoloLens. They also act as a way to balance out the weight of the HoloLens so it doesn’t feel too heavy at the front or rear. The headset should run for around two or three hours before it needs to be recharged via Micro USB.

One of the final parts of the HoloLens is the spatial sound. This works through two speakers that are positioned just above your ears as you wear the headset. Apps and games can take advantage of this to make enemies sound like they’re behind you instead of having to load up visual elements to make you turn around. I’m surprised at just how well this works. It sounds like you’re wearing headphones, but you’re not. Even at full volume, it’s hard for others around you to hear it, so it’s not going to irritate people in a room. You can adjust the volume at the side, and even plug in headphones to get the same spatial effect.

Microsoft is now shipping all of this hardware to developers for $3,000. The software giant isn’t discussing exactly when consumers will be able to experience HoloLens, but it’s likely that the final hardware will differ from what we see today. It took Microsoft several years to shrink this into a unit you can wear on your head, so the hope is that more breakthroughs and time will allow it to shrink this down even further. Developers will now shape the future of the HoloLens software and experiences, and perhaps even its hardware. This is just the beginning for HoloLens, and Microsoft is hoping it’s the future.

Photos by Vjeran Pavic