Microsoft had to blindfold me so I could hear the future

Microsoft’s system works by creating a 3D-soundscape of audio that’s transmitted through your jaw bone. You wear a bone conduction headset that pairs to a smartphone that can pick up nearby beacons and guide you around. The bone conduction here is key, as it means you can still hold conversations with people or hear the environment around you because you’re not wearing headphones that cover your ears, which is crucial if you’re visually impaired and rely on sound as a primary sense. Microsoft has created a clever system of audio cues for nearby stores, points of interest, and journey details, alongside regular GPS instructions and a unique audio ping that keeps you straight on track when you’re navigating. Like Oculus’ Rift, it’s a technology you really need to try to fully experience, so I did.

At first, I walked around blindfolded with only the aid of a white cane. My senses of hearing and smell immediately increased — along with my panic levels. I couldn’t walk straight, I didn’t know where I was going, and I would have walked into a lot of walls if it weren’t for a helpful guide nearby. Mostly, I just felt fear. When my guide asked me to remove the blindfold all I could see was the water of a canal I was about to walk straight into.

Microsoft’s headset is designed to tackle these challenges with technology that won’t cost $80,000 — the lifetime price tag of a guide dog. It won’t replace a guide dog or a white cane, but it might just lower the costs and complexity of navigating cities if you’re visually impaired. 90 percent of the world’s visually impaired live in low-income settings, according to the World Health Organization. Inexpensive technology is badly-needed and a simple headset could be the answer.

After that initial test it was time to try Microsoft’s system. The headset is a prototype. It pairs with a smartphone, which acts as a control center to set navigation points and find information on nearby places. The headset is calibrated by first scanning your entire face and head with Kinect, producing a map for the system to improve the accuracy of the headset and tailor it to your own dimensions. It’s all driven using gestures from a smartphone app that plays back audio menus straight "into" your ears. You simply tap and hold and slide or drag around to navigate, with each move accompanied by a subtle vibration from the handset and the menu item read back to you. If you’re on a train or a bus it will tell you when you’re about to reach a station or stop using GPS or nearby beacons, but it’s blindly walking around environments where it’s especially impressive.

The first step is positioning your head towards a click-clack sound until you hear a small constant pinging noise. That’s the trigger to let you know you’re facing in the right direction for navigation, thanks to a compass embedded in the back of the headset. At first, blindfolded and relying on the headset and cane, I still felt anxious. Within minutes, though, the combination of a white cane and this subtle guide helped me walk straight, around corners, and even cross roads with assistance. If the pinging noise cut out I felt lost, but as soon as I aligned myself again I was back on track. It felt natural, like a guide was placing me in the right direction, and adding the white cane allowed me to sense any obstacles in the way.

"I was a bit mistrustful to begin with," admits Jennifer Bottom, who has been blind from birth and is helping Microsoft with its trials. "What you need as a blind person is very different to what you need as a seeing person." Bottom says she would use the device, "and I don’t normally say that." Microsoft’s solution is different from others she has tested that simply shout GPS instructions at you without any aides, allowing her and others to listen to the environment or talk to a friend while they’re navigating.

The real magic of this system is the 3D audio technology that gives you a real sense of direction. One feature on the headset allows you to push a button and hear a list of nearby places of interest. They’re processed through the headset dependant on the direction you’re facing so that when a store is read aloud you’ll be able to hear the direction of where it’s located. That might be in the rear left or out in front, but the audio gives you a clear sense of where that store is along a route through just sound alone.

Microsoft has teamed up with the Guide Dog charity and Future Cities Catapult in the UK to experiment and test these new ways to help improve the mobility of the blind using various emerging technologies. 3D sound and beacons are key, and Microsoft has also partnered with Reading Borough Council, Network Rail, Barclays, and Tesco in the UK to run trials with the technology. It’s extremely early right now, but it’s clear there’s a big effort to eventually commercialize this and bring it to life in years to come.

Amos Miller, director of enterprise strategy Asia at Microsoft, who is visually-impaired, joined the board for Guide Dogs between 2008 and 2012 and helped push this project inside Microsoft. Miller has faced his own challenges with mobility — like when his guide dog got trapped between a subway platform and the train. The project itself started out life two years ago as a video before it progressed into research and trials. That original concept video caught the attention of Microsoft CEO Satya Nadella years before he took the top role at the software giant. Nadella was the chairman of a disability group inside Microsoft while he was leading the company’s cloud and enterprise group. "Satya presented all of the team with a reward for that video production," Miller says. "That was his first insight into the opportunity. When he became CEO he decided to continue his sponsorship for the project."

Microsoft principal researcher Bill Buxton, who gave a fascinating talk on smartwatches last year, has worked closely on this 3D audio project. Discussing all the research involved, Buxton stresses the importance of sound events as notifications, known as earcons, with the headset for this particular technology. "There’s as much care and art in the design of good earcons as there is in good icons."

This technology feels like a demonstration of what Microsoft is capable of, and a new sign of collaboration across groups within the company. There's an attention to detail here and Microsoft is trying to take technology out of the way to focus on the experience, making this a particularly interesting experiment to see where the company is heading in future. Microsoft is now working on a second phase of the headset, integrating the technology more deeply instead of just bolting it onto the back of an off the shelf AfterShockz headset. I had to push hard to find what technology was involved here, and Microsoft seemed reluctant to discuss it, focusing on its benefits instead. This small gadget, backed by complex algorithms, cloud data, and new technology could just change the world of 285 million visually impaired people. That’s the sort of challenge you expect the new Microsoft to be working on, and it’s exactly what we’re going to see a lot more of with Nadella at the helm.