Three months ago, celebrated video game publisher Valve did something completely out of character: it fired up to 25 workers, in what one employee dubbed the "great cleansing." At the time, co-founder Gabe Newell quickly reassured gamers that the company wouldn't be canceling any projects, but it just so happens that one project managed to get away.
Valve was secretly working on a pair of augmented reality glasses... and those glasses are still being built by two Valve employees who lost their jobs that day.
"This is what I'm going to build come hell or high water."
Former Valve hardware engineer Jeri Ellsworth and programmer Rick Johnson spent over a year working on the project at Valve, and have been putting in six days a week, 16+ hours a day on the project ever since. "We believed in it... that this is going to change the way that people interact with computers and play games," says Ellsworth. "This is what I'm going to build come hell or high water. It was just a no-brainer that when we were not at Valve... we just had to do it." They formed a company, Technical Illusions, to commercialize the tech. This weekend, they flew down to Maker Faire to show their crazy prototype to the world for the very first time.
It's called CastAR, and it's nothing like Google Glass or the Oculus Rift. The idea here is to project a miniature virtual reality, which you can see and interact with in three dimensions, into the real world.
Four key components make it work. First, a pair of miniature projectors attached to the glasses beam images from a connected PC. A special retroreflective projector screen bounces them back to your face. There, the active shutter glasses filter out images for each of your left and right eyes, 120 times a second, so that you see those images in 3D.
Lastly — and this is the tricky part — a camera built into the glasses sees infrared LEDs positioned around the edges of that projector screen so that the glasses can optically track the exact position of your head, allowing the software to adjust the 3D perspective in real time so that you can physically look around objects that don't even exist. It's basically the famous Johnny Chung Lee demo with the Nintendo Wii remote, except the team has custom algorithms for extremely low latency. With additional pennies-on-the-dollar cameras placed around the surface, the system can track other objects as well, like the infrared wand Ellsworth handed me for my first demo.
I played a game of destructive Jenga with the wand, smashing towers of virtual blocks with a virtual wrecking ball. I craned my head over the surface, looked down, and marveled as they fell into the depths below the actual surface of the table. I flew over a Minecraft-like rough landscape, using nothing but my head to steer. I got a taste of how CastAR could work with virtual toys, a la Skylanders, by placing little RFID tokens on a surface and watching them grow into Team Fortress 2's Scout and Heavy characters.
Finally, Ellsworth showed me the best thing yet. We played a multiplayer game with two Xbox controllers where we stood shoulder to shoulder, laughing at one another's antics as we mowed down zombies (and one another) in a zany capture the flag / king of the hill shooting game. There, even though we were looking at the same surface, the glasses gave us our own unique 3D perspective.
Thanks to the retroreflective surface, so much of the light gets reflected directly back to the viewer that there's very little cross-talk between players... you pretty much only see the light intended for you, which makes adding additional friends easy so long as they have their own pair of specs. There's also enough light to use the device in a moderately sunlit room, or from a reasonable distance away. While the Maker Faire demos were like interacting with small dioramas due to the fairly small projector screens, you could theoretically build a holodeck of sorts by putting retroreflective surfaces on every wall.
I should mention here that while the concept was impressive, the glasses themselves are extremely early, not anywhere close to fit for public consumption right now. Even the original Oculus Rift, virtual reality ski goggles held together with duct tape and dreams, felt a little more put-together than the CastAR prototypes that Ellsworth spent 40 hours painstakingly soldering. Hot glue and bare chips are the theme of the day, though that admittedly fits right in here at Maker Faire.
A holodeck isn't necessarily in the cards, by the way: one of the reasons they're bringing the idea to Maker Faire is to solicit ideas and figure out just what kinds of games and experiences would actually be popular. While Johnson envisions little children filling their Tonka trucks with virtual sand; family board games; and incredible sessions of Dungeons & Dragons, it's really still in the experimental phase, and the team says while some of their creations were a blast (hilarious zombie screams distracting the team from their work!) others weren't as entertaining as they thought they might be. "I suspect we're going to be very surprised about what people find fun in this space," Ellsworth said.
After Maker Faire, a Kickstarter project is the next step: in late summer or early fall, interested parties will be able to pledge money towards the system's development. Ellsworth and Johnson think they can get the cost of a basic system below $200 thanks to the commodity components they're using and their own expertise; Ellsworth has built low-cost chips ranging from video decoders to componentized video games, and is personally working on a low-cost infrared tracking chip. Meanwhile, Johnson is building the software to allow people to build games... impressively, he modified the code to change the game world while I was actually playing. Still, that's a good bit of money and a quite of setup (imagine: you have to have a special surface, glasses, and perhaps a wand or other objects) to play what might be very simple games to begin with.
While their current intent is to build a platform, get the hardware into developers' hands, and see what they build, Johnson says it won't be a dev kit for good: "It's a real product that we will commercialize." It's just too early to say how.
So if the idea is so fantastic, why did Valve give it up? The Technical Illusions team wasn't completely comfortable talking about it. "I came on and I recruited all of my friends, and my colleagues, all the people I could find, and a lot of them are still there. I wish them the best in what they're doing, and I'd hate to do anything to hurt them or derail things they've got going," she says.
Still, before long, part of the story comes out. Valve's hardware initiative was researching both augmented reality and virtual reality... and the virtual reality supporters won out. "You can imagine that Valve is known for a certain class of game," says Johnson. Augmented reality simply isn't as suitable for the first-person shooters that Valve has built its reputation on.
And yet, Ellsworth and Johnson didn't want to let go. "I didn't even want to do VR and AR at first, but someone else in our group was really excited about it and I played along," Ellsworth says, but after six months she decided that AR was the future and apparently wound up on the losing side of Valve's internal struggle. Johnson, working on Valve's Linux team at the time, had already been spending his free hours helping Ellsworth after bonding over a shared love of arcade machines, and decided to join her. "I've done a gazillion first-person shooters, and this is the first time where I can totally do a new type of experience. That's where my passion is," he says.
For its part, Valve has released all interest in the project, leaving the Technical Illusions team legally clear to do what they love. Says Ellsworth: "Gabe was completely behind it... I talked to Gabe, and he talked to the lawyers, and he's like, 'It's theirs, make it happen,' because he could see we were passionate about it."
Video by John Lagomarsino