• The Rise and Fall and Rise
      of Virtual Reality


      In the wake of Facebook’s purchase of Oculus VR, can this revolutionary technology triumph anew?

      Enter
  1. Seeing is Believing:

    The State of Virtual Reality


    The promise of virtual reality has always been enormous. Put on these goggles, go nowhere, and be transported anywhere. It’s the same escapism peddled by drugs, alcohol, sex, and art — throw off the shackles of the mundane through a metaphysical transportation to an altered state. Born of technology, virtual reality at its core is an organic experience. Yes, it’s man meets machine, but what happens is strictly within the mind.

    It had its crude beginnings. A definition of virtual reality has always been difficult to formulate — the concept of an alternative existence has been pawed at for centuries — but the closest modern ancestor came to life in the fifties, when a handful of visionaries saw the possibility for watching things on a screen that never ends, but the technology wasn’t yet good enough to justify the idea. The promise of the idea was shrouded, concealed under clunky visuals. But the concept was worth pursuing, and others did (especially the military, who have used virtual reality technology for war simulation for years). The utopian ideals of a VR universe were revisited by a small crew of inventors in the late ’80s and early ’90s. At the time the personal computer was exploding, and VR acolytes found a curious population eager to see what the technology had to offer.

    Not enough, it turned out. Though a true believer could immerse him or herself in the roughly built digital landscape, the chasm between that crude digital experience and the powerful subtly of real life was too great. The vision simply did not match the means. In the mid-’90s, VR as an industry basically closed up shop. Though still used in the sciences, those eager to bring VR to the masses found themselves overshadowed by a glitzier, more promising technological revolution: the internet.

    Then, two years ago, Palmer Luckey, a kid born during the waning days of VR’s late-20th-century golden era, put the pieces together using improved technology. He raised some money and soon developed the Oculus Rift, his own version of a clunky headset. The graphics were still basic but the experience was, surprisingly, lifelike. For the first time ever, one could casually wander through a comically realistic rendering of Jerry Seinfeld’s apartment. Or hack a zombie to death. It didn’t really matter what you did inside the goggles, really, just the act of immersion was awing. Someone at Facebook got the memo, and they purchased Oculus wholesale for $2 billion, signaling a promising, if unclear, future for virtual reality.

    Imagine 10 years ago trying to envision the way we use cellphones today. It’s impossible. That’s the promise VR has today. VR at its best shouldn’t replace real life, just modify it, giving us access to so much just out of reach physically, economically. If you can dream it, VR can make it. It’s a medium for progress, not the progress itself. In celebration of the rise of VR still to come, The Verge investigated its past, present, and future to offer a glimpse of what we feel is enormous possibility.

    The Rise and Fall and Rise of Virtual Reality was created by Katie Drummond, Ellis Hamburger, Thomas Houston, Ted Irvine, Uy Tieu, Rebecca Lai, Dylan Lathrop, Christian Mazza, Casey Newton, Adi Robertson, Matthew Schnipper, Melissa Smith, Sam Thonis, and Michael Zelenko.

  2. Voices from a Virtual Past

    An oral history of a technology whose time has come again

    Kevin Kelly

    1

    When Facebook bought virtual reality company Oculus in early 2014, virtual reality blew up. While game and movie studios began reimagining the future, others looked back at the "old days" of VR — a loosely remembered period in the 1990s when gloves and goggles were super cool and everyone was going to get high on 3D graphics. But things were never so simple. We spoke to 18 key VR innovators about their work and dreams. What follows is over two decades of memories and visions for what the future could be.

    Some people identify the birth of virtual reality in rudimentary Victorian "stereoscopes," the first 3D picture viewers. Others might point to any sort of out-of-body experience. But to most, VR as we know it was created by a handful of pioneers in the 1950s and 1960s. In 1962, after years of work, filmmaker Mort Heilig patented what might be the first true VR system: the Sensorama, an arcade-style cabinet with a 3D display, vibrating seat, and scent producer. Heilig imagined it as one in a line of products for the "cinema of the future," but that future failed to materialize in his lifetime.

    In 1965, Ivan Sutherland — already known as the creator of groundbreaking computer interface Sketchpad — conceived of what he termed "The Ultimate Display," or, as he wrote, "a room within which the computer can control the existence of matter." He demonstrated an extremely preliminary iteration of such a device, a periscope-like video headset called the "Sword of Damocles," in 1968.

    Meanwhile, at the Wright–Patterson Air Force Base in Ohio, military engineer Thomas Furness was designing a new generation of flight simulators, working on a multi-decade project that eventually became the hallmark program known as the Super Cockpit.

    A few years later, in the late ’60s, an artist and programmer named Myron Krueger would begin creating a new kind of experience he termed "artificial reality," and attempt to revolutionize how humans interacted with machines.

    Jaron Lanier Co-founder of pioneering virtual reality company VPL, musician, and technological philosopher Ivan Sutherland proposed a head-tracked head-mounted display in ’63 as part of the initial invention of computer graphics itself. [He] built one, which is on display at the Computer History Museum in Silicon Valley. When I was a teenager in the ’70s, I was so excited by Ivan’s work that I used to almost jump up and down — stop random people on the street like, "Look at this! Look at this!" — and just make an ass of myself.

    Ben Delaney Market researcher and creator of VR industry newsletter CyberEdge Journal It was a head-mount that was suspended from the ceiling because it was just too heavy to wear. The display was all wire frame, but they were 3D models, and you could change your position and see different views thanks to a tracking system built into the head mount. Ivan was really the father of VR.

    Nicole Stenger Digital media artist, creator of influential virtual reality film Angels There were two inventors who basically found the secret of VR in the ’60s: Ivan Sutherland and Tom Furness. Ivan Sutherland started a major computer company, while Tom Furness was developing the technology inside the Air Force. When he started his lab, he had been kicked out of the Air Force because they didn’t believe in it anymore. They were wrong, of course. After the first Gulf War, when his system was being used by the Air Force, everyone realized that [it] was a major breakthrough.

    Stephen Ellis Head of the NASA Ames Research Center’s Advanced Displays and Spatial Perception Laboratory The first time I saw something that provided full immersion experience was here at Ames [NASA’s Ames Research Center]. Though I’d been in aircraft simulators, Ames was at the forefront of developing [VR technology]. In the ’50s and ’60s, they built a model train-like environment, with all the little buildings and aircraft, and flew a miniature camera across the surface to create a visual that could be fed into the cockpit and looked like an out-the-window scene. And then, while I was here, it actually switched to computer-based imagery using Evans’ and Sutherland’s equipment. Ivan Sutherland was involved in developing some of the fast hardware that would make it possible to do the perspective transformations quickly enough so you could have some degree of interactivity in the systems.

    VPL co-founder Jaron Lanier in a DataGlove and head-mounted display (Kevin Kelly)

    Jaron Lanier Flight simulation was really the first practical digital-simulation application. There are people who put a lot of effort into them and some of them are really cool. When I was coming up in the ’70s, the flight simulators were definitely the highest art.

    Scott Fisher Head of interactive media at the University of Southern California, founder of the NASA Ames Research Center’s Virtual Environment Workstation Project, and co-founder of VR company Telepresence Research [Mort Heilig] was just so brilliant and ahead of his time. He just didn’t have good luck with this stuff. There are four [Sensoramas] left. I feel bad; it’s groundbreaking work. He should absolutely be acknowledged and be a common name in these discussion and he’s not.

    Linda Jacobson Author, founding staff member of Wired, and former "virtual reality evangelist" for supercomputer company Silicon Graphics A lot of new science museums have interactive displays: basically, you’re interacting with projection images that are generated by a computer while your body is being tracked by a camera that interpolates where you are in space and alters the graphics accordingly. The intersection of those two technologies really are at the basis of VR as we know it today, and was first developed by Myron Krueger.

    "It just seemed to me that I was important and the computer wasn’t."

    Myron Krueger Groundbreaking early virtual reality artist and innovator When I got to the University of Wisconsin, I decided to find the biggest computer that I could use by myself and make it interactive. Because of my liberal arts background, I had a much different idea about what computers were for, and so I imagined a more romantic search for a relationship between a human and a machine. I decided to try to find the essence of interactivity. Most people were either on the far end of theory — and I mean stuff that would never be practical in a thousand years — or on the other end, making arguments about what was practical at that minute, and there wasn’t much in between. I just imagined what it would be like to use a computer in the extreme, sort of, and I thought that being able to move around physically was one of the things. I don’t know why I thought all of this was important, but it just seemed to me that I was important and the computer wasn’t.

    As I worked towards thinking about what it would mean to do the computer as a full-body experience, I got involved with a dynamic environmental sculpture called "GlowFlow." I decided from then on that I would focus on interactivity. I vowed to create an experience that would allow a person to go into a room and come out with their attitudes about computers changed.

    I didn’t know about Ivan Sutherland’s statement that the ultimate display would have you sitting, but I knew that I wanted to be able to walk around. So that in this environment, everything you saw and everything you heard would be a response to your physical movement.

    2

    As Myron Krueger developed his own system of projector-based VR, the computer was invading American businesses, and eventually, homes. The Apple II was released in 1977, followed two years later by VisiCalc, a groundbreaking software application that moved personal computing beyond the realm of mere curiosity. Atari was at its peak, having expanded from arcade games to home consoles in the late ’70s and early ’80s. Meanwhile, a new generation of researchers was coming of age, crafting successors to Sutherland’s head-mounted display and Heilig’s entertainment supersystem.

    Scott Fisher I’d been working with stereoscopic imaging since I was a teenager in the ‘60s. I’ve always been obsessed with the idea of images that you are immersed in. I worked on many different ways to present 3D imagery. It wasn’t until the late ’70s and early ’80s, when we started getting some new tech to track where you were in a space, that I started prototyping things to take that into account.

    Tom Zimmerman Engineer, co-founder of pioneering virtual reality company VPL, and inventor of its most famous product, the DataGlove In the late ’70s, I was an undergrad at MIT, and me and my classmate were fantasizing about virtual orchestras. We started thinking about how you could play different chords with your fingers, and that’s as far as we got. It was all on paper. I finished MIT and went to the University of Amherst, and then I went to New York to study electronic music. So then, I started thinking about the orchestra thing again. This time, I was thinking about air guitar. Here I am in Queens now, luckily with no job, and I’m living at my parents’ house. The dreamer in me was saying, "Wouldn’t it be cool if you could play air guitar and really hear it come out of the speakers?" So I set out to make a sensor to [measure] finger-bending.

    Howard Rheingold Journalist and author of Virtual Reality, one of the definitive historical accounts of VR The personal computer was getting really mature because of the visual graphic user interface that Macintosh and Windows provided. When VR came along, some people looked at that as okay, the next step. The last step was moving from a command line interface to the visual interface. Maybe the next one was when you might be totally immersed in the world.

    "The last step was moving from a command line interface to the visual interface. Maybe the next one was when you might be totally immersed in the world."

    Tom Zimmerman I was in New York, I’m studying electronic music, and then I decided to study Assembler programming because I got an Atari 400. Someone decided to give this class on Assembler, and there were 20 people in the class and maybe four of us had Ataris. Everyone else had Apples. One of the people was a woman named Nancy Mayer, and she said, "Oh, why don’t you come to my apartment…" (meaning us four people) "…and my husband, he knows Ataris really well, he can help us." Well, her husband happened to be Steve Mayer, one of the major founders of Atari.

    So her husband’s teaching us how to use the Atari computer and I told Nancy, "I’m moving to California, I fell in love with a ballet dancer, and she got into the Oakland ballet, so I’m leaving." Her husband said, "Oh, you know, there is this music research group forming at the Atari research lab in Sunnyvale, California; would you be interested in interviewing?" And I was like, "Damn, yeah." So I got interviewed and they accepted me and flew me out.

    An early omnidirectional treadmill, used for walking in virtual reality (Ben Delaney)

    3

    The Atari Sunnyvale Research Laboratory was founded in 1982 to explore the future of digital entertainment. The lab was headed by Dr. Alan Kay, an influential computer scientist who had previously worked at the Xerox Palo Alto Research Center. It operated for only two years, shuttering in the aftermath of the 1980s "Atari crash" that decimated the video game market. The lab employed several individuals who would go on to play major roles in the development of virtual reality — including Zimmerman, Scott Fisher, Jaron Lanier, and Brenda Laurel.

    Scott Fisher I went to Atari in California to work in corporate research with Alan Kay. I started working on an immersive display for coin-op arcade environments. A head-mounted display wasn’t feasible, so we built it into something more like an arcade cabinet that you just look into.

    [The researchers] were given a big budget and asked, "What’s the home of the future in 20 years? What’s entertainment and education?" It was a wonderful think-tank of brilliant people making stuff and trying things.

    When Atari crashed, NASA Ames offered to put a position together for me as a research scientist observer. I started in ’85 and we built one of the first versions of a head-mounted display using the wide-angle optics I found and was working on at Atari.

    Mark Bolas Director of the University of Southern California’s Mixed Reality Lab and founder of VR hardware company Fakespace My first encounter with VR really was computer music at UC San Diego. It was one of the first systems where you could synthesize sound completely from a computer. And that gave me a taste for this idea that you could create a perceptual experience that nobody had ever had before. To me, VR is just a visual extension of that. So I did graduate work in computer music. And then at the same time I was doing that work, I was building an underwater telepresence system where you would look into a display and see through the eyes of the underwater robot.

    So I’m in grad school, and I’m pretty sure I’m going to get a job as a product designer. I had an offer from the company I really wanted an offer from. Then, a professor said, "Hey, go visit this Scott Fisher guy down at Ames, because he’s interested in your robot work." I visited him and put on his head-mounted display, put on the glove, and I remember staring, wiggling my fingers in front of my face. It was this horrible feeling, because I realized that I was going to have to say no to this job I just had been offered. I just had to play with this system.

    4

    In the early ’90s, virtual reality’s growing appeal created a broad market for related books, magazines, and newsletters. Ben Delaney’s CyberEdge Journal addressed the business of VR, and MIT launched Presence to cover virtual environment research. University of Wyoming masters student Joseph Gradecki, with the help of his wife, produced 17 issues of PCVR, a bimonthly how-to guide for building home VR systems.

    Ben Delaney I was working for PC World magazine and one of my teammates there said, "Hey, there is this interesting presentation going on — it’s this new technology called VR. You want to check it out?" It consisted of about 47 polygons, all in bright primary colors, no curves, and it operated at about 5 or 10 frames per second. It was remarkably crude, but the promise was pretty amazing.

    "It was remarkably crude, but the promise was pretty amazing."

    Sitting in the audience I said, "This is going to be hot and I want to be involved in it." I arranged meetings with the two or three VR companies that were big enough to have marketing directors. One of them said, at the end of a lunch, "You know what we need? We need a newsletter. No one knows what is going on." So, in January of ‘91, the first issue of CyberEdge Journal came out.

    Joe Gradecki Software engineer and creator of homebrew virtual reality magazine PCVR In the beginning [PCVR] was just for other enthusiasts; other people that had an interest in virtual reality. It was the Make Magazine of virtual reality: I’m going to try and experiment with it, build it, and I’ll tell you how to do it. We were playing around with 1½-inch TVs — everything — trying to build a good head-mounted display that gave you reality within that immersive environment.

    Cyberedge Journal editor Ben Delaney (left) presents an award to W Industries/Virtuality founder Jon Waldern (Ben Delaney)

    5

    Arguably the best-known maker of virtual reality entertainment in the 1990s was W Industries (later called Virtuality), a UK-based company that produced arcade headsets. At the time of W’s first official showing in 1990, virtual reality had already received significant mainstream press coverage. Some writers were skeptical of the technology — British journalist Hunter Davies summed up Waldern’s arcade pod by saying "the helmet was annoying and the game boring." But others saw both potential and danger in a technology that could, as one writer put it, "make users’ dreams come true." The New York Times said that virtual workspaces developed by major companies like Autodesk could have "profound implications," but it warned that "psychologists who now worry about children losing themselves in video game fantasy worlds would no doubt find artificial environments a bigger problem."

    Jon Waldern Inventor, entrepreneur, and founder of British virtual reality arcade gaming company W Industries / Virtuality The first show we went to was the computer-graphics show in Alexandra Palace, London. We had two interactive VR systems linked together in big wooden boxes and a little booth. We had queue lines — I’m not kidding you — all the way around the building. The original direction of the company was to use virtual reality as a development tool for computer-aided design. But very quickly somebody suggested, "Hey, this is just an amazing experience, why don’t you make a game out of it?"

    "I think everyone was hopeful, and looking forward to a change in consciousness. Either that, or they thought we were a bunch of crazy hippies."

    Ben Delaney Cybersex was a big titillation. People thought you would be able to put on some sort of tactile suit and have sexual encounters with real or imaginary people in a virtual world.

    Brenda Laurel Author of human-computer interaction text Computers as Theatre, interactive media expert, and co-founder of VR company Telepresence Research The public perception of the medium was that it was powerful, amazingly cool and that we were about to have our heads turned around by it. I think everyone was hopeful, and looking forward to a change in consciousness. Either that, or they thought we were a bunch of crazy hippies.

    Virtual reality glove and full-body datasuit prototypes, developed by VPL (Kevin Kelly)

    6

    Besides head-mounted displays, no technology is so strongly associated with virtual reality as the wired glove, a sensor-equipped device that tracks the wearer’s movement and location. Many companies manufactured such gloves, but the best-known was VPL’s "DataGlove" — created by company co-founder Tom Zimmerman with the help of engineer Young Harvill. VPL, depending on who you ask, stood for Visual Programming Language or Virtual Programming Language. Founded in 1985, the company was the product of a partnership between two former Atari lab employees: Zimmerman and Jaron Lanier, a musician and programmer who is sometimes referred to as the "father of virtual reality."

    Tom Zimmerman I told Atari about my glove, and they offered me ten thousand bucks. I was considering it, and then my friend in New York said, "No, it’s going to be worth more than this dude, don’t license it to them." And I met Jaron Lanier at an electronic music concert at Stanford — apparently he had worked for Atari, but I didn’t know him. At this point, I had left Atari. Jaron told me about his company; he invented a visual program language with the name VPL, and he had a little tablet as the interface to it. I showed him my glove, and he thought, "Wow, this is much better than a tablet."

    "I used to say, if the resurrection happened in black and white, nobody would cover it."

    Me and this marketing guy had this little startup to do a voice-control synthesizer. So I was doing that during the day and making gloves for Jaron at night. And then, it started getting more interesting. Jaron basically said, "Why don’t we start a company? We can get funding. I know some great programmers. This language plus glove, it looks like a hot combination." It turns out not many people wanted to program. But people loved the glove and Jaron just ran with the concept of VR.

    Myron Krueger VPL had color graphics. I used to say, if the resurrection happened in black and white, nobody would cover it. And Jaron might as well have been sent from Central Casting to be the crazy scientist.

    Tom Zimmerman We did a joint project with NASA Ames Research out there in Mountain View. They, and a fellow Scott Fisher, had this head-mounted display and the idea of astronauts fixing satellites [from] inside the space capsule. And so, we provided the gloves and they provided the HM display, and that created the first goggle VR system.

    Scott Fisher The first commercial contract VPL had was from us at NASA; we asked them to build a five-fingered glove so we could take it to interact with the virtual spaces we had. We built one glove out of the initial fiber optic sensor material, then they improved the tech and we had several more built. They went on to sell commercial versions of that of course, which was great to see.

    Tom Zimmerman When we started VPL, I added some ultrasound hand-tracking technology to the glove — little tiny speakers [that] made a little burst of sound that only dogs can hear. We made an inexpensive glove, which Jaron was kind enough to call the Z Glove, and then we made a high-end one with some sensors. Young Harvill was very clever; he figured out how to make them using fiber optics.

    The little Z Glove we licensed to a New York City game company, which licensed it and turned into Mattel. We hit the big time because they made 1.3 million of this Power Glove, based on the Z Glove.

    A Virtual Technologies force-feedback glove (Ben Delaney)

    7

    The Nintendo Power Glove was released in 1989 at an extremely low cost comparative to available tech. Outside of its normal gaming uses, it became a centerpiece of the homebrew virtual reality community.

    Joe Gradecki When I went up to University of Wyoming for my master’s degree, a couple guys in the computer science department had seen a thing on the Nintendo Power Glove. My wife and I were sitting there and said, "Hey, that’s kind of cool." There were schematics published on connecting it to the PC, and all we needed was code. That was probably 1992, and there was nothing like that around, really.

    Tom Zimmerman To me, Power Glove was the real intrigue. That was where a million people touched virtual reality. I don’t think the implementation was that great, but that was probably the widest-distributed VR system.

    A full-body VR capture suit by VPL (Kevin Kelly)

    8

    VPL would go on to develop more hardware, including a full-body motion-tracking suit and a head-mounted display called the Eyephone.

    Stephen Ellis They were probably one of the first companies to hop on the so-called virtual reality bandwagon. CAE, the Canadian Electronics simulation company, had been making head-mounted simulators from the mid-1980s, but those were really expensive. On the order of millions of dollars for the helmet and the computer-based display system — it was a very expensive thing. And VPL, the company where I believe Jaron was CEO, was selling systems costing on the order of hundreds of thousands of dollars.

    Tom Zimmerman The golden year was 1987, at least for me. I had my hand on the cover of Scientific American and National Inquirer. By that time we had production going on, so my role was pretty much done. Now it was a company cranking out products, more software-oriented. Also, I was a bit burnt out from working 14-hour days, so I took off and went to a farm in West Virginia.

    That was kind of it for me in the VR world. I came back to help them out in a lawsuit [over the Power Glove], which was yucky. We were a little research company doing very innovative work. But other people started catching on, and some people started trying to use our IP without licensing it. The company started spending more money on legal defense than R&D.

    Skip Rizzo Head of medical virtual reality at the University of Southern California I started off working in injury rehabilitation, brain injury, and it’s a very difficult process. It can be hard to motivate people with brain injuries to do their rehab exercises. So one of my patients, a young kid, maybe 21 years old, was sitting there one day and he had this thing called a Game Boy. He was bent over, glued to Tetris, focused on it, he couldn’t put it down. And I said, "Ooh shit, what about using this for rehab exercises?"

    "What Jaron brought to the table was the ability to communicate to the public, to the press, these ideas."

    I was driving to the gym in 1992 and Jaron Lanier is on NPR doing a segment where he’s talking about a virtual kitchen in a department store in Japan. Here I am, having one of those driveway moments listening to this interview. I didn’t go into the gym, I just sat in my car because I had to finish the show. I thought shit, we could improve brain function with these patients, we could rehab and train these patients in a way that’s immersive and fun. There was a bookstore next to the gym. I went in and I said, "Give me every book you have about virtual reality." I think they had maybe three books. I went to the gym and read while I worked out.

    VPL datasuit diagram (Kevin Kelly)

    Tom Zimmerman I think if it wasn’t for Jaron, I would have had this invention, but it’s not clear if it would have made the big time. Because what Jaron brought to the table was the ability to communicate to the public, to the press, these ideas.

    Jaron Lanier The profile of VR, VPL, and perhaps of me was relatively… it was pretty intense. I was younger, you know? I think it got to my head a little bit at one point; I had trouble dealing with it. I wasn’t mature enough to be ready for some of it.

    Brenda Laurel It was just an ugly scene. [Jaron Lanier] hired a bunch of successive CEOs but was unable to collaborate. So he had to continue to be the head guy, and he wouldn’t take advice from people about running a company, so there would be this revolving door of people coming in trying to straighten him out.

    9

    During its rise, VPL accepted funding from French electronics and defense company Thomson-CSF and put up its patent portfolio as collateral. But Thomson later foreclosed on its loans, driving VPL into bankruptcy by early 1993 and acquiring its VR patents.

    Ben Delaney VPL went bankrupt in late ’92. They were — and Jaron Lanier is — one of these organizations that’s sort of like the Kardashians: famous for being famous, more so than for what they had actually produced. VPL was, for some reason, a media darling. They did a lot of good stuff, and produced some good systems. But ultimately, much of the most important work was done by other people. And they didn’t last very long. They were a bright star that burned out quickly.

    "They were a bright star that burned out quickly."

    Linda Jacobson Thomson later auctioned off the VPL patent portfolio, I guess to recoup their investment because apparently that company never implemented any of the designs or inventions. A number of companies bid on it, including the company where I worked at the time, Silicon Graphics. But Silicon Graphics was outbid by Sun Microsystems, which did purchase the portfolio. I never heard again of any kind of hardware innovation coming out of that patent portfolio.

    Jaron Lanier The thing to remember about the fall of VPL is that I was gone by then. I left VPL for a lot of reasons — mostly personal — in ’92 or so. VPL fell a year or two after that. If you were to believe what’s in the papers, it’s a pretty exotic thing that happened: the Justice Department uncovered a plot by the French Secret Service to infiltrate it to steal technology. [It] struck me as absurd because there wasn’t much hidden or proprietary. I mean, it wasn’t like there was anything to really steal there. It wasn’t like a death ray or anything, it was really just some code to simulate, like, a gall bladder.

    10

    The Lawnmower Man, released in 1992, introduced VR to a broad audience. The film starred Pierce Brosnan as a scientist testing virtual reality therapy on a mentally disabled man played by Jeff Fahey. Brosnan’s character, Dr. Lawrence Angelo, develops superintelligence, and eventually, virtual godhood. Despite his heavily fictionalized depiction of the technology, writer and director Brett Leonard drew on work by real-world companies like W Industries and VPL.

    Brett Leonard Director of The Lawnmower Man and Virtuosity: When I made my first little zombie movie called The Dead Pit, which did very well and launched my career, they came to me with this seven-page short story by Stephen King. I said, "Well, I don’t really want to make that movie. I actually want to make a movie about virtual reality." I couldn’t really make a feature film out of a seven-page short story about a guy being chased by a lawnmower.

    "I couldn’t really make a feature film out of a seven-page short story about a guy being chased by a lawnmower."

    I wanted to make something around this computer revolution that I was literally living in the middle of. When I moved from Ohio to Santa Cruz, I was hanging out with people that were hanging out with Wozniak and Jobs. As a young filmmaker, I got a view into that culture, into the thought processes that were going on.

    David Traub Film producer and virtual reality consultant on The Lawnmower Man I got hired as head of technology for a children’s company in ’90. In maybe ’92, we hired a young director [Brett Leonard] to do a music video for us. He came up to me and said, "You’re the technology guy here, have you ever heard of something called VR?" I showed him my two-year-old thesis and he goes, "Oh man, I need your help."

    Brett Leonard I actually had to create a video that described what virtual reality was and showed what computer graphic effects were. I mean, the only effects going on at that time were on a 100 million dollar movie. The Lawnmower Man was a $5 million film, and so I had to find a way of putting over 20 minutes of high-end digital effects into a low-budget film.

    Jon Waldern They were a real small production company before that movie. I met Brett Leonard at a rock concert in London, and we agreed to provide all the support and training so that people could get a whole experience of it. For various reasons they didn’t use our headsets in the movie. But we were actually on the lot with the machines training everybody and giving them VR experiences. And of course the movie went on to become a huge hit. It was great for us, because it was right around our IPO time.

    Ben Delaney The Lawnmower Man made [VR] accessible to the general public in a way that it hadn’t been before. People who weren’t familiar with the technology or computing cybernetics took it pretty seriously — there was a lot of talk about how to prevent these terrible things from happening. It was pretty much a joke in the industry. People either liked the movie or not, but no one actually working in the field took it very seriously.

    Jaron Lanier The funny thing about the fall of VPL is it turned into this weird movie called The Lawnmower Man. The director had licensed a Stephen King horror thing, but then thought the VPL story was more interesting. He turned it into this thing and I was played by Pierce Brosnan, who was an unknown. They used real VPL equipment.

    Tom Zimmerman So I’m watching this movie and Pierce Brosnan is putting on my glove on and I’m like, "You’re going break the fiber optics. You’re tugging it too hard." It was really amazing to see my brainchild come to life.

    That was my greatest reward. Not as much the movie, because it was a little too unreal and a little sinister, but seeing a product out there and actually being able to parlay that to help teach kids about inventing and get their excitement up.

    Palmer Luckey Inventor of the Oculus Rift virtual reality headset and co-founder of Oculus VR, acquired by Facebook in March of 2014 I love Lawnmower Man. I think it’s awesome. In most fiction about VR, the technology is causing some terrible problem; or it’s caused the end of the world, or used to enslave humanity. But that’s because it wouldn’t be interesting science fiction if you said, "Well, guess what? There’s this world and they have perfect VR technology, and people have a healthy balance between real life and VR that makes everyone more productive and happy."

    Virtual reality concept art by Jaron Lanier (Kevin Kelly)

    11

    Beside scientific and gaming applications, virtual reality also emerged as a medium for interactive art. In the ’70s, Myron Krueger projected video images over participants in a dark room, creating the illusion of an artificial reality. The dreamlike nature of VR drew artists and thinkers interested in psychedelic experiences, sometimes in surprising ways: Timothy Leary spent years promoting VR, at one point appearing in a demo video for the predominantly business-focused design company Autodesk. After trying the technology, onetime Grateful Dead lyricist and Electronic Frontier Foundation co-founder John Perry Barlow wrote that, "I have become a traveler in a realm which will be ultimately bounded only by human imagination." Through the ’80s and ’90s, a new generation of artists repurposed cutting-edge (and often extraordinarily expensive) technology to craft virtual reality experiences, including Nicole Stenger’s immersive movie Angels, Char Davies’ interactive virtual environment Osmose, and the folklore-inspired Placeholder, made by a team that included former Atari researcher Brenda Laurel.

    Myron Krueger In one exhibit of mine, Metaplay, when people came in the room the lights were on and they could see their own live video image on the screen. Most people had never seen a video projection in any circumstance, and they had not seen themselves on a video projection. Some of the people would simply sit down and gape at themselves in awe. I would use a data tablet to draw on the screen. And the camera picked up what I drew on that computer, superimposed it on the video image from the gallery, and projected it in front of the people.

    "West Coast VR has a lot to do with acid, I think, because people were kind of mind-prepping for alternative realities."

    When I would move the cursor to start drawing on them, they would treat the cursor like it was a living thing: they would hide from it, or try to hit it away. So then I made the cursor into a ball and put the ball on the screen. I had them playing virtual volleyball with a nonexistent ball without saying a word to them. What struck me was that people just accepted this. They had never read about anything like this. They had never thought about this. It was as if in their DNA, they’d been waiting for this moment and knew exactly what to do.

    Char Davies Artist, founding director of 3D graphics company Softimage, later acquired by Autodesk I first became interested in 3D graphics in the mid-’80s. As a painter, I was looking for ways to cross over the 2D picture plane, to find a way to work on the other side of the picture plane. That led me to the virtual space of 3D computer graphics. I got involved in building a software company, Softimage, because I wanted good access to the tools.

    Brenda Laurel [Virtual reality] during the [’90s] was very idealistic. It was consciousness-expanding in a lot of ways. West Coast VR has a lot to do with acid, I think, because people were kind of mind-prepping for alternative realities. But there was a concatenation between the LSD scene, and Mondo 2000 and all those guys. Timothy [Leary] and Terence McKenna even got into it.

    Char Davies I started writing about Osmose in 1993, when the VR that I was aware of and most computer games were shoot-’em-up games. They all involved either a gun, or a pointer, or a mouse. They all involved a hand. I believed that if you made the interface centered on the hand, it reinforced a way of approaching the world in terms of doing things to things as an instrument — either shooting it or manipulating it in some way. So we introduced a hands-free interface that relied on breathing and balance. This shifted the experience from one of "doing" to one of "being".

    "People cried, usually men."

    Brenda Laurel Char’s work, both Osmose and Emphemere, were just poetic. They were a new art form and I take my hat off to her. I was looking at it as a structuralist, "Okay, what can we turn inside out here about this?’ But her sensibility is much more of a fine artist, and so the stuff that she did was just outrageously beautiful. One of the guys who installed it at the museum in Toronto, I think, got his head in it and came out 10 minutes later weeping and saying that his life had been changed forever.

    Char Davies People cried, usually men. I had a number of people come to me and say that when they experienced Osmose they felt that they had experienced dying; they had experienced what death was and they weren’t afraid of death anymore. I think this was due to the sensations of buoyancy, of floating through semi-transparent fluxing landscapes, all very deliberate on our part.

    The only work that I think I found of interest, that actually preceded Osmose, was Brenda Laurel’s work, Placeholder. I think Brenda was one of the few people who was trying to do something with VR outside of the commercial mainstream. Everyone else was basically doing games; entertainment and military, basically.

    Virtuality VR arcade pods (Ben Delaney)

    12

    By the early- to mid-1990s the VR business was in full swing, VPL’s sudden bankruptcy and takeover notwithstanding. Former NASA researcher Mark Bolas moved on to create Fakespace Labs, which released multiple head-mounted displays by 1994. Between 1994 and 1995, Jon Waldern partnered with Atari, Philips, and IBM to produce cheaper business and home-entertainment models of its arcade headsets.

    By the mid-’90s, it became apparent that the virtual reality bubble was about to bust. In 1993, Brenda Laurel added a chapter to her book Computers as Theatre entitled "Post-Virtual Reality: After the Hype is Over." Atari scuttled the partnership with W Industries / Virtuality and merged with a hard drive manufacturer. The headset market as a whole began to collapse, making it harder to pursue VR for art or research. Companies that remained in business, like Mark Bolas’ Fakespace and Silicon Graphics, did so with a lower profile. Once a synonym for the future, virtual reality faded from public consciousness, replaced by booming internet technology.

    Myron Krueger At SIGGRAPH ‘93 or ‘94 [Dan Sandin] demonstrated what was called a CAVE: a stereo projection on all four walls and on the floor. It was incredibly effective visually, very powerful, and it was replicated 60 times within a year. If you had a research lab, and somebody was sponsoring work, you’d show them what you were doing with their money. But then you’d show them the [lab’s] big project, which in this case would be the CAVE. Whatever they thought of what you were doing, they would always be knocked out by the CAVE. When you put on the head mount, it was always a little bit of a disappointment. I used to say that it was vaccinating virgins against virtual reality. Because once you’d done it, you didn’t have an urge to do it again.

    "After 1995, virtual reality was promising as a technology but it was clear that it wasn’t ready for gaming."

    Jon Waldern Our challenge at the time was, how do we now take this from an industrial $60,000–$70,000 machine that sold to arcade operators and Universal Studios and entertainment providers to consumers? We did a joint venture with Atari and showed the consumer version at CES around ‘95, ‘94. Unfortunately our timing was not good, and despite the storied name, [Atari] went bust.

    Skip Rizzo After 1995, virtual reality was promising as a technology but it was clear that it wasn’t ready for gaming. The engineering challenges required Manhattan Project-levels of funding, and that was going to come from the gaming industry.

    And what else happened in 1995? The internet. Suddenly everyone was connected, and virtual reality was this ugly little brother lagging behind. It was the butt of jokes from uninformed fucking idiots.

    VFX1 Headgear virtual reality system (Ben Delaney)

    Linda Jacobson The web really changed things. The fact that there was a proliferation of online access to 3D design tools meant a lot of people starting focusing there and not on the hardware.

    Ben Delaney To a large extent, the internet and the World Wide Web did the VR community a tremendous favor. [The internet] became the new breakthrough technology that was going to amaze everybody. The mainstream press found other, more exciting things to talk about; especially toward the end of the ’90s when very few of the wild [VR] promises had been fulfilled. People just walked away from it.

    Ben Delaney People thought that there would be addictions to VR — that people would end up living in virtual worlds and never coming out. There was concern that there would be physical damage as well as mental damage, and that’s part of the reason that head-mounted displays didn’t take off. Sony came out with the first consumer-oriented head mount in the ’90s. Ultimately, they never released it widely because their lawyers said they faced too much liability if people wore it and did stupid things. People were concerned about the possibility of people trying to drive while wearing head-mounted display. People were concerned that somebody would be using it in their living room and trip over the cord and crash through a table.

    Linda Jacobson Businesses were turned off by the press’s celebration of VR as some kind of mind-blowing trip to another world where you could have sex with avatars and create havoc. One of the interesting discussions we had at Silicon Graphics in the late ‘90s was about not using the phrase VR. It sounded like it was just about games, and everybody knew that that was also a euphemism for pornography.

    Mark Bolas Companies came out with displays that they said were VR, but they [had a] narrow field of view. People tried them, and basically it was this little window. Yeah, the graphics changed when you turned your head, but there is an art to making you feel like you’re immersed, and they weren’t getting that. You’d look around and go, "Oh yeah, well, whatever, I’d rather play on my flat screen; it’s sharper, I still get my 3D graphics." So those hurt the industry rather than helped it.

    Intel virtual reality advertisement (Ben Delaney)

    Scott Fisher I ended up doing more work in Japan than anything else because Japan in general is so tech-smitten and obsessed that they just love the whole VR development. The Japanese government in general was funding research, building huge research complexes just to focus on this. There were huge initiatives while there was nothing happening in the US. I ended up moving to Japan and working there for many years.

    Ben Delaney I could never figure out how to make any money with CyberEdge Journal. Ultimately, the internet did me in because when the World Wide Web happened, people started saying information wants to be free and I kept saying, "But my printer wants to get paid." I offered the mailing list to a competitor and he managed to keep going for another couple of years.

    "VR didn’t bust from my perspective. The VR hype busted."

    Mark Bolas VR didn’t bust from my perspective. The VR hype busted; but frankly Fakespace was doing great. We were selling lofty displays to people solving real problems. They could do 3D visualizations of where an oil well should go; they could get really abstract data and visualize it in a 3D world. For the consumer market, 3D games on a flat screen was a pretty big step. So I never viewed VR as dying; I viewed VR as just being put onto a 2D screen.

    Nicole Stenger [Tom] Furness said there would be a winter of VR and he was absolutely right. But the seeds that have been planted before the winter don’t necessarily die. There’s spring sometime, where they start blooming again. I’m cautiously optimistic. I believe that it will come back and it will bloom and be so much the better.

    A Division Pro-vision 100VPX virtual reality system (Ben Delaney)

    13

    Throughout the late ’90s and early 2000s, virtual reality companies continued to operate, but with a lower, more pragmatic profile. The military became the biggest advocate for VR’s utility. 3D graphics continued to advance, but referring to them as "virtual reality" became increasingly rare. Companies periodically showcased virtual reality systems and peripherals, but despite protests from Lanier and others, the "death of VR" had become a standard narrative.

    Then, in 2012, a young entrepreneur named Palmer Luckey revealed a $300 virtual reality headset called the Oculus Rift. While the Rift became a symbol of VR’s resurgence, the groundwork had been laid years before. Luckey had worked with researchers like Skip Rizzo, who used VR to treat cognitive and motor rehabilitation — including post-traumatic stress disorder — and Mark Bolas, who had moved to the University of Southern California’s interactive media program. Improvements in computing power and display technology, meanwhile, had solved some of the problems that had proved intractable in the 1990s.

    Skip Rizzo The war broke out, and I had the idea for virtual reality and PTSD, knowing that this was going to be a health problem for the military. My boss said, "I think the military will push this under the rug anyways, so don’t spend time on it." So I said fuck it, and took the idea to ICT [Institute for Creative Technologies]. They jumped on it.

    1995 to 2000 era was a dark period. But then, in 2000, you could start doing virtual reality with a PC. That brought costs down, and it gave us some momentum. From 2002 onwards, we started seeing a lot more innovations that showed promise. But the military, the war, that’s what really catapulted my own work — they had the need, and they had deep pockets, and I was able to do cool shit.

    Mark Bolas In 2006 we came up with this thing called the Wide Five display, and that changed everything for me. It had a really wide field of view, like 150 degrees, and it could use off-the-shelf LCD displays. And we had a contract with the Office of Naval Research to figure out a way to do virtual reality for training. We basically bet that things like mobile phones would come out and drop the price of larger displays.

    Palmer Luckey I think the first head-mounted display I tried was a Myvu Solo. I was like, "This isn’t very good. This is pretty bad." I didn’t know that that was actually the cutting edge for consumer head-mounted displays at that point. It wasn’t until I was in my mid-teens that I started researching it more and realized that actually, VR was not as great as I thought it was when I was six or seven.

    In late 2009, when I was more into PC gaming, I thought, "Well, what’s the best?" I spent months and months researching everything on the market. I just kind of went crazy and started buying all kinds of different HMDs in government auctions and eBay. I kept aiming higher and higher. I was like, "Well okay, consumer VR isn’t that good, but the military uses this for training, so surely the military gear is the real sweet stuff." And then as I got my hands on some of that stuff, I realized that even that wasn’t very good. So I decided that I was going to try to do something myself.

    Tron (1982)

    Mark Bolas The one thing I do think that is getting lost in the VR story is the role of research. It sounds like it all just happened all of a sudden. There has been this consistent thread of funding, largely government-funded projects and European-funded projects, that has been pushing it along step-by-step, and has been creating these labs where these things could happen. It didn’t just come out of thin air.

    Jaron Lanier Lately, Mark has been a professor at USC. In a class, he came up with most of the distinguishing features of the Oculus Rift and posted them as an open resource. The things that distinguish Oculus Rift from other head-mounted displays is basically a use of computation. What they do is compute some changes to the image to make it workable with a cheaper lens system. But Mark kind of did that, with some students — it was pretty much Mark’s work. Then Facebook buys it for billions of dollars. I don’t know what happened, but I would imagine that some administrator at USC probably wanted to talk to Mark after that.

    "I just wish people would dig a little bit and build on stuff."

    Scott Fisher I just wish people would dig a little bit and build on stuff. Most of all, I would hope for the acknowledgment for the work that Mark has done in this area.

    Palmer Luckey I spent a huge amount of time reading through basically every single published piece of literature on VR. I think that there were a lot of people that were giving VR too much credit, because they were working as VR researchers. You don’t want to publish a paper that says, "After the study, we came to the conclusion that VR is useless right now and that we should just not have a job for 20 years." There were a few people that basically came to that conclusion. They said, "Current VR gear is low field of view, high lag, too expensive, too heavy, can’t be driven properly from consumer-grade computers, or even professional-grade computers." It turned out that I wasn’t the first person to realize these problems. They’d been known for decades.

    Ben Delaney I got a real chuckle out of Oculus’ literature when they were claiming an extraordinarily wide 100 degree field of view — a lot of head mounts had better 100 degree field of view 20 or 30 years ago. That’s not adequate. It still feels like you’re looking through a tube. So, none of those problems were solved when I was active in the field; and as far as I know, they’re not really solved yet.

    Stephen Ellis I just saw the Morpheus system that Sony is developing for video games, and it’s extremely high quality. The people working on it have actually been doing it for some years — they’ve developed displays and tracking technology that’s really, really good. It makes Oculus look like Model T in terms of the quality of the tracking, and the imagery is very good. An interesting thing about it is they’re using viewing optics that were used in some of the very early virtual reality headsets — these optics harken back to some early patents that were used in viewing systems that we had in our lab 25 years ago.

    Palmer Luckey Here’s a secret: the thing stopping people from making good VR and solving these problems was not technical. Someone could have built the Rift in mid-to-late 2007 for a few thousand dollars, and they could have built it in mid-2008 for about $500. It’s just nobody was paying attention to that.

    Jaron Lanier I’d predicted, based on Moore’s law type of thinking, that [VR] would become cheap enough for consumers by 2020. That was my sort of my standard answer. And I think it’ll turn out that I was a few years pessimistic. I think that it’s coming a little faster than that.

    "Look, even if we sell 300 developer kits, that’s a huge success for VR."

    Palmer Luckey I reached out to Mark Bolas in July of 2011. I had actually purchased a Fakespace Boom 3C on eBay really cheaply. It looked like I was missing a part that only someone at Fakespace would have. We ended up hanging out and I was able to get a job there as a lab technician.

    My first job was supposed to be sorting and stacking boxes in their storage room, because I knew quite a bit about old VR systems. Later I got the chance to work on a lot of different VR HMD hardware, especially with Skip Rizzo’s team on the post-traumatic stress disorder treatment project. It was a blast working there. There are a lot of really smart, brilliant people at ICT.

    Jaron Lanier and Howard Rheingold (Kevin Kelly)

    I left in May or June of 2012. I had been working on the Rift in my own time, and I’d announced the Kickstarter a couple months earlier, but I kept working with ICT on the side. Right around June, I decided I was going to do this Oculus thing full-time and give it a real go.

    I was worried about the number of people who would back [the Kickstarter]. I think we were going to have a goal of maybe $500,000. The night before, we had a discussion and I was like, "Guys we need to push this down." I was like, "Look, even if we sell 300 developer kits, that’s a huge success for VR. No one has gotten that many game developers interested in VR." We compromised at $250,000. It was kind of a silly worry in hindsight.

    Char Davies I was very pleased when the Oculus Rift appeared, because I have always believed that a sense of immersion is most fully enabled via a headset with a wide field of view. I had been waiting a long time for such technology to reappear. I say "re-appear" because the last wide-field-of-view helmets I used were from the mid-90’s, and nothing suitable has come along since, until the Oculus Rift. But when I heard that Facebook had bought the company, I was disappointed. Twenty years after Osmose, I still believe that immersive virtual space has rich potential for enabling people to step outside their habitual assumptions about reality to see freshly: that the technology will be used instead to maximize advertising revenue is a profound shame.

    Palmer Luckey A lot of people have asked, "Why Facebook? Why not a company like Microsoft or Google, or a company that does hardware?" And to be honest, we’re not looking for a partner who knows hardware, because we have an incredible hardware team. We didn’t want to be bought by somebody who was going to shred us apart and make us part of their product line.

    Jaron Lanier There is something that kind of made me happy about Facebook buying Oculus because there wasn’t some big population of people, like there was with WhatsApp, that you could spy on because you captured them. It was the first major acquisition in Silicon Valley in a while that was really motivated by just pure technology love. And I love that. That’s why things should happen in Silicon Valley. The fact is they tried the demo, they said, "Oh my god, this is the future, it’s incredible, let’s get into this." That’s what should happen.

    "It just feels like they’re recycling the same old press releases and nonsense that people were talking about 20 years ago."

    Ben Delaney I used to make these buttons that said, "VR hype" with the negatory crossbar on it. It was always a big deal. And it never stopped. People imagine this as a technologically manifested dream world, and want to do everything they can do in their dreams. I’ve been really, really getting a chuckle out of reading the hype about the Oculus, because it just feels like they’re recycling the same old press releases and nonsense that people were talking about 20 years ago.

    Jaron Lanier The thing that’s really struck me is how many of the little tropes and stories and controversies and press reports and little weird clichés of the saga of the Oculus company are so similar to the ones that happened for VPL so many years earlier. It would be a project for someone someday [to] correlate ’80s press with the press of the last couple years. You could find a lot of the same headlines.

    Linda Jacobson I really believe that we were talking about a new medium of artistic and creative expression. And that’s my personal thing, rather than it being just a game machine. I still kind of get that sense that when Oculus talks about consumer VR, that’s a euphemism for gaming and pornography, which was definitely the albatross around VR’s neck in the early days.

    "To me it was a beautiful idea."

    Skip Rizzo There’s a risk if all this is only driven by gaming. I just hope that doesn’t happen, because with these headsets we’ve got now, there’s so much other stuff you can do. I can see a renaissance, in education, in health, in so many things. If you’ve got a $300 head-mounted display, and you can run it off a laptop or even a computer in the headset itself, you’ve got a low-cost device that will change the way we do clinical things: for rehab, for psychologists, for educators. It could be transformative.

    Myron Krueger It’s just a sin that virtual reality wasn’t recognized early on. I mean, the gesture interface is the signature technology of Apple, it is the Kinect system, and every news program has a map where people are drawing with their fingers. Virtual reality is, if not exactly an industry, certainly an idea that permeates everything.

    I’m not sure where people think virtual reality is today; it may be where it’s always been: full of promise and a little slow on delivery. But it was the idea. To me it was a beautiful idea.

    Compiled by Adi Robertson and Michael Zelenko with additional reporting and interviews by Katie Drummond, Casey Newton, and Melissa Smith

    • Imminent reality


      Three artists offer far-out concepts of a VR-saturated world

    When it comes to imagining the future, talk is cheap. We can study trends, consult algorithms and heed history, but the coming decades will always be a mystery to the logical mind. Not so to those right brained enough to envision something fresh and colorful and maybe a little scary. Seeking the wisdom of the visually enlightened, we asked three illustrators with very different points of view to imagine a future saturated with virtual reality.

    Andrew Archer

    Melbourne, Australia

    “The process of the illustration was initially just to show an infinite reflected perspective of a person proceeding to put on VR goggles / display but then I realized the human element was missing. The image of a human falling through the various levels of VR then strengthened the concept and connection between oneself and the thought of falling within a virtual reality.”

    Jim Stoten

    London, England

    “I would like to show a futuristic street scene, showing people of the future using virtual reality technology in an everyday way. Some people will be walking alongside their own projected pet dinosaur … Some will be talking to hologramatic projections of business colleagues or family members on their telephone — eye projectors. Some will be playing huge interactive video games on screens the size of buildings, with large VR hands. Some will be watching TV shows as they walk along.”

    Maiko Gubler

    Berlin, Germany

    “Virtual reality is already a reality. A gestural iPhone emoticon or a simulated packaging foam — all is a 100 percent actual and based on very physical resources. Increasingly transparent and immersive wearable devices will force our notion of the "real" materiality. I wanted to expose the constructedness of our realities and, at the same time, make the dissolving seams more apparent again.”

  3. An Oculus Rift in every home

    Cory Ondrejka on Facebook's $2 billion bet

    It’s still early days in the VR game, and only two companies stand ahead of the pack, ready to put a virtual reality kit in the house of every American: Sony and Oculus, now owned by Facebook. We sat down with the heads of VR at both companies to talk about making Virtual Reality everyday reality.

    Cory Ondrejka, the co-creator of Second Life, is now VP of engineering at Facebook. Ondrejka introduced Mark Zuckerberg to Oculus, and spearheaded Facebook’s acquisition of the up-and-coming company. Now, it’s Ondrejka’s job to help Oculus build out its virtual reality vision as quickly as possible, and distribute it to millions upon millions of people. He spent a few minutes telling us about his virtual dreams — and how they’re being turned into reality.

    Why is Facebook so interested in Oculus?

    Cory Ondrejka: Our mission is to make the world more open and connected. That means people need tools to connect and communicate, and be able to share their experiences with their friends and each other. What VR opens up is an entirely new way to share those experiences by letting you share a sense of place and presence, and really move beyond what we’re used to with what we can share from a cellphone or photo. That opportunity to be able to partner with a team that’s built this incredible technology, that pushes us across the discontinuity from malfunctioning VR to functioning and amazing VR, and looking at what that platform creates for communication, creativity, expression, it’s right in the middle of our mission.

    What was your first experience with the Oculus Rift like?

    Ondrejka: My first experience was this amazing sense of presence. I’d been tracking Oculus all the way back to its Kickstarter days because it’s a space I’ve been in for almost 20 years. For all of us who grew up around the waves of VR, every time a new wave would come out, you’d hope we finally crossed the threshold of performance and weight and comfort. And putting Oculus on, it had crossed those thresholds. It was this moment, like, they’ve done it.

    What impressed you most about it?

    Ondrejka: First, visual performance. The difference between a slightly laggy experience or bad registration, or an experience with slightly malfunctioning optics, makes you half feel like you’re in VR and your brain has to adapt to it. With Oculus you didn’t have that. Instead, it was able to cross that threshold into presence where your brain is saying "Well, this is real," and that difference is fundamentally the difference between VR that’s a promise and VR that’s actually here.

    "It was able to cross that threshold into presence where your brain is saying "Well, this is real," and that difference is fundamentally the difference between VR that’s a promise and VR that’s actually here."

    It’s solving 1,000 hard problems, and solving each one of them well enough to cross that threshold. This has always been the challenge of VR. You can’t just get one piece perfect, you have to get them all over the line. The amazing thing about the Oculus team is how they were able to bring a level of focus to hardware and software, and to push all the pieces across the line.

    Is VR going to bring people closer together or push them further apart?

    Ondrejka: So let’s say you have a friend getting married and you can’t be there. Today you’re going to get texts and videos. But what if you could put a 360 video camera in the audience? Then, what you have is an ability to really feel like you’re there, and look around and see what’s going on in a way that’s making an incredible connection. Moviemakers will take advantage of this. You think about what you could do with director’s commentary or additional scenes. What if you could turn around and have these 360-degree views of the set while they’re shooting? Wouldn’t that be the coolest thing ever?

    Today, we capture places in a few ways. I can compress a space through text, send it to you, and you reproduce it in your head. I can take a photo that can be incredibly powerful, but it captures a singular moment or singular view. Then you move to video and you have time, you’re moving forward. The next step is actually being able to capture all 360 degrees and depth and everything that’s going on. If I want to get you excited about visiting my favorite spot in Hawaii, looking down at this caldera, that sense of being able to look over the edge is in VR. You’ll say I need to get on a plane and go there tomorrow. That’s what we’re on the cusp of.

    Whichever [company] will be dominant we don’t know, but the good part is that we don’t have to know. We’re opening a new palette for creators.

    The core of Facebook was the original profile, this idea of it being the first five minutes of a conversation you have when you reconnect with somebody. It’s who you are, where you are, what you’re doing, and a picture of you. As that moved forward through time we added more details about you, more photos, videos, and the leap of being able to say, "This was the place where I was," or, "Here’s where I am." Bringing that through in VR is gonna be a much richer way to share my experience with my friends.

    Do you think it could also replace how we use the basic internet?

    Ondrejka: [It’s not clear if] the right way to represent the web is some very literal spatial model. That’s where things get more complicated because the web is so vast that spatial models tend to break down. You can’t put enough things close to you if you’re trying to do a truly spatial model. You don’t want text floating in space in random orientations — you want it where you’d hold a book. There are limitations in how we’d consume 2D content in 3D.

    "You can name and locate 1,000 things scattered around an apartment, but you can’t remember the file structure on your computer or the files on your desktop."

    We are evolved to operate in 3D spaces. You can name and locate 1,000 things scattered around an apartment, but you can’t remember the file structure on your computer or the files on your desktop. We are very good at remembering spatial environments. VR has the advantage of being able to plug into that.

    The other input is what we’ve all seen in science fiction movies. The great thing about science fiction movies is that we have a 30-40 year heritage of people exploring how you’d represent imaginary content. The challenge there is that while those imagined interfaces tend to be exciting visually, they are built to be entertaining, they’re not built to be useful. No one’s had to live with a sci-fi interface for eight hours a day at their desk.

    How soon is VR going to be in the average American home?

    Ondrejka: As soon as we can get it there. When you think about companies acquiring other companies, everyone focuses on numbers, but the real meat is, how do you accelerate them, scale them, and enable them to move faster? For [Facebook], with Instagram, it was about how we can help Kevin [Systrom, Instagram founder] and his team really be able to grow quite a bit faster than they could as an independent company. The focus is how do we let [Oculus] actually get to what you’re asking even faster than they could’ve otherwise, and with a better product? We’ll get it out there as quickly as we can.

    What’s exciting for us is between Oculus and Facebook we have the opportunity to make it available really broadly. When you think about all the content created and games invented, having a larger, more available audience is critical for this expansion. What’s going to be the killer app? While we don’t know exactly what it’s gonna be, we’re confident there will be a bunch of them. And that killer app is gonna be so much better than anything you can experience on a screen or phone.

  4. The virtual reel

    Two documentaries about real-world problems solved in the virtual universe

  5. Digital natives

    A conversation between virtual reality visionaries Jaron Lanier and Kevin Kelly

    Kevin Kelly

    In 1989, Kevin Kelly, an itinerant writer, photographer, and future founding editor of Wired magazine, traveled to Redwood City, California, to meet Jaron Lanier in his office. Four years prior, Lanier — a computer scientist with a mane of thick blonde dreadlocks — had founded VPL Research in an effort to bring virtual reality hardware to the marketplace.

    "Right before my eyes, Jaron Lanier built an artificial reality and then climbed into it," Kelly wrote for his Whole Earth Review magazine at the time. Kelly wrote an introduction to Lanier’s work at VPL, capturing the growing optimism about the technology as well as fears about its implications. He also organized an accompanying interview with Lanier, in which the programmer attested that, "We’re witnessing the birth of a culture here."

    Lanier may have been overly optimistic: what followed was a decades-long period where virtual reality continually fell short of sci-fi visions. A quarter century later, Kelly and Lanier met again in the light of all that has happened since. I sat in on the conversation, which took place at Lanier’s home in Berkeley, and tossed out the occasional question. But mostly I listened as two of technology’s leading voices brought real scrutiny to virtual reality’s past, present, and future.

    Highlights from Jaron Lanier's end of the discussion are below. And for those with a deeper interest in virtual reality, we’ve transcribed the full interview.

    Highlights from the conversation

    On our disappointment with technology

    I wonder if the reason we keep on cycling back to hope about cool things like VR is that for all the tech news and our fetishizing about our touch devices, we’re still a little disappointed in the menu of tech items that we have at this late date. It’s 2014 and you can buy a robot to clean your house, but it doesn’t really work that well yet. We all think it will, some day, but it’s not quite there. We have some demos of cars that drive themselves but you can’t really buy one. Everything is kind of taking so long. So I feel like, in a way, we keep on cycling through the same tech hope stories because there’s an impatience and frustration. We wait until we’ve forgotten one of them, then we rediscover it. So I think there’s a little bit of that going on.

    On spurred progress

    The components have finally gotten cheap enough that we can start to talk about them as being accessible in the way that everybody’s always wanted. Although, it’s one thing to just have an affordable headset that’s decent, it’s another thing to have a whole system. Moore’s law is so interesting because it’s not just the same components getting cheaper, but it really changes the way you do things. For instance, in the old days, in order to tell where your head was so that you could position virtual content to be standing still relative to you, we used to have to use some kind of external reference point, which might be magnetic, ultrasonic, or optical. These days you put some kind of camera on the head and look around in the room and it just calculates where you are — the headsets are self-sufficient instead of relying on an external reference infrastructure. That was inconceivable before because it would have been just so expensive to do that calculation. Moore’s law really just changes again and again, it re-factors your options in really subtle and interesting ways.

    "Moore’s law is so interesting because it’s not just the same components getting cheaper, but it really changes the way you do things."

    On how his feelings about VR have changed

    In the ’80s, I had maybe an outright mystical approach to it. For me, the very most important thing about VR was that when you were in it, you’d feel your own existence in the sense that if all the sensory input is artificial, then what’s floating there, that’s your consciousness. So to me, it was sort of proof that subjectivity is real; that consciousness is real, that it’s not just a construct that we put on things. Just to notice that you really exist, to me was the very, very core of it. There were a zillion and one variations on that, that can become really vivid and colorful in different ways. But that was always the core for me. And extending from that, this possibility of a kind of communication that would involve directly creating what people sense in common instead of relying as much on symbols such as words.

    KK: The post-symbolic…

    J: Post-symbolic communication, yeah. I used to go on and on about that stuff. And I can still do it, if asked. At the time, I shared a kind of idealism about what digital stuff would do to the world with my friends. And I’ve actually been connecting with a lot of people from those days. I’ve just recently started emailing with Richard Stallman again after years of being out of touch; and many other people from those early days. You know, I think all of us had the sense of mission that we were really doing something that would open up the world, and that a lot of mankind’s problems were kind of just artificial and due to inadequate technology: if we could just have better communication and all this stuff, a lot of problems would clear up.

    I had to reconsider that ideology at great personal pain because I didn’t want to question it. For me, it just took a lot of individual people not doing so well as digital stuff rose up. And, in my case, the particular thing that bothered me was initially seeing musicians not do as well as I thought they would. There was a time, up until around the turn of the century, I was writing fire-breathing essays like, "Piracy is your friend" and "Open everything up and it’ll work out." Then, when I started looking at the numbers of people who were benefitting, I realized that what was actually happening was the loss of the middle hump of outcomes; we were concentrating people into winners and losers, which is the worst outcome. I’ve also become really concerned about VR’s role in that.

    On the strange things researchers are doing with VR

    A few researchers started to do experiments that I would have been terrified to do myself. I’m thinking of a person who has been a research partner, a collaborator for many years — Jeremy Bailenson at Stanford. He started to just sort of see how he could screw with people in VR. I was always like, "Can we give them better math abilities by changing how their bodies work?" — that was the kind of thing I was interested in. [Bailenson] was like, "Hey, I want to see if I can screw with their self-esteem by making them gradually shorter during an interaction, or turn gradually more black during an interaction." And he can. This notion that you could see VR as a way to screw with people without their awareness, crossed with our current business model where everything is about advertising and manipulation and spying — we [will] have a surveillance economy in the online world. It’s been very painful to see that potential unfolding.

    "We [will] have a surveillance economy in the online world. It’s been very painful to see that potential unfolding."

    The whole spectrum is true at once. I think all the mystical, ideological stuff is still as true as it ever was. The potential is all there, and the beauty is all there. But this potential for manipulation is also there.

    On how virtual reality has evolved over the past 25 years

    The biggest thing that’s happened is that the industrial use of VR — as opposed to consumer and entertainment use — matured, and has become ordinary enough to be boring. But it hasn’t happened in this sort of big, unified way; it’s a bunch of little pockets that are each very specialized.

    The major ones are surgical simulations and training — in fact, that one got good enough that at this point our main concern is overuse; there is a danger that surgeons are spending a little too much time in simulators and there might be some damage done. That’s not known, but that’s been a concern. We know it’s true for flight simulators. For instance, the crash that happened here in San Francisco, the Korean airliner, has been attributed in part to an overreliance on simulator training and an over-reliance on automation in the cockpit, although I think there were some other issues too.

    Another one is vehicle design — you basically cannot buy anything that can surround and move you that isn’t first prototyped in VR now. Every car, boat, plane, civilian, military — everything is designed in VR now.

    On trying new models with VR

    Every time there is a new platform, there is a change to create a new economic model. 3D printers for hobbyists have kind of come out on a Linux model, where everyone shares their models. What if that had happened on a pay-per-model business where everyone could pay and get some money out of it? It would be an experiment. A lot of people would really be offended by the very idea of it. But what if it actually generated a lot of cool models and if it gave some people the ability to pay for their kids’ college educations? What if that stuff actually worked out really well and everyone ended up happy?

    I don’t know. I mean, look, with these things, ideology isn’t a good measure. You have to be empirical. Starting up a VR platform is another one of these points where we could empirically test it and say, let’s do this one in a whole new way that’s never been done before: [create a] really easy, convenient universe of micropayments where everybody is on equal footing, everybody is a buyer and seller, everybody is a first-class citizen, and just see what happens. It might be great. I have a feeling with VR that it could be great — the thing about it is it does take a lot of effort and craft to make good stuff; to make a good Minecraft world, or to make a good world in whatever thing you do. It would make sense for society to design that as a way [in which] you can actually directly make a living.

    On the pleasures of VR, past and present

    The first thing I’ll say is I use VR a lot these days in research of different kinds — cognitive science research and some visualization stuff. I also love working with exotic optics and sensors and I still play around with that stuff. And I absolutely take time to play beyond what’s needed for the research. I still find great pleasure in screwing around with it. But I’ve always felt, from the very beginning that it’s — how would I say it — you know, every musician I know prefers to live without the radio on, enjoys the silence and contrast. The best way to use VR, not in some moralistic or judgmental sense, but just in terms of my experience, is [to] use it as little as possible and enjoy the contrast that it gives me to reality.

    "I’m kind of bracing myself; waves of teenagers whose bodies have become so inactive from being immersed that hospitals have to sever their limbs or something."

    At the old lab, VPL, one of the things we’d sometimes have is just a flower sitting there. So if somebody was in a demo for 20 minutes, we’d come out and say, just look at this flower. And you suddenly saw this flower in this hyperreal way because your senses had adjusted to this sort of lower resolution of a virtual world. Then, when you see reality, you suddenly see it with this kind of detail and this density. You see just the sheer reality of it. You just feel things from it. It’s really incredible. To me, that contrast, that feeling that you have when you’re out of it after you’ve used it, has universally been more precious than what happens in it. So, yeah, I like it. I might be wrong about this, but I suspect that a lot of people will find what I found: that the coolest thing to do with [VR] is not to be in there for hours the way people are with their pocket devices these days, just staring at the screen…

    KK: Or playing games.

    J: ...or playing games. The coolest thing is to come out of it for hours. I think that’s really just the most amazing thing.

    On the importance of experimenting

    J: As for what people actually do, I’m kind of bracing myself; waves of teenagers whose bodies have become so inactive from being immersed that hospitals have to sever their limbs or something. I’m very much hoping that it isn’t that bad but the thing about reality is that it’s not fully predictable. We have to dive in and learn. I think it’s important to make experiments, and it’s really important even to experiment in a way that can be a little dangerous. It’s important to take risks. The thing that’s really a sin is to not learn from them, to ignore the results. That's when you really lose it. As long as we’re awake and paying attention and we learn and get better, that’s what matters.

    On the future of VR and commerce

    KK: As we build VR systems, what things should people keep in mind? Now that we know what we know, how should we build these differently?

    J: I think every technical person is obliged to think about how we can move towards a world that really serves people, rather than splitting us into an elite and everybody else. It’s funny how old all these ideas are. So much of this goes back to 19th-century thinking, and it’s correct thinking, it turns out. It’s just the strangest thing. At any rate, I’m really concerned by the way tech culture has evolved since the clouds got rich, you know? We’re seeing a kind of a tech-supremacy feeling.

    "Facebook is kind of painting itself into a corner where both it and Google are in this mutual embrace of making each other more and more creepy in battle."

    KK: My friends who are making more VR worlds or gear, what do you think they should not be doing that they are doing right or that they should be doing that they’re not doing?

    J: We have to evolve out of what we’re calling the advertising business model. If you extend the idea of advertising to total surveillance in the way that we’re doing it, it doesn’t result in a stable, serviceable way to build a society. We have to all come to that recognition, find an alternative, and it’s never more true than with the VR stuff.

    Obviously, I’m hoping Facebook’s business model will evolve by the time they ship something. Facebook is kind of painting itself into a corner where both it and Google are in this mutual embrace of making each other more and more creepy in battle. And they have to find some way out of that.

    Full interview

    Jaron Lanier tests an environment in VPL’s virtual reality goggles

    KK: The question I’m interested in, Jaron, is whether anything’s changed in 25 years about VR. Did we just sort of skip 25 years and this is a couple months later? Or has there been actual innovation?

    J: Well, no, there’s been a lot of motion on a lot of levels. I should say that in the time between now and then, I haven’t been continuously active in virtual reality. I left the virtual reality field, more or less, around ’92 — I was the chief scientist at the Internet2 organization, which was working on how to make the internet scale; I was a musician in New York and this and that. For me, the 25-year mark isn’t such a thing. [Our] interview was actually ’87, and I believe your visit that you described was, as well. We’d started the company, VPL research, the first VR company, in ’83. My first systems with my friends were something like ’81. And of course, the first head-mounted display was ’65. So to me the timeline goes back a little further. And my personal involvement goes back to about ’79 or so. I tend to think on a 35-year sequence rather than a 25-year sequence.

    The biggest thing that’s happened is that the industrial use of VR — as opposed to consumer and entertainment use — matured, and has become ordinary enough to be boring. But it hasn’t happened in this sort of big, unified way; it’s a bunch of little pockets that are each very specialized.

    "The biggest thing that’s happened is that the industrial use of VR ... has become ordinary enough to be boring."

    The major ones are surgical simulations and training. In fact, that one got good enough that at this point our main concern is overuse; there is a danger that surgeons are spending a little too much time in simulators. That’s not known, but that’s been a concern. We know it’s true for flight simulators. For instance, the crash that happened here in San Francisco, the Korean airliner, has been attributed in part to an overreliance on simulator training and automation in the cockpit, although I think there were some other issues too.

    Another one is vehicle design: you basically cannot buy anything that can surround and move you that isn’t first prototyped in VR now. Every car, boat, plane, civilian, military — everything is designed in VR now.

    Manufacturing processes are generally modeled in a simulation environment, where workers are observed in a simulated environment before machinery is finalized. Another one is security and defense applications, which is huge.There are very elaborate VR tools for all kinds of military scenario training. There are dozens of sub-specialties of different kinds of virtual reality systems for different applications. There’s a maturity, I’d say, to the point where it’s kind of boring.

    Another thing that’s happened is the components have finally gotten cheap enough that we can start to talk about them as being accessible in the way that everybody’s always wanted. Although, it’s one thing to just have an affordable headset that’s decent, it’s another thing to have a whole system. Moore’s law is so interesting because it’s not just the same components getting cheaper, but it really changes the way you do things. For instance, in the old days, in order to tell where your head was so that you could position virtual content to be standing still relative to you, we used to have to use some kind of external reference point, which might be magnetic, ultrasonic, or optical. These days you put some kind of camera on the head and look around in the room and it just calculates where you are — the headsets are self-sufficient instead of relying on an external reference infrastructure. That was inconceivable before because it would have been just so expensive to do that calculation. Moore’s law really just changes again and again, it re-factors your options in really subtle and interesting ways.

    "The components have finally gotten cheap enough that we can start to talk about them as being accessible in the way that everybody’s always wanted."

    KK: Right, I call it the computational x. Anything you can imagine x is, is going to be done by computers instead of whatever it is. So instead of a lens, you can do computation.

    J: Listen, computational displays are a fantastic, emerging area now and those are still hidden from us by the veil of inadequate progress on Moore’s law. There’s still a computational overhead that’s really challenging. But that’s definitely coming down the pike. It’s amazing.

    KK: Explain what computational display is.

    Credit: Ben Delaney

    J: Well, that’s where you do heavy-duty computation to get magic out of pretty simple display components. The terminology is still evolving, actually — different communities use different terms for overlapping meanings. Computational holography is where you compute fringe patterns on a display instead of the actual image you’ll see. So, you’re computing all these tiny edges and because of the quantum nature of life, when light encounters an edge, it can bend a little bit. If you can calculate very carefully how it’ll bend, you can take a simple display, with a whole bunch of little ridges rendered on it, and bend light to create a 3D effect: you can turn a regular display into a 3D display. I’m oversimplifying it, but it’s an amazing thing.

    Another incredible one is usually called a compressed light field — although, once again, there is different terminology in play — where you do a few layers of displays, and by calculating very carefully how they combine together, you can create this magical, encompassing 3D effect where you can really focus into the image even though there are really only a few layers of display. And once again, the computation overhead is pretty significant. We’re getting to the point where we can really calculate fields of energy instead of dealing with just the bulk manipulation of a field, like with a lens. That’s transformative. It really just opens up a huge territory.

    KK: One of the important things that we should keep in mind, [is] that there is not going to be a single version of VR, that there will be multiple types.

    J: It’s going to be like everybody talks about a computer; the term computer doesn’t exactly mean anything anymore. Instead we talk about having a phone, or a tablet, or a cloud service.

    KK: That’s the technology, but maybe we can explore some of the consequences of the technology. Have you changed your mind about [the impact of VR] very much?

    J: Sure, I mean, it’s broadened quite a bit. In the ’80s, I had maybe an outright mystical approach to it. For me, the very most important thing about VR was that when you were in it, you’d feel your own existence, in the sense that if all the sensory input is artificial, then what’s floating there, that’s your consciousness. So to me, it was sort of proof that subjectivity is real; that consciousness is real, that it’s not just a construct that we put on things. Just to notice that you really exist, to me, was the very, very core of it. There were a zillion and one variations on that that [could] become really vivid and colorful in different ways. But that was always the core for me. And extending from that, this possibility of a kind of communication that would involve directly creating what people sense in common instead of relying as much on symbols such as words.

    "So to me, it was sort of proof that subjectivity is real; that consciousness is real, that it’s not just a construct that we put on things."

    KK: The post-symbolic…

    J: Post-symbolic communication, yeah. I used to go on and on about that stuff. And I can still do it, if asked. At the time, I shared a kind of idealism about what digital stuff would do to the world with my friends. And I’ve actually been connecting with a lot of people from those days. I’ve just recently started emailing with Richard Stallman again after years of being out of touch; and many other people from those early days. You know, I think all of us had the sense of mission that we were really doing something that would open up the world, and that a lot of mankind’s problems were kind of just artificial and due to inadequate technology: if we could just have better communication and all this stuff, a lot of problems would clear up.

    I had to reconsider that ideology at great personal pain because I didn’t want to question it. For me, it just took a lot of individual people not doing so well as digital stuff rose up. And, in my case, the particular thing that bothered me was initially seeing musicians not do as well as I thought they would. There was a time, up until around the turn of the century, I was writing fire-breathing essays like, "Piracy is your friend" and "Open everything up and it’ll work out." Then, when I started looking at the numbers of people who were benefitting, I realized that what was actually happening was the loss of the middle hump of outcomes; we were concentrating people into winners and losers, which is the worst outcome. I’ve also become really concerned about VR’s role in that.

    A few researchers started to do experiments that I would have been terrified to do myself. I’m thinking of a person who has been research partner, a collaborator for many years — Jeremy Bailenson at Stanford. He started to just sort of see how he could screw with people in VR. I was always like, "Can we give them better math abilities by changing how their bodies work?" — that was the kind of thing I was interested in. [Bailenson] was like, "Hey, I want to see if I can screw with their self-esteem by making them gradually shorter during an interaction, or turn gradually more black during an interaction." And he can. This notion that you could see VR as a way to screw with people without their awareness, crossed that with our current business model where everything is about advertising and manipulation and spying, we [will] have a surveillance economy in the online world. It’s been very painful to see that potential unfolding.

    "The VR world is sort of like the most saturated tracking environment you can imagine."

    The whole spectrum is true at once. I think all the mystical, ideological stuff is still as true as it ever was. The potential is all there, and the beauty is all there. But this potential for manipulation is also there.

    KK: Actually, I hadn’t thought of that, but you’re right. If you’re in a virtual world, everything is being tracked, in a certain sense; almost by definition, everything is trackable. The VR world is sort of like the most saturated tracking environment you can imagine.

    J: Absolutely.

    KK: And what’s interesting to me is that in our [previous] discussions, I don’t think that ever even came up.

    J: I don’t know if we talked about it. I wasn’t ready to contemplate anything short of hippie optimism in those years. I had confronted it, actually. Just to be clear, Norbert Wiener talked about this stuff in his own way. I mean, maybe not quite as head-on as we might expect today, but from very early on, people saw this possibility.

    KK: I don’t recall that — I mean I know he had lots of complaints but I don’t recall this particular complaint.

    J: The very, very first issue of Whole Earth Review

    "All panaceas become poison: computers as poison."

    KK: "All panaceas become poison: computers as poison."

    J: "Computers as poison" was the cover. These ideas had currency from the very origin of computer science. Norbert Wiener’s book, The Human Use of Human Beings, essentially lays out the terms of the problem — not quite in the language that we’d understand today, but it’s there if you read it carefully.

    In the practically accessible picture, the way we used to put it was like in a virtual world, art is still art, but bullets aren’t bullets. It would naturally pull people away from harming each other into a realm that was still consequential in good ways but less consequential in bad ways. That was the idea — that it would be inherently beneficial. The way events have unfolded, the empirical evidence is not supporting that level of idealism. I feel like I have to adjust.

    Jaron Lanier and Howard Rheingold at VPL

    KK: But at the highest level you say you are still an optimist and we should go and build these systems. As we build VR systems, what things should people keep in mind? Now that we know what we know, how should we build these differently?

    J: I think every technical person is obliged to think about how we can move towards a world that really serves people, rather than splitting us into an elite and everybody else. It’s funny how old all these ideas are. So much of this goes back to 19th-century thinking, and it’s correct thinking, it turns out. It’s just the strangest thing. At any rate, I’m really concerned by the way tech culture has evolved since the clouds got rich, you know? We’re seeing a kind of a tech-supremacy feeling.

    KK: My friends who are making more VR worlds or gear, what do you think they should not be doing that they are doing right or that they should be doing that they’re not doing?

    J: We have to evolve out of what we’re calling the advertising business model. If you extend the idea of advertising to total surveillance in the way that we’re doing it, it doesn’t result in a stable, serviceable way to build a society. We have to all come to that recognition, find an alternative, and it’s never more true than with the VR stuff.

    Obviously, I’m hoping Facebook’s business model will evolve by the time they ship something. Facebook is kind of painting itself into a corner where both it and Google are in this mutual embrace of making each other more and more creepy in battle. And they have to find some way out of that.

    "Facebook is kind of painting itself into a corner where both it and Google are in this mutual embrace of making each other more and more creepy in battle."

    KK: A counter to that is the success of Minecraft, which, as far as I know, is not running on an advertising model; that kind of captures that sense of technology for good, and being creative, and going into these worlds. I wouldn’t call it a VR, but there’s an immersive-world aspect to it. What’s actually impressed me most about it [is that] when I saw it, I would never have guessed that eight-year-old girls would be playing this on their phones — in an addictive way. I would not have believed that. Is that evidence this could work in that way?

    J: Yeah, I absolutely think it is. It’s a great example, and it’s not the only one. There are others in the gaming community that have tremendously admirable qualities. In my utopia, there would be a larger number of people achieving middle-class stability in the Minecraft ecosystem than there are. But, on the other hand, everything you said is absolutely correct. It’s wonderful to see kids work with it, and to my knowledge, nobody is using it to spy on the kids and create behavioral models of them and all that. One of the things about our world today is that in the attempt to create the omni-transparent world, we created the super-opaque, creepy world, you know? It’s just absolutely backfired. Everything that we thought created transparency had precisely the opposite effect.

    KK: [Minecraft] captures a lot of what you were hoping for VR at that time, which is that you have this sort of open-ended sandbox in which you could make anything you could imagine, play in it, and share that world with other people. Now Minecraft, it’s pixelated, it’s blocked…

    J: Well, it’s a start.

    Event producer Sally Rosenthal demonstrates a head-mounted display at NASA’s Ames Research Center

    KK: But it’s very much in that vein. If you just amped up the resolution, all these other factors, then you’d have something very similar to what you’re talking about.

    J: The particular thing is that Minecraft is probably not going to scale a whole lot further. It has a particular identity, which is just great. I have a feeling there will be other designs that scale further. I might be wrong about that, we’ll see. At any rate, I totally agree. It is very much along the lines of what I imagined back then as a form of communication that was presenting forms and dynamics that people invented for each other. So that has come to pass.

    And not just in Minecraft. I think there are a lot of other gaming platforms where you see that sort of thing emerging. There’s a bit of an unfortunate thing that happens in the gaming world — which is a mirror of what happens in the cinema world — where the very biggest budgets and promotions are reserved for what’s perceived as the most conservative bets, which are inevitably these macho destruction fantasies. I suspect that those decisions are actually not the optimal ones from the business point of view; that if people had the courage to tunnel through and try to do more things like Minecraft, they would actually make more money than with more shoot-’em-up things.

    KK: I think the people who are making a living from Minecraft are the people making tutorials and the millions of YouTube videos and the books.

    J: Absolutely. I absolutely celebrate all of that. I just think at some point, a stable civilization would have to generate more means to sustenance from that larger community than our current system is.

    KK: Right. And those occupations, I think, are still opaque to us. In a sense, they’re hard for us to imagine. Who would have guessed?

    C: In the case of Facebook and Oculus in particular, have you thought of a business model direction that wouldn’t be based on advertising and surveillance?

    Jaron Lanier at VPL

    J: Absolutely. This gets to my last book about that stuff. To my mind, there are a few proposals that are on the table that are interesting. The one that I’ve been trying to push the hardest was the origin point for networking when it started — Ted Nelson’s idea of the universe of micropayments. The thing about micropayments is they have to be really universal or else there are only tiny pockets of people who get anything and everyone else gets frustrated and pissed off. It has to be big enough to really send some benefits around to everybody. That’s a hard gap to leap over to get to that point. Facebook has a very tough road to hoe here because they have no background in this stuff. Neither does Oculus. They are kind of starting from scratch, but I certainly wish them the very best success. What they can do is create a system where anybody who wants to can set up a super easy system for micropayments and then add to it. They might see this exploding economy that generates a lot of economic growth, a lot of profit, and a lot of distributed benefits for their users.

    "Every time there is a new platform, there is a change to create a new economic model."

    Every time there is a new platform, there is a change to create a new economic model. 3D printers for hobbyists have kind of come out on a Linux model, where everyone shares their models. What if that had happened on a pay-per-model business where everyone could pay and get some money out of it? It would be an experiment. A lot of people would really be offended by the very idea of it. But what if it actually generated a lot of cool models and if it gave some people the ability to pay for their kids’ college education What if that stuff actually worked out really well and everyone ended up happy?

    I don’t know. I mean, look, with these things, ideology isn’t a good measure. You have to be empirical. Starting up a VR platform is another one of these points where we could empirically test it and say, let’s do this one in a whole new way that’s never been done before: [create a] really easy, convenient universe of micropayments where everybody is on equal footing, everybody is a buyer and seller, everybody is a first-class citizen, and just see what happens. It might be great. I have a feeling with VR that it could be great — the thing about it is it does take a lot of effort and craft to make good stuff; to make a good Minecraft world, or to make a good world in whatever thing you do. It would make sense for society to design that as a way [in which] you can actually directly make a living.

    KK: But didn’t Second Life have a peer-to-peer...?

    J: Yeah, I was an advisor to Second Life. Second Life, to my mind, was a really worthy experiment that yielded mixed results.

    KK: The economic part of it seemed to be good, right?

    J: If you want to move towards an economy that could be sustainable when Moore’s law has moved to some ultimate point — some decades from now — then Second Life didn’t give results strong enough to support that. But [it came] a hell of a lot closer than most other things. It was really a step forward. Second Life was a failure in terms of a design that could interface with existing laws and existing economic systems. It had a huge problem related to taxation and regulation. [With] technology and idealism, you wish away these things, but you can't. There was a degree of fraud, there was a degree of bullying, but overall Second Life was kind of impressive. The biggest failure of Second Life is that it didn’t succeed in holding its audience's attention beyond the initial years. I mean, it did pretty well for a few years but then it kind of fell out of public consciousness.

    KK: It’s still going, yeah.

    J: But I would say it’s not going with the intensity of something like Minecraft. Minecraft has proven some sort of longevity. Second Life, for a brief moment, was probably as big as Oculus or VPL in terms of public attention. It didn’t sustain that. Another thing with Second Life — I don’t think it was extensible enough. There was this problem that a lot of this stuff started to look the same because the tools didn’t have that much breadth. And that was a big issue.

    KK: I think it’s a scalable thing. It’s centralized in a certain sense. With Minecraft, everybody can put it up on their own server and that really helped spread it so you can have a little bit more decentralized, scalable input.

    The Cave Automatic Virtual Environment at the University of Illinois

    J: Yeah, that’s a really interesting question, too. This whole issue about who should own the servers, where they should be, and what works best is actually a very empirical question and is still evolving. I don’t think there is any absolute answer because the right answer could vary with a particular design.

    KK: If we can go back to the idea of there being varying, different species of VR: so some will be more centralized, and some will be more distributed and they’ll just be different forms; they’ll just be different media, almost, in some ways.

    J: Oh definitely. There is absolutely no question that there will be. There are about four major variations on VR: one where you’re all in there, which is like Oculus; a heads-up thing, which would be like Google Glass; another would be mixed reality where you see stuff overlaid and mixed in with the real world; another would be telepresence where you feel that you’ve embodied some device; and another is tele-immersion, where you might [be] in a remote location but with sufficient illusion that it transcends video conferencing.

    KK: What’s the difference between tele-immersion and inclusive?

    J: As a practical matter, you can’t detail immersion with a heads-up display because your face is covered. I’ve tried and other people have tried. Let’s say I’m wearing some kind of head-mounted display and you’re wearing a head-mounted display and we both want to have the experience that we’re in the same room with neither of us wearing them so we can see each other’s faces. We have to have sensors trying to sense our face, and then we have to reconstruct it well. You get the uncanny valley issues. It’s not technically impossible, but it’s a pretty long-term goal to do that. The more likely thing is to have neither person wearing them but instead have them sense from remote sensors and have some sort of display that has a volumetric quality. In practice, they’re distinct.

    KK: So the immersive one is where you are experiencing something. The other one is where someone else is experiencing you.

    J: The totally immersive one, which I call classic VR, [is] like what we were selling in the ’80s. You can see other people but you have to be avatars, you have no choice. In the other space you're attempting realism. You could attempt any degree of avatasation, or whatever the word should be but the...

    KK: Let’s make up that word right now. Avatize!

    C: Avatizing?

    J: Avatized. I’m an avatizer. I don’t know. The terminology just becomes so insane. But I would say the five I just listed are probably the big five.

    KK: So it’s just a matter of degrees in terms of how much of the overlays are visible most of the time?

    J: You could say it’s a matter of degree, but in practice, it requires a pretty different strategy. For instance, with Google Glass, the focal plane you focus on doesn’t really have to match what’s there. Another issue is one eye versus two. If you really want to do mixed reality, you kind of have to do it with both eyes. [While] heads up is actually much, much, much more sensible with one eye. You can say it’s degrees, but in practice they really diverge into two different designs.

    KK: You said you’ve been involved with [VR] since the '70s into the '80s, thinking about it and very actively building stuff. But it doesn’t strike me that you have any interest in spending time in VR now. Am I wrong about that? If there was the right system with the right form, can you imagine yourself spending some significant time there?

    An image from VPL’s "reality built for two," a virtual telepresence project

    J: The first thing I’ll say is I use VR a lot these days in research of different kinds — cognitive science research and some visualization stuff. I also love working with exotic optics and sensors and I still play around with that stuff. And I absolutely take time to play beyond what’s needed for the research. I still find great pleasure in screwing around with it. But I’ve always felt, from the very beginning that it’s — how would I say it — you know, every musician I know prefers to live without the radio on, enjoys the silence and contrast. The best way to use VR, not in some moralistic or judgmental sense, but just in terms of my experience, is [to] use it as little as possible and enjoy the contrast that it gives me to reality.

    At the old lab, VPL, one of the things we’d sometimes have is just a flower sitting there. So if somebody was in a demo for 20 minutes, we’d come out and say, just look at this flower. And you suddenly saw this flower in this hyperreal way because your senses had adjusted to this sort of lower resolution of a virtual world. Then, when you see reality, you suddenly see it with this kind of detail and this density. You see just the sheer reality of it. You just feel things from it. It’s really incredible. To me, that contrast, that feeling that you have when you’re out of it after you’ve used it, has universally been more precious than what happens in it. So, yeah, I like it. I might be wrong about this, but I suspect that a lot of people will find what I found: that the coolest thing to do with [VR] is not to be in there for hours the way people are with their pocket devices these days, just staring at the screen…

    "When you see reality, you suddenly see it with this kind of detail and this density. You see just the sheer reality of it."

    KK: Or playing games.

    J: ...or playing games. The coolest thing is to come out of it for hours. I think that’s really just the most amazing thing. I still really do. Here we are, surrounded by all these acoustic musical instruments and in a way, building these devices and building virtual stuff has just helped me appreciate physical stuff more and more. Physicality is just so astonishing.

    KK: It’s underappreciated.

    J: Yeah, maybe we’ll appreciate it more when we have something to compare it to.

    KK: That’s certainly what I call the third way — the nerd way: you investigate real things [by] making an artificial something, which can both give you some evidence about how the real world works and also better questions, more interesting views about the real world when you come back from those. Whether it be VR, artificial intelligence, or artificial democracy, you investigate the real world by making synthetic things.

    J: I totally agree. As for what people actually do, I’m kind of bracing myself; waves of teenagers whose bodies have become so inactive from being immersed that hospitals have to sever their limbs or something. I’m very much hoping that it isn’t that bad but the thing about reality is that it’s not fully predictable. We have to dive in and learn. I think it’s important to make experiments, and it’s really important even to experiment in a way that can be a little dangerous. It’s important to take risks. The thing that’s really a sin is to not learn from them, to ignore the results. That's when you really lose it. As long as we’re awake and paying attention and we learn and get better, that’s what matters.

    "As long as we’re awake and paying attention and we learn and get better, that’s what matters."

    KK: Our sense of history in this world is very dim and very short. We were talking about the past: VR wasn’t talked about for a long time, right? Thirty-five years. Most people have no idea that this is 35 years old. Thirty years later, it’s the same headlines. Was the technological power just not sufficient 30 years ago?

    The Nintendo Power Glove, based on a VPL dataglove design

    J: Both I and a lot of other people really, really wanted to get a consumerable version of this stuff out. We managed to get a taste of the experience with something called the Power Glove. Remember that? You could put on this big glove and reach into things on the screen. Sony actually brought out a little near-eye display called Virtual Boy; not very good, but they gave it their best shot. And there were huge projects that have never been shown to the public to try to make a consumable [VR product], very expensive ones. Counting for inflation, probably more money was spent [than] than Facebook just spent on Oculus. We just could never, never, never get it quite there.

    KK: Because?

    J: The component cost. It's Moore’s law. Sensors, displays… batteries! Batteries is a big one.

    KK: So the vision was there, but the technology wasn’t.

    J: It wasn’t there. Whether it’s there [now], we’ll find out in the next year or two. We’re sort of saying it’s there but nobody’s actually proven it yet. Proof is in the actual doing.

    KK: You can’t discount, 20 years from now, some headline saying, "Okay this time it’s real, we’re looking at VR in 20 years."

    J: I wonder if the reason we keep on cycling back to hope about cool things like VR is that for all the tech news and our fetishizing about our touch devices, we’re still a little disappointed in the menu of tech items that we have at this late date. It’s 2014 and you can buy a robot to clean your house, but it doesn’t really work that well yet. We all think it will, some day, but it’s not quite there. We have some demos of cars that drive themselves but you can’t really buy one. Everything is kind of taking so long. So I feel like, in a way, we keep on cycling through the same tech hope stories because there’s an impatience and frustration. We wait until we’ve forgotten one of them, then we rediscover it. So I think there’s a little bit of that going on.

    "I feel like, in a way, we keep on cycling through the same tech hope stories because there’s an impatience and frustration."

    C: Reading the original interview, one of the things I found so exciting was your excitement about VR as a way to temporarily escape the physical bounds of your body, to escape all reality. I’m not sure that you would characterize it as an escape at the time, but reading it I just thought, my god, what a different way of being.

    J: If you were to try to enumerate what was actually invented at VPL, there are a few different things. Arguably, the most important one was avatars, because nobody had experienced that before. There were representations of people on the screen, but as far as being immersed in an alternate body and being with other people in a simulation, there had never been a social, immersive simulation before. That was such a striking and remarkable experience. To this day, it’s still very rarified. I’ve noticed there is a revival of experiments in which people look out at each other’s eyes and that sort of thing. There are a bunch of videos about that online, which is one of the things we were really into in the old days. As far as becoming a full-bodied avatar and being with somebody else, seeing each other as avatars and interacting that way, that’s still an incredibly rarified experience. I don’t know why it should be, because we have all the tools now. Just put your damn body in front of a Kinect and put on the Oculus and you should be able to do it. But for some reason, people haven’t rediscovered that, and when they do, there are going to be blown minds because it’s just so intense. It’s just really amazing.

    KK: I know that that’s what High Fidelity is working on in some ways. What they’re doing is capturing all the emotional contortions of your face so they can actually present that to an avatar.

    J: This is one of those moments where I feel a bit of jerk saying, "Oh yeah, I’ve been doing that for decades," but yeah, I’ve done that a lot. And yeah, it’s absolutely true. In fact, face-tracking and mapping the avatar's face is absolutely a core thing. That first started to work in the ’90s, and I and some friends had a little startup that was doing it. I suspect [that] there are a few code packages that can do that and I have a feeling that they might be using our old one because a lot of people do.

    "This is one of those moments where I feel a bit of jerk saying, "Oh yeah, I’ve been doing that for decades," but yeah, I’ve done that a lot."

    If you can capture as many nuances of your body as possible and transfer them into the avatar, every extra parameter you can measure enhances it in a synergistic way where it gets cooler and cooler. The other thing that’s really interesting is to see how weird an avatar can get and still be you. The classic early experiment was the famous lobster avatar that Ann Lasko made. Jeremy Bailenson over at Stanford has [tried] super minimal [designs] where you just turn into a cube you can stretch. The interesting thing is even if you just turn yourself into a cube, you can still convey affect and attention and all sorts of things.

    KK: Ask any animator at Pixar who does this all the time. They take a rug or a carpet and they make it into somebody with a personality and a persona.

    J: For you to become that thing in real time is something different, and I think a lot of people will find this to be a really delightful zone of experience.

    KK: Also, multiple people can share an avatar in interesting ways. The other thing is in a virtual world, this question of which is the avatar and which is the environment can be shifting over time. All kinds of things can happen: you can turn into the clouds and what not. It gets very, very interesting and very fun. Are you still working in some of these different directions, even at Microsoft? With the various elements — whether it’s avatars or face recognition. What’s your role now?

    "Hey look at Microsoft Research; every bad thing that can be said about us has already been said, we don’t care what you say."

    J: One of the reasons I ended up at Microsoft Research is incredibly simple. When Google bought Eyematic [Interfaces, which developed ways for computers to understand images], at one point Sergey Brin told me, "At Google we’re just not that comfortable with people being out there [and] having outspoken opinions; we really don’t want people to be sort of blogging about this and that." I was like, are you kidding? That's what I do. It was at some event and Bill Gates was there too and he said, "Hey look at Microsoft Research; every bad thing that can be said about us has already been said, we don’t care what you say." It's like, well, that's kind of cool. It’s given me a chance to be part of a large-scale, great lab without the constraints that Silicon Valley companies often put on people. But the other thing that was magical about Microsoft was Kinect. The notion that you could, using depth cameras, turn into an avatar and interact with worlds.

    KK: So are you continuing [work on Kinect] or are you doing anything new in that direction?

    J: I might not choose to fully disclose everything I’m doing right now because that’s part of the fun. But I do a lot of stuff. I’ve always been a pretty promiscuous technical person. I have one collaboration with a cosmologist on a digital model of the earliest moments of the universe. I’m trying to come up with an alternative to inflation, just to see where it goes.

    In terms of VR? Sure, I’m still working on it. I’m still really interested in all the things that I’ve always been interested in — like where can optics go if you have computation? Where can sensors go if you have computation? All that kind of stuff.

    KK: That’s probably what we’re going to see in the next 15 years or so; continuing [to develop] the variety of sensors that can generate information for a virtual world.

    A SimEye helmet-mounted virtual display (Ben Delaney)

    J: We seem to be able to keep Moore’s law going, but we haven’t quite seen a battery's Moore’s law keep up. Energy technology is probably going to be more the impediment than anything else. And also there is the tooling cost; to keep Moore’s law going we might be moving to very different kinds of material. If we were seeing something as dramatic as a Moore’s law-level exponential improvement in batteries, we would happily explain it as a sort of "know-how" improvement based on information access. The only problem is that we’re not seeing. The problem with energy is it’s closer to the constraints of reality; it’s a little further from our fantasy worlds. I’m still optimistic that we’ll figure it out. I’m optimistic that we’ll come up with better ways of generating and storing energy, but it’s an actual hard problem.

    KK: Right, exactly. If Moore’s law was to cease or slow down, it would have tremendous affect on our lives because right now; we expect things to get cheaper and better all the time. If that wasn’t true, if your little devices weren’t getting cheaper and better all the time, that would have a huge hit on the culture and the economy.

    J: It would be giant. Well, it has to happen at some point. Reality doesn’t have infinite resolution. There is going to be a Moore’s law limit. The question is, where it is and when is it? One of the things that Microsoft Research is particularly great about is exploration at the edge of Moore’s law.

    "Reality doesn’t have infinite resolution. There is going to be a Moore’s law limit."

    And there are a few of them right now. One is trying to use silicon in more flexible ways. Doug Burger, a colleague there, just demonstrated using field programmable data rays as a cloud architecture so you can just reconfigure the whole cloud all the time instead of having a fixed processor design. The benefits are really amazing; it’s going to totally change what we expect from clouds. That’s an example of noticing an opportunity for more flexible architecture so that you can simulate a jump in Moore’s law even though you’re still stuck with the basic stuff.

    KK: What happens is that we just redefine Moore’s law. We’ve been doing that all along.

    J: Of course. That’s a game that you can play.

    KK: Like cameras: for a while, the number of pixels was important. Then we realized, omigosh, at a certain point, the number of pixels isn’t important, it’s the speed.

    J: In the next few years we’re going to start to really understand what we can do with quantum computing. We're so close now, empirically, to getting to something on that. I’m pretty enthused about it. I think what it’ll look like is a cloud service that can do more than we could have plausibly expected it to do otherwise. I think we’re going to start to see much better machine learning. But all of it is going to be module — all the same latency and clogging and political issues that darken our cloud services now… none of that is going away with technology.

    KK: Going back to this issue of the reality of virtual things and what we might want to think about as we make these worlds larger and more encompassing and more persistent; have you had any ideas about making the distinction between how we treat things and who gets to decide about them?

    J: I have been part of a lot of conversations about what the laws should be and trying to come up with regulations. How do you protect the kid who's being bullied without impinging on freedom of speech? How do you prosecute revenge porn without empowering some politician? Or the right to be forgotten — how you do that without empowering some politician? It’s these kinds of discussions that led me to become more interested in Ted Nelson’s original ideas about micropayments. In a lot of ways, if you can make an economy adjust for these sorts of things instead of adjudication with rules, it just works better for everybody.

    KK: Really?

    Jaron Lanier’s "Ritual World" VR environment

    J: Yeah, I think so. For instance, if there’s a grocery store that has some weird tart that I want, instead of arguing with people about whether I should have it, if l have money I could get it. It creates a very simple system where there is only one varying parameter: price. It removes, in most cases, all these legal things. There is a lot to be said for it — it’s a good simplifying principle for human affairs. That’s why it exists, you know?

    KK: It seems as if you’re saying the market is going to solve everything.

    J: Not everything, but maybe some things.

    KK: Okay, so let’s say the right to be forgotten; how would having a market and people’s money solve that issue? If you pay enough, you erase [them]?

    J: No, I think if people can set a price for their information and have a single parameter adjustment for how private they want it to be and people can find points of equilibrium.

    KK: So you pay for privacy.

    J: And you also earn for it. It would be a balance point. There could be people who say: I like living in a barter world. I will give up my privacy by setting the cost of my information to be really cheap: have at it. It’s the bargain I want, and I like it. And somebody else might say: actually, I don’t like being modeled. I don’t like being manipulated. I don’t like being known. I want to be off the grid. I'm going to set my price really high, even though that means not many people are ever going to pay for it.

    But most important, the thing that I want people to understand, is that if there were a society like that — and this is speculative without test of course — [it] would put a price on government spying on people, too. We'd put a check on it. And, of course, law enforcement would be able to get warrants to intervene in some cases. There’d be all the adjustments that one imagines, but it might very well work better.

    This idea of openness and prioritizing free speech is wonderful in theory, but it has two practical flaws that have emerged empirically that I certainly hadn’t predicted theoretically. One of them is ever more income concentration, because nobody benefits from information in an economic sense, except for the pleasure of the people who have the biggest and most influential computers. The other thing is that at random, people are victimized by it very terribly. An example would be revenge porn. Another example is people whose personal information was used to target them with effective but malicious financial offers and practices.

    KK: I certainly endorse Esther’s idea of having to pay people to read your email.

    J: That’s a great example. For people who don’t know the story: Esther Dyson proposed, at one point that if we put the tiniest postage, a micropostage, on email, it would solve spam.

    A CAVE virtual reality installation at the Ars Electronica Futurelab

    KK: But it’s important that the postal payment goes to the person, the receiver; that’s who’s getting the money. It’s not the post office.

    J: Right, there’s no third party. There’s not some entity that’s growing rich on postage. It’s person-to-person in distributing.

    KK: That’s very important.

    J: Absolutely correct. I think there is some kind of a setup where Mark Zuckerberg will read your email for $100 or something now. So that exists in a couple isolated cases. But she was shouted down because now everything has to be open and free. The beautiful thing about that, is that for anybody doing normal communication, the cost is so marginal that it really shouldn’t have any effect. And since it doesn’t go to some third party, it doesn’t create some new, centralized powerhouse. I think it was an interesting proposal and I wish it had been tried more.

    KK: I do too. The question is: how do you implement it? Either the whole system has it or it doesn’t. It’s very hard to implement incrementally. That’s one of the issues about this world is that there are lots of things you can implement incrementally, but there’s a whole set of things that you can’t.

    J: Micropayments are like that. If micropayments aren’t universal, their benefits aren’t spread around enough to generate support for it. Maybe if you could get over the valley, there would be. The only other thing I can say about that is because of Moore’s law, there are always new platforms coming out, like 3D printing, virtual worlds. Every time a new platform comes about, there is a new opportunity to experiment. And there will be dozens of them.

    KK: You can actually say it slightly differently: there are natural monopolies, and each time a monopoly comes up, that's an opportunity. Monopolies are much more ephemeral because they unravel almost as fast as they build up. Basically, we have a future of one natural monopoly after another. Each time one comes up, there is an opportunity for doing something that’s ubiquitous. So there’s hope.

    Early VPL concept drawings of virtual reality

    "Just to make people not become nauseous, you have to really get into it and become human-centered."

    J: I think there is tremendous hope. The stuff that scares me most these days is the same stuff that would scare most people, which is whether we’ll sort out some way to deal with climate change in time. And whether we can reconcile modernity and human nature in some workable, sustainable way. People are tribal and modernity wants us to be globalized. How do we reconcile those things? People are biological, but we want to be immortal. How do we reconcile those things? Those are the kinds of big-picture issues that are really scary and really challenging.

    KK: And is VR a solution or just another part of the problem?

    J: In the old days, I used to think of VR as a technology that turned technologists into humanists because you had to work with people so much. It’s one thing to put stuff on the screen and ask people to project themselves onto the screen, get lost in it, but as soon as they are wearing the stuff, you just have to work with a human body; you have to think with human factors. Just to make people not become naufseous, you have to really get into it and become human-centered. You have to become super human-centered. You can't ask people to meet you halfway anymore. You have to really go to the people to get VR to work. So I always thought it helped you notice people and human nature more and it should make engineers more empathetic and more sympathetic to people. And I still think that’s kind of true.

    Among the post-idealist ideologues, VR has come to this completely different role where the great AI in the future will somehow recreate all our consciousness in VR. In the extremes of the movement, people are worried about. There’s this idea that the virtual copies of your consciousness will be created in VR by the big AI in the future [and it] will torture your copies if it finds out that you didn’t give all your money to some AI company to help bring about the singularity.

    C: Roko's basilisk?

    J: Exactly, that kind of stuff. There’s a lot of variations of that. There’s this sort of insane way that VR is being used to turn technology into a new medieval church of some kind. It’s sort of like the worst of scholasticism in Catholic tradition or, maybe, the most nerdy and controlling version of Islam, or something like that, but from 1,000 years ago. That’s certainly not what I was expecting. That’s a little different.

  6. Step into Sony’s virtual world

    Shuhei Yoshida on the Project Morpheus Virtual Reality Headset

    It’s still early days in the VR game, and only two companies stand ahead of the pack, ready to put a virtual reality kit in the house of every American: Sony and Oculus, now owned by Facebook. We sat down with the heads of VR at both companies to talk about turning Virtual Reality into everyday reality.

    Shuhei Yoshida, president of Worldwide Studios for Sony Computer Entertainment, was part of the team that brought the original Sony Playstation to life. His first games gave players a glimpse of farflung environments, from Gran Turismo’s Nürburgring raceway to Final Fantasy VII’s dystopian universe. Twenty years later, Yoshida is defining the next age of gaming with Project Morpheus, Sony’s new virtual reality headset. Yoshida recently took a few minutes to explain what that future might resemble

    What do the next few years of VR look like?

    Yoshida: We have something here that we’ve never experienced before. Working on game development, we always try to create a new kind of experience, and having VR technology is almost unfair. If you are a programmer, one person can create an experience with game engines like Unity. It’s a great time to be a programmer.

    What’s been your best experience with VR?

    Yoshida: I’m a good swimmer, but I’ve never tried diving, so the experience of The Deep was eye-opening for me. When the demo starts you’re in a cage. You can look up and see the reflection of the sun on the water. It’s really beautiful. After a few days, looking back on that experience is interesting because it’s kind of raw in my memory — as if i was actually in the sea.

    "You can look up and see the reflection of the sun on the water. It’s really beautiful."

    Were you scared?

    Yoshida: Once you try it, somehow everybody forgets that there are other people around you. People start to talk very loud. You forget about being seen by other people.

    What will it take to actually get Morpheus into people’s homes?

    Yoshida: The first thing we strongly believe, and Oculus says the same thing, is that we need to deliver the core technology first. We want to make the technology before we bring it to market. The product we have we’re pretty happy with, but there are many areas we can improve on to bring it to an even higher level. And the technology advances every month.

    But we also have to nail it from a system-development standpoint. People don’t buy hardware just to have hardware. It needs strong content. Every developer who has started working on VR learns that they have to relearn what they have learned over the years making games. A lot of tech and new assets can be used again, but the approach has to be pretty different.

    "If the games or apps are designed poorly, it makes people sick, which is a really, really bad thing to have as an introduction."

    You’re building a location-based environment, so the experience has to be good. If the games or apps are designed poorly, it makes people sick, which is a really, really bad thing to have as an introduction. We have to work hard to share the knowledge that we have gained.

    And finally, the product has to be affordable. All three categories have to work.

    Sony’s philosophy behind Project Morpheus is "VR is a medium, not a peripheral." What does that mean?

    Yoshida: When you talk about VR, I liken it to 3D graphics technology. When we introduced the Playstation, it was a revolution in terms of 3D graphics. It was in the hands of all game developers to deliver 3D games to your home instead of to the arcade. The 3D graphics card had a huge impact, but developers are still learning how to use it in games like Grand Theft Auto or The Last Of Us, 20 years after the launch of PS1.

    3D graphics are now used everywhere: on the web, in movies, and even in the anime industry. Nobody would’ve conceived of 3D anime before. In a similar way, VR can be way too many different things. Once it becomes very affordable to bring to millions of people, more uses will be created. What I’m really excited about is that VR puts you in the scene. You can be a witness to some great event, historical, dramatic — it’s going to be a powerful medium to bring storytelling to.

    "You can be a witness to some great event, historical, dramatic — it’s going to be a powerful medium to bring storytelling to."

    Games like The Last of Us already have really strong stories and character development. We start with games because game developers know what it takes to bring VR to market. Also, you have to buy PS4 to use Morpheus, so our initial target audience is naturally people who have PS4.

    What will the killer app be for Morpheus?

    Yoshida: A sense of presence is something that only VR can bring to the market, an experience that gives you a really strong sense of presence somewhere else. That somewhere else has to be really attractive to you, so you want to go back to that space again and again. That will be the killer app for that platform.

    I spent three months with a backpack in Europe going from one city to another. That was a great experience, but it’s really hard to do getting older—you don’t have the time or money. Well, I have more money now but I don’t have time [laughs]. You get to be someplace and an accident might happen, but in VR you can set a perfect location. It could be a fantastic space like Star Wars or Game of Thrones, or another place you want to immerse yourself in. An anime IP in Japan would be attractive to a Japanese audience — the place that you want to come back to will be different for different people.

    How big a priority is Morpheus for Sony? Is this a product that’s going to be around for the next 20 years?

    Yoshida: For Sony Computer Entertainment, VR and Morpheus is one of the areas we are very excited about, like cloud gaming with PS now, or other networked services. We’re really focused on bringing that experience to consumers.

    Are we ever going to see big games on Morpheus, from Hideo Kojima [the creator of the Metal Gear series] or any others?

    Yoshida: That would be awesome. Kojima-san has always liked gadgets. He always has some future vision for some gadgets.

    Have you played the Destiny beta? Half of the time I was just hiding out in the Tower, just chilling, looking at the great vista. That would be great in a VR setup.

    Will we have to simulate all of our senses for VR to be truly great? Is it about more than just sight?

    Yoshida: I believe that 3D sound is an essential part of creating a sense of presence. This is the safest way to avoid technical problems. Without 3D sound it’s very easy to break the immersion. When you see an object to your left, you want to hear the sound that the object creates from your left. So we designed a system to handle all the work of creating 3D sound so that game developers can pretty easily map sounds in the 3D space.

    Each sound source has 3D location data so that for every frame, the proper sound is computed to your left ear and right ear. That’s part of Morpheus, and we believe having 3D sound is essential.

    "One of the people who tried our Castle demo, instead of attacking the dummy he was trying to pat the dummy on the shoulder, or shake hands with it."

    Other [inputs] like interaction — one of the really strong experiences is when you see your arm, and when you hold a gun or a sword and you flick your wrist and see it in the VR space. In the meantime, the Dual Shock 4 has LEDs so the controller’s position can also be tracked by Playstation Camera, like PS Move. It’s not good for holding a sword, but when game designers design a mechanism like you’re in a cockpit, you can create a pretty strong sense of immersion. One of the people who tried our Castle demo, instead of attacking the dummy he was trying to pat the dummy on the shoulder, or shake hands with it. He was really having a blast doing that, so we encourage those kinds of experiences too.

    We used to say seeing is believing. Now we have to say experiencing is believing.


    Photo credit: Ben Lang, Executive Editor at Road to VR

    • Virtual insanity


      The peculiar portrayals of VR through pop culture history

    1950

    September 23, 1950

    The Veldt

    Ray Bradbury wrote "The Veldt" long before the term "virtual reality" was coined, but his dark short story foreshadows the idea of a virtual place becoming all too real. Like much of his work, it explores the consequences of creating an automated world -- in this case, a house that will raise your children with seductive video footage of the African veldt.

    1981

    February 01, 1981

    True Names

    The dreamlike "Other Plane" from Vernor Vinge's novella shares much with the cyberspace of William Gibson's later and better-known Neuromancer. But its hackers or "warlocks" experience VR through the trappings of myth and fantasy, reminiscent of the Dungeons & Dragons-esque role-playing games that were just beginning to appear on personal computers.

    1982

    July 09, 1982

    Tron

    Tron is the cult film that brought virtual reality to the masses, in the form of a wireframe world and guys in glowing suits riding light cycles. It bombed at the box office, but the movie's influence carried far and wide into other fictional virtual worlds. A long-awaited sequel failed to live up to the promise of the original.

    1983

    September 30, 1983

    Brainstorm

    An underwhelming film that follows a pair of scientists (Christopher Walken and Natalie Wood) developing a brain-computer interface to allow people to share feelings and experiences using virtual reality. Despite including a virtual, nonstop orgasm (oh la la!), the movie was a commercial disappointment and carries a dark legacy: not only did turbulent production sour director Douglas Trumbull on making movies, it was also Natalie Wood's last project — she died before the film wrapped.

    1984

    July 01, 1984

    Neuromancer

    One of the ur-texts of cyberpunk, Neuromancer gave us the "consensual hallucination" of the matrix, where computer hackers got rich by infiltrating corporate data centers, as well as "simulated stimulation" cassettes that could record and play back experiences. Gibson's influence would later earn him a cameo in VR-themed TV series Wild Palms, in which he's introduced as the man who coined the word "cyberspace." "And they won't let me forget it," he responds.

    1992

    March 06, 1992

    The Lawnmower Man

    When people think of “virtual reality,” this is often the first movie that comes to mind. It uses real gear from famed VR company VPL, but in our own reality, that equipment rarely gives wearers the power of superintelligence, telekinesis, and total control over the internet. The movie is nominally based on a Stephen King story, but King later sued to get his name removed, noting that his story has nothing to do with VR and in fact includes few similarities beyond the appearance of lawnmowers.

    1993

    October 31, 1993

    Murder She Wrote

    As Wired correctly noted in a blog post celebrating this incredible episode of Murder She Wrote, Angela Lansbury ends the episode literally smelling the flowers, and lamenting technology’s inability to replicate such a feeling. It's a heavy moment for a light show, which is perfect, because Jessica Fletcher wanders around in a VR headset looking like everyone’s grandmother just discovering the future. Adorable.

    January 14, 1993

    The Simpsons, "Marge vs the Monorail"

    Lisa conjures up what is described as a virtual reality kit but is really more like a time machine, as it transports her back to Mongolia circa 1200 AD.

    October 08, 1993

    Demolition Man

    Cryogenically frozen in 1996, Sylvester Stallone wakes up in 2036. Lots of crap happens, but mostly he is eager to bone Sandra Bullock. After a successful date, they have sex. Unfortunately for Stallone, that means it happens via VR helmet and a little imagination, as “fluid transfer” is off-limits in the future (and apparently disgusting).

    June 11, 1993

    Jurassic Park

    Crackpot scientists use VR displays to complete gaps in dinosaur DNA, which makes no sense and is awesome.

    May 16, 1993

    Wild Palms

    Often compared to David Lynch’s Twin Peaks, this surreal miniseries imagined virtual reality as the tool of a sinister, Scientology-like conspiracy. VR visionary Brenda Laurel consulted for the series, which also features a small cameo by William Gibson. Kathryn Bigelow — who would go on to make Strange Days — got an early taste of VR by directing one episode.

    1994

    February 03, 1994

    Mad About You

    Paul Riser invests in a virtual reality system made by a preteen. He then uses it to virtually hang out with Christie Brinkley on a park bench. She tells him everything he does is “just right” and then asks about his work. Helen Hunt uses the VR to get Paul Riser to say he was wrong.

    September 03, 1994

    VR Troopers

    Made by the same production company as Mighty Morphin Power Rangers, it was a similar live-action show. Three teens become VR troopers doing their best to stop the virtual and real worlds from being overtaken by mutants. 90 episodes were produced.

    December 09, 1994

    Disclosure

    In this hard-boiled movie based on a Michael Crichton novel, Michael Douglas straps on a headset to access "The Corridor," a 2D data stream where his corrupt employer is inexplicably keeping sensitive files.

    1995

    March 10, 1995

    VR.5

    A Fox TV show that lasted just over two months. It featured a convoluted plot about rewiring the real world via a virtual one, accessed through a wrinkle in time. Or something.

    May 26, 1995

    Johnny Mnemonic

    This isn’t really a story about virtual reality — it’s about an emotionless cyber-courier who enlists the help of Henry Rollins and Ice-T to find a drug-addicted dolphin. But the image of Keanu Reeves wearing a huge silver visor and gesticulating wildly in a virtual info-city has become one of the most memorable depictions of VR.

    October 13, 1995

    Strange Days

    Kathryn Bigelow’s film lacks a solid plot, but its dark take on what would happen if we could tap into others' brains feels prescient. Basically we would do really bad stuff. Angela Bassett is not feeling it and helps kick ass to rescue VR experience dealer Lenny Nero (Ralph Fiennes) from a downfall of his own making. Potentially the best movie currently available to stream on Netflix.

    August 04, 1995

    Virtuosity

    Denzel Washington is a vengeful cop in a futuristic Los Angeles who must stop a young Russell Crowe playing a virtual reality killer named SID, who somehow escapes into the real world. The movie didn’t make much sense by its own logic, and it was a huge flop.

    1996

    December 05, 1996

    Otherland

    The first book in what would become an underrated, four-book sci-fi opus centers upon a future where the internet is made up of a series of connected virtual worlds accessed by virtual reality implants and bathtubs. Oh, and there is a mysterious disease that traps people in virtual comas.

    1998

    March 28, 1998

    Daria, "The New Kid"

    Daria, MTV's animated ’90s anti-hero, goes on an unsurprisingly awkward date to an arcade. She ends up as a knight in a VR environment, which aptly foreshadows Sony's initial Project Morpheus game 16 years later. Potentially not a coincidence.

    1999

    Trailer:

    April 23, 1999

    eXistenZ

    David Cronenberg looked at television and got Videodrome, so just imagine what he made of virtual reality. If you imagined an endless series of nesting video games, filled with grotesque organic computers and exploration of what "reality" even means, you’re absolutely right.

    November 13, 1999

    Batman Beyond, "Hooked Up"

    This episode shows a bunch of teens becoming so violently addicted to virtual reality they start stealing and committing crimes to feed their habit.

    October 08, 1999

    Harsh Realm

    X-Files creator Chris Carter’s short-lived series about a virtual reality simulation operated by the US Army was ahead of its time: the US Navy is now looking into its own virtual reality simulations.

    March 31, 1999

    The Matrix

    Keanu Reeves again plays the lead role, this time as a cynical ’90s hacker who “wakes up” to the fact that reality as we know it is all just a sophisticated virtual reality simulation created by artificial machines who enslave an unwitting human race as a source of energy on a ruined Earth. Hyper-stylized violence ensues.

    2000

    February 27, 2000

    X Files, “First Person Shooter”

    Co-written by William Gibson, this episode sees Mulder and Scully get stuck in a VR game chasing the Lone Gunman. AV Club called the episode “legendarily bad” and gave it a D+.

    2001

    December 10, 2001

    Vanilla Sky

    Cameron Crowe’s Vanilla Sky is actually a fairly close remake of late-’90s sleeper hit Open Your Eyes. Both movies follow an attractive bachelor horribly disfigured in a car accident whose reality becomes increasingly unhinged. Virtual reality doesn’t make an appearance until late in the film, but the twist ending changes everything.

    2002

    June 21, 2002

    Minority Report

    Although most people recall the holographic, gesture-controlled computers and creepy eye-scan advertising in this Tom Cruise sci-fi thriller (adapted from a short story by Philip K. Dick), this movie also features virtual reality as a prison sentence for future felons.

    2009

    June 21, 2002

    Chronic City

    Called one of the 10 best books of the year by the New York Times, Jonatham Lethem's novel features a host of surreal elements, including the Second Life-ish computer universe, "Yet Another World." He's said the book was influenced by virtual reality and much of the real reality in the book was meant to mimic regular life, but suckier. Kind of like virtual reality.

    2011

    August 16, 2011

    Ready Player One

    The bulk of this seminal geek-culture novel takes place in a futuristic virtual reality multiplayer game called OASIS, which happens to be littered with ’80s geek references.

  7. How to build your own VR headset

    Trust us. It’s easy.

    In March, Facebook announced its intention to acquire Oculus VR for a mind-numbing $2 billion. Twenty years after its initial rise and fall, virtual reality was once again lauded as the technology of the future. “Imagine,” Mark Zuckerberg wrote in a much republished post at the time, “enjoying a courtside seat at a game, studying in a classroom of students and teachers all over the world, or consulting with a doctor face-to-face — just by putting on goggles in your home.”

    For now, most consumers will have to keep imagining — Oculus doesn’t intend to release a commercial version of its Rift headset until late 2015. But we couldn’t wait, and decided to create our very own VR headset instead. Low-cost headset alternatives are already filling the Rift vacuum: at this summer’s I/O Google introduced Cardboard, a DIY VR headset kit that can be purchased for as little as $20. But we opted for a sturdier and, dare we say, more svelte design: the VR2GO headset created by researcher and professor Mark Bolas, Perry Hoberman, and Thai Phan at Bolas’ Mixed Reality Lab at the University of Southern California.

    As it turns out, all it takes to create your own VR headset is a 3D printer, a couple magnifying lenses, and a smartphone. Just don’t tell Facebook that.

    The VR2GO headset design requires a 3D printer. Luckily for us, the UPS store in downtown Manhattan is among a handful of UPS stores around the country offering 3D printing services. If you’re not near a 3-D printer, you can turn to services like Shapeways, which will print and mail your headset anywhere in the world.

    Download the 3D STL file of the VR2GO headset for free from the MxR Lab. A bit larger than the size of a juice box, the headset’s design is cleaner than Google’s Cardboard and far nimbler than the Oculus Rift.

    The UPS store uses a uPrint SE Plus machine — our headset took just over five hours to print and cost $105.

    uPrint SE uses a thermoplastic filament that is heated to a malleable state, and finely extruded through a small nozzle.

    Unlike more basic printers, the uPrint SE encases the final product in a soluble support structure. This allows for greater detailing, and the construction of simple moving parts, like gears.

    The support structure is dissolved in a liquid bath, leaving a picture-perfect headset behind.

    The design calls for two 7X aspheric magnifying lenses, which retail for $10. Pop the lenses into the headset, concave side in.

    Bottom Right: MxR Lab offers VR2GO designs for iPhone 4, iPhone 5, and Android phone headsets. MxR has also developed its own simple VR demo, FOV2GO, which drops you into an alchemist’s workshop. It’s available for free at the App Store and on Google Play. Other games and demos, like those developed for the Hasbro My3D headset, are also compatible with VR2GO. Download some apps, clip the phone into the back of the headset, and leave the earthly world behind.