How Harmonix put Rock Band into a virtual reality world

A long road to a simple fantasy


When Oculus fans unwrap their new motion controllers sometime later this year, they’ll find a strange little accessory alongside them: a chunk of black plastic with an oddly shaped hole in the middle. This is Oculus’ invitation to try out Rock Band VR, one of the biggest, most potentially crowd-pleasing Oculus Touch launch titles. A game franchise that lets people play out their rock n roll fantasies sounds like such a natural fit for VR that the core game would barely need to change. As a launch trailer last year simply put it, “We wanted to make the Rock Band experience on Oculus so realistic that the player actually feels like they are a rock star.” But for all the video’s easy jokes about groupies, headbanging, and smashed guitars, the reality is far more complicated — and more interesting.

As a company, Rock Band developer Harmonix Music Systems has always focused on building for odd new platforms. The company was founded in 1995 by a pair of MIT Media Lab students, Eran Egozy and Alex Rigopulos. Its early projects weren’t games so much as experimental electronic instruments, including projects like The Axe, which let players make music by moving a PC joystick, or an Epcot Center installation that produced a similar effect with infrared sensors. Besides Guitar Hero and Rock Band, with their custom plastic peripherals, Harmonix produced one of the first games for Sony’s EyeToy camera, as well as one of the most successful Xbox Kinect game franchises, Dance Central. In 2015, the studio announced a music visualizer called Harmonix Music VR for Sony’s virtual reality platform.

Rock Band VR

"We’ve always been excited about the kind of new creative outlets that new tech affords us," says Greg LoPiccolo, a Harmonix veteran and Rock Band VR’s creative lead. Last year, Oculus Studios head Jason Rubin approached the studio about making a new Rock Band installation, specifically for the then-unreleased Rift headset. The companies quickly came to an agreement, and LoPiccolo’s team began work on what would become Rock Band VR in August. As LoPiccolo puts it, they figured out two things right away: if done right, Rock Band in virtual reality would be amazing, and they had no idea how to make it work.

The first version of Rock Band VR looked a lot like the original series. After attaching a motion controller to a standard Harmonix guitar peripheral and putting on the Rift, players would see a five-lane "note highway" with beats to hit at specific moments, layered over a concert scene. "It feels great to stand on stage with a screaming crowd in front of you with your band behind you," says LoPiccolo. But the system turned one of virtual reality’s biggest selling points, a feeling of physical presence, into a distraction. Unless players had memorized a song, they would find themselves spending the whole game staring at the highway, trying to block out everything going on around them.

Behind the scenes, says audio lead Steve Pardo, Harmonix was throwing out idea after idea to fix the problem. They stripped the five-note system down to three notes, vastly simplifying the beats that players had to match. They tried moving the position of the highway, so it ran out of the guitar’s neck instead of across the crowd. The game became easier to play, and more visually distinct from its flatscreen counterpart. But nothing changed the fact that Rock Band VR was a virtual reality experience that effectively penalized exploring its world. Then, the team made a fundamental change: instead of celebrating the dexterity of the perfectly executed solo, their game would reward the steady, low-key improvisation of rhythm guitar. The new system replaced individual notes with periodic chord markers, then encouraged players to try out their own patterns in the gaps.

"It was something I kind of have always wanted to do in a way," says Pardo. "I remember thinking like — man, it would be cool if I could just play a power chord, and strum it, and it would be a power chord, right?" He built on a freestyle system that was already being put into Rock Band 4, which procedurally generated music based on users’ input, subtly changing the song based on how they played. "It was really kind of indulging in this fantasy I wanted to have for myself."

When I take a train out to Harmonix’s Boston offices, the latest build of Rock Band VR is set up in a little conference room in one corner of the office, undergoing final tests before its debut at Seattle’s PAX gaming show. Rock Band already requires a lot of hardware, but with the Rift and Touch, Rock Band VR takes it to another level. I step into the center of the room and hesitate, unsure whether I should put the guitar on over the Rift, or vice versa. (The best order, I decide, is controller and then headset, so you can pull off the Rift before removing the guitar at the end of a song.) Everything is unfamiliar enough that I don’t even recognize when things go wrong — it takes me most of the tutorial to realize that the virtual guitar floating several feet away is actually the result of a bug, and supposed to be in my hands.

Like a lot of VR projects, Rock Band VR may actually be easier for people who haven’t internalized the assumptions behind traditional video games. Hitting button sequences won’t take you as far as just following your air-guitar instincts, including mimicking the swaggering body language of a rock star, which the Rift headset and motion controller can detect and add to your score. Even if the set list were exactly the same as an earlier Rock Band game, the experience would be completely different — the way that performing hip swings and hand gestures in front of a Kinect in Dance Central was different from hitting directional pads in its ancestor Dance Dance Revolution.

When I convince myself to relax a little, despite knowing that cameras are rolling outside the headset, this turns out to be a lot of fun. I don’t exactly feel like I’m part of a real band, although the final game is supposed to add a level of narrative development that no other Rock Band game has had. But it’s easier to slip into the fantasy of the digital stage, to get past the inner voice whispering that playing with a fake musical instrument is a little silly. Its virtual world adds a whole new answer to the eternal question: why don’t you just learn guitar? Because no matter how good you get, few people will ever get so close to the adulation of a real crowd.

But as with any game, Rock Band VR introduces layers of abstraction, and occasional frustration. As I try to shake my head in time to David Bowie’s "Suffragette City," the headset doesn’t pick it up, and I’m not sure if I’ve totally missed the cue or I’m just learning the system’s quirks. Since the Rift doesn’t detect foot motion, I have to hit effects pedals by looking at them. And most confusing of all, I can’t see my fingers on the guitar frets. That last problem, actually, is eminently fixable. Oculus’ own controllers feature capacitive surfaces that detect touch, and a custom Rock Band VR guitar could incorporate the same technology, giving you virtual fingers inside the game. The fact that there isn’t one — that the extent of the new hardware is a little Oculus Touch holster for older generations of guitar — speaks to how much any VR development team still has to hedge its bets.

Oculus needs games like Rock Band VR. Unlike the HTC Vive, the Rift currently features few experiences that could only take place in virtual reality, and unlike PlayStation VR, it doesn’t have a lot of big names to lean on. The Facebook-owned company can offer game designers like Harmonix lots of money for bold, experimental projects. But that doesn’t change the fact that the Rift is still a tiny platform. Asking people to buy one weird new accessory — two, if you count the sold-separately Oculus Touch — is difficult enough. Staking a whole new line of peripherals on an already marginal product would be almost suicidally optimistic.

If any company is in a position to understand the challenges of niche hardware, it’s Harmonix. Despite its contributions, the EyeToy and Kinect never developed a robust game catalog, and the Kinect in particular has become a cautionary tale for how VR could end up: as a creative novelty that never quite finds its place. "I think everyone’s feeling out the waters, because nobody knows how to do anything," says LoPiccolo of the VR industry. "We’re making all our first mistakes just like everybody else. Nobody knows what’s going to work and what’s not going to work, and I think we feel like we’ve stumbled into this corner of VR that works incredibly well."

Rock Band VR Harmonix

Greg LoPiccolo

This uncertainty extends to basic elements of the experience. We don’t know how the physical isolation of the Oculus Rift will change the party-game appeal of Rock Band, or how the system might translate to other instruments. Will people still want to hold impromptu concerts when they can’t see who’s watching? Even outside the virtual reality aspect of Rock Band VR, Harmonix is giving players unprecedented latitude to make their own music. "This is the closest we have ever gotten to what it really feels like to playing guitar. Partly just because the guitar mechanic itself is so much more expressive — it gives you so much more creative freedom — but also because you’re on stage and we track your movements," says LoPiccolo. "There’s a whole aspect of gameplay that really has to do with performance, it’s not just about playing the guitar."

There may be something about VR that inherently encourages experience over mechanics. At the same time, the idea that people want open-ended self-expression goes counter to one of Harmonix’s earliest lessons. In a 2009 interview with CNN, Rigopulos and Egozy recounted the "horrendous failure" of The Axe in a way that’s surprisingly reminiscent of many VR experiences. "The product made an incredible demo — everyone who stepped up to try it thought it was magical," recalled Rigopulos. "But then, after 15 minutes, they lost interest." Then, after moving to Japan to work in its lucrative karaoke industry, they made a counterintuitive discovery. "Karaoke isn't about personal expression. It's about people reproducing the songs they know as accurately as they can," said Rigopulos. "The whole notion of adding improvisation elements just wasn't connecting." It was this revelation, effectively, that turned Harmonix from an electronic music company to a game development studio.

But as risky as working with virtual reality still is, the medium changes things. Rock Band VR doesn’t leave players adrift to improvise on their own. It encourages creativity within guidelines, while providing a sensory experience that’s totally different from a karaoke bar or your own living room. And if it does succeed, the reward will be a fantasy unlike anything else in VR — perhaps even the Oculus Rift’s saving grace.

Disclosure: Matt Boch, a designer on Rock Band VR, is a visiting professor at the NYU Game Center, my husband's workplace.