Inside Industrial Light & Magic's secret Star Wars VR lab

ILMxLab isn't just exploring the future of entertainment... they're already making it

I’m on Industrial Light & Magic’s motion capture stage, standing inside what they call “the cave.” It’s not much to look at: two big screens angled at 90 degrees, awash in a smeary blur of images. But put on a pair of modified 3D glasses, and bam — it’s the Holodeck, and I’m on Tatooine standing face to face with one of the most famous robots in movie history. I walk around C-3PO, crouching one moment then jumping the next. The mo-cap performer across the room raises his hand, and the CG Threepio waves. It’s exhilarating and immersive, and it’s all happening in real time.

The cave is a place for filmmakers to test out worlds that don’t exist yet, and for ILM to demo and build augmented reality experiences for its recently unveiled skunkworks division, ILMxLab. The lab is a developmental playground for any and all kind of interactive or immersive experience. Virtual reality, AR, theme park attractions; it’s all up for grabs, uniting decades of visual effects expertise, computer wizardry, and Lucasfilm’s own creative team into a self-contained entertainment studio of the future.

The goal isn’t to just create what people will be trying out on their Oculus Rift next year. It’s to come up with the interconnected virtual experiences we’ll be having 10 years from now. And you’d better believe they’re starting with Star Wars.

Watch our featurette about the digital masterminds at ILMxLab, and their vision for the future of entertainment:

When Rob Bredow left Sony Pictures Imageworks for ILM in 2014, it didn’t take him long to realize what he’d gotten himself into. "It was like my first week here, and an assistant knocks on the door real urgently. ‘Yeah, did someone request a lightsaber from the showroom down the hall?’" He grins, bright eyes sparkling behind his glasses. "I’m definitely working at Lucasfilm now."

Bredow had spent the previous five years as Imageworks’ chief technology officer, a role that followed years of VFX experience on movies like Independence Day and The Polar Express. But as we walk through the halls of ILM, he tells me it was the chance to work on the new Star Wars films that lured him away. Now, as Lucasfilm’s head of New Media and vp of its Advanced Development Group, he oversees the intersection of technology and storytelling that is ILMxLab.

The group is a collaboration between all arms of the Lucasfilm empire, taking the visual effects expertise of ILM, the latest in real-time computer graphics from the Advanced Development Group, and the audio prowess of Skywalker Sound, and mixing it all together under the incomparable appeal of the Star Wars franchise. It’s a unique effort; while other movie studios are teaming with tech companies to dabble in VR, and VR companies are spinning up their own mini production houses to dip their toes into film, ILM already has the necessary departments integrated under one roof — not to mention a highly-anticipated slate of films in production to play off of. It allows Bredow’s team to experiment with multiple types of experiences simultaneously, in a kind of next-generation entertainment sandbox.

ILMxLab iPad demo

With near-cinematic-quality video streaming to an iPad at 24 frames per second, the short looks like it could be a lost scene from a movie — only I could then slow down time, use the tablet to frame my own shots, or see what characters were doing before they were even introduced in the edited version.

During my visit, the team has three projects to show off: the holographic cave, an iPad augmented reality movie with C-3PO and Boba Fett, and an Oculus Rift port of the same short film (with a VR speederbike "bonus" mode). According to Bredow, each of the experiences is about a year old, and represent proofs of concept rather than shipping products. In the case of the Oculus demo, that’s clear; while it takes advantage of ILM’s digital character models, it suffers from the same middling resolution and interface problems that any year-old Oculus experience does. But the iPad demo is something different, providing a glimpse at what true movie-quality, real-time computer graphics can make possible.

On the first pass, I just watch the short like a traditionally shot and edited scene from a movie: some Stormtroopers on Tatooine are on a search, C-3PO and R2-D2 are in hiding, and eventually Boba Fett shows up for a big reveal. But the quality of the visuals is arresting. It doesn’t just look like a CG cutscene; from the quality of the motion to the sunlight glinting off the droids, it nearly looks like a movie shot on film that I’m watching on an iPad. And it’s not a simple video file being played back either; the footage is being rendered in real-time on an ILM server and then streamed to the tablet (ILM later tells me they’ve run successful tests over LTE, as well). With a tap I can slow down time, change perspective, or watch the entire scene from a Stormtrooper’s point of view. In another mode I can use the iPad as a virtual viewfinder, scoping around the world to see whatever I like, or even join the droids before they even appear in the edited version of the scene. Instant prequel.

It’s all too easy to imagine going to a theater to see Episode VIII, then firing up an iPad to explore what happened just before some big climactic Sith showdown or lightsaber duel. "Instead of cutting a scene because the running time is too long — or only getting to see that on the Blu-ray release as a special feature — what if there was a version where it’s more like, ‘I still want to tell that story, it would just be better in VR?’" Bredow smiles. "Let’s let this show as a parallel story, let’s release this clip with multiple things you can explore and give the audience something that they can sink their teeth into if they want to engage in the movie in that way."

ILMxLab Oculus demo

The short film from the iPad demo was ported to the Oculus, and frame rate and motion blur were removed to make it look less like film and more like a real-life environment. I could move around and explore different parts of the story, or start up a speederbike mode and get a real bird’s-eye view as an AT-AT attacked down below.

If Bredow is the focused, technological wizard that brings the disparate parts of ILMxLab together, then the group’s blue-sky shaman is creative director John Gaeta. A visual effects legend in his own right, Gaeta was the person that invented bullet time for The Matrix, a film he credits for shaping his own philosophy about VR. "The actual experience of The Matrix itself started expanding my thought on the very topic that the movie was about," he says. "I joined that effort with the Wachowskis partly because I thought it was so well-written, how they considered virtual reality completely sublime. In the perfect state, it’s a flawless simulation."

Hearing Gaeta riff on the possibilities of entertainment in the future is like going through a portfolio of Syd Mead’s best futurist work: the ideas are wild and inspiring, with an air of inevitability about them no matter how far the reach. He sees a world where cinema becomes real-time and reactive, where audiences can virtually inhabit the worlds of their favorite characters and use them to tell their own tales, and where the the failure of today’s transmedia — "it’s very antiseptic in a franchised way" — is washed away in favor of an interconnected universe of story experiences that let audiences immerse themselves in a world to whatever degree they want. And it’s not as far away as you might think. "Building block one is to allow people to understand that there is no longer a wall between cinema, or storytelling," he explains. "It could just be the opening gateway, right? The portal."

"There is no longer a wall between cinema, or storytelling. It could just be the opening gateway."

It’s aspirational stuff, and as he riffs I notice the team shares an unexpected point of inspiration: Disneyland. "Walt Disney believed that he could take a fantasy world from cinema that he was making, and basically create a physical experience that you could walk into," Gaeta says. "This is like a 2.0, supercharged version of the same philosophy." It makes sense; when you zoom out, Gaeta’s grand vision resembles nothing so much as a digital Disneyland, and Walt Disney Imagineering even has a satellite team embedded at ILM for working on Disneyland attractions. "There’s a lot to be learned from those formats in the parks."

My full chat with John Gaeta (produced by Ryan Manning)

ILMxLab The Cave viewfinder

The cave serves as a virtual holodeck, and the technology can be used for exploring and designing sets, mocking up AR projects, or even to shoot scenes. Here I'm using a handheld display to line up a shot, while the dinosaur is animated via real-time motion capture by an actor across the stage.

As we walk through Lucasfilm, Gaeta points out a particular floor. That, he says, is where the writers and directors of the new Star Wars movies work — and it’s a place he and Bredow visit regularly to give demos and conceive of experiences for movies that are still years away. "We have this sort of bedrock underneath us, of evolving story over time that we can keep thinking about." Gaeta says. "So imagine going into story sessions in which you actually talk about something that could be happening eight years from now. Literally, we have these meetings, where we’re all, ‘Well, Moore’s law, plus these possible breaks, plus new inputs which seem to be evolving….’" The trick of having the right technology, at the right time in the master Star Wars narrative, is constantly in question.

The "Star Wars Floor" is xLab's secret weapon

In a way, that Star Wars floor — home of Lucasfilm’s Story Group — is xLab’s secret weapon. George Lucas’ space opera is more than just a series of movies at this point; it’s a cultural media platform unto itself, and it’s immensely important that all its disparate pieces fit together. Bredow tells me he reports to both the general manager for Lucasfilm and ILM, and head of development Kiri Hart. "Specifically on the story side, Kiri has a real vision for how the Star Wars world should kind of reach out and be told through these different media," he says. Even something as minor as the iPad demo is made in concert with Hart’s team.

Hannah Gillis of ILMxLab

As the Star Wars universe progresses and technology evolves, Lucasfilm’s hope is that the new mediums the lab is working on become viable outlets for Star Wars storylines and experiences — and in turn, new ways for audiences to consume. xLab could be the roadmap to no less than the future of franchised entertainment, something that is likely not lost on Disney’s CEO.

"Bob Iger sees the potential, and I think that’s why he’s really been willing to take a long-term view of this," says Vicki Beck, ILM executive in charge of strategic planning. By its very nature, something like xLab has to be a long-term investment; not only is there no business model in place for VR or AR experiences — Beck tells me they’ve discussed everything from freemium to subscription models — but there isn’t even consumer hardware for sale yet.

In terms of diversification, however, the appeal for ILM is obvious. Visual effects is a notoriously tough business, with studios looking for quicker work at a cheaper price to such a degree that smaller companies like Rhythm & Hues have winked out of existence. These new mediums will allow ILM to repurpose the assets it creates for movies, wringing additional value out of them, and with many different types of xLab projects in development Bredow is bullish on his team’s prospects. "We know that these foundational technologies have tons of applications," Bredow says. "If AR ends up being huge and VR ends up being medium-sized, that works out just fine for us. We’ve got lots of plays in both. Even if we were to only leverage this technology for the filmmakers to be immersed in their environments, that’s really useful too."

John Gaeta at ILMxLab

John Gaeta plays with a velociraptor.

That trickle-up effect is already happening. Director Gareth Edwards has been using an xLab Oculus Rift set-up to preview and tweak set designs for his upcoming film Star Wars: Rogue One, Bredow says. "When he, two minutes into having the headset on, referred to it as ‘real life,’ I was like, ‘okay, we’re on to something here.’"

"I want to be in the X-Wing. I want to be piloting the AT-AT."

Bredow hopes to release some sort of xLab demo experience within the next six to 12 months — my money’s on a piece tied to The Force Awakens — but some of the group’s early work is already available for public consumption. The lab worked with filmmakers Felix & Paul for a Jurassic World VR tie-in, and also contributed to Warcraft: The Skies of Azeroth, which debuted during San Diego Comic-Con. But those are things VR dabblers are already familiar with: relatively static, on-rails experiences. Baby steps, necessary to introduce people to an entirely new medium, but nothing near what Gaeta and Bredow are really dreaming of… yet.

"What I’m talking about is being in the shoes of a character like you saw in the story," Gaeta tells me. "I want to be in the X-Wing. I want to be piloting the AT-AT. Or be where the droids are hiding. We think those are experiences that are just one step deeper than consuming the story. And what’s wonderful about that is that once the destination is built you can wander around in it, you can be social in it, you can do other things. You can play in it."

Video edited by Ryan Manning. Shot by Tom Connors and John Lagomarsino.