Nonny de la Peña was the first person to bring modern virtual reality to Sundance, when she debuted Hunger in Los Angeles in 2012. Since then, one of her former interns at USC's mixed reality program — a one-time journalism student named Palmer Luckey — has helped turn VR into a global phenomenon. This year, de la Peña's work is returning to Sundance alongside over two dozen other virtual reality experiences, spanning a range of genres and formats.
But de la Peña's two pieces still aren't quite like anything in the show. While Across the Line and Kiya, the former about anti-abortion protestors and the latter about intimate partner violence, chronicle real events, they don't do so through film. Instead, she and her team record real audio and motion capture data, then work it into a digital recreation of an event. It's an unusual technique that lets people not just see an event, but explore it.
After having a strong reaction to both pieces earlier this week at Sundance, I sat down with de la Peña to talk about her work, the value of interactivity, and the rise of virtual reality journalism.
Interview has been condensed and edited for clarity.
Adi Robertson: Has it changed how you approach things, now that VR is becoming more democratized and more people can see your work?
Nonny de la Peña: I don't think it changes my approach other than that I'm a little more thoughtful on how I design a piece that can be experienced on different platforms, and when I limit it to one platform. When is the piece really much better as a sit-down thing, and when is it much better if it’s a walk-around? So that is a consideration that definitely has been affected by the way that Google Cardboards are out there en masse now.
To have them come in your Sunday morning newspaper, that's sort of a minor miracle to me. Especially because it's been journalism in virtual reality — having people think I was completely mad, and now having an unbelievable explosion of artists interested in using the medium for telling stories. It's pretty extraordinary.
You got a Knight Foundation grant last year to help establish best practices for VR journalism, right?
We just did a kickoff with Frontline maybe two weeks ago in New York, so we're just starting. There are some great folks that have put together some handbooks on best practices for approaching GoPro camera shooting with 360, but we will also look at, what does it mean to think about [non-video] spatial narrative?
I think for journalists, initially, everybody felt very, very comfortable with 360-degree video, because you're filming realism, even though it's got its own encumbrances for the way that you have to shoot. But when you think about, how do we start recreating [an environment] in real time, so you can do it in a gaming engine, sometimes people get a little nervous. And I find it kind of surprising.
"Everybody felt very comfortable with 360-degree video, because you're filming realism."
I wasn't feeling so well last night, so I decided to stay in bed and watch Making a Murderer. And when you're making documentary films, you have to cut to people working in the field, or cows, or reflections on cars. You're just not really showing anything that's happening based on what people are saying. Somehow, people feel that doing that in a VR way can be less ethical. But it isn't. It's just a different approach.
I remember The Thin Blue Line — people had huge reactions towards that, right? People thought the recreations were somehow scandalous. And nobody thinks that anymore. They recognize that there was legitimacy in what he did, and good research, and good thinking through how he made the pieces. So I think we're going through a little of these growing pains in VR.
Have you gotten to try any of the 8i stuff yet?
Yes, I saw #100humans here and some pieces a few months ago.
That's going to be very interesting. I've got an LGBTQ project I'm working on — 80 percent of homeless youth are LGBTQ, homeless youth that have been tossed out by their parents. And we're working with a kid who actually recorded the moment when he gets thrown out of the house and his family beats him.
And we're going to recreate what happened to him. We're going to scan him and put him in as himself. But we're also going to bring him in and shoot him with 8i and have him tell about his life and how he's managed to pull himself together anyway, as a kind of hopeful end to the piece.
For things like the LBGTQ project, how do you find people and help them work in this medium?
A lot of guidance. I fly the individual out and help guide them through the process so they understand, so they know what they're going to go through, what it means for them.
With Use of Force, which is a piece I did about the border patrol beating [migrant Anastasio Hernandez Rojas] to death, I brought the main witness in and I scanned her face and body, and I motion captured her face and body, recreating her whole memory of the night.
So instead of her saying to me, "And then I yelled down at them ‘Stop, stop, what are you doing,'" she was pretending she had her camera in her hand filming, and she was yelling "Stop, stop" — and then I'd put her in next to you. She's telling her story as much through her body as through her words, which is sort of native to this medium.
How do we put these tools into the hands of other people, the way that now anyone can be a filmmaker?
That is going to happen. It is true we're at a new juncture, but it was only four years ago that I showed up with Palmer [Luckey] with a duct-taped pair of goggles, right?
I'm old enough that I had to change film in a bag when I was making documentary films. Even just a few years ago, working as a journalist and talking about what kind of internet piece you can do, people would say, "Well, it's got to be on a non-smartphone." Now everybody's got a smartphone. So in so few years, the advances are happening so fast.
"It won't be that long before the tools of this trade are going to be available to many, many people."
After Palmer sold the [Oculus Rift] goggles for $2 billion, I texted him and said, "Thank you very much," because it was very clear to me that this was going to change everybody's lives. And what happened was HTC and Valve got involved, and Sony got involved, and all these artists got involved. We just saw this explosion of energy thinking about VR. Something's changed, right?
So maybe it's not so easy for somebody yet to go out and approach making a VR piece yet, maybe it's still a little bit cumbersome. But even last year, this space had a fraction of the artists that it has today. There's a lot of people that are turning their energy and resources towards it. It won't be that long before the tools of this trade are going to be available to many, many people. And we're going to continue to see, it's just beginning.
Is there still impact in people making immersive work in non-VR worlds, like Second Life?
Yeah! One of the things that has yet to be cracked [in VR], of course, is shared spaces. The story I love to tell is — my daughter was out in my garage office, and my son was in the living room. My daughter walks into the living room, and he looks up and goes, "Why did you leave?" But she'd just walked into the room. And what he meant was, why did you leave the Minecraft server we're playing on?
And I think people are always going to like having representations of themselves that aren't necessarily photoreal, that there’s plenty of playful space for this.
It feels like a few years ago, we had a big conversation about empathy in games, and empathy was about interacting with a game system. Now it seems like we’ve shifted that to VR.
It’s empathy because of the content people have been making. And it's a full body thing. But what we haven't played with at all yet is that interactivity. Did you listen to the Serial podcast? If you could imagine standing in the back of the Best Buy parking lot, and you're trying to figure out time — and you can rewind time and fast-forward time and participate in how you engage with the story. It doesn't mean that the story changes. But the way that maybe you want to rehear what somebody just said, or freeze one character and try figuring out what everybody else was doing at the time.
"As human beings we have this need to create."
There's so many implications with the controllers that we haven't even begun to integrate yet with VR. Of course, [Vive painting game] TiltBrush is great. There's something really fundamentally interesting about TiltBrush that I think about a lot, because my daughter's very much an artist, my son doesn't think about himself as an artist, but he's been playing for literally hours. So what does that mean?
As human beings, we have this need to create. Anybody and everybody goes into TiltBrush and engages with it in a way that taps into some inner creative thing. So when I think about controllers and interactivity — wow. I don't know what's going to happen next.