Skip to main content

'Clouds': an interactive documentary shot on Kinect proves the future of video is not at CES

'Clouds': an interactive documentary shot on Kinect proves the future of video is not at CES

Share this story

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

Jonathan Minard & James George (RGB+D)
Jonathan Minard & James George (RGB+D)

The camera of the future is not at CES — it's at Eyebeam, an art and technology incubator in the Chelsea neighborhood of New York City, where two artists-in-residence are currently shooting a documentary that's probably better described as interactive software than a feature film.

Ever since their first meeting at a coding conference in Pittsburgh, James George and Jonathan Minard have been perfecting their experimental filmmaking technique, which uses a DSLR + Kinect hybrid image capture process in conjunction with custom editing software. The results are two-fold: a distinct set of stuttered images that can be spatially manipulated by the viewer, and a free dev kit that allows anyone with the right equipment to play along — like Hollywood 3D's dirtier, open-source cousin made into a video game.

Clouds, an upcoming documentary that was just funded on Kickstarter, is being shot entirely using this method, which George and Minard call RGB+D. Set within the framework resembling a modern "choose your own adventure" game, the duo says the finished film (if you could call it that) will be almost game-like — an interactive narrative offering a reflective look at the culture of creative coding and software art, as told by a sizable cast of artists, journalists, and curators.

George and Minard explain that the concept came from a keynote address from sci-fi author Bruce Sterling — one of the subjects of the documentary — in which he tried to predict what the cameras of the future would be like, describing a device "that absorbed photons from all angles at all times, turning the act of taking a picture into a computational problem of choosing which angle at which moment to visualize."

"We want 'Clouds' to feel like a continuous revelation in a space of ideas."

With its noted hacker-friendliness and the ability to capture 3-dimensional images in 30 frames-per-second, the Kinect seemed an obvious choice to try and make that fiction a reality. George and Minard began beta testing their methods, using the process to document a Reddit AMA and an art hackathon at nearby gallery space 319 Scholes. But eventually the stylized footage led to bigger ideas.

"The fact that we were already visualizing this data in a virtual environment resembling a video game, with unlimited possible camera positions and visual styles, had opened up our thinking," write George and Minard in an email interview. "It dawned on us that the conversation we were recording had begun to resemble an interconnected network."

The two artists have begun parsing the various nodes of that network into conversation "seeds," allowing the viewer to ask questions and follow various digressions as the conversation progresses. The end result will be an executable program generating a virtual space in which viewing angles can be changed and individual ideas pursued. Suddenly, the videography ceases to be a lightly-interactive depiction of events and starts to become more like a computer simulation, navigated in real time.

"The goal of building such a system is to leave the possibility space wide open, to create an experience that feels spontaneous and full of surprises ... We want Clouds to feel like a continuous revelation in a space of ideas."

Today’s Storystream

Feed refreshed An hour ago Dimorphos didn’t even see it coming

J
External Link
Jess WeatherbedAn hour ago
Won’t anyone think of the billionaires?

Forbes reports that rising inflation and falling stock prices have collectively cost members of the Forbes 400 US rich list $500 billion in 2022 with tech tycoons suffering the biggest losses.

Jeff Bezos (worth $151 billion) lost $50 billion, Google’s Larry Page and Sergey Brin (worth a collective $182b) lost almost $60b, Mark Zuckerberg (worth $57.7b) lost $76.8b, and Twitter co-founder Jack Dorsey (worth $4.5b) lost $10.4b. Former Microsoft CEO Steve Ballmer (worth $83b) lost $13.5b while his ex-boss Bill Gates (worth $106b) lost $28b, albeit $20b of that via charity donations.


T
Thomas Ricker6:45 AM UTC
Check out this delightful DART Easter egg.

Just Google for “NASA DART.” You’re welcome.


M
The Verge
We’re about an hour away from a space crash.

At 7:14PM ET, a NASA spacecraft is going to smash into an asteroid! Coverage of the collision — called the Double Asteroid Redirection Test — is now live.


E
Twitter
Emma RothSep 26
There’s a surprise in the sky tonight.

Jupiter will be about 367 million miles away from Earth this evening. While that may seem like a long way, it’s the closest it’s been to our home planet since 1963.

During this time, Jupiter will be visible to the naked eye (but binoculars can help). You can check where and when you can get a glimpse of the gas giant from this website.


E
Twitter
Emma RothSep 26
Missing classic Mario?

One fan, who goes by the name Metroid Mike 64 on Twitter, just built a full-on 2D Mario game inside Super Mario Maker 2 complete with 40 levels and eight worlds.

Looking at the gameplay shared on Twitter is enough to make me want to break out my SNES, or at least buy Super Mario Maker 2 so I can play this epic retro revamp.


R
External Link
Russell BrandomSep 26
The US might still force TikTok into a data security deal with Oracle.

The New York Times says the White House is still working on TikTok’s Trump-era data security deal, which has been in a weird limbo for nearly two years now. The terms are basically the same: Oracle plays babysitter but the app doesn’t get banned. Maybe it will happen now, though?


R
Youtube
Richard LawlerSep 26
Don’t miss this dive into Guillermo del Toro’s stop-motion Pinocchio flick.

Andrew Webster and Charles Pulliam-Moore covered Netflix’s Tudum reveals (yes, it’s going to keep using that brand name) over the weekend as the streamer showed off things that haven’t been canceled yet.

Beyond The Way of the Househusband season two news and timing information about two The Witcher projects, you should make time for this incredible behind-the-scenes video showing the process of making Pinocchio.


R
External Link
Russell BrandomSep 26
Edward Snowden has been granted Russian citizenship.

The NSA whistleblower has been living in Russia for the 9 years — first as a refugee, then on a series of temporary residency permits. He applied for Russian citizenship in November 2020, but has said he won’t renounce his status as a U.S. citizen.


E
External Link
Emma RothSep 26
Netflix’s gaming bet gets even bigger.

Even though fewer than one percent of Netflix subscribers have tried its mobile games, Netflix just opened up another studio in Finland after acquiring the Helsinki-based Next Games earlier this year.

The former vice president of Zynga Games, Marko Lastikka, will serve as the studio director. His track record includes working on SimCity BuildIt for EA and FarmVille 3.


A
External Link
Vietnam’s EV aspirant is giving big Potemkin village vibes

Idle equipment, absent workers, deserted villages, an empty swimming pool. VinFast is Vietnam’s answer to Tesla, with the goal of making 1 million EVs in the next 5-6 years to sell to customers US, Canada and Europe. With these lofty goals, the company invited a bunch of social media influencers, as well as some auto journalists, on a “a four-day, multicity extravaganza” that seemed more weird than convincing, according to Bloomberg.