Skip to main content

This app uses a neural network to describe the world around you

This app uses a neural network to describe the world around you

/

It's wrong more often than it's right

Share this story

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

"A tree climbing a piece of wood. A large bird sitting on a mountain. A bike with a bike leaning against it."

This is the world according to AI Scry, a new iOS app that uses a neural network to identify whatever's in front of your phone's camera and describe it to you. The results veer between uncannily accurate and enjoyable nonsense (tilted heavily towards the latter), but still manage to encapsulate both the promise and limitations of neural networks. It's also just very fun to wander around while a computer narrates what's in front of you in a rambling, free associative style.

AI Scry uses a well-know neural network

AI Scry isn't an entirely new concept. It's built by an art and technology studio named disk cactus (stylized as a pair of emoji that our site can't render), but is powered by a well-travelled neural network known as Neural Talk. This program was developed by a computer scientist from Stanford University named Andrej Karpathy, who later open-sourced the code to let anyone play with it. We've seen a Neural Talk-enabled laptop go on a stroll around Amsterdam, and there have been similar AI projects, such as an app for Android, and Facebook's efforts to automatically caption photos for the blind. AI Scry, though, embraces the weirdness of machine vision wholeheartedly.

AI Scry takes on The Verge's offices and neighboring streets.

Sam Kronick of disk cactus tells The Verge that he and his colleagues want the app to introduce people to the problems and benefits of AI and machine learning without being "too robotic." He says that "whether it’s Spotify building custom playlists or companies deciding your credit score," these sorts of hidden systems are increasingly common, but not well understood. "When you set one free and unconstrained in the world (as with AI Scry), it’s plain to see that it has a ‘mind’ of its own," says Kronick. "[It's] complete with weird quirks and idiosyncrasies that can be traced back to the chain of humans who designed, programmed, and trained the system."

These quirks become clear when using the app, with AI Scry managing to be both very clever and extremely dumb. It's nowhere near smart enough to tell the difference between, say, a pen and a pencil, or even recognize a $5 bill on a table, but it's more successful on a macro scale. It can tell the difference between the contents of a fridge and a park full of trees, for example, or spot that this is a car and that is a person. And occasionally, it makes imaginative leaps that seem ingenious.

At one point the app described a particularly dense and bushy-looking bush as "a close up of a garden of broccoli," which borders on the poetic. But these moments are definitely few and far between, and more frequently the app is just dead wrong, often falling back on the same few incorrect guesses. (In my experience AI Scry particularly liked to identify things as scissors, mirrors, and bananas. Imagine how terrifying a world made up of only these three objects would be.)

You can adjust what the AI Scry team call the "Attention Aperture" to tweak the accuracy of these results. This slider — the name of which was inspired by the US military's research into psychic powers — gives the captioning algorithm more or less free rein. Set to the far left, you only get the most obvious and safe answers (which can still be wrong), but set it to the far right, and the computer will take more chances on what it's seeing.

"Further and further down that probability chain."

"The Attention Aperture lets you go further and further down that probability chain until it just sounds like random noise," explains Kronick. "In the human brain, we do something like this when using similes or metaphors… but this is an alien brain that we don’t fully understand, so you are welcome to perform your own experiments and come up with your own diagnosis of what’s going on."

Kronick says that this part of the app is best described as "alien psychology" — probing the oddities of the neural network by turning its free-association up to max. It's an interpretation that might be a little too grandiose for some people, but it's an interesting and empowering way to relate to a world full of AI. It's certainly better than resigning ourselves to the idea that we'll never understand our computers, even if they are spouting nonsense.