Skip to main content

Google’s AI has learned how to draw by looking at your doodles

Google’s AI has learned how to draw by looking at your doodles

Share this story

Image: Google

Remember last year when Google released an AI-powered web tool that played Pictionary with your doodles? Well, surprise! Those doodles you drew have now been used to teach Google’s AI how to draw. The resulting program is called Sketch-RNN and, frankly, it draws about as well as a toddler. But like any new parents, Google’s AI scientists are proud as punch.

To create Sketch-RNN, Google Brain researchers David Ha and Douglas Eck collected more than five million user-drawn sketches from the Google tool Quick, Draw! Each time a user drew something on the app, it recorded not only the final image, but also the order and direction of every pen stroke used to make it. The resulting data gives a more complete picture (ho, ho, ho) of how we really draw.

All in all, Ha and Eck gathered 70,000 training doodles for 75 different categories, including “cat,” “firetruck,” “garden,” “owl,” “pig,” “face,” and “mermaid.” Their goal? To create a machine that can “draw and generalize abstract concepts in a manner similar to humans.” And it can! After studying this data, it learned first to draw based on human input, as seen below:

The human doodles on the left; the computer’s imitations are on the right.
The human doodles on the left; the computer’s imitations are on the right.
Image: Google

Notice, as seen most clearly in the penultimate row, the AI is not just copying line for line the human doodle. The input on the left-hand side shows a cat with three eyes — but the AI is copying the concept, not the sketch itself, and it knows enough to know three eyes is one too many.

Next, Sketch-RNN learned to draw the objects without copying a starting sketch. (For more on how deep neural networks process and imitate data, check out our AI explainer.)

From left to right: frogs, cats, and crabs.
From left to right: frogs, cats, and crabs.
Image: Google

But what’s the benefit of getting neural networks to sketch things in the first place, when they’re already pretty good at making photo-realistic images? Well, as Ha and Eck explain, although doodles look childish to us, they’re also masterpieces of abstraction and data compression. Doodles, they say, “tell us something about how people represent and reconstruct images of the world around them.” In other words, they’re more human. And once you’ve taught an AI to sketch, you can deploy it in all sorts of fun ways. Sketch-RNN can complete doodles started by someone else:

And it can combine different doodles together. So, in the picture below, the neural network has been asked to draw some combination of the category “cat” and “chair.” The result? Weird cat-chair chimeras:

It can also create what are called “latent space interpolations” — looking at any number of doodle subjects, and combining them together in different ratios to create new sketches with multiple characteristics. In the group of drawings on the left, below, the AI has combined four different doodles: the pig, rabbit, crab, and face.

These drawings are obviously quite basic, but the methods used to create them are so interesting — and, so potentially useful. In the future, AI programs like Sketch-RNN could be used as creative aids for designers, architects, and artists. If someone is struggling with a certain picture or design, they could get an AI to absorb their work and spit out a few more suggested variations. The images the computer produces might not be useful in themselves, but they could spark something in the human. Is this AI creativity? It’s difficult to know what else to call it.