Skip to main content

This MIT neural network translates pictures of food into recipes

This MIT neural network translates pictures of food into recipes


Take a pic, get a recipe

Share this story


Researchers from the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a neural network that can, in theory, “look” at an image and find the recipe for the food that’s depicted. It’s the latest addition to an increasing list of similar projects, following the likes of both Pinterest and, yes, Silicon Valley’s actually real hot dog-identifying app. 2017 is apparently the year of using the major advancements in AI to identify food.

The CSAIL team calls the network Recipe1M, and it’s described in detail in a paper that was published this week. Simply put: the researchers fed the AI more than 1 million recipes and nearly 1 million images. Over the course of that training, it made and refined associations between what goes into a recipe and how that relates to a food photo.

The result is an interface called Pic2Recipe that’s reminiscent of the lo-fi goofy TensorFlow projects we’ve seen, like edges2cats or Pix2Pix. Instead of generating nightmare fuel, though, this web portal grabs recipes based on the food photos that you upload and ranks them based on how sure the system is that the results are correct.

The idea is fun, but don’t expect good results just yet

But I wrote “in theory” before because, in my short round of testing, Pic2Recipe mostly failed at correctly making those associations. Pictures I’ve taken of bowls of ramen, bags of potato chips, teriyaki beef ribs, and even rice and beans were all met with a response of “no matches.” A picture of the Impossible Burger at Nishi fetched a recipe for “baby bagel sandwiches” (though that one was admittedly tricky).

A blurry photo of my grandmother’s Chex Mix returned some closer results, like recipes for “ginger almonds” and “spicy chili peanuts.” But this isn’t meant to be a working product just yet, and I think it’s easy to see how a project like this could actually blossom into a real version of the idea as the whole system goes through more refinement. Funny enough, the only result that Pic2Recipe got right on the nose was when I uploaded a picture of a hot dog. I’m not saying that projects like these are only good for identifying hot dogs. I’m just being frank.