Skip to main content

Twitter is using machine learning to crop photos to the most interesting part

Twitter is using machine learning to crop photos to the most interesting part

/

The machine knows what you want to look at

Share this story

The lure of machine learning isn’t always about big new features; often, what it does best are small tweaks that subtly improve user experience. So it is with Twitter’s use of neural networks to automatically crop picture previews to their most interesting part.

The company’s been working on this tool for a while, but described its methods in detail in a blog post yesterday. It’s an interesting little read, with ML researcher Lucas Theis and ML lead Zehan Wang explaining how they started just using facial recognition to crop images to faces, but found that this method didn’t work with pictures of scenery, objects, and, most importantly, cats.

Their solution was “cropping using saliency” (saliency here meaning whatever’s most interesting in a picture — faces or not). To define this they used data from academic studies into eye-tracking, which record what areas of images people look at first. “This data can be used to train neural networks and other algorithms to predict what people might want to look at,” write Theis and Wang.

More examples of before and after using Twitter’s ML-powered auto-cropper.
More examples of before and after using Twitter’s ML-powered auto-cropper.
Image: Twitter

Once they’d trained a neural network to identify these areas, they needed to optimize it to work in real time on the site. Luckily for them, the cropping needed for a photo preview is pretty broad — you’re only narrowing down an image to maybe its most interesting third. You don’t need to target in on specifics. That means Twitter could pare down and simplify the criteria the neural network was judging using a technique called “knowledge distillation.”

The end result was a neural network ten times faster than its original design. “This lets us perform saliency detection on all images as soon as they are uploaded and crop them in real-time,” write Theis and Wang. 

This new feature is currently being rolled out on desktop, iOS, and Android apps to all users says the company. So next time you see a photo preview on Twitter that invites you to click remember to thank a neural network.