Skip to main content

Google and MIT’s new machine learning algorithms retouch your photos before you take them

Google and MIT’s new machine learning algorithms retouch your photos before you take them

Share this story

It’s getting harder and harder to squeeze more performance out of your phone’s camera hardware. That’s why companies like Google are turning to computational photography: using algorithms and machine learning to improve your snaps. The latest research from the search giant, conducted with scientists from MIT, takes this work to a new level, producing algorithms that are capable of retouching your photos like a professional photographer in real time, before you take them.

The researchers used machine learning to create their software, training neural networks on a dataset of 5,000 images created by Adobe and MIT. Each image in this collection has been retouched by five different photographers, and Google and MIT’s algorithms used this data to learn what sort of improvements to make to different photos. This might mean increasing the brightness here, reducing the saturation there, and so on.

Using machine learning to improve photos has been done before, but the real advance with this research is slimming down the algorithms so that they are small and efficient enough to run on a user’s device without any lag. The software itself is no bigger than a single digital image, and, according to a blog post from MIT, could be equipped “to process images in a range of styles.”

A composition created by MIT showing the original 12-megapixel image (left) and the retouched version produced by the new algorithm (right).
A composition created by MIT showing the original 12-megapixel image (left) and the retouched version produced by the new algorithm (right).

This means the neural networks could be trained on new sets of images, and could even learn to reproduce an individual photographer’s particular look, in the same way companies like Facebook and Prisma have created artistic filters that mimic famous painters. Of course, it’s worth pointing out that smartphones and cameras already process imaging data in real time, but these new techniques are more subtle and reactive, responding to the needs of individual images, rather than applying general rules.

In order to slim down the algorithms, the researchers used a few different techniques. These included turning the changes made to each photo into formulae and using grid-like coordinates to map out the pictures. All this means that the information about how to retouch the photos can be expressed mathematically, rather than as full-scale photos.

“This technology has the potential to be very useful for real-time image enhancement on mobile platforms,” Google research Jon Barron told MIT. “Using machine learning for computational photography is an exciting prospect but is limited by the severe computational and power constraints of mobile phones. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.”

Will we be seeing these algorithms pop up in one of Google’s future Pixel phones? It’s not unlikely. The company has previously used its HDR+ algorithms to bring out more detail in light and shadow on mobile devices since the Nexus 6. And speaking to The Verge last year, Google’s computational photography lead, Marc Levoy, said that we’re “only beginning to scratch the surface” with this work.

Today’s Storystream

Feed refreshed 31 minutes ago Yes, it happened again.

D
External Link
David Pierce31 minutes ago
Bluetooth speakerphones are an underrated accessory.

Wirecutter has a good roundup of options, which are as good for Zoom calls as advertised, but they’re also just useful in life! I use mine roughly as a desk phone, and also connect it every time we’re doing the “everyone gather around the phone to talk to Grandma” thing. I have the Jabra Speak 510, and love it.


E
External Link
Elizabeth Lopatto54 minutes ago
The SPAC-lash is here.

So remember the craze for SPACs, an alternative way to take a company public? The times, they have a-changed: the SPAC King of Silicon Valley, Chamath Palihapitiya, couldn’t find acquisition targets for two of his piles of cash. One of them, Social Capital Hedosophia VI, is the largest-ever SPAC.


R
Twitter
Richard Lawler58 minutes ago
Use every (mega) pixel.

Halide is touted as one of the best photography apps on the iPhone, and as usual, it has a new update out to help you get the most out of the best camera available on the platform — that 48-megapixel sensor on the iPhone 14 Pro.

Version 2.9.0 of Halide is live in the App Store, with quick 48MP shooting directly to HEIC or JPG and many other new features.


Welcome to the new Verge

Revolutionizing the media with blog posts

Nilay PatelSep 13
N
Quote
Nilay Patel60 minutes ago
Speaking of CarPlay.

Dan posted about CarPlay earlier and it reminded me that wireless CarPlay has actually set Apple back in its quest to somehow take over the car market — problems with it disconnecting are the number one complaint from new car owners, according to JD Power. Something to think about as Apple continues its relentless quest to remove all the ports from our phones.

The most frequent problem reported across the industry regardless of powertrain type or brand cachet related to smartphone connectivity. Dropped connections with Apple CarPlay was the number one problem, as Android Auto complaints leveled off from last year. Further, the number of reported problems with wireless Apple CarPlay increased as well, due to the feature being offered in more cars. 


R
Twitter
Richard LawlerTwo hours ago
PC gamers showed up for a PlayStation hit.

NPD’s August update for videogame sales has arrived, noting that PS5 was number one in hardware sales for the month and that new-gen hardware sales are up significantly from last year due to improved supply for both the PS5 and Xbox Series X / S.

But the most astounding leap came from Marvel’s Spider-Man, which jumped from 84th on the chart the month before to number 3, thanks to Sony re-releasing the PlayStation exclusive on PC via Steam (and Steam Deck).


D
External Link
Dan SeifertTwo hours ago
Apple’s CarPlay is still frustratingly basic.

Stephen Hackett at 512 Pixels has blogged about the frustrations he’s had migrating to a new iPhone and not having his CarPlay preferences carry over, despite every other app on his phone copying over correctly.

I’m with him on that, but I’m more annoyed by the second point he highlights: CarPlay still treats every vehicle you connect to as a different thing. Instead of preserving your preferences when you plug in to a different car, it makes you set it all up again for each new vehicle. This is annoying for families with more than one car; it’s downright maddening for frequent users of rental cars. Seems like a simple thing to fix!


R
Quote
Richard Lawler12:49 PM UTC
Adnan is out.

Yesterday, a Baltimore City Circuit judge overturned the murder conviction of Adnan Syed, setting him free — for the moment — after serving 23 years in a case documented by the podcast Serial. This morning, host Sarah Koenig released Serial’s first new episode in seven years.

It’s Baltimore, 2022. Adnan Syed has spent the last 23 years incarcerated, serving a life sentence for the murder of Hae Min Lee, a crime he says he didn’t commit. He has exhausted every legal avenue for relief, including a petition to the United States Supreme Court. But then, a prosecutor in the Baltimore State’s Attorney’s office stumbled upon two handwritten notes in Adnan’s case file, and that changed everything.


J
External Link
James Vincent11:48 AM UTC
For every living human there are 2.5 million ants, say scientists, unprompted.

I honestly don’t know what to do with this information, which comes via The Washington Post. This is just one guy’s opinion, but it seems like an awful lot of ants. Like God accidentally maxed out the ant-slider or spilled a bag of “Oops! All ants!” into the biosphere during Creation. What I need is a lie down and to not think about the millions — sorry, 20 quadrillion — of ants out there.