Google has open-sourced an AI tool that helps phones shoot pictures in portrait mode

Google’s Pixel phones have some of the best cameras on the market.
Photo by James Bareham / The Verge

Google’s Pixel phone has one hell of a camera, and one of the reasons for this is AI. Google has used its machine learning talent to squeeze better images out of a tiny smartphone lens, including its portrait mode shots, with blurred backgrounds and pin-sharp subjects.

Now, Google has open-sourced a lump of code named DeepLab-v3+ that it says will help others recreate the same effect. (Although, this is not the same tech that Google itself uses in the Pixel phones — see the correction note at the bottom of the article.) DeepLab-v3+ is an image segmentation tool built using convolutional neural networks, or CNNs: a machine learning method that’s particularly good at analyzing visual data. Image segmentation analyzes objects within a picture, and splits them apart; dividing foreground elements from background elements. This can then be used to create ‘bokeh’ style photographs.

A diagram showing how image segmentation works for a typical photograph.
Image: Google

As Google software engineers Liang-Chieh Chen and Yukun Zhu explain, image segmentation has improved rapidly with the recent deep-learning boom, reaching “accuracy levels that were hard to imagine even five years [ago].” The company says it hopes that by publicly sharing the system “other groups in academia and industry [will be able] to reproduce and further improve” on Google’s work.

At the very least, opening up this piece of software to the community should help app developers who need some lickety-split image segmentation, just like Google does it.

Correction: Google contacted The Verge to clarify that DeepLab-v3+ is not the exact same technology used in the Pixel’s portrait mode, as the company’s original blog post had implied. Portrait mode on the Pixel is just an example of the sort of features DeepLab-v3+ can enable. We regret the error.

Comments

I want to see goggle release the whole the thing.

I’m kind of surprised they’re releasing even this. Seems like a proprietary advantage, and they are giving it away.

It could result in more sales for Android.

Or something for Amazon to fork and then ban sales of Google Pixel phones on Amazon.com

#wouldntputitpastthem

Considering android’s market share, I’m not sure it’s possible to sell more android phones.

People could start buying 2-packs?

Advertise them for use in "True Stereo Calling", one for each ear.

Perhaps they don’t see it as an advantage (either because it’s a temporary crutch toward multi-lens SKUs, or because despite being impressive – just one lens! – it only looks as good as iPhone or competing implementations for the final product, which is what matters to customers).

(As opposed to their continuous HDR mode, which is technically impressive, matters to customers, and significantly better by nearly all criticism, and doesn’t already have a just-as-good alternative on iPhone).

Perhaps the advantage is that they’ve already replaced it with a significantly better implementation so here is some free code to divert some of the resources of all of our competitors for a couple months.

I agree, that it’s like giving away an advantage. Couldn’t Apple just take the code and use it to sell even more iPhone 8 devices that have a single lens?

Yes but, why loose the opportunity to sell more expensive phones using the feature as a deal breaker?

Besides, most people seems to buy smartphones based on brand loyalty instead of which features are more beneficial to them, thus the feature being available in most android devices won’t drive that much of a market share away.

The usual Google move in the software/AI space seems to be. 1 – Invent some amazing software that gives a market advantage. 2 – Iterate and improve on this idea until the first idea becomes irrelevant or the rest of the market catches up. 3 – Release a paper detailing your original breaktbrough to show how amazing you were all the while holding onto the next big idea to have as a future advantage. This makes me pretty excited for what their next photography plays and tech will be.

Not exactly the code, but you can get the apk file of the Pixel’s camera from XDA.

Exactly and I did just that for my Mi A1.
The results are impressive, put’s the Mi A1 close to flagship levels of quality in photography for both the front and rear camera in both good and low light. The selfie Portrait mode with the Pixel Camera app is simply staggering. Beats any similar solution by miles.

Weird, this article should explain that semantic image segmentation engine is used for other things besides separating foreground from background in portrait mode??? Maybe they’re opening it up so other developers can come up with new ways to use it?

Hopefully it can be used to isolate things in images and make them look bigger. I could use that for science and stuff.

I think some form of this is also applied in autonomous navigation?

This gives every Android phone a leg up over iPhone, potentially.. which then comes back full swing to benefit Google/Android when these devices all sport a competitive advantage over an iPhone. Seems like a smart call to me.

Yep, also don’t underestimate the effect of "Google is nice"

Yea but what’s stopping Apple from benefitting too?

This is pretty huge! Things like this is what makes a technology company like Google amazing.

View All Comments
Back to top ↑