Google Lens actually shows how AI can make life easier

During Google’s I/O developer conference keynote, artificial intelligence was once again the defining theme and Google’s guiding light for the future. AI is now interwoven into everything Google does, and nowhere is the benefits of CEO Sundar Pichai’s AI-first approach more apparent than with Google Lens.

The Lens platform combines the company’s most cutting-edge advances in computer vision and natural language processing with the power of Google Search. In doing so, Google makes a compelling argument for why its way of developing AI will generate more immediately useful software than its biggest rivals, like Amazon and Facebook. It also gives AI naysayers an illustrative example of what the technology can do for consumers, instead of just for under-the-hood systems like data centers and advertising networks or for more limited hardware use cases like smart speakers.

Lens is effectively Google’s engine for seeing, understanding, and augmenting the real world. It lives in the camera viewfinder of Google-powered software like Assistant and, following an announcement at I/O this year, within the native camera of top-tier Android smartphones. For Google, anything a human can recognize is fair game for Lens. That includes objects and environments, people and animals (or even photos of animals), and any scrap of text as it appears on street signs, screens, restaurant menus, and books. From there, Google uses the expansive knowledge base of Search to surface actionable info like purchase links for products and Wikipedia descriptions of famous landmarks. The goal is to give users context about their environments and any and all objects within those environments.

Image: Google

The platform, first announced at last year’s I/O conference, is now being integrated directly into the Android camera on Google Pixel devices, as well as flagship phones from LG, Motorola, Xiaomi, and others. In addition to that, Google announced that Lens now works in real time and can parse text as it appears in the real world. Google Lens can even now recognize the style of clothing and furniture to power a recommendation engine the company calls Style Match, which is designed for helping Lens users decorate their home and build matching outfits.

Lens, which before today existed only within Google Assistant, is also moving beyond just the Assistant, camera, and Google Photos app. It’s also helping power new features in adjacent products like Google Maps. In one particular eye-popping demo, Google showed off how Lens can power an augmented reality version of Street View calls out notable locations and landmarks with visual overlays.

In a live demo today at I/O, I got a chance to try some the new Google Lens features on a LG G7 ThinQ. The feature now works in real time, as advertised, and it was able to identify a number of different products from shirts to books to paintings with only a few understandable hiccups.

For instance, in one situation, Google Lens thought a shoe was a Picasso painting, only because it momentarily got confused about the location of the objects. Moving closer to the desired object I wanted to recognize, the shoe in this case, fixed the issue. Even when the camera was too close for Lens to identify the object, or if it was having trouble figuring out what it was, you could tap the screen and Google would give you its best guess with a short phrase like, “Is it... art?” or, “This looks like a painting.”

Image: Google

Most impressive is Google Lens’ ability to parse text and extract it from the real world. The foundational groundwork for this has already been laid with products like Google Translate that can turn a street sign or restaurant menu in a foreign language into your native tongue by just snapping a photo. Now that those advancements have been refined and built into Lens, you can do this in real time with dinner menu items or even big chunks of text in a book.

In our demo, we scanned a page of Italian dishes to surface photos of those items on Google Image Search, in addition to YouTube videos for how to make the food too. We could also translate the menu headers from Italian into English by just selecting the part of the menu, an action that automatically transforms the text into a searchable format. From there, you can copy and paste that text elsewhere on your phone and even translate it on the fly. This is where Google Lens really shines by merging the company’s strengths across a number of products simultaneously.

We unfortunately didn’t get to try Style Match or the Street View features that were shown off during the keynote, the latter of which is a more experimental feature without a concrete date for when it will actually arrive for consumers. Still, Google Lens is much more powerful one year into its existence, and Google is making sure it can live on as many devices, including Apple-made ones, and within as many layers of those devices as possible. For a company that’s betting its future on AI, there’s few examples as compelling for what that future will look like and enable for everyday consumers than Lens.

Comments

*shown off (not "shone off")

Take that Apple, I think Apple lost this year I don’t think Apple would have anything that can trump on this level with all those cash.

Concerning Google lens and much of what they announced in the keynote, You’d have a point if Google weren’t making all these services available to iOS users.

But they are, and all Apple needs to do is keep making great phones.

Hardware wise yes you’re right. OS wise well Apple have already said they’re holding back on bigger changes until next year… I think stability is the way to go though, the experience on iPhone X definitely needs refining (I’ve had more little bugs and glitches on this thing than ever before)

Apple isn’t trying to compete in this area. They are even marketing NOT doing this as a feature. Building this type of knowledge base requires largely unfettered access to users personal data (reading your emails, always knowing your location, seeing all of your photos, etc).
It’s a fun tech demo for now, but I don’t see it as being a killer-app any time soon.
If this ever becomes a true financial impact for Apple, they have an easy solution. Give third party apps more access to the OS core UI (default apps, assistant, etc). Google doesn’t care what OS you use, so long as they can reach you. They would jump at the opportunity to gain more access to Apple’s users. I know this is cliched, but it is true that for Google, Android isn’t the product, you are.

Thanks for the video. Can you please post it to YouTube though? That player just isn’t as nice a user experience.

it’s just horrible.

I can’t even play the video on mobile (I use Chrome and I’m running Android 8.1)

I had the Google Lens app on my iPhone about 5 years ago, and it did very similar things. I remember taking random photos of stuff and having Amazon links come up. I’ve been waiting for it to come back and now it looks much better.

Eh, I had it too. It could recognize types of objects, sometimes. But it would totally fail at exactly identifying what it was. Like a dog vs German Shepherd ( I actually don’t think it could even get the dog part right)

Thats because the technology 5yrs back had a error rate of 30% and now the error rate is 2%. so its becoming more accurate.
The way they were able to achieve such low error rates by using CNN by having a very very deep network with many hidden layers

This tells the importance of persistence. The old "Nokia city lens" hanged up gloves long before it reached its potential.

I hope they try to bring Google Glass back

The ironic thing about Google glass was that it never was this. People thought it was but in actuality it was just a display for stuff you saw similarly on Android wear devices. It never was augmented reality basically

View All Comments
Back to top ↑