Skip to main content

Google now lets you search for things you can’t describe — by starting with a picture

Google now lets you search for things you can’t describe — by starting with a picture

/

Starting with a US beta

Share this story

You like the way that dress looks but you’d rather have it in green. You want those shoes but prefer flats to heels. What if you could have drapes with the same pattern as your favorite notebook? I don’t know how to Google for these things, but Google Search product manager Belinda Zeng showed me real-world examples of each earlier this week, and the answer was always the same: take a picture, then type a single word into Google Lens.

Today, Google is launching a US-only beta of the Google Lens Multisearch feature it teased last September at its Search On event, and while I’ve only seen a rough demo so far, you shouldn’t have to wait long to try it for yourself: it’s rolling out in the Google app on iOS and Android.

Take a screenshot or picture of a dress, then tap, type “green,” and search for a similar one in a different color.
Take a screenshot or picture of a dress, then tap, type “green,” and search for a similar one in a different color.
GIF: Google

While it’s mostly aimed at shopping to start — it was one of the most common requests — Google’s Zeng and the company’s search director Lou Wang suggest it could do a lot more than that. “You could imagine you have something broken in front of you, don’t have the words to describe it, but you want to fix it... you can just type ‘how to fix,’” says Wang.

In fact, it might already work with some broken bicycles, Zeng adds. She says she also learned about styling nails by screenshotting pictures of beautiful nails on Instagram, then typing the keyword “tutorial” to get the kind of video results that weren’t automatically coming up on social media. You may also be able to take a picture of, say, a rosemary plant, and get instructions on how to care for it.

Google’s Belinda Zeng showed me a live demo where she found drapes to match a leafy notebook.
Google’s Belinda Zeng showed me a live demo where she found drapes to match a leafy notebook.
GIF by Sean Hollister / The Verge

“We want to help people understand questions naturally,” says Wang, explaining how multisearch will expand to more videos, images in general, and even the kinds of answers you might find in a traditional Google text search.

It sounds like the intent is to put everyone on even footing, too: rather than partnering with specific shops or even limiting video results to Google-owned YouTube, Wang says it’ll surface results from “any platform we’re able to index from the open web.”

When Zeng took a picture of the wall behind her, Google came up with ties that had a similar pattern.
When Zeng took a picture of the wall behind her, Google came up with ties that had a similar pattern.
Screenshot by Sean Hollister / The Verge

But it won’t work with everything — like your voice assistant doesn’t work with everything — because there are infinite possible requests and Google’s still figuring out intent. Should the system pay more attention to the picture or your text search if they seem to contradict? Good question. For now, you do have one additional bit of control: if you’d rather match a pattern, like the leafy notebook, get up close to it so that Lens can’t see it’s a notebook. Because remember, Google Lens is trying to recognize your image: if it thinks you want more notebooks, you might have to tell it that you actually don’t.

Google is hoping AI models can drive a new era of search, and there are big open questions about whether context — and not just text — can take it there. This experiment seems limited enough (it doesn’t even use its latest MUM AI models) that it probably won’t give us the answer. But it does seem like a neat trick that could go fascinating places if it became a core Google Search feature.

Today’s Storystream

Feed refreshed Sep 24 Striking out

E
External Link
Emma RothSep 24
California Governor Gavin Newsom vetoes the state’s “BitLicense” law.

The bill, called the Digital Financial Assets Law, would establish a regulatory framework for companies that transact with cryptocurrency in the state, similar to New York’s BitLicense system. In a statement, Newsom says it’s “premature to lock a licensing structure” and that implementing such a program is a “costly undertaking:”

A more flexible approach is needed to ensure regulatory oversight can keep up with rapidly evolving technology and use cases, and is tailored with the proper tools to address trends and mitigate consumer harm.


A
Youtube
Andrew WebsterSep 24
Look at this Thing.

At its Tudum event today, Netflix showed off a new clip from the Tim Burton series Wednesday, which focused on a very important character: the sentient hand known as Thing. The full series starts streaming on November 23rd.


A
The Verge
Andrew WebsterSep 24
Get ready for some Netflix news.

At 1PM ET today Netflix is streaming its second annual Tudum event, where you can expect to hear news about and see trailers from its biggest franchises, including The Witcher and Bridgerton. I’ll be covering the event live alongside my colleague Charles Pulliam-Moore, and you can also watch along at the link below. There will be lots of expected names during the stream, but I have my fingers crossed for a new season of Hemlock Grove.


A
Andrew WebsterSep 24
Looking for something to do this weekend?

Why not hang out on the couch playing video games and watching TV. It’s a good time for it, with intriguing recent releases like Return to Monkey Island, Session: Skate Sim, and the Star Wars spinoff Andor. Or you could check out some of the new anime on Netflix, including Thermae Romae Novae (pictured below), which is my personal favorite time-traveling story about bathing.


A screenshot from the Netflix anime Thermae Romae Novae.
Thermae Romae Novae.
Image: Netflix
J
Twitter
Jay PetersSep 23
Twitch’s creators SVP is leaving the company.

Constance Knight, Twitch’s senior vice president of global creators, is leaving for a new opportunity, according to Bloomberg’s Cecilia D’Anastasio. Knight shared her departure with staff on the same day Twitch announced impending cuts to how much its biggest streamers will earn from subscriptions.


T
Twitter
Tom WarrenSep 23
Has the Windows 11 2022 Update made your gaming PC stutter?

Nvidia GPU owners have been complaining of stuttering and poor frame rates with the latest Windows 11 update, but thankfully there’s a fix. Nvidia has identified an issue with its GeForce Experience overlay and the Windows 11 2022 Update (22H2). A fix is available in beta from Nvidia’s website.


A
External Link
If you’re using crash detection on the iPhone 14, invest in a really good phone mount.

Motorcycle owner Douglas Sonders has a cautionary tale in Jalopnik today about the iPhone 14’s new crash detection feature. He was riding his LiveWire One motorcycle down the West Side Highway at about 60 mph when he hit a bump, causing his iPhone 14 Pro Max to fly off its handlebar mount. Soon after, his girlfriend and parents received text messages that he had been in a horrible accident, causing several hours of panic. The phone even called the police, all because it fell off the handlebars. All thanks to crash detection.

Riding a motorcycle is very dangerous, and the last thing anyone needs is to think their loved one was in a horrible crash when they weren’t. This is obviously an edge case, but it makes me wonder what other sort of false positives we see as more phones adopt this technology.


A
External Link
Ford is running out of its own Blue Oval badges.

Running out of semiconductors is one thing, but running out of your own iconic nameplates is just downright brutal. The Wall Street Journal reports badge and nameplate shortages are impacting the automaker's popular F-series pickup lineup, delaying deliveries and causing general chaos.

Some executives are even proposing a 3D printing workaround, but they didn’t feel like the substitutes would clear the bar. All in all, it's been a dreadful summer of supply chain setbacks for Ford, leading the company to reorganize its org chart to bring some sort of relief.