Skip to main content

Google search’s next phase: context is king

Google says search is still far from solved

Share this story

Illustration: Alex Castro / The Verge

At its Search On event today, Google introduced several new features that, taken together, are its strongest attempts yet to get people to do more than type a few words into a search box. By leveraging its new Multitask Unified Model (MUM) machine learning technology in small ways, the company hopes to kick off a virtuous cycle: it will provide more detail and context-rich answers, and in return it hopes users will ask more detailed and context-rich questions. The end result, the company hopes, will be a richer and deeper search experience.

Google SVP Prabhakar Raghavan oversees search alongside Assistant, ads, and other products. He likes to say — and repeated in an interview this past Sunday — that “search is not a solved problem.” That may be true, but the problems he and his team are trying to solve now have less to do with wrangling the web and more to do with adding context to what they find there. 

AI will help Google explore the questions people are asking

For its part, Google is going to begin flexing its ability to recognize constellations of related topics using machine learning and present them to you in an organized way. A coming redesign to Google search will begin showing “Things to know” boxes that send you off to different subtopics. When there’s a section of a video that’s relevant to the general topic — even when the video as a whole is not — it will send you there. Shopping results will begin to show inventory available in nearby stores, and even clothing in different styles associated with your search.

For your part, Google is offering — though perhaps “asking” is a better term — new ways to search that go beyond the text box. It’s making an aggressive push to get its image recognition software Google Lens into more places. It will be built into the Google app on iOS and also the Chrome web browser on desktops. And with MUM, Google is hoping to get users to do more than just identify flowers or landmarks, but instead use Lens directly to ask questions and shop.

“It’s a cycle that I think will keep escalating,” Raghavan says. “More technology leads to more user affordance, leads to better expressivity for the user, and will demand more of us, technically.” 

Google Lens will let users search using images and refine their query with text.
Google Lens will let users search using images and refine their query with text.
Image: Google

Those two sides of the search equation are meant to kick off the next stage of Google search, one where its machine learning algorithms become more prominent in the process by organizing and presenting information directly. In this, Google efforts will be helped hugely by recent advances in AI language processing. Thanks to systems known as large language models (MUM is one of these), machine learning has got much better at mapping the connections between words and topics. It’s these skills that the company is leveraging to make search not just more accurate, but more explorative and, it hopes, more helpful.

One of Google’s examples is instructive. You may not have the first idea what the parts of your bicycle are called, but if something is broken you’ll need to figure that out. Google Lens can visually identify the derailleur (the gear-changing part hanging near the rear wheel) and rather than just give you the discrete piece of information, it will allow you to ask questions about fixing that thing directly, taking you to the information (in this case, the excellent Berm Peak Youtube channel).

Multimodal search requires entirely new input from users

The push to get more users to open up Google Lens more often is fascinating on its own merits, but the bigger picture (so to speak) is about Google’s attempt to gather more context about your queries. More complicated, multimodal searches combining text and images demand “an entirely different level of contextualization that we the provider have to have, and so it helps us tremendously to have as much context as we can,” Raghavan says. 

We are very far from the so-called “ten blue links” of search results that Google provides. It has been showing information boxes, image results, and direct answers for a long time now. Today’s announcements are another step, one where the information Google provides is not just a ranking of relevant information but a distillation of what its machines understand by scraping the web. 

In some cases — as with shopping — that distillation means you’ll likely be sending Google more page views. As with Lens, that trend is important to keep an eye on: Google searches increasingly push you to Google’s own products. But there’s a bigger danger here, too. The fact that Google is telling you more things directly increases a burden it’s always had: to speak with less bias. 

By that, I mean bias in two different senses. The first is technical: the machine learning models that Google wants to use to improve search have well-documented problems with racial and gender biases. They’re trained by reading large swaths of the web, and, as a result, tend to pick up nasty ways of talking. Google’s troubles with its AI ethics team are also well documented at this point — it fired two lead researchers after they published a paper on this very subject. As Google’s VP of search, Pandu Nayak, told The Verge’s James Vincent in his article on today’s MUM announcements, Google knows that all language models have biases, but the company believes it can avoid “putting it out for people to consume directly.”

A new feature called “Things to know” will help users explore topics related to their searches.
A new feature called “Things to know” will help users explore topics related to their searches.
Image: Google

Be that as it may (and to be clear, it may not be), it sidesteps another consequential question and another type of bias. As Google begins telling you more of its own syntheses of information directly, what is the point of view from which it’s speaking? As journalists, we often talk about how the so-called “view from nowhere” is an inadequate way to present our reporting. What is Google’s point of view? This is an issue the company has confronted in the past, sometimes known as the “one true answer” problem. When Google tries to give people short, definitive answers using automated systems, it often ends up spreading bad information

Presented with that question, Raghavan responds by pointing to the complexity of modern language models. “Almost all language models, if you look at them, are embeddings in a high dimension space. There are certain parts of these spaces that tend to be more authoritative, certain portions that are less authoritative. We can mechanically assess those things pretty easily,” he explains. Raghavan says the challenge is then how to present some of that complexity to the user without overwhelming them. 

Can Google remain neutral if it’s delivering answers to users directly?

But I get the sense that the real answer is that, for now at least, Google is doing what it can to avoid facing the question of its search engine’s point of view by avoiding the domains where it could be accused of, as Raghavan puts it, “excessive editorializing.” Often when speaking to Google executives about these problems of bias and trust, they focus on easier-to-define parts of those high-dimension spaces like “authoritativeness.” 

For example, Google’s new “Things to know” boxes won’t appear when somebody searches for things Google has identified as “particularly harmful/sensitive,” though a spokesperson says that Google is not “allowing or disallowing specific curated categories, but our systems are able to scalably understand topics for which these types of features should or should not trigger.”

Google search, its inputs, outputs, algorithms, and language models have all become almost unimaginably complex. When Google tells us that it is able to understand the contents of videos now, we take for granted that it has the computing chops to pull that off — but the reality is that even just indexing such a massive corpus is a monumental task that dwarfs the original mission of indexing the early web. (Google is only indexing audio transcripts of a subset of YouTube, for the record, though with MUM it aims to do visual indexing and other video platforms in the future). 

Often when you’re speaking to computer scientists, the traveling salesman problem will come up. It’s a famous conundrum where you attempt to calculate the shortest possible route between a given number of cities, but it’s also a rich metaphor for thinking through how computers do their machinations. 

“If you gave me all the machines in the world, I could solve fairly big instances,” Raghavan says. But for search, he says that it is unsolved and perhaps unsolvable by just throwing more computers at it. Instead, Google needs to come up with new approaches, like MUM, that take better advantage of the resources Google can realistically create. “If you gave me all the machines there were, I’m still bounded by human curiosity and cognition.”

Google’s new ways of understanding information are impressive, but the challenge is what it will do with the information and how it will present it. The funny thing about the traveling salesman problem is that nobody seems to stop and ask what exactly is in the case, what is he showing all his customers as he goes door to door?

Today’s Storystream

Feed refreshed 20 minutes ago The tablet didn’t call that play by itself

James Vincent20 minutes ago
Today, 39 years ago, the world didn’t end.

And it’s thanks to one man: Stanislav Petrov, a USSR military officer who, on September 26th, 1983, took the decision not to launch a retaliatory nuclear attack against the US. Petrov correctly guessed that satellite readings showing inbound nukes were faulty, and so likely saved the world from nuclear war. As journalist Tom Chivers put it on Twitter, “Happy Stanislav Petrov Day to those who celebrate!” Read more about Petrov’s life here.

Soviet Colonel who prevented 1983 nuclear response
Photo by Scott Peterson/Getty Images
The Verge
James Vincent35 minutes ago
Deepfakes were made for Disney.

You might have seen the news this weekend that the voice of James Earl Jones is being cloned using AI so his performance as Darth Vader in Star Wars can live on forever.

Reading the story, it struck me how perfect deepfakes are for Disney — a company that profits from original characters, fans' nostalgia, and an uncanny ability to twist copyright law to its liking. And now, with deepfakes, Disney’s most iconic performances will live on forever, ensuring the magic never dies.

External Link
Elizabeth Lopatto56 minutes ago
Hurricane Fiona ratcheted up tensions about crypto bros in Puerto Rico.

“An official emergency has been declared, which means in the tax program, your physical presence time is suspended,” a crypto investor posted on TikTok. “So I am headed out of the island.” Perhaps predictably, locals are furious.

The Verge
Richard LawlerAn hour ago
Teen hacking suspect linked to GTA 6 leak and Uber security breach charged in London.

City of London police tweeted Saturday that the teenager arrested on suspicion of hacking has been charged with “two counts of breach of bail conditions and two counts of computer misuse.”

They haven’t confirmed any connection with the GTA 6 leak or Uber hack, but the details line up with those incidents, as well as a suspect arrested this spring for the Lapsus$ breaches.

Welcome to the new Verge

Revolutionizing the media with blog posts

Nilay PatelSep 13
The Verge
Richard LawlerTwo hours ago
Green light.

Good morning to everyone, except for the intern or whoever prevented us from seeing how Microsoft’s Surface held up to yet another violent NFL incident.

Today’s big event is the crash of a NASA spaceship this evening — on purpose. Mary Beth Griggs can explain.

David PierceTwo hours ago
Thousands and thousands of reasons people love Android.

“Android fans, what are the primary reasons why you will never ever switch to an iPhone?” That question led to almost 30,000 comments so far, and was for a while the most popular thing on Reddit. It’s a totally fascinating peek into the platform wars, and I’ve spent way too much time reading through it. I also laughed hard at “I can turn my text bubbles to any color I like.”

Thomas Ricker7:29 AM UTC
Table breaks before Apple Watch Ultra’s sapphire glass.

”It’s the most rugged and capable Apple Watch yet,” said Apple at the launch of the Apple Watch Ultra (read The Verge review here). YouTuber TechRax put that claim to the test with a series of drop, scratch, and hammer tests. Takeaways: the titanium case will scratch with enough abuse, and that flat sapphire front crystal is tough — tougher than the table which cracks before the Ultra fails — but not indestructible.

Emma RothSep 25
Rihanna’s headlining the Super Bowl Halftime Show.

Apple Music’s set to sponsor the Halftime Show next February, and it’s starting out strong with a performance from Rihanna. I honestly can’t remember which company sponsored the Halftime Show before Pepsi, so it’ll be nice to see how Apple handles the show for Super Bowl LVII.

Emma RothSep 25
Starlink is growing.

The Elon Musk-owned satellite internet service, which covers all seven continents including Antarctica, has now made over 1 million user terminals. Musk has big plans for the service, which he hopes to expand to cruise ships, planes, and even school buses.

Musk recently said he’ll sidestep sanctions to activate the service in Iran, where the government put restrictions on communications due to mass protests. He followed through on his promise to bring Starlink to Ukraine at the start of Russia’s invasion, so we’ll have to wait and see if he manages to bring the service to Iran as well.

External Link
Emma RothSep 25
We might not get another Apple event this year.

While Apple was initially expected to hold an event to launch its rumored M2-equipped Macs and iPads in October, Bloomberg’s Mark Gurman predicts Apple will announce its new devices in a series of press releases, website updates, and media briefings instead.

I know that it probably takes a lot of work to put these polished events together, but if Apple does pass on it this year, I will kind of miss vibing to the livestream’s music and seeing all the new products get presented.

External Link
Emma RothSep 24
California Governor Gavin Newsom vetoes the state’s “BitLicense” law.

The bill, called the Digital Financial Assets Law, would establish a regulatory framework for companies that transact with cryptocurrency in the state, similar to New York’s BitLicense system. In a statement, Newsom says it’s “premature to lock a licensing structure” and that implementing such a program is a “costly undertaking:”

A more flexible approach is needed to ensure regulatory oversight can keep up with rapidly evolving technology and use cases, and is tailored with the proper tools to address trends and mitigate consumer harm.