Skip to main content

    Google personalizes search results even when you’re logged out, new study claims

    Google personalizes search results even when you’re logged out, new study claims

    /

    A study, albeit from competitor DuckDuckGo, finds that Google search results can vary significantly

    Share this story

    Illustration: Alex Castro / The Verge

    The amount of personalization inherent in any one of Google’s many massive software products runs deep, based on everything from your search history to your location to every single search link you might have clicked. And avoiding that personalization seems to have become more difficult over the years. According to a new study conducted by Google competitor DuckDuckGo, it does not seem possible to avoid personalization when using Google search, even by logging out of your Google account and using the private browsing “incognito” mode.

    DuckDuckGo conducted the study in June of this year, at the height of the US midterm election season. It did so with the ostensible goal of confirming whether Google’s search results exacerbate ideological bubbles by feeding you only information you’ve signaled you want to consume via past behavior and the data collected about you.

    The study found that Google search results varied significantly, even in private mode

    It’s not clear whether that question can be reliably answered with these findings, and it’s also obvious DuckDuckGo is a biased source with something to gain by pointing out how flawed Google’s approach may be. But the study’s findings are nonetheless interesting because they highlight just how much variance there are in Google search results, even when controlling for factors like location.

    DuckDuckGo found that a majority of participants in its study saw different results when searching three divisive terms: “gun control,” “immigration,” and “vaccinations.” According to the company, “these discrepancies could not be explained by changes in location, time, by being logged in to Google, or by Google testing algorithm changes to a small subset of users.” DuckDuckGo says it controlled for location by treating local results that varied across regions as though they were identical. “Interestingly, this adjustment didn’t affect overall variation significantly,” the study reads.

    Image: DuckDuckGo

    For the study, DuckDuckGo compiled 87 result sets (76 on desktop and 11 on mobile), and it conducted the searches consecutively and simultaneously starting at 9PM ET on June 24th, 2018. It did one private, logged-out test and then a logged-in test immediately after, so as not to influence the private test with prior results. What DuckDuckGo found was that using private browsing and logging out of Google had almost no effect on the variation in search results: users saw a roughly equitable amount of variation across all three searches and when searching privately and while logged in.

    Some key elements to the variation included changes in news sources and the placement of sometimes identical links in different positions, which has a drastic impact on the likelihood that they get clicked. The study also found variations in how news articles and videos were laid out among standard text links, and as many as 22 different domains showing up in the first page of results for “vaccinations,” with a standard search result page typically containing 10 organic links.

    Image: DuckDuckGo

    Perhaps most importantly, there doesn’t seem to be any way to get a single, objective search result from Google that can be easily replicable across users or locations. To further drive home the point that private and logged-out searches are still equally variable, DuckDuckGo calculated the difference between text results for private, logged-out users and compared them to both the results of other anonymous users and the standard, logged-in search results for those initial users.

    “We saw that when randomly comparing people’s private modes to each other, there was more than double the variation than when comparing someone’s private mode to their normal mode,” the study reads. The data indicates that between standard, logged-in search results and private, logged-out ones, there was typically a difference of only one or two domains, while the variation between anonymous users was from three (for gun control) to five (for vaccinations) domain changes. That indicates that there is some heavy personalization going on for people using private browsing when logged out, and it’s not clear why that is or what effect it’s having on user behavior.

    Image: DuckDuckGo

    Ultimately, the study seems to prove that there’s no easy way to use Google search without the software seemingly trying to determine who you are and whether it can better serve your needs. That’s not necessarily a bad thing; technology companies all over the world built businesses around being smarter, faster, better automated, and more personal. (And by giving away really useful, ad-supported products for free.)

    A majority of internet users wouldn’t use Google search if it wasn’t constantly a best-in-class search engine, with a number of tangential benefits, like the Chrome browser tie-ins, that make it a lot easier and more convenient than, say, DuckDuckGo. But it’s still unsettling to know that there’s so much unseen, algorithmic tinkering going on when you perform a basic action, like searching one or two-word terms on Google. And that there’s really nothing we can do to get a neutral result.

    Google says incognito mode doesn’t base results on signed-in search histories

    Google could not comment on a study it did not have access to, but the company says that search results can change by the minute and sometimes even by the second, especially for news topics. Google also says that personalization is done on only a small fraction of the total number of queries entered into search, and relying on recent queries is often to determine the context for a search, like when a word may apply to a sports team and a city simultaneously. The company did confirm that it does not personalize results for incognito searches using signed-in search history, and it also confirmed that it does not personalize results for the Top Stories row or the News tab in search.

    Following the study’s publication this morning, Google told The Verge in a statement that it found the methodology flawed and the findings misleading. “This study’s methodology and conclusions are flawed since they are based on the assumption that any difference in search results are based on personalization. That is simply not true,” a Google spokesperson said. “In fact, there are a number of factors that can lead to slight differences, including time and location, which this study doesn’t appear to have controlled for effectively.”

    In a separate series of statements, Danny Sullivan, who ran the influential Search Engine Land blog and now works as Google’s public liaison for search, echoed similar sentiments on Twitter about the nature of personalization in Google Search:

    It’s important to note that any kind of personalization, even based on just location, isn’t a new feature or something Google has been doing behind the scenes without anyone’s knowledge. The company began personalizing search results for every user, even those without a Google account, way back in 2009 using an anonymous cookie that would take into account information like your location, language, search history, and other factors.

    More recently, the company says it’s moved away from personalizing search results, even as it increasingly uses personalization for products like Google Assistant and Gmail because it didn’t seem to really improve the search experience. “A query a user comes with usually has so much context that the opportunity for personalization is just very limited,” Pandu Nayak, the head of ranking at Google search, told CNBC in September. Google says it still personalizes for location and “immediate context from a prior search.”

    Regardless, Google has come under fire in the last couple of years for engaging in practices that, while not exactly clandestine, were certainly not widely known and likely kept out of the limelight to avoid too much public scrutiny.

    One notable example, surfaced by an investigation from The Wall Street Journal, was how app developers were often able to read your personal Gmail messages, which prompted concern from some users and a congressional inquiry. Google also felt the need to stop scanning Gmail messages to target advertisements last year because it was worried that the practice, when it came to light in contract negotiations, might scare away enterprise companies.

    Google has come under fire for not being transparent about how its products work

    Google also failed to disclose a flaw in a Google+ API that could have exposed hundreds of thousands of users’ personal information for roughly six months, an action that might result in a Federal Trade Commission investigation. And in September, Google was forced to reverse a controversial Chrome login feature that would help it better target ads by automatically logging users into the browser without their express permission.

    The common thread here is that Google’s products, while often free to use and usually quite well-designed, only make money by using a vast trove of personal information to sell targeted online advertisements. And to improve how much money it makes from each ad sale, and the total quantity of ads it sells, Google often resorts to tactics its users have little insight into or deep understanding about.

    Personalized search results aren’t necessarily nefarious on their own. But the ways Google’s algorithms function — especially those that are now aided by complex and often unexplainable artificial intelligence software — are and always will be outside the understanding of the common user. By definition, that means most people have no idea how they work, including lawmakers and members of privacy watchdogs and advocacy groups. That could be a problem for a company like Google as it tries to maneuver a consumer market and a legislative environment that’s not quite as forgiving or negligent as it used to be.

    Update 12/5, 2:05PM ET: Added statement from Google calling the study’s methodology flawed. The headline has been updated to reflect that Google’s personalization of search results is a claim being made by its competitor DuckDuckGo.

    Update 12/5, 3:34PM ET: Included public statements on the nature of personalization in Google search from Danny Sullivan, the company’s liaison for search, published on Twitter.

    Today’s Storystream

    Feed refreshed Two hours ago Striking out

    A
    Youtube
    Andrew WebsterTwo hours ago
    Look at this Thing.

    At its Tudum event today, Netflix showed off a new clip from the Tim Burton series Wednesday, which focused on a very important character: the sentient hand known as Thing. The full series starts streaming on November 23rd.


    A
    The Verge
    Andrew Webster4:28 PM UTC
    Get ready for some Netflix news.

    At 1PM ET today Netflix is streaming its second annual Tudum event, where you can expect to hear news about and see trailers from its biggest franchises, including The Witcher and Bridgerton. I’ll be covering the event live alongside my colleague Charles Pulliam-Moore, and you can also watch along at the link below. There will be lots of expected names during the stream, but I have my fingers crossed for a new season of Hemlock Grove.


    J
    Twitter
    Jay PetersSep 23
    Twitch’s creators SVP is leaving the company.

    Constance Knight, Twitch’s senior vice president of global creators, is leaving for a new opportunity, according to Bloomberg’s Cecilia D’Anastasio. Knight shared her departure with staff on the same day Twitch announced impending cuts to how much its biggest streamers will earn from subscriptions.


    T
    Twitter
    Tom WarrenSep 23
    Has the Windows 11 2022 Update made your gaming PC stutter?

    Nvidia GPU owners have been complaining of stuttering and poor frame rates with the latest Windows 11 update, but thankfully there’s a fix. Nvidia has identified an issue with its GeForce Experience overlay and the Windows 11 2022 Update (22H2). A fix is available in beta from Nvidia’s website.


    A
    External Link
    If you’re using crash detection on the iPhone 14, invest in a really good phone mount.

    Motorcycle owner Douglas Sonders has a cautionary tale in Jalopnik today about the iPhone 14’s new crash detection feature. He was riding his LiveWire One motorcycle down the West Side Highway at about 60 mph when he hit a bump, causing his iPhone 14 Pro Max to fly off its handlebar mount. Soon after, his girlfriend and parents received text messages that he had been in a horrible accident, causing several hours of panic. The phone even called the police, all because it fell off the handlebars. All thanks to crash detection.

    Riding a motorcycle is very dangerous, and the last thing anyone needs is to think their loved one was in a horrible crash when they weren’t. This is obviously an edge case, but it makes me wonder what other sort of false positives we see as more phones adopt this technology.


    A
    External Link
    Ford is running out of its own Blue Oval badges.

    Running out of semiconductors is one thing, but running out of your own iconic nameplates is just downright brutal. The Wall Street Journal reports badge and nameplate shortages are impacting the automaker's popular F-series pickup lineup, delaying deliveries and causing general chaos.

    Some executives are even proposing a 3D printing workaround, but they didn’t feel like the substitutes would clear the bar. All in all, it's been a dreadful summer of supply chain setbacks for Ford, leading the company to reorganize its org chart to bring some sort of relief.


    E
    TikTok
    Spain’s Transports Urbans de Sabadell has La Bussí.

    Once again, the US has fallen behind in transportation — call it the Bussí gap. A hole in our infrastructure, if you will.


    J
    External Link
    Jay PetersSep 23
    Doing more with less (extravagant holiday parties).

    Sundar Pichai addressed employees’ questions about Google’s spending changes at an all-hands this week, according to CNBC.

    “Maybe you were planning on hiring six more people but maybe you are going to have to do with four and how are you going to make that happen?” Pichai sent a memo to workers in July about a hiring slowdown.

    In the all-hands, Google’s head of finance also asked staff to try not to go “over the top” for holiday parties.


    E
    External Link
    Insiders made the most money off of Helium’s “People’s Network.”

    Remember Helium, which was touted by The New York Times in an article entitled “Maybe There’s a Use for Crypto After All?” Not only was the company misleading people about who used it — Salesforce and Lime weren’t using it, despite what Helium said on its site — Helium disproportionately enriched insiders, Forbes reports.