Google’s silent Chrome experiment crashes thousands of browsers and angers IT admins

Illustration by Alex Castro / The Verge

Google left thousands of machines in businesses with broken Chrome browsers this week, following a silent experimental change. Business users accessing Chrome through virtual machine environments like Citrix kept seeing white screens on open Chrome tabs, blocking access to the browser and leaving it totally unresponsive. It left many IT admins confused over the problem, as businesses typically manage and control Chrome updates.

After complaints, Google was forced to reveal it had launched an “experiment” on stable versions of Chrome that had changed the browser’s behavior. The experiment was made silently, without IT admins or users being warned about Google’s changes. Google had simply flipped the switch on a flag to enable a new WebContents Occlusion feature that’s designed to suspend Chrome tabs when you move other apps on top of them and reduce resource usage when the browser isn’t in use.

“The experiment / flag has been on in beta for ~5 months,” explained David Bienvenu, a software engineer at Google, in a Chromium bug thread. “It was turned on for stable (e.g., m77, m78) via an experiment that was pushed to released Chrome Tuesday morning. Prior to that, it had been on for about one percent of M77 and M78 users for a month with no reports of issues, unfortunately.”

Google rolled back the change late on Thursday night, following multiple reports from businesses with thousands of users affected. “I’ll rollback the launch of this experiment and try to figure out how to deal with Citrix,” noted Bienvenu in the bug thread.

“This has had a huge impact for all our Call Center agents and not being able to chat with our members,” explained a Costco IT admin in the Chromium thread. “We spent the last day and a half trying to figure this out.”

One IT admin that alerted The Verge to the issue said “we felt that this is a shady thing that Google can update Chrome silently without announcing anything and can impact 100,000+ people on a whim.” Those concerns are mirrored by hundreds of replies on Google’s support forum, the bug tracker thread, and on Twitter and Reddit.

It has left IT admins angry that they’ve wasted valuable resources and time on trying to fix issues in their environment, and questions over why Google decided to make a silent change to Chrome in the first place. “I am stunned by your response,” said one IT admin in response to Bienvenu’s confirmation on the issues. “Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects. We are running professional services for multi million dollar programs.”

Google confirmed the testing and rollout of this feature in a statement to The Verge, and apologized for the issues:

“After the rollout, we received reports that in some virtual environments, Chrome on Windows displays a blank page, which may be because Chrome mistakenly believes it’s covered by another window. As soon as we confirmed the reports, the feature was disabled.

If Chrome on Windows is displaying blank pages, restart Chrome. On the next start, this feature will be disabled.

We also want to provide an explanation of how this change was rolled out. For some features, Chrome uses a gradual rollout process that happens more slowly than the main rollout. This allows us to quickly revert a change if we discover a bug that wasn’t uncovered in prior testing.

Once we received reports of the problem, we were able to revert it immediately. We sincerely apologize for the disruption this caused.”

Update, November 15th 4PM ET: Article updated with a statement from Google.

Comments

And if I’m Microsoft I’m approaching every affected IT department and selling them Chromium Edge right now…

Chrome might have been better than IE, but Edge is pretty good now. I only use Chrome now for Google’s services.

but isn’t that simply because Edge is now Chromium based?

Firefox ftw.

Microsoft is sneakily playing a really clever game. Edgium is so good and so well integrated, it doesn’t matter if you have default settings, every HTML attachment in the email opens in Edgium.

I’m really confused by this comment:

Firefox [for the win].

How/why is this evidence of Firefox winning?

Then it’s followed by a statement that Microsoft’s new Chromium based Edge works really well and for some reason alwaysonline finds that to be sneaky behavior on Microsoft’s part.

Microsoft is sneakily (?) playing a really clever game. Edgium is so good and so well integrated, it doesn’t matter if you have default settings, every HTML attachment in the email opens in Edgium.

The fact that a majority of companies and web developers build and test on Chrome and for Chrome has major risks.

The advantage is less time testing and building for other browsers but when something like this happens, and they don’t have a fallback plan, everything goes south. It is either everything or nothing.

Also, Firefox is the only other REAL browser out there. IE is soon to be sunset and the new Edge will be using Chromium minus the Google stuff, but basically Chromium.

Any issues in Chromium means every browser except Firefox will fail. That’s the reason why Firefox should be considered. For the win doesn’t mean winning.

Install Edge beta and see why it is sneaky. It is sneaky for the below reasons:
1. Imports Chrome data without even asking
2. Sets itself as default application for opening email attachments
3. Sets itself as the default browser when the user clicks on the Windows lock screen and the link opens.

To me, if I set a browser as default, everything that opens a web page should be using the default browser, unless you right click on a HTML document and specifically set it to open in a different browser.

MSFT hasn’t changed their playbook. Integration with their OS to favor their apps. Sure, in most cases, the user can change the behavior but the extra friction is enough to deter many users from doing so.

Not sure how that would help… It’s a 5 month old feature. That’s the reason I run Chrome Beta. Are you suggesting Microsoft not send updates and turn Chrome into the new Internet Explorer 6?

IT admins should take accountability. You should be reviewing beta versions for potentially disruptive updates… do your job

So many large enterprises are 10+ years behind in a lot of technology. If it’s not broke, don’t fix it. And when their employees call about something being broke, it’s user error and escalation of the ticket means ticket is escalated to the trash.

"Everything is a beta, Nothing is ever committed"
source: Google

Even their roll out method sucks. I paid $1000 for a new unlocked Pixel 4 and have been hitting the check for update button daily for a week but can’t see the November update yet while it’s been out for a week. Guessing by this article, only 1% of pixel 4s will have the update and if they miss a bug there then it should nicely impact the 99% on day 1. At least increase the effing roll out percentage and let people update if they’re manually checking for an Ota update. I don’t wanna plug USB cables and sideload updates.

There was most likely an issue with the update that had to be corrected, considering basically no Pixel 4’s got the update (Just carrier models which received a different build than unlocked models).

The check for update button has been working for months and worked for all other Pixel variants for this rollout.

Of course they’re committed. They git commit all the time.

Google is kind of annoying with the uncontrollable (for the users) server-side switches. When the Play Store redesign started rolling out, it changed back and forth on my phone for weeks, sometimes even several times within the same day.

This incident is exactly why you need choices. If only the companies had ensured that they had supported multiple browsers, this could have been avoided.

Company:
Let’s save dollars and support only Chrome.

Chrome:
I am going for vacay.

Company:
Shiiiii. Saved dollars lost!

it would be fine if they did this in their beta or experimental versions. They should NEVER do this on the stable release, unless it’s in an update and documented.

Honestly, I think it’s about time to do away with indemnification clauses in clickthrough EULAs. When a company can cost businesses millions of dollars by haphazardly flipping a proverbial switch, they need to be able to be held liable for the damages they cause their users. This whole all profit, no liability business model needs to be nipped in the bud, even if it is a decade or two late.

For the free browser that you totally aren’t forced to use in any way?

It’s not like these businesses are paying for Chrome Enterprise or something…

Google is distributing this software and making money off of it. If they don’t want to be responsible, they can stop distributing it, and nothing bad will happen to users, it’s open source anyway.

I’m pretty sure that somewhere in their terms it’s clearly stated that they’re not responsible in cases like this one.. No one is forcing those IT experts to use it – there are plenty of browsers out there.

Did you…did you even read the original comment?

Honestly, I think it’s about time to do away with indemnification clauses in clickthrough EULAs.

It’s ok because once Microsoft launches the Chrome version of edge we will do away with Google Chrome!

Just because it’s free doesn’t mean Google is free of any responsibility. Can a free TV station advertised as a kids channel suddenly broadcast porn during school hours?!

That stupid argument about free of responsibility because the product is free is non-sense, especially in 2019. Besides, free is not really free because all these "free" services and products are actually part of an ecosystem that makes money off of user time and user data.

For this stunt, some government agency needs to step in and lay down some hefty fines, significant punitive fines too. Commerce of this country and world depend on fair play. Company’s should be thinking twice before trying these kinds of experiments without clear and adequate notice and requiring 100% agreement from the user base. I think this is worthy of $1B fine to set an example.

View All Comments
Back to top ↑