Locked out and totally down: Facebook’s scramble to fix a massive outage

Illustration by Alex Castro / The Verge

A prolonged, global outage of Facebook’s apps sent the company’s engineers scrambling to fix the issue at one of its data centers in California, according to two people familiar with the situation.

The outage, which began around 11:40AM ET on Monday, brought down all of Facebook’s apps — including Instagram and WhatsApp — globally, affecting billions of users and millions of advertisers. Inside Facebook, the outage also broke nearly all of the internal systems that employees use to communicate and work. As of 6PM ET, it appears that most of the services are back online.

Several employees told The Verge they resorted to talking through their work-provided Outlook email accounts since Facebook mainly runs on an internal version of the social network that is currently not accessible. While employees could email each other, they couldn’t send or receive emails from external addresses.

Since Facebook requires employees to log in with their work accounts to access tools such as Google Docs and Zoom, those services also weren’t working, leading some employees to use alternative services like Apple’s FaceTime and Discord. Employees who were already authenticated with non-Facebook tools like Google Docs before the outage began still had access.

Facebook engineers were sent to one of its main US data centers in California to restore service, meaning the fix couldn’t be done remotely. Further complicating matters, the outage temporarily broke the ability for some employees to access company buildings and conference rooms with their badges, according to The New York Times, which first reported that engineers had been dispatched to the data center.

In an email to employees sent shortly after service was restored, CTO Mike Schroepfer said the issue was “affecting our networking backbone that connects all our data centers together.”

“If you are not actively working on the recovery, please be patient and don’t rush to reload everything to prevent slowing down the bring up” of the network, he cautioned in the memo that was seen by The Verge.

Facebook hasn’t provided a detailed explanation of the outage, though outside experts are saying it was due to an issue with the networking technology BGP, or Border Gateway Protocol.

Late Monday, Facebook’s VP of Infrastructure, Santosh Janardhan, published a corporate blog post saying the outage was the result of a “faulty configuration change,” adding that the company has “no evidence that user data was compromised as a result of this downtime.”

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Janardhan wrote. “This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”

Update October 4th, 6:33PM ET: Noted that the outage is ending as Facebook and its other services are coming back online.

Update October 4th, 8:05PM ET: Added more information about the outage that was shared with Facebook employees.

Update October 4th, 9:06PM ET: Added the confirmed report that an angle grinder was used to access the server cages.

Correction October 4th, 9:25PM ET: A previous version of this story included a confirmed report that Facebook used an angle grinder to access the server cages. The reporter has since withdrawn confirmation and we have removed the claim from this story.

Update October 4th, 10:29PM ET: Added more details from Facebook about the outage.

Comments

Facebook-gloating aside, it would actually be quite interesting to know how a huge website manages to blow up like this in this day and age.

Yep. And there’s going to be major fall out.

it’s the major ad buyers- depending on 24/7 exposure they paid for- that are going to drive massive concessions and penalties

Ooh, yeah I’d actually love an in depth piece or video breaking down how this happens.

This isn’t everything, but it’s still a good read. From Cloudflare: Understanding How Facebook Disappeared from the Internet

Netflix doc series coming soon

I’d love for this to be the basis of "The Social Network Pt.2: Global Outage"

In theory you’d think huge companies should mitigate all these issues, but in reality even day to day things like certificate renewals are missed and can take down major services. Sometimes the tools there to help are also your own worst enemy, such as replication of bad config which seems to have happened here.

My biggest fear about all of this playing out is that for people like us, we will be like LOL Facebook, but like the general public saying "Yeah we need Facebook" and we get close to the WeChatification of the internet.

Agreed! Maybe Biden or Warren can wake the f up and start regulating something?
Another big fear I have is that nobody is talking about the whistle-blower, ex Facebook product manager, Frances Haugen anymore. I mean, is this project Amplify at work? Control the narrative at all costs, even if we have to make up positive stories on the newsfeed or pull the plug?

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀

Holy shit this is… something !
I’m personnally not deeply affected (I just had to text my non-tech-savy parents to explain why the family WhatsApp has gone silent) but a 6h downtime for such a massive network is properly insane.
Entire businesses rely on Facebook and Instagram. WhatsApp is absolutely massive in some countries. I don’t understand how this is even possible, and I certainly wouldn’t want to be working at Facebook right now.

If entire business relies on Facebook than this is the execs fault. You have to be idiot to put all your money info FB

Why ? That’s like saying a developper is an idiot for selling their apps on the App Store
Facebook/Instagram are ecosystems now, whether we like it or not

Any computer science majors around here? Can this affect privacy? Is this just a "malfunction" or our data can be compromised because of this? asking out of ignorance but I just have a bad feeling and can see hackers rubbing their hands…. hope I’m just being paranoid…

If it’s just that their networks are down, nothing to worry about – as much as you can’t get on, neither can anyone else.

Hackers already obtained 1.5 billion Facebook account details within the past few days; name, email, phone number, gender, etc.

I trust random hackers more than I trust Facebook. Whatever you gave to FB has already been shared with countless third parties.

Facebook privacy lol

This is a full on routing issue, not a security or privacy issue. The data FB has on you is still on their servers, apparently due to poor network design it isn’t even accessible by FB staff internally. The issue is that your browser or app isn’t finding FB servers when they ask for them, because the routing (BGP info) was erased all over the world at the same time, likely due to a mis-configured update. BGP is designed to erase routes when a bad update comes across and trickles down to all the internet routers around the world. If you can’t get to the servers, it’s likely bad actors can’t either. Reloading the routing info on a global scale takes hours to a day. When the proper BGP info is sent out, services will gradually come back up by geographic location blocks..

Got it. thanks.

Who runs their entire company and all communications, even key cards for rooms , on the back end of their website? Surely there can’t be many companies that stupid ?

It’s very likely that they aren’t on "the backend of their website" as you put it..
The info that was deleted in the BGP routing wasn’t likely just public domain routes, but also non-published domains that FB uses for internal stuff.. This issue likely took out every domain FB has control over – not just the public ones. I have a fairly good feeling someone is going to very quickly segregate the internal and external domains into separate lists for the next update.

View All Comments
Back to top ↑