Skip to main content

How do you fix two-thirds of the web in secret?

How do you fix two-thirds of the web in secret?

Share this story

When word of the Heartbleed bug first came out, news spread like a fire alarm — but it didn’t spread evenly. The vulnerability was spread across as many as two out of every three servers, which made a standard disclosure impossible. Some companies like Facebook got the news early, either from Google or OpenSSL itself, and were already patched when Monday’s news broke. Others, like Amazon and Yahoo, were left scrambling to protect themselves. But why did some companies have advanced warning while others got left in the cold? How did Facebook find out while Yahoo was left out of the loop?

"Antitrust laws do not stand in the way."

From a certain angle, it seems like picking favorites — so much so that the FTC issued a statement this morning "making it clear that antitrust laws do not stand in the way of legitimate sharing of cybersecurity threat information." But there’s a complicated etiquette for sharing this information within the industry, generally known as "responsible disclosure." The idea is to share bugs with service providers before the exploits become public knowledge, which means separating out the good guys from the bad guys. In a perfect world, you'd let all the good guys know before a single bad guy had a chance to attack. But like any secret, every new insider increased the risk that the news would leak. The worst case scenario was Heartbleed leaking out to a black-hat forum, where the news would spread to attackers first. At a certain point, researchers inevitably decide the risk of a leak is too great and they have no choice but to publish the leak in advance.

Every new insider increased the risk that the news would leak

With Heartbleed, that's about what happened. Early last week, a security engineer at the content-distribution network CloudFlare got an alarming message from a friend: send me your PGP encryption key as soon as you can. Only once a secure channel was established and a non-disclosure agreement was in place could he share the alarming news about Heartbleed. After that, the real work could begin.

Send me your PGP keys as soon as you can

Facebook got a similar message and patched its services before word leaked, but by Monday, the OpenSSL project was getting nervous. According to a Wall Street Journal report, it was a warning from the Finnish cybersecurity agency that forced the disclosure. No one knew who had told the the agency, so OpenSSL had no choice but to assume the secret was out, and issue a public security advisory, directly alerting all the affected clients. That left a lot of people out in the cold, including giants like Amazon and Yahoo, which were left to scramble for an update. Even some ancillary Google services were caught off-guard by the news, simply because the engineers couldn't risk spreading the secret to every corner of the company.

Could they have handled it better?

Could they have handled it better? Even now, many are skeptical. "This is a very hard case to generalize from," says CloudFlare CEO Matthew Prince. "If the bug was just with one company, it's easy to do responsible disclosure, but when it affects all those different services, it's really hard." Disclosures had to balance the potential harm against the possibility of a leak, reaching out only to trusted sources with immense precautions.

There's no roadmap for this kind of disclosure

And while Yahoo users surely felt the sting from being shut out of the early disclosures, many think Google and the OpenSSL project were right to prioritize content distribution networks like CloudFlare and Akamai. "CDNs in particular needed to get a jump on this early," says ICSI's Nicholas Weaver, "because someone could use this in an attack on the server, steal your server's private key, and you never knew they had it." Those private-key attacks are less visible than a Yahoo login, but they affect many more people. Shutting down those channels early may have prevented huge amounts of damage down the road.

In the end, there’s no roadmap for this kind of disclosure because there’s never been a problem this big before. Fixing the bug in secret means setting priorities, choosing which companies to loop in and which to keep out, and there’ll always be reason to second-guess those decisions. But as the web grows, it’s a problem the industry may need to confront again.