When Russian antivirus company Dr.Web discovered a botnet running on over 600,000 Mac OS X computers, it sparked attention even among those not normally interested in computer security. The scope of the infection, along with criticism of Apple’s response, offered another example of a persistent problem. Because they’re profitable, relatively easy to create, and only intermittently targeted by law enforcement, botnets have increased in size and sophistication. That demands constant vigilance from researchers, who are always looking to disrupt and shutdown emerging threats.
Recent weeks have been flush with botnet takedowns, showing two different approaches to the problem. Microsoft launched a legal strike against the Zeus botnet, accompanying US Marshals who seized servers in Pennsylvania and Illinois. Just days earlier, another botnet had fallen thanks to a tactic that required no legal intervention and no seizures — “sinkholing.” Using this strategy, a team of researchers from the CrowdStrike Intelligence Team, Kaspersky Lab, the Honeynet Project, and Dell SecureWorks commandeered the Kelihos.B botnet. Last September they’d done the same with the original Kelihos network, estimated at around 40,000 computers, only to see a new variant emerge with almost 140,000 infected IP addresses, with that number still creeping up.
This second takedown illustrates several things about the battle against botnets, including the tenacity of their creators and the diligence of researchers fighting them. To get an inside look at the ongoing struggle, we talked to two members of the CrowdStrike Intelligence Team, director of intelligence Adam Meyers and senior research scientist Tillmann Werner, who also helped sinkhole the first Kelihos.
Kelihos.B rose from its predecessor’s ashes with the same raison d'etre: blasting out spam and conducting DDOS attacks. Like the earlier incarnation it could also pilfer any valuable information stored on infected machines, but one new wrinkle was its targeting the Bitcoin cyber-currency, including mining for coins and stealing user's wallets. And researchers say that it returned with the same fundamental design, including the familiar vulnerabilities. "When I got a sample of Kelihos.B, I instantly confirmed that the new version could be attacked again," said Tillmann Werner via email, "No technical hurdles — we simply needed some time to set up the sinkhole infrastructure and to coordinate the people involved."
Botnet creators have long recycled code and techniques; analysts believe the original Kelihos succeeded the Storm botnet, which peaked in size around late-2007. Both use a peer-to-peer infrastructure that, theoretically, adds resiliency. Infected machines typically receive commands from other infected machines — this makes it more difficult to "decapitate" the network by eliminating a single command-and-control server. The peer-to-peer network can also change quickly in response to threats; each node can propagate a list of new peers if there’s an intrusion.
"We were actually a little surprised that it worked so well."
But it’s exactly this capability that enables the "sinkhole" technique. If researchers can crack the communications protocol used among the peers, they can create "poison" data that will propagate through the whole botnet. The data forces all peers to connect to a single machine. That machine, of course, belongs to the white hats, who now control the botnet.
This technique has worked several times, and proved successful with the first Kelihos. But even given the minor changes to Kelihos.B, researchers couldn’t be absolutely sure about their plan. They’d spent a lot of time preparing, dismantling and examining the code, crafting the poison, and setting up the injection. But it was impossible to eliminate every uncertainty. "You don't really know how good it's gonna work," says Werner, "as you cannot test it with the real botnet, obviously, and lab tests might miss something or the botmaster might take counteractions of some sort." A savvy botmaster might notice his dwindling control and try to fight back. In 2009, for example, researchers temporarily seized the Torpig botnet, only to watch the original owners distribute a new binary that wrested back control.
With the sinkholing scheduled for Wednesday, March 21, Werner and his colleague, Brett Stone-Gross (who’d helped crack Torpig), worked late nights on Monday and Tuesday honing their tools. At 10:00 AM PST, they were ready to go. They hit the enter key, injecting the poison into the botnet. The effect was immediate: a second screen began to fill with information, including unique bot IDs, number of IP addresses, and simultaneously open connections.
"The sinkhole was flooded with requests only a second after we pressed enter," says Werner. "We were actually a little surprised that it worked so well, even better than for Kelihos.A, where it took a few minutes for the poison to propagate." Within an hour they’d collected 50,000 machines — 10,000 more than they’d expected the entire botnet to contain. Marco Preuss at Kaspersky Lab had begun a coordinated poisoning effort and saw similar results; soon the number of sinkholed machines topped 100,000.
And what about a challenge from the botmaster? In a worst-case scenario, the botmaster(s) — thought to be from a Russian-speaking country, in part because Kelihos.B spewed out Russian language spam and targeted Russian political activists with DDOS attacks — could have pushed an update to the infected machines not yet sinkholed. That would have quarantined them from the poisoned group, enabling a splintered botnet to survive. But while Werner and his colleagues saw some responses during the effort, none were effective.
Why not just force the botnet to self-destruct?
One question for the team is what to do next. The sinkhole only shifts control of the botnet; it doesn’t cure the zombie computers. Contacting owners and ISPs about infections is a slow, laborious process, and requires the sinkhole to remain in place until every infection is remediated. Many of the sinkholed computers remain so because their owners, often unidentifiable, don’t realize the machines are compromised, and so they never get fixed. (The FBI’s Operation Ghost Click faces a similar problem, having liberated thousands of infected machines from a compromised DNS server, then getting stuck running a substitute DNS server to keep the zombie machines online.) Wouldn’t it just be easier to push an update, just like the botmaster might’ve, but one that would remove the offending software? Why not just force the botnet to self-destruct?
CrowdStrike’s Adam Meyers audibly cringes at the idea. "Can you imagine pushing code onto someone else’s machine and something goes wrong? That would be very, very bad." Forcing updates onto more than 100,000 machines running a menagerie of Microsoft operating systems (84 percent using Windows XP) would be a technical possibility, but comes with a host of legal concerns. Questions of risk and liability limit the range of action, even for seemingly unresponsive machines.
Despite the legal questions, remediation "for their own good" might seem like a tempting option. But beneficent tampering is still tampering, and security professionals continue to debate the ethics of their interventions. The Honeynet Project, a research organization that participated in the Kelihos.B sinkholing, recently released a draft code of conduct. It emphasizes understanding and respecting the relevant laws, and, wherever possible, minimizing the impact on end users.
Actively disrupting botnets represents something of a shift among researchers, according to Meyers. "We've spent a lot of time over the last 15-20 years categorizing and generating information about this stuff," he says, "but we haven't really done anything from an action standpoint. I think we're seeing a lot more of this activity, and we're seeing people who might not necessarily talk to each other come together in groups. They're brought together in these malware working groups. They spend a lot of time and effort trying to figure out how to take these things down."
Of course, as might have been predicted, soon after the sinkholing began, a new variant of Kelihos appeared. Kelihos.C spread through social media, using the familiar strategies of its predecessor. So the cat-and-mouse game continues. But Meyers, Werner, and their colleagues keep watching, probing for weaknesses and finding new methods of intervention. "You’ll get a lot of people saying, ‘Oh, they were up and running within a few hours,’" Meyers says. "Sure, but now they have to build that whole infrastructure back up. At the end of the day, if it’s death by a thousand cuts, at least we’re able to make life harder for them."