000.000.0.1 A hacker is made
Autumn 1998. A first-floor room in McLaughlin Hall. Dan Kaminsky is a second-year information systems major. A lifelong tech geek, he’s made fast friends as the go-to guy for dorm-room computer emergencies. Your hard drive gives up the ghost in the midst of a term paper? Your laptop is dead? Dan’s your man.
But there’s one problem that requires a bigger fix: printing. To send papers to the residence hall network printers, students have to use a program that is crashing their Windows 95/98 operating systems nearly as fast as Kaminsky can revive them. Kaminsky’s solution is to write a new program, DoxPrint, which allows Kaminsky’s own computer, functioning as a server, to relay jobs to the printers. Problem solved.
Three months into his sophomore year, Kaminsky has, according to his own estimates, half the print jobs in the dorms going through his room in McLaughlin. But while his tinkering is sophisticated, his tact isn’t. Conceived without consulting officials, DoxPrint raises alarms with SCU’s Information Technology managers. After months of operation, Kaminsky’s program is eliminated just before finals.
“I was being this punk kid who thought, ‘I’ll show IT how smart I am with this awesome code and they’ll just be blown away,’” Kaminsky recalls years later. “I totally bungled the politics of it.”
Manoochehr Ghiassi has taught at Santa Clara for nearly 30 years. A professor of information systems, he served as Kaminsky’s mentor at SCU and judges him one of his top three students ever. But Kaminsky’s genius was, he says, difficult to detect at first. Unchallenged, Kaminsky was the kind of kid who could fade in a class he was in fact capable of teaching. Once Ghiassi figured out Kaminsky needed pushing, Kaminsky responded—with sometimes surprising results.
Indeed, Kaminsky used what he learned in Ghiassi’s systems programming class to craft DoxPrint, which Kaminsky turned in as a class project.
Ghiassi fielded the concerns about DoxPrint from administrators. He also invited Kaminsky to watch a presentation by a professor whom the department was considering hiring. The presence of Kaminsky the undergrad was barely noted until the floor opened for questions, and Kaminsky’s incisions began.
“A few faculty heads turned to see who this fellow was at the end of the conference room,” Ghiassi says.
A decade later, such precociousness would look like the 25-cent lemonade-stand beginnings of a Wall Street heavy-hitter. Kaminsky is now paid by the biggest names in the business to find problems in their software. But even they don’t expect him to unearth a problem that goes beyond the concerns of any one company to the well-being of the Internet itself.
000.000.0.2 License for mayhem
Early winter, 2008. Kaminsky is 28 years old and in demand. As an Internet security consultant, he is hired by the likes of Microsoft to punch holes in applications and operating systems like Vista so they can get fixed before the bad guys exploit the same weaknesses. In one sense, he is a white wizard versed in the often dark arts of hacking. But Kaminsky isn’t trying to break anything when he stumbles onto a flaw of horrific proportions. He’s measuring the speed at which Internet requests are processed by different servers, looking for faster ways to host data—a task that has him messing in the Internet’s basic plumbing. But what he finds is a gaping hole in the very pipes connecting computers across the Web—a weakness so fundamental that the security of practically everything online seems suddenly at risk.
“At first I thought I must be missing something,” Kaminsky says, looking back. “I thought, ‘This can’t work—because if it worked, the Internet would be in so much trouble.’ Then it worked.”
Kaminsky found a way to fool servers that relay requests for Internet addresses. Whether a server is looking for Google, Bank of America, or Facebook, he can steer the requester down a false path. The hack takes seconds and leaves almost no trace. And anyone seeking those sites via compromised servers can be diverted to Kaminsky’s world—totally unaware of the misdirection.
The hack is a license for mayhem. In analog terms, it is as if Kaminsky has found a way to reroute your bank’s telephone number—so that when you call to speak to a customer service agent, you are instead redirected to an impostor. The crook-masquerading-as-teller can ask for necessary personal information. With security silently breached, the safety of e-mail, bank accounts, passwords—even top secret government information—is in peril.
The threat is as clear as day to Kaminsky, who thankfully has learned something about diplomacy since his days unilaterally trying to fix SCU’s IT system. Quietly, he starts to mobilize an unprecedented coalition of companies to fix the flaw, rallying them with a chilling warning: not that the enemy is at the gates, but that there are no gates.
000.000.0.3 Sound the alarm
For Paul Vixie, one of the first to get the message, hearing Kaminsky’s discovery is like being handed a stick of dynamite. He is president of the Internet Systems Consortium, based in Redwood City. Vixie is also something of an institution himself in the realm of the Domain Name System (DNS)—the system that essentially functions as the Internet’s phone book. DNS connects the userfriendly name you type in—such as santaclaramagazine.com—with its numeric Internet Protocol (IP) address—a sequence of numbers separated by three periods. Vixie’s company maintains and distributes the most popular software for carrying out the protocols that Kaminsky has been manipulating.
To Vixie, the gravity of the situation is obvious. This isn’t a localized bug in Cisco, Apple, or Microsoft coding, but an ancient hole in DNS itself, the foundation of the Internet, affecting millions of machines.
“I was freaked out,” Vixie says. “I locked it down as tight as I could without making it totally impractical or impossible to talk at all.”
Inadvertent spills are more of a danger than spies. Kaminsky’s brilliance is in discovering a flaw that had gone unseen since the early 1980s, but which is as plain as the nose on your face once you know what to look for. Vixie tells Kaminsky never again to speak about his discovery over a cell phone. From now on, they are to discuss things only by landline or encrypted e-mail—or in person.
Vixie rouses the small community of DNS vendors and experts, ignoring the grumbles of those who have to dig up old phones or commandeer fax machines to get to a landline. Not that Vixie is sharing much information. In most cases, he simply tells people to get on a flight to Seattle, no questions asked. A few grumble that they aren’t in a spy novel. But they come: 16 DNS experts and vendors from Internet giants like Microsoft, Cisco, Sun, and Linux as well as the small universe of leading DNS engineers arrive at the emergency summit on March 31, 2008, at a Microsoft office in Redmond, Wash. Some travel from as far as Finland. Many of those attending still have no clue for the reason of the meeting until Kaminsky takes center stage with a modest-looking PowerPoint presentation. His message: Practically every server out there can be hacked in less than 10 seconds.
“Everyone agreed this is really, really bad,” Kaminsky says. “It was definitely, ‘Ah, crap…. We have to fix this.’”
000.000.0.4 Timing is everything. Or at least half of it.
To understand the flaw Kaminsky found, it helps to think of the Internet as an old telephone exchange staffed by millions of operators. When a user types in a website—for example, www.scu.edu—the request passes to the local Internet Service Provider and then bounces up a hierarchy of operators until it gets to one that knows the correct numerical address. The answer is then sent back down to the initial operator, who connects the user to the site in question.
The process goes on millions of times a second across the Internet, generally without hitch. But Kaminsky has found a way to silently slip a wrong answer into the exchange, redirecting anyone asking the infected server for a particular site wherever he chooses. Most dangerous would be a false destination designed to look like the real thing.
The basic idea of the hack has been around for years, but the long odds of success rendered it a minor concern. Servers randomly number their requests with an identification number from one to approximately 65,000, and only an answer with the same number included is accepted as legitimate.
Still, one in 65,000 isn’t exactly astronomical odds. What makes the servers hard to fool is the fact that hackers have to time the attack to the millisecond that they update their records. For efficiency’s sake, servers don’t go scurrying to look up www.scu.edu every time someone asks for it. Instead, they rely on their memory of previous requests. Only after a set period of time do they “forget” the location and go looking for the answer anew. Only in that fleeting moment is the server vulnerable to false information.
In other words, a hacker trying to misdirect Santa Clara University’s Web traffic to her own site has to hit the targeted server precisely at the moment it updates its memory of SCU’s address. Even then, the hacker has to guess the ID number before someone else sends the correct address. It is like trying to win a lottery. But Kaminsky has found a way to buy as many tickets as he wants and ramp up the odds to where they’re in his favor.
The secret is to ask for a spoof page—for example, www.xyz.google.com. Since the name is made up, it is almost assured of not being in the server’s memory, forcing the server to go ask for help. It is as if Kaminsky is calling up the local operator and asking for a nonexistent employee at Google. Even though the server knows Google’s general numbers from countless previous requests, it has never heard of this invented person, forcing it to go looking for the operator officially in charge of Google. With no end of fake Google addresses to ask for, Kaminsky can command the server to go for help on cue, subjecting it to ambush after ambush of false responses, each with a guess at the correct ID number.
If the old way of trying the hack was like deer hunting, where the attacker waits long hours for a chance at a solitary shot, this is a fully automatic assault on a sitting duck. Indeed it’s usually a matter of seconds before the blizzard of fake replies gets the ID number correct. Then, as far as that server is concerned, Kaminsky’s site is Google. It all depended on Kaminsky asking for a fake page—something a savvy teenager in a cybercafé anywhere in the world can do.
Kaminsky added that one little twist that brought the whole house of cards down. Sandy Wilbourn, one of the witnesses to Kaminsky’s 2008 presentation, says, “There was a huge sense of urgency because there was no patch for it.”
Wilbourn is vice president of engineering for Nominum, which helps provide broadband access to 200 million customers. In his mind, Kaminsky’s biggest accomplishment isn’t just finding the bug; rather, it’s building consensus for a response, avoiding the chaos that would have been crippling if the vendors left Redmond without a joint plan of action.
000.000.0.5 The fix
In the Microsoft gathering, Kaminsky advocates for “source port randomization”—basically adding another layer of numeric matching between the query and the answer. The fix changes the odds so that, instead of there being a one in 65,000 chance of guessing a correct ID number to impersonate a server, the odds are now more than one in several billon. The solution is far from perfect; even those odds can be cracked with enough computing power. But as battlefield medicine, it gives everyone the chance to regroup and prepare for a longer fight with better armor.
Still, weaving the change into existing products made by unrelated companies requires unprecedented cooperation. Normally, companies work on development cycles of months and years; they fix their own vulnerabilities on their own time, testing patches in trials. But the vastness of the hole across all platforms requires companies to unite in a secret pact. If one vendor announces a fix for their own products or divulges the problem before the others are ready, they’ll invite an onslaught of attacks that could affect everyone. Collaboration—and secrecy—are key.
The companies agree on a drop-dead release date for the fix: July 8, 2008, when patches will be available for the world to download. It’s up to each company to solve the problem in their own products. Meeting the deadline is a major undertaking for the companies involved. But any time one of them bellyaches about costs or headaches, Vixie points to Kaminsky.
That’s because Kaminsky—the man who had raised the alarm privately—has also set a time and place for making the flaw very public. On Aug. 6, 2008, Kaminsky will disclose the full details of the bug at the Black Hat conference in Las Vegas, an annual hackers gathering where he has held court since his days at Santa Clara.
In Kaminsky’s view, there is no way to keep the flaw secret for much longer. Full disclosure at Black Hat—a month after releasing the July 8 repairs—is the best way to get the rest of the good guys up to speed, even if it inevitably means divulging information to the bad guys as well. That pending speech provides crucial motivation to keep companies focused on meeting the patch-release deadline. As Vixie puts it, “It turned out to be good to light the fuse before wandering around showing people the stick of dynamite.”
The July 8 release of security patches globally makes, not surprisingly, global news as well. Suddenly folks who’d never even heard of DNS know there’s an open hole in the Internet’s fundament. But even as Cisco, Sun, Microsoft, and other tech behemoths beseech people to implement the new patches, the companies stay silent on what exactly is being fixed.
Nominum’s Wilbourn recalls, “We’d tell our clients we have this new version of our server you really, really, really need to deploy, but we can’t tell you exactly what for. But you really need to do it.”
The patch-without-explanation approach creates tension, to be sure—or more like boiling frustration among Kaminsky’s fellow hackers, who don’t enjoy being told to do something just because “Dan said so.” But even the U.S. Department of Homeland Security is vouching for his advice; the department’s U.S. Computer Emergency Readiness Team has joined the July 8 chorus calling for urgent responses to the threat.
For Kaminsky, the process offers the same kind of lesson as the DoxPrint debacle so many years earlier. Being right isn’t enough if you don’t bring the right people into the process. He enlisted Vixie to help orchestrate the involvement of DNS vendors, but he hadn’t involved any hackers like himself. With each online posting calling him a grandstanding idiot, he is learning that “You can’t vouch for your own bug.”
On July 9, Kaminsky arranges a call with a handful of top hackers, swearing them to secrecy before giving them a full briefing that silences their doubts. “Dan has the goods,” Thomas Ptacek, a fellow security researcher, writes on his blog. “Patch now, ask questions later.”
Yet less than two weeks later, Ptacek leaves the details of the hack on a public part of his website. He claims it was meant to be released only after Kaminsky’s Black Hat speech. But the information is out, just 13 days after the release of the patches.
Almost immediately, hackers hit an Internet Service Provider in Texas with the very attack that Kaminsky had warned against, with the very result he had predicted. The hack redirects Google searches to a mock page in an apparent attempt to drive up clicks on ads. It’s relatively harmless, but the implications are frighteningly clear. How many more malicious uses of the hack have gone unreported or unnoticed?
Vixie is furious that the details had been disclosed, though Kaminsky emphasizes the positive. They had expected to get 50 percent of vulnerable servers patched over the year. They have two-thirds fixed within a month, most before the leak occurred.
“We managed 13 days of opportunity to patch without the underlying vulnerability being known,” Kaminsky says. “It wasn’t the 30 I was hoping for, but I’ll take 13 over zero any day of the week.”
000.000.0.6 What happened in Vegas
The intrigues in the run-up to the Black Hat Conference only whet people’s appetite for Kaminsky’s full disclosure. The likes of The New York Times and the BBC are taking notice; Wired magazine is collecting pieces for a 4,000-word profile.
With so much attention, Kaminsky’s speech is ripe for disappointment. Before Kaminsky’s appearance, fellow hackers even nominate him for the conference’s Most Overhyped Bug award. But it’s a standing-room-only crowd in the ballroom at Caesar’s Palace when he takes the stage. And Kaminsky fulfills the overstuffed expectations. His long-awaited full disclosure earns ovations and cheers from fellow hackers. Not only has he found the bug, but he has marshaled the forces to fix it—even if his success in doing just that has made it harder to measure just how close the Internet came to the chaos he warned of.
“Kaminsky has done more than any other human in history to raise awareness on a single security issue,” trumpets one story on InternetNews.com. “The silent threat that had been lurking in the plumbing of the Internet is silent no more.”
One journalist writing for CNET opines, “He has changed Internet security, and done so for the better of us all.”
Two years later, the spotlight has waned, Kaminsky says, gladly. Chief scientist and co-founder of a firm called Recursion Ventures, he is not sure he’d want to weather such a media storm again, though he says the attention served its purpose, making the flaw virtually useless to hackers.
But he’s the first to admit that the fix amounted to a Band-Aid. The Internet is so rife with holes and security lapses, it is best to think of your cyber exchanges as postcards in the mail, he says. There’s little stopping someone from reading them—or even rewriting them. He was just able to find a way to make it harder to redirect them. “On the Internet, we haven’t really invented the envelope yet,” he says.
Much of his work now is getting people to grasp the importance of making overdue steps in security. Indeed, in May 2009, Kaminsky took his act to Capitol Hill to testify before the House Subcommittee on Communications, Technology and the Internet. Not surprisingly, he offered a grim portrayal of the serious threats abounding in cyberspace. “The problems that we are trying to solve are smeared across company boundaries, are smeared across individual boundaries and, indeed, are smeared across the public-private boundary,” he told the committee in his closing remarks. “There actually does need to be a coordinating authority across all of these disparate actors to guide the public partnership toward actually fixing the scale of the problems,” he said.
Kaminsky, the diplomat, at your service.