Patrick Peterson spent a year obsessively pursuing the criminals behind a giant spam attack. He never found the bad guys, but he learned a ton about fighting this e-mail scourge.
Patrick Peterson was at home in San Francisco on Memorial Day last year, idly checking work e-mail, when he received an alarming message on his
The e-mail said that IronPort's operations center in San Bruno, Calif. was seeing spam activity like it never had before. On a typical day IronPort's hardware, deployed at companies around the world, catches 5 billion spam e-mails, or 16% of all spam on the Internet. But by noon on Memorial Day the volume was already double the norm, and new varieties were mutating rapidly to avoid detection.
The surge went on for two more weeks and turned out to be a single, coordinated blast--20 billion messages in all--designed to drive gullible buyers to 14 e-commerce sites, like MyCanadianPharmacy.info and ExclusiveCaviarOnline.com, hawking fake Viagra, Rolexes and Russian caviar.
Peterson, who's been in Internet security for seven years, figured it had to be the largest spam attack in history, likely the work of an organized crime ring out of eastern Europe. Estimates based partly on typical consumer response rates to spam offers peg this ring's annual revenue at $100 million. "This was an entirely new level of sophistication," says Peterson.
He started to obsess about who could produce such a spam onslaught without being detected. Then he dug a little, then more, then even more. So began the strangest and most frustrating year of Peterson's 39-year-long life. In the course of his spam hunt he broke several laws, engaged the help of shadowy hackers, hid months of activity from his wife and boss, neglected work and ended up losing half his annual bonus. He contacted 20 agents in nine government agencies in the U.S., China and India but failed to engage any of them in a serious quest for the culprits.
In the end he never found out who did it. The only tangible evidence from his dive into the spammer's world are 17 piles of paper on his office floor and an envelope stuffed with fake Viagra pills shipped from phony addresses in India and China.
Peterson joined the security industry wanting to be a hero. After graduating from Stanford University with two degrees in electrical engineering, he went to work designing wireless chips but switched fields after feeling lost in a crowd. "You're one of ten thousand engineers working on some small fraction of the challenge. In security, you can have big impact," he says.
His recent hunt was not a total waste of time. IronPort, which was acquired last month by
It was unclear a few hours into that attack whether this particular barrage was coming from a bunch of different spammers using similar techniques, or a single spammer. One clue that made it feel like one hulking missive: Most of the messages were sprinkled with lines from J.R.R. Tolkien's The Hobbit. This helps them pass through spam filters designed to block e-mails populated with words like "Viagra" and "Cialis" or variations such as "V!agra" or "Ci@lis."
Peterson was stunned at how rapidly the spam in this attack was mutating. IronPort's quarantine system, which logs new spams sent to Peterson's e-mail, usually isolates 120 messages per day. But this attack had spawned 350 before the first day was over, with new versions of spam appearing every 12 minutes. Peterson wanted to see if all these iterations were being generated by the same automated software program. He had IronPort's bank of 100 servers start running dozens of "cluster analyses," or scans that look for frequently used combinations of letters and symbols among millions of messages. It turned out the proliferating varieties shared a similar vocabulary. Every message had one or more of these triplets: AMB, VAL, GRA, NAX, MER and LEV.
Peterson presumed the spam was, like most spam nowadays, coming from a botnet, a ring of hacked PCs and servers co-opted into churning out e-mails, with their owners unaware. Some 11% of all PCs are botnet zombies. Every computer that sends an e-mail has to reveal its 32-bit Internet Protocol address, which indicates geographic location. Peterson filled an Excel spreadsheet 28,756 rows long with the unique IP addresses that were trying to flood IronPort's customers. The network he mapped rivaled a large company's in scale: 110,000 computers connected in 119 countries, including Spain, South Korea, Brazil and Israel. He'd never seen an operation this massive.
Then Peterson began clicking on the Web links advertised in the e-mails. Using a software program called Traceroute, he timed how long it took to load each phony-pill-selling Web site. Typically such requests take 50 milliseconds and make about 15 stops along the Internet. But the requests going to sites like MyCanadianPharmacy took more than 200 milliseconds and were making 43 stops, some of them unidentifiable, before reaching the final server. The spammers had set up an elaborate ricochet to cover their tracks and disabled the protocol that lets outsiders identify the final destination. Peterson could see only the beginning of the traffic's path, not the end.
In early June Peterson spent a week trolling hundreds of the phony spamvertised Web sites, looking up their registration information in the "Whois" database. Every new Web site has to provide a name, address and phone number to this database. He thought he might find the repetition of a name or number that could yield a clue.
No luck. His criminals had filled the entries for 2,100 Web addresses with gobbledygook, essentially thumbing their noses at one of the few ways the Web aims to keep track of who owns what site. "At this point I had lots of dead ends," he says.
By now Peterson was spending half his workday on the spam hunt. He began to ignore big projects, such as the one he started in January 2006 to bring outside security experts to speak at IronPort. He used to organize one meeting a month but over the summer didn't organize any, and failed to submit papers to five important security conferences. He also missed five meetings of the Internet Engineering Task Force, an influential, standards-setting group. "I kept thinking, 'I need to get back to my real job,' but then I'd get one more thread," he says.
One promising lead came the afternoon he ran a simple test to see if the phony pharmacy Web sites were filling orders or just stealing credit card numbers. Using a bogus credit card number, he placed orders on Pharmashop, a site that looked as though it ran on a server in Hungary, and MyCanadianPharmacy, which seemed to be run out of China.
Bingo. Although the sites looked different and were run by machines at opposite ends of the world, he got identical customer service e-mails, seconds apart, telling him to try another card. And both e-mails originated from a server at a hosting outfit called InterCage in Concord, Calif., 40 miles east of IronPort's offices.
Peterson had an assistant drive there, but the address was just a post office box. InterCage had been accused in chat rooms and blog posts of hosting lots of spam-related sites. So Peterson called the San Francisco office of the Federal Bureau of Investigation. The next day agent Christopher Sadlowski was in Peterson's office taking detailed notes on the InterCage link. The firm's president, Emil Kacperski, says that like any Internet provider, InterCage might unwittingly be allowing spam to pass through. InterCage has cooperated with subpoenas before, says Kacperski.
Peterson felt a sense of validation. He was onto something. "It was a rush for me. I felt like I was getting closer." But Sadlowski was soon transferred to another division and thus went the first in a string of disappointing attempts to enlist law enforcement. "No one is approaching this from both the technical- and physical-world angles," says Peterson. "That's why the bad guys continue to get away with it."
In mid-June Peterson ran out of leads and, feeling frustrated, made a buy on MyCanadianPharmacy. Now, Americans are forbidden to import prescription drugs, but there's an exception: a three-month supply is permitted. To buy the pills, he created a one-time-use
A London address appeared at the bottom of one message. Peterson had an IronPort employee there visit the building. It housed several offices, but a receptionist had never heard of the Web company. The address for the world headquarters of MyCanadianPharmacy is an empty parking lot in Toronto.
Peterson's credit card was charged to an account in Russia. He called MasterCard for more details, but, without a subpoena, the card company could reveal only the merchant's account name, #Pharmacyclient1.com, and asked him if he had a complaint. He said he didn't, but a few weeks later his card was mysteriously reimbursed for the full charge.
MasterCard has ceased processing transactions for 500 fake pharma Web sites over the past three years. But the sites close and reopen with another bank under a totally different name. "It is a game of Whack-a-Mole," says Joshua L. Peirez, a security and policy chief at MasterCard.
Subscribe to Forbes and Save. Click Here.
By August Peterson was getting so frustrated that he was considering breaking into some of the spammers' botnet machines. Then he got a phone call from a man in Canada who identified himself only by his online handles "Spam Killrz" or "SiL" (as in "Spam is Lame"). SiL had heard Peterson speaking about the spam attack on the radio and, as it turned out, was tracking the same criminals, but with less-than-savory tactics.
SiL's day job is with a Toronto software firm, but at night he hacks around the Web trying to get into computers compromised by spam rings. Most of the time he informs the server's owners, he says, "though sometimes I operate in a legal gray area."
SiL told Peterson that MyCanadianPharmacy and Pharmashop were linked to a dozen other sites selling illegal drugs, watches and caviar. SiL also remotely dissected a server at a Greek university that had been tallying pill orders at the rate of one every 30 minutes. "Suddenly, my investigation felt even bigger than I'd imagined," recalls Peterson.
In late August Peterson's Viagra order arrived, wrapped in a long, thin envelope with a Mumbai return address on the back. He paid investigators in India $3,000 to snoop around. They visited the address but turned up nothing.
In October lab results from Ohio's Toxicology Associates proved that Peterson's pills were filler. (FORBES duplicated Peterson's order and got pills from Shanghai that contained the honest-to-God sildenafil used in real Viagra. But
At work, Peterson was perpetually distracted. "Hours slipped," he says. He flunked his 2006 performance review; his boss, IronPort's marketing chief, Thomas Gillis, gave him a bonus of 10% of salary instead of the usual 20%. "It is frustrating to be his manager. Pat has to understand the whole situation. This is one of his greatest strengths, but it's also a real weakness sometimes," says Gillis.
At a January Web-identity conference, Peterson rambled through a presentation without slides. Then the following month, a day before the security industry's biggest conference, he met with IronPort's top engineers to go over his hourlong presentation. They found dozens of glaring flaws.
IronPort executives in March told Peterson to stop wasting time on the pill chase and refused his request to rope in other staffers. So Peterson began hiding his sleuthing at work and sneaking onto his home computer after his wife had gone to bed. "She hasn't asked why, so I don't tell," he says.
Peterson made another online drug buy in April with a
Recently he's been getting "404" error messages when he points his work PC at MyCanadianPharmacy. He didn't think much of it because the spammers are bouncing traffic through so many stolen machines. Then he got a call from his confidant in Canada, SiL. IronPort's IP address was now intentionally blocked from that Web site, as are IP addresses at the FBI, FDA and MasterCard. Others soon told him: You're not in the game unless you're on that list. Says Peterson: "It was a victory of sorts."
The greater victory was learning--the hard way--how large spam networks operate. IronPort now regularly checks traffic lags between its computers and the computers sending spam mail. A delay probably means the computer trying to send the e-mail is hijacked. Now those machines, and the URLs promoted within spam, get flagged in IronPort's system. Then it tries to tell if that zombie computer is communicating with others that should also be flagged.
IronPort also scans data on newly registered Web domains. There it looks for clusters of attributes like shady registrars, and repetitive names and ZIP codes in the Whois database. Peterson's engineers update customers' spam filters every two minutes instead of every five because of the rapid morphing he witnessed last June. He says: "If I can't find them, at least I can make their lives harder."
Sp@mmerz in the Slammer
The CAN-SPAM Act of 2003 has done nothing to thwart spam volumes. But it has put a few scamsters behind bars.
Jeremy Jaynes
Sold software, porn and work-at-home schemes. Made $24 million in total. Convicted, now appealing nine-year sentence on free- speech grounds.
Daniel Lin
Sold weight-loss patches and "generic Viagra." Made $350,000 in a three-month stretch. Convicted, now serving three years in federal prison.
Chris Smith (a.k.a. Rizler)
Operated illegal online pharmacy. Made up to $24 million. Convicted of distributing controlled substances, money laundering. Sentencing set for July; currently being held for smuggling a computer into a halfway house.
Jeffrey A. Kilbride and James R. Schaffer
Sent spam laced with dirty photos to spur traffic to porn sites. Made more than $2 million. Both face maximum of 30 years, plus up to $500,000 fine. Sentencing in September.
Robert Soloway
"Spam King" peddled software that made spamming easy. Accused of money laundering, identity theft, mail and wire fraud. Prosecutors trying to seize $773,000. Ordered to pay