When e-mail spam first came about, the primary delivery method was to buy a T1 and start spewing out the spam. The original RBL centralized a method to block such spam sources quickly and efficiently. Spammers then progressed from using throw-away dialup accounts paid for with stolen credit cards, to using open relays, to using throw-away broadband lines, to using open proxies. At least when they sent it from their own networks they could claim some amount of legitimacy.
Since the Sobig virus appeared in 2003, and most noticeably after the big outbreak in August 2003, the major source of spam has been sent through what are known as Zombie PCs — end-user computers that have been infected with a virus that allows a spammer to use their computer to send out email anonymously. In fact there’s a fascinating paper claiming that the author of the Sobig viruses is Ruslan Ibragimov, who also is the author of the Send Safe program which is designed to send spam, including through Zombie PCs. Though Ibragimov denies it, it’s quite a coincidence that Sobig and Send Safe both contain a large amount of code in common.
While estimates vary as to how many infected PCs are being abused by spammers, the reliable figures range from 4 million to 10 million. I like to say 5 million since it’s on the lower end of the scale and is a nice round figure. The initial pattern was these PCs would be instructed by a central controller — often over a private IRC channel — to send mail directly to one of the target user’s incoming mail servers.
This had the advantage that spam would be coming from so many different sources that it would be hard to block. Except that the CBL, Spamhaus XBL, and others have found ways to identify and block zombie PCs reasonably quickly, and ISPs have finally started to block the ability of client PCs to send out email directly.
Therefore it should come as no surprise at all that spammers are looking for new ways to send out their scams. As CNET reports, the new method is for Zombie PCs to send out mail through the PC’s ISP’s mail server. Actually, abuse managers at major ISPs have been warning about this as a growing problem for at least a month, but the current publicity is raising awareness more generally now.
What is alarming anti-spammers about this change in tactics is that it makes it harder to block spam coming into your mail servers, because any ISP mail server will also be a source of a lot of legitimate mail. Despite criticism from some spammers, most anti-spammers try to be very conservative about not blocking legitimate mail. Ironically though, this also makes it hard for an ISP to ignore the problem of spam from their network, because their mail servers will be overloaded handling the extra traffic, and it will be more apparent where the spam is coming from.
The fundamental disconnect here is that there has been a tremendous effort on the side of preventing spam from coming into a network. There’s been relatively little action on the prevention of spam leaving a network. What efforts have been made have been slow to adapt to the rapidly changing tactics of spammers. Now that the ISP’s own mail servers are involved in distributing spam, it is time to step back from the old tactics of preventing spam entering your network and instead demand that ISPs prevent spam from leaving their networks. Ultimately they will be forced to because when it gets bad enough, even an ISP’s mail server will get blocked if it is a large source of spam.
Another disconnect is the current efforts to authenticate email. I don’t want to get too down on these efforts, but most of them only authenticate mail sent from server to server at the domain level and not end to end. I don’t want to get into details, but take a look at what Dave Crocker has said about the difference. With these approaches mail going through authorized channels such as an ISP’s mail server is presumed to be OK. The problem being that most ISP’s mail servers will gladly relay any mail from any of their clients without any attempt to authenticate who it came from.
The problem is that a little old standard commonly called SMTP AUTH has not been very widely adopted. With this standard one can submit mail to a server using either an SSL certificate or a username/password so that the mail server knows who sent it. If a mail server does not do at least this, it really has no idea who the mail came from without someone manually checking logs to determine who was on an IP at a certain time. If it does do this, then accountability is possible.
But but but, the anti-spammers cry: a trojan running on a Zombie PC will have access to login credentials whether it is a certificate or a plaintext password, and will be able to use those to send out spam anyways! In response, I posted a bit of a rant to one of the anti-spam lists which I’ve included here with some editing:
I’m not that worried. There’s still ways to combat this for all consumer-oriented accounts (which accounts for the vast majority of zombie PCs). It will require some compromises that will bother some people but which will have zero impact for the vast majority of users, have workarounds for those who care, and ultimately have very little downside.
Before we start some definitions:
Consumer-oriented account: The type of service most people (outside this audience) have, a dynamic IP address, an ISP-provided email account, a mail reader set to use just that email account and a no-servers policy.
Business/power-user account: Usually a static IP, often reverse DNS, often running their own mail server or at least allowed to run servers.
The following applies *only* to consumer-oriented accounts (though it could be made optional for business/power-user accounts that don’t need outgoing mail capability):
Block port 25 in and out, for both source and destination (so asymmetrical routing won’t work). This will mean these users can’t run mail servers. This is your first compromise. Most of these services already have explicit no-servers policies but just haven’t been good about enforcing it. The few people that need it will need to get service from a power-user ISP like Speakeasy or contract out for mail service (like they already typically do for dynamic dns service). Tough. Most users will never know the difference. Most ISPs are already here or getting here.
These ISPs MUST NOT block mail submission(port 587 and the older port 466), vpn ports, ssh, imap, ssl imap, pop2, pop3, ssl imap, etc. so that those people that need alternative mail service and can’t get other business/power-user service can contract out their mail service and access it through one of these methods.
ISPs should implement SMTP AUTH, then help their users transition over to sending out only SMTP AUTH mail. This is your second compromise. Yes, this will be hard. Tough. At the end of this transition, you should block every non-mailserver on your network from relaying unauthenticated email. No, pop before smtp is not adequate. No, whitelisting your IPs is not adequate. Some ISPs are already here or planning on it.
The current zombies look up incoming MX for the PC’s domain and send out through them. These ISPs should block every system on their networks except known authorized mail servers from sending anything to these servers. One might be tempted to just take away the ability to relay through these hosts, but then it leaves you open to zombie machines seeding spam to your network through zombies in your own user population. These servers should thus only accept incoming mail from outside the network and authorized email servers inside the network (all of which should eventually only accept SMTP AUTH submissions). This is your third compromise, but it should have little impact.
Incidentally, you could use a separate domain for your client PCs that isn’t used for mail to block or catch this technique: e.g. if your user’s mail is all username@yourisp.com, set up your client reverse DNS to use hosts in the otherdomain.net which has no incoming mail servers, or dummy ones set to catch zombies. This will only work until the Zombies get smarter, but it shouldn’t break anything else.
Now that you have SMTP AUTH, zombies will probably migrate to hijacking credentials. But since you’ve already read this, you know there’s a variety of way to combat that, but basically it boils down to this: knowing who is sending which emails allows you to identify problems, contain them and correct them. This is your fourth compromise. There’s lots of choices depending on what is palatable to you and your users (or a combination of these is even better):
The easiest is to set limits on how much mail any account can send over various periods of time. After you implement SMTP AUTH, examine your logs to determine what sort of legitimate mail volumes a single account will generate over a period of a month, a week and a day. Add on a generous amount on top to account for growth and set these as per-account limits. Initially you’ll catch a lot of zombies quickly because they’ll blow the daily quota quite easily. Over time they will probably get smarter and limit volume, but to be useful they will still send out way more volume than a real user would over longer periods like a week or a month. If Zombies are limited to the amount of traffic a real user would produce, they will be ineffective as a vehicle for spam. You can either make this a hard limit that stops further mail when the threshold is reached, or exceeding the limit can trigger an alert that requires account review by a technician.
If you don’t want to impose limits on your users, another method would be to track which users generated which messages. An easy way to do this is throw a message-id to username table in a db somewhere based on what goes out through your mail servers. For all complaints coming into abuse@yourisp.com, have it automatically find the message-id in the complaint and identify the responsible account to the technician reviewing the complaint. You can even consolidate and prioritize complaints by volume with excellent accuracy that will identify your biggest problems. You can even have it automatically suspend an account that goes over a certain complaint threshold, and you can do this accurately as well since you know how much mail they’ve sent out. Accuracy of this will be excellent because you will see all outgoing mail through your network and know which particular user’s credentials were used. You will only be missing mails sent from outside your network using the address of your user.
You could stick an anti-spam filter on outgoing mail and block and/or track how many possible spam messages were sent from a certain account. Over a certain threshold, and you trigger a review or automatic suspension. As well, outgoing messages MUST be filtered for viruses and any detected should be blocked and trigger a review or automatic suspension.
You can even force messages sent out to be sent from the authorized email address of the user. You don’t have to do this, because some combination of the above should be effective. And SPF will make it difficult for 3rd party mails to get out. But if that’s your only option, it will certainly work.
With some combination of the above, SMTP AUTH allows you to quickly and accurately identify exactly which accounts are causing the most abuse on your network. There’s probably other options I haven’t thought of as well.
Yes, this means you will have to actually deal with Zombies and spammers on your network. Tough. You might even have some whiny journalist write an article about your ‘draconian’ actions. Tough. You will have to disable a user’s outgoing mail until they fix their PC. Tough. Most of the pain will be up front and will diminish over time.
Yes it sucks to have to do all this, but if all the above is done, there’s no way a spammer can send out adequate volumes from a consumer-oriented ISP for it to be worth the while. Will that completely solve the problem? Maybe, but probably not. There’s still breaking into and hijacking real servers, hijacking networks and ASNs and other routing tricks, poorly secured corporate networks (wi-fi?), and a whole lot of other stuff. Not to mention that most ISPs will only implement the above kicking and screaming. But at least we’ll get rid of a large portion of the 5 million plus vectors we currently need to worry about.
I expect a lot of hand-wringing from some of you about how it makes it hard for the Linux hobbyist running their own server. Tough. These people know how to use VPNs, ssh tunneling, port 587 rerouting, fetchmail, and a variety of other ways to do whatever they want to do. Yes, that’s extra work they’ll have to do to get the functionality. Tough. The other 99% of Internet users won’t even notice the difference. I do not think Mr. Linford (reference above CNET article) is at all exaggerating that the email infrastructure is nearing crisis, and worrying about this issue when faced with a crisis is really not productive.
This will also put a big dent in the ability for viruses to spread in email. It won’t completely destroy it, but the volume will diminish dramatically if every message has to go through anti-virus filtering on the way out. And even now, virus writers are coming up with innovative new ways to distribute viruses. In the past months we’ve seen the rise of infected banner ads and web pages using browser flaws, infected JPEG image and mp3 files, and even propagation of viruses over instant messaging.
It is time for us to focus on preventing spam from getting out instead of just getting in. Yes, this will be an added burden on ISPs. The spammers aren’t giving up without a fight, and the weakest ISPs will only end up losing the game if they are complacent.
(*) FUSZP stands for Final Ultimate Solution to the Zombie Problem, in reference to FUSSP. And yes, I mean it ironically.