For those who haven’t clawed their eyes out after watching the previous Numa Numa video, there’s an updated version of Numa Numa Dance at Newgrounds or if you don’t like tons of flashing crud in the browser, click on the direct link to the flash.

For those who actually like the song, it’s real name is Dragostera Din Tei by Haiducii/O-Zone. If you’re really addicted to it, there’s several versions up in the iTunes Music Store, including some really horrid English versions.

Wired News is reporting in their article Shady Web of Affiliate Marketing one of the good parts of the awful CAN-SPAM law. Affiliate programs are now getting some well deserved scrutiny because CAN-SPAM says they are liable if they know or should have known that their affiliates are spamming. Bravo.

Maggie’s coworker’s son got his Windows XP box 0wn3d quite well recently and they asked me to come take a look at it and fix it up. He had been running XP SP1 and had only a couple of additional patches installed, so it was pretty ripe for the picking. He didn’t want to upgrade to SP2 because he was worried it would make his computer too slow. He *was* running an Norton anti-virus, but still got loaded up with spyware, adware, clickbots, trojans and ratware. Why anti-virus stuff refuses to handle anything except viruses remains a mystery to me. Anti-virus stuff should block ANY malicious software. He was probably being used to send spam too, because his first-hop ping round trip was 2 seconds almost immediately on bringing up the net connection.

Usually to clean up a box I’d run McAfee Stinger, AVG Free Edition, Ad-Aware SE, and Spybot Search & Destroy and that’d take care of things. Not in this case.

I ran McAfee Stinger which was able to find a couple of copies of Korgo in the cache, but they didn’t look like they were active infections. Ad-Aware and Spybot both found tons of malware and AVG found a few, but a lot of it kept re-appearing after being removed. I found that a program called Golden Retriever Cash Back was reinstalling new malware each time. I was able to find registry entries to disable it, and then it was easier to make progress. Ad-Aware and Spybot were able to clean up many of the rest but there were still a few things that they and AVG still didn’t find. So then I went through Task Manager to look at each process name in google and see whether it was friend or foe. The bad stuff I’d kill and then search for their files and registry entries and manually remove it.

The real break-through was finding out about a program called Hijack This. It’s not for the novice, but it was able to find out how some of the stuff I couldn’t find with search were getting invoked and was able to disable them. The ones they couldn’t find were running as winx69.exe, ryhpka.exe, winagent.exe, mcafee32.exe and navprotect.exe. Some of the other malware running that I removed included clfmon.exe, Sygate.exe, elitevcy32.exe, pwn.exe, mssce.exe, msfwel.exe, gamma.exe, jah.exe, mssw32.exe, istsvc[1].exe. And that’s just the stuff that they could find but couldn’t remove automatically.

All in all this took most of the afternoon and evening on New Years Day (Feb 9). At the end, updated all security patches through present including SP2, and he had a well machine and good network performance again.

I dunno how regular people are able to get rid of this crap when they get infected this badly. The easy to use “click here” software was easily fooled, and several of the infestations were able to elude me for a while. And on top of that, five of them weren’t caught at all by anything except me manually going through Task Manager and cleaning up startup programs.

For those of you who aren’t being careful, here’s some advice:

1) Run good anti-virus software that updates *at*least* once a day. Make sure your mail server uses anti-virus software as well. (I use the pay version of AVG7 and have it set to update every 6 hours. My mail server runs ClamAV and clamassassin and updates every hour.)

2) Have a hardware firewall, or at least a software firewall. (I use a unix box as a firewall, but a USD30 broadband router is usually pretty good as a hardware firewall.) XP SP2’s firewall is getting better, but there’s still better options.

3) Don’t use Internet Explorer. Mozilla Firefox has had much less frequent and less serious security bugs than Internet Explorer. There’s a reason IE has the nickname Internet Exploiter. Whatever you do, make sure popups are disabled and ActiveX is highly restricted.

4) Don’t use Outlook Express. Also avoid Outlook, or at least make sure it is Outlook 2003 and is patched to latest update. The Outlook mail readers make it way too easy for malware to get through. If you insist on using these programs, you must disable the preview pane, and don’t click on any attachments you didn’t expect to come. Even if you get something from someone you know, if you weren’t expecting it, don’t open the attachments until you’ve confirmed that the sender actually sent it.

5) If you get a message from your bank, auction web site, personal payments site, or any other sensitive service saying you need to do something, don’t click on any links in the email. Instead, manually go to their website and see if there’s something there you really need to do. If in doubt, call them up and ask before you do anything.

6) Don’t respond to, or click on any links in any spam messages — messages you did not request from companies you don’t have a business relationship with. Just don’t. They are either gonna rip you off or infect you.

7) Run Ad-Aware SE and Spybot Search & Destroy regularly. Make sure Spybot immunizes your system each time. If you are technical enough, run Hijack This as well.

8) Run Windows Update frequently and always keep up to date with all critical patches. Better yet, set Windows Update to run automatically. (However, be aware that it will automatically reboot your system after being patched, so you don’t want to do this if you run things continuously.)

9) If you have any part of Office/Word/Excel/Powerpoint/Outlook installed, you’ll need to keep it updated separately at http://office.microsoft.com/officeupdate/

10) Be careful about pirated software, p2p downloads and porn. Some of it includes malware.

Maggie wanted me to make some cookies to take to Germany with her for her sister, so I made today:

2 dozen green tea shortbread
2 dozen oatmeal pecan chocolate chip

Seems like a lot of effort to carry cookies all the way to Germany, but I’m getting to be a very lean traveller. I hardly take anything not strictly necessary. On the other hand my wife, daughter and mother in law had 3 big suitcases and 2 medium suitcases plus a few carry-ons for this trip. Now I do tend to bring a fair bit of stuff back to Taiwan with me when I go to US, but otherwise I don’t bring much. And even coming back to Taiwan I set a strict limit of 2 bags, and if it doesn’t fit it gets left behind.

Maggie just called that her plane is boarding soon. She said one of the bags was too heavy so she had to move some stuff to carry ons. 🙁

Maggie, Emily and my mother in law are off to Germany tonight to visit Maggie’s sister there. They’ll be there through Feb 28 and then returning. I’m not going because I’ve got plenty of traveling of my own to do coming up.

First I’m going to Japan Feb 20-25. I’ll be going to Kyoto to the APRICOT2005 conference to talk about spam issues as part of my chair of APCAUCE duties. While I’m there I’ll take the Shinkansen over to Nagoya for a Dreams Come True concert on Feb 23. Then there’s the LONGEST TRAVEL DAY EVER on Feb 25 where I will start from the hotel in Kyoto in the morning, take the subway and train to Kansai International Airport (Osaka), fly to Taipei, go home and repack, clean up and grab a bite, then back to the airport that night to fly to San Francisco, getting there in the evening (still the same day but much later), drive down to Santa Clara and in all likelihood collapse into a puddle for a couple of days. The next week I’m off to Kansas for classes Mar 1-4, back to Santa Clara, followed by a drive down to Santa Barbara after that and return back to Taipei on March 13 (really it’s late night on March 12, but since it is just past midnight departure, it says March 13 on the ticket), arriving on March 14 early morning.

I’m getting tired just thinking about all that. Whew.

http://funpic.hu/swf/numanuma.html

Today is the official start of the Year of the Chicken on the lunar calendar. Happy New Year and all that.

I made a chocolate cake with chocolate frosting from scratch yesterday. Once again had some oven calibration problems so it came out a bit undercooked, but not so much it was a disaster.

Still battling the cold.

I managed to catch Emily’s cold so been pretty miserable the last couple of days. It really hit me after going out on Sunday to buy some baking stuff at Jasons Marketplace and was having cold sweats, shaking and felt terrible. Came home and went straight to bed at around 8pm and went fast asleep. Felt better yesterday and was able to crank out 10 dozen more cookies to complete Chinese New Year orders, but still under the weather enough that it sucked. Today’s New Year’s Eve, so will be able to relax tomorrow, but today’s also my mother in law’s birthday, so I need to make a birthday cake.

4 dozen Oatmeal Milk Chocolate
4 dozen Oatmeal Raisin
2 dozen Oatmeal Pecan Raisin

Lots of orders coming in for Chinese New Years

Today’s production:

4 dozen Oatmeal Milk Chocolate Chip
4 dozen Oatmeal Cranberry Orange
4 dozen Oatmeal Raisin
4 dozen Peanut Butter
4 dozen Green Tea Shortbread

I need to get a commercial mixer and commercial oven if this keeps up.

I’ll have to do this again on Monday because already more orders coming in.

When e-mail spam first came about, the primary delivery method was to buy a T1 and start spewing out the spam. The original RBL centralized a method to block such spam sources quickly and efficiently. Spammers then progressed from using throw-away dialup accounts paid for with stolen credit cards, to using open relays, to using throw-away broadband lines, to using open proxies. At least when they sent it from their own networks they could claim some amount of legitimacy.

Since the Sobig virus appeared in 2003, and most noticeably after the big outbreak in August 2003, the major source of spam has been sent through what are known as Zombie PCs — end-user computers that have been infected with a virus that allows a spammer to use their computer to send out email anonymously. In fact there’s a fascinating paper claiming that the author of the Sobig viruses is Ruslan Ibragimov, who also is the author of the Send Safe program which is designed to send spam, including through Zombie PCs. Though Ibragimov denies it, it’s quite a coincidence that Sobig and Send Safe both contain a large amount of code in common.

While estimates vary as to how many infected PCs are being abused by spammers, the reliable figures range from 4 million to 10 million. I like to say 5 million since it’s on the lower end of the scale and is a nice round figure. The initial pattern was these PCs would be instructed by a central controller — often over a private IRC channel — to send mail directly to one of the target user’s incoming mail servers.

This had the advantage that spam would be coming from so many different sources that it would be hard to block. Except that the CBL, Spamhaus XBL, and others have found ways to identify and block zombie PCs reasonably quickly, and ISPs have finally started to block the ability of client PCs to send out email directly.

Therefore it should come as no surprise at all that spammers are looking for new ways to send out their scams. As CNET reports, the new method is for Zombie PCs to send out mail through the PC’s ISP’s mail server. Actually, abuse managers at major ISPs have been warning about this as a growing problem for at least a month, but the current publicity is raising awareness more generally now.

What is alarming anti-spammers about this change in tactics is that it makes it harder to block spam coming into your mail servers, because any ISP mail server will also be a source of a lot of legitimate mail. Despite criticism from some spammers, most anti-spammers try to be very conservative about not blocking legitimate mail. Ironically though, this also makes it hard for an ISP to ignore the problem of spam from their network, because their mail servers will be overloaded handling the extra traffic, and it will be more apparent where the spam is coming from.

The fundamental disconnect here is that there has been a tremendous effort on the side of preventing spam from coming into a network. There’s been relatively little action on the prevention of spam leaving a network. What efforts have been made have been slow to adapt to the rapidly changing tactics of spammers. Now that the ISP’s own mail servers are involved in distributing spam, it is time to step back from the old tactics of preventing spam entering your network and instead demand that ISPs prevent spam from leaving their networks. Ultimately they will be forced to because when it gets bad enough, even an ISP’s mail server will get blocked if it is a large source of spam.

Another disconnect is the current efforts to authenticate email. I don’t want to get too down on these efforts, but most of them only authenticate mail sent from server to server at the domain level and not end to end. I don’t want to get into details, but take a look at what Dave Crocker has said about the difference. With these approaches mail going through authorized channels such as an ISP’s mail server is presumed to be OK. The problem being that most ISP’s mail servers will gladly relay any mail from any of their clients without any attempt to authenticate who it came from.

The problem is that a little old standard commonly called SMTP AUTH has not been very widely adopted. With this standard one can submit mail to a server using either an SSL certificate or a username/password so that the mail server knows who sent it. If a mail server does not do at least this, it really has no idea who the mail came from without someone manually checking logs to determine who was on an IP at a certain time. If it does do this, then accountability is possible.

But but but, the anti-spammers cry: a trojan running on a Zombie PC will have access to login credentials whether it is a certificate or a plaintext password, and will be able to use those to send out spam anyways! In response, I posted a bit of a rant to one of the anti-spam lists which I’ve included here with some editing:

I’m not that worried. There’s still ways to combat this for all consumer-oriented accounts (which accounts for the vast majority of zombie PCs). It will require some compromises that will bother some people but which will have zero impact for the vast majority of users, have workarounds for those who care, and ultimately have very little downside.

Before we start some definitions:

Consumer-oriented account: The type of service most people (outside this audience) have, a dynamic IP address, an ISP-provided email account, a mail reader set to use just that email account and a no-servers policy.

Business/power-user account: Usually a static IP, often reverse DNS, often running their own mail server or at least allowed to run servers.

The following applies *only* to consumer-oriented accounts (though it could be made optional for business/power-user accounts that don’t need outgoing mail capability):

Block port 25 in and out, for both source and destination (so asymmetrical routing won’t work). This will mean these users can’t run mail servers. This is your first compromise. Most of these services already have explicit no-servers policies but just haven’t been good about enforcing it. The few people that need it will need to get service from a power-user ISP like Speakeasy or contract out for mail service (like they already typically do for dynamic dns service). Tough. Most users will never know the difference. Most ISPs are already here or getting here.

These ISPs MUST NOT block mail submission(port 587 and the older port 466), vpn ports, ssh, imap, ssl imap, pop2, pop3, ssl imap, etc. so that those people that need alternative mail service and can’t get other business/power-user service can contract out their mail service and access it through one of these methods.

ISPs should implement SMTP AUTH, then help their users transition over to sending out only SMTP AUTH mail. This is your second compromise. Yes, this will be hard. Tough. At the end of this transition, you should block every non-mailserver on your network from relaying unauthenticated email. No, pop before smtp is not adequate. No, whitelisting your IPs is not adequate. Some ISPs are already here or planning on it.

The current zombies look up incoming MX for the PC’s domain and send out through them. These ISPs should block every system on their networks except known authorized mail servers from sending anything to these servers. One might be tempted to just take away the ability to relay through these hosts, but then it leaves you open to zombie machines seeding spam to your network through zombies in your own user population. These servers should thus only accept incoming mail from outside the network and authorized email servers inside the network (all of which should eventually only accept SMTP AUTH submissions). This is your third compromise, but it should have little impact.

Incidentally, you could use a separate domain for your client PCs that isn’t used for mail to block or catch this technique: e.g. if your user’s mail is all username@yourisp.com, set up your client reverse DNS to use hosts in the otherdomain.net which has no incoming mail servers, or dummy ones set to catch zombies. This will only work until the Zombies get smarter, but it shouldn’t break anything else.

Now that you have SMTP AUTH, zombies will probably migrate to hijacking credentials. But since you’ve already read this, you know there’s a variety of way to combat that, but basically it boils down to this: knowing who is sending which emails allows you to identify problems, contain them and correct them. This is your fourth compromise. There’s lots of choices depending on what is palatable to you and your users (or a combination of these is even better):

The easiest is to set limits on how much mail any account can send over various periods of time. After you implement SMTP AUTH, examine your logs to determine what sort of legitimate mail volumes a single account will generate over a period of a month, a week and a day. Add on a generous amount on top to account for growth and set these as per-account limits. Initially you’ll catch a lot of zombies quickly because they’ll blow the daily quota quite easily. Over time they will probably get smarter and limit volume, but to be useful they will still send out way more volume than a real user would over longer periods like a week or a month. If Zombies are limited to the amount of traffic a real user would produce, they will be ineffective as a vehicle for spam. You can either make this a hard limit that stops further mail when the threshold is reached, or exceeding the limit can trigger an alert that requires account review by a technician.

If you don’t want to impose limits on your users, another method would be to track which users generated which messages. An easy way to do this is throw a message-id to username table in a db somewhere based on what goes out through your mail servers. For all complaints coming into abuse@yourisp.com, have it automatically find the message-id in the complaint and identify the responsible account to the technician reviewing the complaint. You can even consolidate and prioritize complaints by volume with excellent accuracy that will identify your biggest problems. You can even have it automatically suspend an account that goes over a certain complaint threshold, and you can do this accurately as well since you know how much mail they’ve sent out. Accuracy of this will be excellent because you will see all outgoing mail through your network and know which particular user’s credentials were used. You will only be missing mails sent from outside your network using the address of your user.

You could stick an anti-spam filter on outgoing mail and block and/or track how many possible spam messages were sent from a certain account. Over a certain threshold, and you trigger a review or automatic suspension. As well, outgoing messages MUST be filtered for viruses and any detected should be blocked and trigger a review or automatic suspension.

You can even force messages sent out to be sent from the authorized email address of the user. You don’t have to do this, because some combination of the above should be effective. And SPF will make it difficult for 3rd party mails to get out. But if that’s your only option, it will certainly work.

With some combination of the above, SMTP AUTH allows you to quickly and accurately identify exactly which accounts are causing the most abuse on your network. There’s probably other options I haven’t thought of as well.

Yes, this means you will have to actually deal with Zombies and spammers on your network. Tough. You might even have some whiny journalist write an article about your ‘draconian’ actions. Tough. You will have to disable a user’s outgoing mail until they fix their PC. Tough. Most of the pain will be up front and will diminish over time.

Yes it sucks to have to do all this, but if all the above is done, there’s no way a spammer can send out adequate volumes from a consumer-oriented ISP for it to be worth the while. Will that completely solve the problem? Maybe, but probably not. There’s still breaking into and hijacking real servers, hijacking networks and ASNs and other routing tricks, poorly secured corporate networks (wi-fi?), and a whole lot of other stuff. Not to mention that most ISPs will only implement the above kicking and screaming. But at least we’ll get rid of a large portion of the 5 million plus vectors we currently need to worry about.

I expect a lot of hand-wringing from some of you about how it makes it hard for the Linux hobbyist running their own server. Tough. These people know how to use VPNs, ssh tunneling, port 587 rerouting, fetchmail, and a variety of other ways to do whatever they want to do. Yes, that’s extra work they’ll have to do to get the functionality. Tough. The other 99% of Internet users won’t even notice the difference. I do not think Mr. Linford (reference above CNET article) is at all exaggerating that the email infrastructure is nearing crisis, and worrying about this issue when faced with a crisis is really not productive.

This will also put a big dent in the ability for viruses to spread in email. It won’t completely destroy it, but the volume will diminish dramatically if every message has to go through anti-virus filtering on the way out. And even now, virus writers are coming up with innovative new ways to distribute viruses. In the past months we’ve seen the rise of infected banner ads and web pages using browser flaws, infected JPEG image and mp3 files, and even propagation of viruses over instant messaging.

It is time for us to focus on preventing spam from getting out instead of just getting in. Yes, this will be an added burden on ISPs. The spammers aren’t giving up without a fight, and the weakest ISPs will only end up losing the game if they are complacent.

(*) FUSZP stands for Final Ultimate Solution to the Zombie Problem, in reference to FUSSP. And yes, I mean it ironically.