I found this out the hard way, although the solution quickly became obvious. Fortunately, it's something that almost every company can easily fix, in case the problem is affecting you.
If you have your own Internet domain name, or your company runs an e-mail server, this applies to you. If not, you probably know someone who could use the information below.
Secondary MX Records Rear Their Heads
I'm currently involved in a major project that involves hands-on testing of numerous hardware-based antispam appliances. The full results will be posted at WindowsSecrets.com on Jan. 26, 2006.
Meanwhile, you may be able to eliminate a lot of the spam your company receives by taking advantage of a little-known back door that I unwittingly stumbled across but that spammers are all too familiar with.
Here's the concept in a nutshell: Every Internet domain name that accepts e-mail must post information called an MX record (Mail Exchanger record). The MX record lets any mail server in the world know where to send messages addressed to, say, email@example.com.
It's common for companies to post a record for a second server as well. This server is said to have a secondary MX record. If the primary mail server is down or too busy to accept messages, the secondary server will be contacted instead by machines trying to send mail. The secondary server will accept the messages and eventually route them to the primary server when it's once again available. (For technical details, see this Sendmail tutorial.)
Now that you know the basics, you'll be able to see how spam can pour in if all of this isn't properly set up.
Opening the Back Door to Spam
I run my own mail server, which sits on a machine called northwind.windowssecrets.com. ("Northwind" is a dummy name that Microsoft commonly uses in demos.) My server uses a static IP address of 220.127.116.11, which we'll call 50 for short. By the way, I'm not telling you any confidential information here. Anyone on the Internet can ping my server and see all of this in plain sight.
Laying out a plan to test antispam appliances, a developer who works in my company decided to assign each device, one at a time, to a second IP address. We duly changed our primary MX record to that second address: 18.104.22.168. (Let's call this 51.) Within a few days, every mail server on the Internet had cleared the old number out of its cache and was sending our e-mail to the new number.
At this point, any antispam appliance being tested on our 51 address would do its thing, quarantining messages judged to be spam. It would then route the remaining good mail to our real mail server on the 50 address. So far, so good.
Unfortunately, my developer had innocently posted the 50 address of the real mail server as a secondary MX record. Big mistake. More spam than we'd ever seen started pouring in.
The spammers were working the angles like this:
Primary mail server. Spammers were sending messages to numerous different e-mail addresses, most of them nonexistent, via our primary MX record. Since the primary MX record was now an antispam appliance we were testing, this spam was mostly detected by the appliance and blocked.
Secondary mail server. Since the spam hadn't been delivered to the primary MX record, spam servers simply looked up our secondary MX record and delivered the messages there. The real mail server at 50 readily accepted the messages, since the antispam appliance at 51 had been circumvented.
Old subdomain names. As a result, we quickly deleted 50 as a secondary MX record. Mail servers should then try to send mail only to our primary MX record at 51, where our antispam appliance operates. But many spam servers simply clung to the fully qualified name of our mail server, Northwind. The spam servers continued sending spam directly to the machine at that subdomain name -- ignoring the Internet mail protocol, which establishes the MX record as the standard.
A Fast Fix to a Frustrating Problem
Prior to our latest device tests, our mail server had been defended by no antispam appliance. We had configured it to avoid most spam by rejecting any mail sent to nonexistent addresses. It had also been told to automatically drop connections from known spam servers found on Spamhaus.org's SBL+XBL block list. This reduced the spam we received to manageable levels.
Now that our mail server was on a separate IP address from whatever antispam appliance was currently being tested, however, stronger measures were needed. If your company has a secondary MX record pointing to a backup mail server, you may need to take the following steps, just as we did:
Delete the secondary MX record. It's usually not necessary for smaller companies to post a secondary MX record these days, with a mail server that's at all reliable. Servers that send Internet mail will generally re-try a down server for up to 72 hours before giving up. Unless your mail server conks out for days at a time, these re-tries should be sufficient for you not to miss any mail, even if you have brief outages.
Protect your secondaries. If your company truly does have redundant mail servers, go ahead and post secondary MX records for them. Any secondary machine, though, must have antispam protection that's just as strong as the primary machine's.
Block inbound mail except from your filter. We configured our firewall to accept e-mail connections only from the antispam appliance at 51. Doing this clobbered the spam servers that were "remembering" our mail server's IP address and subdomain name (after its secondary MX record was deleted). Good mail servers observed the standard Internet protocol and sent mail through the primary MX record. Spam servers, which don't follow the rules, couldn't get in through the back door.
With all of the obscure conventions and jargon of the Internet mail system, it's not surprising that mail servers can get set up wrong. If that's the case in your company, spammers are happy to drive right through the hole and deliver plenty of junk to your inboxes.
(For more on protecting your mail server, see my Jan. 24 column.)
An Executive Tech update on "Fixing Elections"
I wrote on Dec. 20, 2005, that two Florida counties had scrapped Diebold-brand electronic vote-counting equipment after computer experts found that results could be changed without detection. I noted later in that article, "Diebold memory cards became an issue in Ohio after the 2004 Presidential election. Secretary of State Kenneth Blackwell ordered the cards and other election records sealed from public inspection until after the state's electors were sworn in."
On Dec. 21, Carlo LoParo, a spokesman for the Ohio Secretary of State, e-mailed me, saying, "Diebold electronic voting machines were NOT in use in Ohio during the 2004 election. Further, the story you referenced relates to Ohio law requiring ballots be sealed for a period of 11-15 days after an election to allow for the arrival of overseas absentee ballots. (Please see Ohio Revised Code 3505.32.) This process does not relate to 'electors' or the Electoral College."
I respectfully disagree with these statements.
According to a study by VerifiedVoting.org, a national nonprofit organization that advocates reliable vote counting methods, Diebold election equipment was in use in the 2004 elections in Ohio's Lucas and Hardin Counties and possibly others. Additional counties used electronic equipment, including memory cards, from Triad Governmental Systems and other vendors. At least 41 of Ohio's 88 counties adopted Diebold electronic voting equipment prior to the state's 2005 elections.
The Dayton Daily News and other news outlets reported on Dec. 12, 2004, that election records had been sealed by the Ohio Secretary of State. This was 40 days after the Presidential election, long after the normal 11-to-15-day period. The Daily News reporter wrote: "This period usually lasts about 10 days after the election, but was being extended because an official recount is under way in the state, according to Blackwell spokesman Carlo LoParo." The final determination of contests for Presidential electors was on Dec. 7, 2004, based on federal law, and Presidential electors voted in each state on Dec. 13, 2004.
Bob Fitrakis, an Ohio attorney who formerly was involved in litigation with the Secretary of State, said in an interview regarding Diebold election services, "GEMS, which is Diebold software, was used to count votes on equipment in about half the counties." Fitrakis edits the Columbus Free Press.
I would also like to acknowledge the research assistance provided by Mark Crispin Miller, the author of Fooled Again, an October 2005 book that criticizes election practices observed in Ohio and other U.S. states in the 2004 elections.