Filtering Spam with Blocklists

For business professionals, spam is a huge problem that causes clogged in-boxes and wastes many hours of productive time to sort through it. Once you have an email address, it’s only a matter of time before you start getting unwanted email.

Even with email blocking software, it might be necessary to close the account and start again if the problem becomes too severe, but then you have to update everyone in your address book. And even then, it won’t be long before your new address winds up on a spammers list. So what do you do?

To stem the tide, many companies resort to blocklists, or blacklists as they are sometimes called, for filtering spam. Some of these companies such as Brightmail or SpamCop are commercial software companies.

Others, such as MAPS (Mail Abuse Prevention System a.k.a. The Realtime Blackhole List) are designed to create intentional network outages (a.k.a. blackholes) to limit the transport of unwanted mass e-mails.
Spamhaus, a European resource, lists IP addresses of spammers and uses their Register of Known Spam Operations (ROKSO), which they believe are behind 90% of American and European spam.

“The whole concept of blacklists is essentially giving out IP addresses or top-level domains that are known or in most cases suspected sources of spam,” says Linda Munyan, marketing communications manager for Brightmail. “These can be permission-based marketing groups or open relays that are being used by spammers. Often you hear these days about companies or ISPs who are blocking anything coming in from certain countries, such as Russia, China or Korea.

“Top-level domain refers to blocking from a certain country as opposed to IP addresses which are pointing to a certain machine. As an example, anything that comes in with a .ru in the ‘From’ portion of the header information is not accepted because there’s so much spam coming from that country (Russia),” she says.

When filtering spam, an issue is ‘false positives,’ something filtered as spam, when in fact it is legitimate mail. In the case of Brightmail, their rate of false positives is one in 100,000. But in some cases, spam has become such a huge problem that service providers resort to drastic measures, taking the risk of blocking legitimate mail from getting through to their end users.

Decoy Accounts And Attack Algorithms

“With Brightmail, we install filtering software at our customer sites,” says Munyan. “Our solution is comprised of three components. The first part is called the Probe Network, a network of decoy email accounts that we’ve put out across the Internet. Each customer that we bring on board is required to provide us with a certain percentage of statistical representation of their total email box universe. These are brand new accounts that have never been used.

“At Brightmail, our business intelligence team creates names for those new, unused accounts and they seed them across the Internet, into places such as Usenet groups where spammers are known to harvest email addresses,” Munyan says. “Our Probe Network has a statistical reach of 100 million email boxes and that’s what we use as our net (sometimes known as a spam trap or honey pot). The Probe Network attracts an incredible amount of spam and our business intelligence team constantly monitors the probes, refining them and making sure that they’re productive.

“Then we have a pretty sophisticated set of algorithms which groups this spam into attacks. A spam attack is any group of messages that are similar in nature. Many spammers try to randomize their messages so that filtering software techniques such as ours are thwarted. But our grouping algorithms strip out all that randomizing stuff, (numbers, spaces, dashes, dots, the university diploma, etc.) to its core essence and groups it into a spam attack. From there, we can write rules against it,” she says.

“The third component is called bloc. That’s the Brightmail statistics and operations center, where we have a group of people who work 24/7 and who service a QA function, making sure that the rules are working properly, that we haven’t caught anything that’s legitimate mail and that the rules are delivered out to our customers in a timely manner.

“The original focus of Brightmail was with very large ISPs, such as Earthlink, AT&T and Worldnet,” Munyan says. “We’ve just recently started moving into enterprises, companies like Cypress Semiconductor, Cisco, Motorola, and that’s where our focus is today. For ISPs, the fee for using the software would be $1-$2/user/year. For enterprises, the fee would be $5-$15/user/year (comparable to AV pricing). As an example, the fee for an enterprise with 1,000 employees would be around $15,000.”

“Both pricing structures are dependent upon the number of end-users (employees in the case of enterprises or subscribers in the case of ISPs). Obviously, ISPs are larger than enterprises, so their average pricing structure will be overall lower.”

How to Avoid Getting Blacklisted

To create legitimate mailing lists, the double opt-in approach is now considered to be the best way to go as opposed to a single opt-in, where you click once or something is already filled in for you and then you’re considered to have opted into a mailing list.

With double opt-in, you have to actively take the first step of clicking a box saying that you want to be a part of a list. Next, you have to respond positively back to an email message that says: ‘You indicated that you wanted to join this user group, newsletter, etc. If so, please reply back to this message.’ This is the second piece of the double opt-in approach.

To solve the problem of being blacklisted, MAPS is trying to drive businesses toward the double opt-in approach so that they don’t get blacklisted. But sometimes, people do get blacklisted for a variety of reasons. An example is if the packet size of their bulk email is too large and it’s caught by some of MAPS customer base, etc.

To get off a blocklist, you have to contact the service in question. According to information on the MAPS Web site: “Your first step in getting off of the MAPS RBL is to contact us, by phone if necessary (650-779-7080) or by e-mail if possible. Tell us the IP address you think is being affected since we can’t do anything that could affect your domain name, and tell us what you’re doing or would like our help to start doing to become less friendly to spammers. The moment you demonstrate favorable intentions toward stopping spam from using your resources, we will as a good faith gesture take you off of the MAPS RBL.”

Another interesting wrinkle is being falsely accused of spamming. According to SpamCop: “False spam reports are not tolerated. Users who file false reports will be banned from the SpamCop service and/or fined. If you would like to take action against a user who has erroneously accused you of spamming, please forward the entire SpamCop report, including full headers and the entire spam and proof that the user in question did subscribe for your list.

“If you would like to pursue action with the user’s Internet provider, SpamCop reports include all the information you need, including the IP address and datestamp of the complainant. You can even feed a SpamCop report to SpamCop to determine the originating point.”

The big question though, is how a fine of the above nature would be enforced. That information was not found on the SpamCop website, despite a thorough search.