Greylisting With Exim

Introduction

I have used this implementation for 4+ years now and so far I have had to do very few adjustments on it. It just works - the SPAM reduction is by 90-95%. Combined with spamhaus it yields 98%+ SPAM reduction prior to any form of filtering based on content.

Greylisting with Exim

My implementation is based on the "easy" implementation as described on here. Some of the ideas may be applicable to some of the more complex implementations as well:

• Spam-Filtering-for-MX HOWTO
• List of Greylisting Implementations on Spamlinks

Principle of Operation

Greylisting relies on the fact that SPAM is predominantly sent by Zombie botnets which do not follow standard SMTP conventions. Specifically, zombies violate the following parts of the SMTP protocol:

• Zombies do not retry to retransmit the email at regular intervals
• Zombies identify transient failures as permanent and attempt to redeliver using a different FROM email address

It is a common misconception that greylisting works simply because there are too few hosts implementing it. That is not true. Spammers cut corners in the smtp spec predominantly because the originating zombies try to spew as much SPAM as possible before it shows on the radar of checksumming and blacklisting services like Pyzor and spamcops. If they retry like a proper SMTP relay they are likely to be blacklisted after sending a considerably smaller amount of messages.

Greylisting takes advantage of the SMTP protocol violations by keeping a transient whitelist for hosts which have behaved like proper SMTP implementations and rejecting temporarily email from all that do not.

The first connection from an unknown host is checked versus a whitelist of allowed legitimate senders and if it is not in the list is deferred with a temporary error. All normal SMTP implementations will retry after a time. The retry time in a normal mail server is nearly always above 10 minutes. At that time the mail will be allowed through. There are some exemptions to this:

• Google - their corporate mail and hosted mail services are OK. However, the classic gmail does not use a persistent outgoing IP address. As a result it may be delayed for days unless whitelisted.
• Linkedin - they run on a cloud similar to google and their outgoing IP address is not persistent either. However, the actual range of IPs in use is fairly small so they all end up in the database in about a week.
• RIM - the registration emails sent by RIM for a new blackberry have no retry. Fantastic - same as they used to invent GPRS, 2-way pagers and so on, they now "invent" the SMTP. Applause.

Once the connection has been permitted by the greylist, it goes through autoblacklist checking. Autoblacklisting works on a principle very similar to the one used by Spamhaus and many other blacklisting services. Trap email addresses (for example ca@sigsegv.cx) are seeded on webpages for harvesting by spammers. These addresses are all listed in a local file /etc/exim4/spamtrap.lst for matching.

Spammer databases also contain large number of email IDs which have been misidentified as email addresses. These can be picked up out of the logs and added to the blacklist. If an incoming email is listed in the smaptrap file the host sending it is blacklisted straight away. Any mail from blacklisted hosts is rejected with a permanent reject.

Implementation

The implementation uses two tables:

exim_greylist | CREATE TABLE `exim_greylist` (
  `id` int(11) NOT NULL auto_increment,
  `relay_ip` varchar(64) default NULL,
  `from_domain` varchar(255) default NULL,
  `block_expires` datetime NOT NULL default '0000-00-00 00:00:00',
  `record_expires` datetime NOT NULL default '0000-00-00 00:00:00',
  `origin_type` enum('MANUAL','AUTO','UPDATED') NOT NULL default 'AUTO',
  `create_time` datetime NOT NULL default '0000-00-00 00:00:00',
  `status` enum('FRESH','UPDATED','CONFIRMED') NOT NULL default 'FRESH',
  `attempts` int(11) default '0',
  PRIMARY KEY  (`id`)
)

exim_blacklist | CREATE TABLE `exim_blacklist` (
  `id` int(11) NOT NULL auto_increment,
  `relay_ip` varchar(64) default NULL,
  `from_domain` varchar(255) default NULL,
  `block_expires` datetime NOT NULL default '0000-00-00 00:00:00',
  `record_expires` datetime NOT NULL default '0000-00-00 00:00:00',
  `origin_type` enum('MANUAL','AUTO','UPDATED') NOT NULL default 'AUTO',
  `create_time` datetime NOT NULL default '0000-00-00 00:00:00',
  `status` enum('FRESH','UPDATED','CONFIRMED') NOT NULL default 'FRESH',
  `attempts` int(11) default '0',
  PRIMARY KEY  (`id`)
)

exim4.conf needs the following "test conditions" added before any of the ACLs in the beginning of the file:

• Greylist Check

GREYLIST_TEST = \
SELECT CASE \
    WHEN now() - block_expires > 0 THEN 2 \
    ELSE 1 \
  END \
  FROM exim.exim_greylist \
  WHERE relay_ip = '${quote_mysql:$sender_host_address}' \
   AND from_domain = '${quote_mysql:$sender_address_domain}';

• Blacklist Test

BLACKLIST_TEST = \
SELECT CASE \
    WHEN now() - block_expires > 0 THEN 2 \
    ELSE 1 \
  END \
  FROM exim.exim_blacklist \
  WHERE relay_ip = '${quote_mysql:$sender_host_address}' ;

• Greylist Add

GREYLIST_ADD = \
INSERT INTO exim.exim_greylist (relay_ip, from_domain, \
   block_expires, record_expires, create_time) \
 VALUES ( '${quote_mysql:$sender_host_address}', \
   '${quote_mysql:$sender_address_domain}', \
   DATE_ADD(now(), INTERVAL 10 MINUTE), \
   DATE_ADD(now(), INTERVAL 7 DAY), \
   now() \
 );

* Blacklist Add

BLACKLIST_ADD = \
INSERT INTO exim.exim_blacklist (relay_ip, from_domain, \
   block_expires, record_expires, create_time) \
 VALUES ( '${quote_mysql:$sender_host_address}', \
   '${quote_mysql:$sender_address_domain}', \
   DATE_ADD(now(), INTERVAL 1 DAY), \
   DATE_ADD(now(), INTERVAL 1 DAY), \
   now() \
 );

• Greylist Attempts add - used to blacklist idiots which launch a DOS if you greylist them

GREYLIST_ATTEMPTS_ADD = \
UPDATE exim.exim_greylist  \
   set attempts=attempts+1 \
 WHERE relay_ip = '${quote_mysql:$sender_host_address}' \
  AND from_domain = '${quote_mysql:$sender_address_domain}';

• Housekeeping

GREYLIST_UPDATE = \
UPDATE exim_greylist SET record_expires = DATE_ADD(now(), INTERVAL 7 DAY), status = 'CONFIRMED' \
 WHERE  relay_ip = '${quote_mysql:$sender_host_address}' AND \
   from_domain = '${quote_mysql:$sender_address_domain}';

GREYLIST_PURGE = \
 DELETE from exim_greylist \
 WHERE record_expires < now();

BLACKLIST_PURGE = \
 DELETE from exim_blacklist \
 WHERE record_expires < now();

• Mysql access info

hide mysql_servers = SQLhost/Database/User/Password

These prepared statements are used in the check_rcpt ACL (modified from debian so some debianisms remaining in it). If you feel that 98%+ SPAM reduction is not enough you can push it a bit further with reverse DNS checking and SPF. However I have found that there are so many broken hosts for both of these out there that it better to leave these to an "advisory" check instead of outright rejecting them at the MTA.

acl_check_rcpt:

  accept
    hosts = :

  .ifdef CHECK_RCPT_LOCAL_LOCALPARTS
  deny
    domains = +local_domains
    local_parts = CHECK_RCPT_LOCAL_LOCALPARTS
    message = restricted characters in address
  .endif

  .ifdef CHECK_RCPT_REMOTE_LOCALPARTS
  deny
    domains = !+local_domains
    local_parts = CHECK_RCPT_REMOTE_LOCALPARTS
    message = restricted characters in address
  .endif

  warn set acl_m2 = ${lookup mysql{GREYLIST_TEST}{$value}{0}}
  warn set acl_m3 = ${lookup mysql{BLACKLIST_TEST}{$value}{0}}

  accept
    .ifndef CHECK_RCPT_POSTMASTER
    local_parts = postmaster
    .else
    local_parts = CHECK_RCPT_POSTMASTER
    .endif
    domains = +local_domains : +relay_to_domains

  .ifdef CHECK_RCPT_VERIFY_SENDER
  deny
    message = Sender verification failed
    !acl = acl_whitelist_local_deny
    !verify = sender
  .endif

  deny
    !acl = acl_whitelist_local_deny
    senders = ${if exists{CONFDIR/local_sender_callout}\
                         {CONFDIR/local_sender_callout}\
                   {}}
    !verify = sender/callout

  accept
    hosts = +relay_from_hosts
    control = submission/sender_retain

  accept
    authenticated = *
    control = submission/sender_retain

  deny  message = we do not accept relaying from our own addresses externally
    sender_domains = +relay_to_domains

  defer message = Client host rejected: Server undergoing maintenance.
         !acl = acl_whitelist_local_deny
    recipients = /etc/exim4/spamtrap.list
    condition = ${if eq{$acl_m3}{0}{1}}
    condition = ${lookup mysql{BLACKLIST_ADD}{yes}{no}}
    
  defer message = Client host rejected: Server undergoing maintenance.
         !acl = acl_whitelist_local_deny
    condition = ${if eq{$acl_m2}{0}{1}}
    condition = ${lookup mysql{GREYLIST_ADD}{yes}{no}}

  defer message = Client host rejected: Server undergoing maintenance.
         !acl = acl_whitelist_local_deny
    condition = ${if eq{$acl_m2}{1}{1}}
    condition = ${lookup mysql{GREYLIST_ATTEMPTS_ADD}{yes}{no}}

  deny  message = We do not accept mail from spamhaus SBL sources. See http://www.sigsegv.cx/policy.html for details 
    dnslists = sbl-xbl.spamhaus.org
    
  deny message = Your server is blacklisted here. 
    condition = ${if eq{$acl_m3}{1}{1}}
  
#  deny  message = We do not accept mail from SORBS  blacklisted sources. See http://www.de.sorbs.net/overview.shtml for details 
#    dnslists = dnsbl.sorbs.net 
    
#  defer message = We do not accept mail from spamcop listed sources.  See http://spamcop.net/
#    dnslists = bl.spamcop.net


  require
    message = relay not permitted
    domains = +local_domains : +relay_to_domains

  require
    verify = recipient

  deny
    !acl = acl_whitelist_local_deny
    recipients = ${if exists{CONFDIR/local_rcpt_callout}\
                            {CONFDIR/local_rcpt_callout}\
                      {}}
    !verify = recipient/callout

  deny
    message = sender envelope address $sender_address is locally blacklisted here. If you think this is wrong, get in touch with postmaster
    !acl = acl_whitelist_local_deny
    senders = ${if exists{CONFDIR/local_sender_blacklist}\
                   {CONFDIR/local_sender_blacklist}\
                   {}}

  deny
    message = sender IP address $sender_host_address is locally blacklisted here. If you think this is wrong, get in touch with postmaster
    !acl = acl_whitelist_local_deny
    hosts = ${if exists{CONFDIR/local_host_blacklist}\
                 {CONFDIR/local_host_blacklist}\
                 {}}

  .ifdef CHECK_RCPT_REVERSE_DNS
  warn
    message = X-Host-Lookup-Failed: Reverse DNS lookup failed for $sender_host_address (${if eq{$host_lookup_failed}{1}{failed}{deferred}})
     condition = ${if and{{def:sender_host_address}{!def:sender_host_name}}\
                      {yes}{no}}
  .endif

  .ifdef CHECK_RCPT_SPF
  deny
    message = [SPF] $sender_host_address is not allowed to send mail from ${if def:sender_address_domain {$sender_address_domain}{$sender_helo_name}}.  \
              Please see http://www.openspf.org/why.html?sender=$sender_address&ip=$sender_host_address
    log_message = SPF check failed.
    condition = ${run{/usr/bin/spfquery --ip \"$sender_host_address\" --mail-id \"$sender_address\" --helo-id \"$sender_helo_name\"}\
                     {no}{${if eq {$runrc}{1}{yes}{no}}}}

  defer
    message = Temporary DNS error while checking SPF record.  Try again later.
    condition = ${if eq {$runrc}{5}{yes}{no}}

  warn
    message = Received-SPF: ${if eq {$runrc}{0}{pass}{${if eq {$runrc}{2}{softfail}\
                                 {${if eq {$runrc}{3}{neutral}{${if eq {$runrc}{4}{unknown}{${if eq {$runrc}{6}{none}{error}}}}}}}}}}
    condition = ${if <={$runrc}{6}{yes}{no}}

  warn
    log_message = Unexpected error in SPF check.
    condition = ${if >{$runrc}{6}{yes}{no}}

# guessing in SPFquery is no longer supported
#  warn
#    message = X-SPF-Guess: ${run{/usr/bin/spfquery --ip \"$sender_host_address\" --mail-from \"$sender_address\" \ --helo \"$sender_helo_name\" --guess true}\
#                                {pass}{${if eq {$runrc}{2}{softfail}{${if eq {$runrc}{3}{neutral}{${if eq {$runrc}{4}{unknown}\
#                               {${if eq {$runrc}{6}{none}{error}}}}}}}}}}
#    condition = ${if <={$runrc}{6}{yes}{no}}
#
  defer
    message = Temporary DNS error while checking SPF record.  Try again later.
    condition = ${if eq {$runrc}{5}{yes}{no}}
  .endif

# basic housekeeping

  warn
    condition = ${lookup mysql{GREYLIST_UPDATE}}

  warn
    condition = ${lookup mysql{GREYLIST_PURGE}}

  warn
    condition = ${lookup mysql{BLACKLIST_PURGE}}

  accept
    domains = +relay_to_domains
    endpass
    verify = recipient

  accept

Maintenance

The biggest disadvantage of the original "easy" Exim Greylist with MySQL? is the ever growing cruft in the database. This can be solved easily using a simple script which periodically cleans up the table of old entries.

The most trivial way of doing it is by running the following SQL statements periodically (f.e. once a day):

      DELETE from exim_greylist
         where record_expires < now();
      DELETE from exim_blacklist
         where record_expires < now();

I have now integrated it into the basic config using the housekeeping statements

Enhancements

One of the biggest hallmarks of SPAM is that SPAMmers nearly always attempt MX-es in reverse order. If you are fortunate enough to have multiple IP addresses which can safely and securely share a database you can improve your greylisting performance by a few more percent and bring the kill rate to 99% on greylisting + spamhaus alone.

In order to do it, you need to increase the greylisting timeout and set it instead of 10 minutes to let's say at least 1 hour on the secondary MX. Normal SMTP hosts will try MX-es in the correct order so the timeout will be set by the primary. If however the secondary is accessed before the primary the timeout will go straight up.