qpsmtpd/plugins/karma

#!perl -w

=head1 NAME

karma - reward nice and penalize naughty mail senders

=head1 SYNOPSIS

Karma tracks sender history, allowing us to provide differing levels
of service to naughty, nice, and unknown senders.

=head1 DESCRIPTION

Karma records the number of nice, naughty, and total connections from mail
senders. After sending a naughty message, if a sender has more naughty than
nice connections, they are penalized for I<penalty_days>. Connections
from senders in the penalty box are rejected per the settings in I<reject>.

Karma provides other plugins with a karma value they can use to be more
lenient, strict, or skip processing entirely.

Karma is small, fast, and ruthlessly efficient. Karma can be used to craft
custom connection policies such as these two examples:

=over 4

Hi there, well known and well behaved sender. Please help yourself to greater concurrency (hosts_allow), multiple recipients (karma), and no delays (early_sender).

Hi there, naughty sender. You get a max concurrency of 1, max recipients of 2, and SMTP delays.

=back

=head1 CONFIG

=head2 negative <integer>

How negative a senders karma can get before we penalize them for sending a
naughty message. Karma is the number of nice - naughty connections.

Default: 1

Examples:

   negative 1:  0 nice - 1 naughty = karma -1, penalize
   negative 1:  1 nice - 1 naughty = karma  0, okay
   negative 2:  1 nice - 2 naughty = karma -1, okay
   negative 2:  1 nice - 3 naughty = karma -2, penalize

With the default negative limit of one, there's a very small chance you could
penalize a "mostly good" sender. Raising it to 2 reduces that possibility to
improbable.

=head2 strikes <integer>

How many strikes are needed to consider the connection nice or naughty.
Various plugins can adjust the karma (see USING KARMA IN OTHER PLUGINS).
For example, with the default value of 3, if the karma of this message is 3
or above, the connection is considered to be a nice one. If it's -3 or less,
it's considered a naughty one. Between -2 and +2 it's neutral

Default: 3

=head2 penalty_days <days>

The number of days a naughty sender is refused connections. Use a decimal
value to penalize for portions of days.

  karma penalty_days 1

Default: 1

=head2 reject

  karma reject [ 0 | 1 | connect | naughty ]

I<0> will not reject any connections.

I<1> will reject naughty senders.

I<connect> is the most efficient setting.

To reject at any other connection hook, use the I<naughty> setting and the
B<naughty> plugin.

=head2 db_dir <path>

Path to a directory in which the DB will be stored. This directory must be
writable by the qpsmtpd user. If unset, the first usable directory from the
following list will be used:

=over 4

=item /var/lib/qpsmtpd/karma

=item I<BINDIR>/var/db (where BINDIR is the location of the qpsmtpd binary)

=item I<BINDIR>/config

=back

=head2 loglevel

Adjust the quantity of logging for this plugin. See docs/logging.pod

=head1 CONFIG FILES

This plugin uses the following configuration files. All are optional.

=head2 karma_tlds

This file can contain semicolon separated tlds and the corresponding
karma adjustment to apply when the envelope sender match. It can be used to
penalize "spammy" tlds, or to raise the karma from (mostly) good tlds.

jp:-4
ch:-3
fr:+1

=head1 BENEFITS

Karma reduces the resources wasted by naughty mailers.  When used with
I<reject connect>, naughty senders are disconnected in about 0.1 seconds.

The biggest gains to be had are by having heavy plugins (spamassassin, dspam,
virus filters) set the B<karma> connection note (see KARMA) when they encounter
naughty senders. Reasons to send servers to the penalty box could include
sending a virus, early talking, or sending messages with a very high spam
score.

This plugin does not penalize connections with transaction notes I<relayclient>
or I<whitelisthost> set. These notes would have been set by the B<relay>,
B<whitelist>, and B<dns_whitelist_soft> plugins. Obviously, those plugins must
run before B<karma> for that to work.

=head1 KARMA

No attempt is made by this plugin to determine karma. It is up to other
plugins to reward well behaved senders with positive karma and smite poorly
behaved senders with negative karma. See B<USING KARMA IN OTHER PLUGINS>

After the connection ends, B<karma> will record the result. Mail servers whose
naughty connections exceed nice ones are sent to the penalty box. Servers in
the penalty box will be tersely disconnected for I<penalty_days>. Here is
an example connection from an IP in the penalty box:

 73122 Connection from smtp.midsetmediacorp.com [64.185.226.65]
 73122 (connect) ident::geoip: US, United States
 73122 (connect) ident::p0f: Windows 7 or 8
 73122 (connect) earlytalker: pass: 64.185.226.65 said nothing spontaneous
 73122 (connect) relay: skip: no match
 73122 (connect) karma: fail
 73122 550 You were naughty. You are cannot connect for 0.99 more days.
 73122 click, disconnecting
 73122 (post-connection) connection_time: 1.048 s.

If we only set negative karma, we will almost certainly penalize servers we
want to receive mail from. For example, a Yahoo user sends an egregious spam
to a user on our server. Now nobody on our server can receive email from that
Yahoo server for I<penalty_days>. This should happen approximately 0% of
the time if we are careful to also set positive karma.

=head1 KARMA HISTORY

Karma maintains a history for each IP. When a senders history has decreased
below -5 and they have never sent a good message, they get a karma bonus.
The bonus tacks on an extra day of blocking for every naughty message they
send.

Example: an unknown sender delivers a spam. They get a one day penalty_box.
After 5 days, 5 spams, 5 penalties, and 0 nice messages, they get a six day
penalty. The next offense gets a 7 day penalty, and so on.

=head1 USING KARMA

To get rid of naughty connections as fast as possible, run karma before other
connection plugins. Plugins that trigger DNS lookups or impose time delays
should run after B<karma>. In this example, karma runs before all but the
ident plugins.

 89011 Connection from Unknown [69.61.27.204]
 89011 (connect) ident::geoip: US, United States
 89011 (connect) ident::p0f: Linux 3.x
 89011 (connect) karma: fail, 1 naughty, 0 nice, 1 connects
 89011 550 You were naughty. You are penalized for 0.99 more days.
 89011 click, disconnecting
 89011 (post-connection) connection_time: 0.118 s.
 88798 cleaning up after 89011

Unlike RBLs, B<karma> only penalizes IPs that have sent us spam, and only when
those senders have sent us more spam than ham.

=head1 USING KARMA IN OTHER PLUGINS

This plugin sets the connection note I<karma_history>. Your plugin can
use the senders karma to be more gracious or rude to senders. The value of
I<karma_history> is the number of nice connections minus naughty
ones. The higher the number, the better you should treat the sender.

To alter a connections karma based on its behavior, do this:

  $self->adjust_karma( -1 );  # lower karma (naughty)
  $self->adjust_karma(  1 );  # raise karma (good)


=head1 EFFECTIVENESS

In the first 24 hours, B<karma> rejected 8% of all connections. After one
week of running with I<penalty_days 1>, karma has rejected 15% of all
connections.

This plugins effectiveness results from the propensity of naughty senders
to be repeat offenders. Limiting them to a single offense per day(s) greatly
reduces the resources they can waste.

Of the connections that had previously passed all other checks and were caught
only by spamassassin and/or dspam, B<karma> rejected 31 percent. Since
spamassassin and dspam consume more resources than others plugins, this plugin
seems to be a very big win.

=head1 DATABASE

Connection summaries are stored in a database. The database key is the integer
value of the remote IP. The DB value is a : delimited list containing a penalty
box start time (if the server is/was on timeout) and the count of naughty,
nice, and total connections. The database can be listed and searched with the
karma_tool script.

=head1 BUGS & LIMITATIONS

This plugin is reactionary. Like the FBI, it doesn't do anything until
after a crime has been committed.

There is little to be gained by listing servers that are already on DNS
blacklists, send to invalid users, earlytalkers, etc. Those already have
very lightweight tests.

=head1 TODO

 * Avoid storing results for DNSBL listed IPs
 * some type of ASN integration, for tracking karma of 'neighborhoods'

=head1 AUTHOR

 2013 - MS - Addeded penalty for spammy TLDs
 2012 - Matt Simerson - msimerson@cpan.org

=head1 ACKNOWLEDGEMENTS

Gavin Carr's DB implementation in the greylisting plugin.

=cut

use strict;
use warnings;

use Qpsmtpd::Constants;

use Net::IP;

sub register {
    my ($self, $qp) = (shift, shift);
    $self->log(LOGERROR, "Bad arguments") if @_ % 2;
    $self->{_args} = {@_};
    $self->{_args}{negative}     ||= 1;
    $self->{_args}{strikes}      ||= 3;
    $self->{_args}{penalty_days} ||= 1;
    $self->{_args}{reject_type}  ||= 'disconnect';

    if (!defined $self->{_args}{reject}) {
        $self->{_args}{reject} = 'naughty';
    }

    $self->init_db();
    #$self->prune_db();    # keep the DB compact
    $self->register_hook('connect',       'connect_handler');
    $self->register_hook('mail',          'from_handler');
    $self->register_hook('rcpt',          'rcpt_handler');
    $self->register_hook('data',          'data_handler');
    $self->register_hook('data_post',     'data_handler');
    $self->register_hook('disconnect',    'disconnect_handler');
}

sub init_db {
    my ( $self ) = @_;
    $self->db( name => 'karma' );
    return if ! $self->db->can('path');
    my $cdir = $self->{_args}{db_dir};
    # karma-specific hints for where to store the DB
    $self->db->dir( $cdir, '/var/lib/qpsmtpd/karma' );
    $self->db->nfs_locking( $self->{_args}{nfslock} );
}

sub hook_pre_connection {
    my ($self, $transaction, %args) = @_;

    $self->connection->notes('karma_history', 0);

    my $remote_ip = $args{remote_ip};

    $self->db->lock or return DECLINED;
    my $key  = $self->get_karma_key($remote_ip) or do {
        $self->log(LOGINFO, "skip, unable to get DB key");
        return DECLINED;
    };

    my $value = $self->db->get($key);
    if ( ! $value ) {
        $self->log(LOGDEBUG, "pass, no record");
        return $self->cleanup_and_return();
    }

    my ($penalty_start_ts, $naughty, $nice, $connects) =
      $self->parse_db_record($value);
    $self->calc_karma($naughty, $nice);
    return $self->cleanup_and_return();
}

sub connect_handler {
    my $self = shift;

    $self->connection->notes('karma', 0);    # default

    return DECLINED if $self->is_immune();

    $self->db->lock or return DECLINED;
    my $key  = $self->get_karma_key() or do {
        $self->log(LOGINFO, "skip, unable to get DB key");
        return DECLINED;
    };

    my $value = $self->db->get($key);
    if ( ! $value) {
        $self->log(LOGINFO, "pass, no record");
        return $self->cleanup_and_return();
    }

    my ($penalty_start_ts, $naughty, $nice, $connects) =
      $self->parse_db_record($value);
    my $summary = "$naughty naughty, $nice nice, $connects connects";
    my $karma = $self->calc_karma($naughty, $nice);

    if (!$penalty_start_ts) {
        $self->log(LOGINFO, "pass, no penalty ($summary)");
        return $self->cleanup_and_return();
    }

    my $days_old = (time - $penalty_start_ts) / 86400;
    if ($days_old >= $self->{_args}{penalty_days}) {
        $self->log(LOGINFO, "pass, penalty expired ($summary)");
        return $self->cleanup_and_return();
    }

    $self->db->set( $key, join(':', $penalty_start_ts, $naughty, $nice, ++$connects) );
    $self->cleanup_and_return();

    my $left = sprintf "%.2f", $self->{_args}{penalty_days} - $days_old;
    my $mess = "You were naughty. You cannot connect for $left more days.";

    return $self->get_reject($mess, $karma);
}

sub from_handler {
    my ($self,$transaction, $sender, %args) = @_;

# test if sender has placed an illegal (RFC (2)821) space in envelope from
    my $full_from = $self->connection->notes('envelope_from');
    $self->illegal_envelope_format( $full_from );

    my $karma_tlds = $self->get_karma_tlds() or return DECLINED;
    foreach my $tld ( keys %$karma_tlds ) {
        my $len = length $tld;
        my $score = $karma_tlds->{$tld} or next;
        $len ++;
        if ( $sender->host && ".$tld" eq substr($sender->host,-$len,$len) ) {
            $self->log(LOGINFO, "adjusting karma for .$tld envelope sender");
            $self->adjust_karma($score);
        }
    }

    return DECLINED;
}

sub rcpt_handler {
    my ($self,$transaction, $recipient, %args) = @_;

    $self->illegal_envelope_format( 
        $self->connection->notes('envelope_rcpt'),
        );

    my $count = $self->connection->notes('recipient_count') || 0;
    $count++;
    if ( $count > 1 ) {
        $self->log(LOGINFO, "recipients c: $count ($recipient)");
        $self->connection->notes('recipient_count', $count);
    }

    return DECLINED if $self->is_immune();

    my $recipients = scalar $self->transaction->recipients or do {
        $self->log(LOGDEBUG, "info, no recipient count");
        return DECLINED;
    };
    $self->log(LOGINFO, "recipients t: $recipients ($recipient)");

    my $history = $self->connection->notes('karma_history');
    if ( $history > 0 ) {
        $self->log(LOGINFO, "info, good history");
        return DECLINED;
    }

    my $karma = $self->connection->notes('karma');
    if ( $karma > 0 ) {
        $self->log(LOGINFO, "info, good connection");
        return DECLINED;
    }

    #  limit # of recipients if host has negative or unknown karma
    return DENY, "too many recipients for karma $karma (h: $history)";
}

sub data_handler {
    my ($self, $transaction) = @_;

    return DECLINED if $self->is_immune();
    return DECLINED if $self->is_naughty();   # let naughty do it

# cutting off a naughty sender at DATA prevents having to receive the message
    my $karma = $self->connection->notes('karma');
    if ( $karma < -4 ) {     # bad karma
        return $self->get_reject("very bad karma: $karma");
    }

    return DECLINED;
}

sub disconnect_handler {
    my $self = shift;

    my $karma = $self->connection->notes('karma') or do {
        $self->log(LOGDEBUG, "no karma");
        return DECLINED;
    };

    $self->db->lock or return DECLINED;
    my $key  = $self->get_karma_key();

    my ($penalty_start_ts, $naughty, $nice, $connects) =
      $self->parse_db_record( $self->db->get($key) );
    my $history = ($nice || 0) - $naughty;
    my $log_mess = '';

    if ($karma <= -$self->{_args}{strikes}) {    # Enough negative strikes ?
        $history--;
        my $negative_limit = 0 - $self->{_args}{negative};
        if ($history <= $negative_limit) {
            if ($nice == 0 && $history < -5) {
                $log_mess = ", penalty box bonus!";
                $penalty_start_ts = sprintf "%s", time + abs($history) * 86400;
            }
            else {
                $penalty_start_ts = sprintf "%s", time;
            }
            $log_mess = "negative, sent to penalty box" . $log_mess;
        }
        else {
            $log_mess = "negative";
        }
    }
    elsif ($karma >= $self->{_args}{strikes}) {
        $nice++;
        $log_mess = "positive";
    }
    else {
        $log_mess = "neutral";
    }
    $self->log(LOGINFO, $log_mess . ", (msg: $karma, his: $history)");

    $self->db->set( $key, join(':', $penalty_start_ts, $naughty, $nice, ++$connects) );
    return $self->cleanup_and_return();
}

sub illegal_envelope_format {
    my ($self, $addr) = @_;

# test if envelope address has an illegal (RFC (2)821) space
    if ( uc substr($addr,0,6) ne 'FROM:<' && uc substr($addr,0,4) ne 'TO:<' ) {
        $self->log(LOGINFO, "illegal envelope address format: $addr" );
        $self->adjust_karma(-2);
    }
}

sub get_karma_tlds {
    my $self = shift;

    my %karma_tlds =
      map { (split /:/, $_, 2)[0, 1] } $self->qp->config('karma_tlds');
    if (!%karma_tlds) {
        $self->log(LOGDEBUG, "no specific karma for tlds defined");
        return;
    }

    return \%karma_tlds;
}

sub parse_db_record {
    my ($self, $value) = @_;

    my $penalty_start_ts = my $naughty = my $nice = my $connects = 0;
    if ($value) {
        ($penalty_start_ts, $naughty, $nice, $connects) = split /:/, $value;
        $penalty_start_ts ||= 0;
        $nice             ||= 0;
        $naughty          ||= 0;
        $connects         ||= 0;
    }
    return $penalty_start_ts, $naughty, $nice, $connects;
}

sub calc_karma {
    my ($self, $naughty, $nice) = @_;
    return 0 if (!$naughty && !$nice);

    my $karma = ($nice || 0) - ($naughty || 0);
    $self->connection->notes('karma_history', $karma);
    $self->adjust_karma(1) if $karma > 10;
    return $karma;
}

sub cleanup_and_return {
    my ( $self, $return_val ) = @_;

    $self->db->unlock;
    return $return_val if defined $return_val;    # explicit override
    return DECLINED;
}

sub get_karma_key {
    my $self = shift;
    my $ip   = shift || $self->qp->connection->remote_ip;
    my $nip  = Net::IP->new($ip) or do {
        $self->log(LOGERROR, "skip, unable to determine remote IP");
        return;
    };
    return $nip->intip;                             # convert IP to an int
}

sub prune_db {
    my $self = shift;

    $self->db->lock or return DECLINED;
    my $count = $self->db->size;

    my $pruned = 0;
    foreach my $key ( $self->db->get_keys ) {
        my $ts       = $self->db->get($key);
        my $days_old = (time - $ts) / 86400;
        next if $days_old < $self->{_args}{penalty_days} * 2;
        $self->delete($key);
        $pruned++;
    }
    $self->log(LOGINFO, "pruned $pruned of $count DB entries");
    return $self->cleanup_and_return(DECLINED);
}