qpsmtpd/plugins/ident/geoip

498 lines
14 KiB
Plaintext
Raw Normal View History

#!perl -w
=head1 NAME
geoip - provide geographic information about mail senders.
POD corrections, additional tests, plugin consistency on files in plugins dir: fixed a number of POD errors formatted some # comments into POD removed bare 1; (these are plugins, not perl modules) most instances of this were copy/pasted from a previous plugin that had it removed instances of # vim ts=N ... they weren't consistent, many didn't match .perltidyrc on modules that failed perl -c tests, added 'use Qpsmtpd::Constants;' Conflicts: plugins/async/check_earlytalker plugins/async/dns_whitelist_soft plugins/async/dnsbl plugins/async/queue/smtp-forward plugins/async/require_resolvable_fromhost plugins/async/rhsbl plugins/async/uribl plugins/auth/auth_checkpassword plugins/auth/auth_cvm_unix_local plugins/auth/auth_flat_file plugins/auth/auth_ldap_bind plugins/auth/auth_vpopmail plugins/auth/auth_vpopmail_sql plugins/auth/authdeny plugins/check_badmailfromto plugins/check_badrcptto_patterns plugins/check_bogus_bounce plugins/check_earlytalker plugins/check_norelay plugins/check_spamhelo plugins/connection_time plugins/dns_whitelist_soft plugins/dnsbl plugins/domainkeys plugins/greylisting plugins/hosts_allow plugins/http_config plugins/logging/adaptive plugins/logging/apache plugins/logging/connection_id plugins/logging/transaction_id plugins/logging/warn plugins/milter plugins/queue/exim-bsmtp plugins/queue/maildir plugins/queue/postfix-queue plugins/queue/smtp-forward plugins/quit_fortune plugins/random_error plugins/rcpt_map plugins/rcpt_regexp plugins/relay_only plugins/require_resolvable_fromhost plugins/rhsbl plugins/sender_permitted_from plugins/spamassassin plugins/tls plugins/tls_cert plugins/uribl plugins/virus/aveclient plugins/virus/bitdefender plugins/virus/clamav plugins/virus/clamdscan plugins/virus/hbedv plugins/virus/kavscanner plugins/virus/klez_filter plugins/virus/sophie plugins/virus/uvscan
2012-04-08 02:11:16 +02:00
=head1 SYNOPSIS
2014-11-04 23:43:54 +01:00
Use MaxMind's GeoIP databases and the GeoIP2 or Geo::IP perl modules to report
geographic information about incoming connections.
=head1 DESCRIPTION
Save geographic information about the sender in the following connection notes:
geoip_country - 2 char country code
geoip_country_name - english name of country
geoip_continent - 2 char continent code
geoip_city - english name of city
geoip_distance - distance in kilometers
geoip_asn - network number
And adds entries like this to your logs:
(connect) ident::geoip: NA, US, United States, 1319 km
(connect) ident::geoip: AS, IN, India, 13862 km
(connect) ident::geoip: fail: no results
(connect) ident::geoip: NA, CA, Canada, 2464 km
(connect) ident::geoip: NA, US, United States, 2318 km
(connect) ident::geoip: AS, PK, Pakistan, 12578 km
(connect) ident::geoip: AS, TJ, Tajikistan, 11965 km
(connect) ident::geoip: EU, AT, Austria, 8745 km
(connect) ident::geoip: AS, IR, Iran, Islamic Republic of, 12180 km
(connect) ident::geoip: EU, BY, Belarus, 9030 km
(connect) ident::geoip: AS, CN, China, 11254 km
(connect) ident::geoip: NA, PA, Panama, 3163 km
Calculating the distance has three prerequsites:
1. The MaxMind city database (free or subscription)
2. The Math::Complex perl module
3. The IP address of this mail server (see CONFIG)
Other plugins can utilize the geographic notes to alter the
connection, reject, greylist, etc.
=head1 CONFIG
The following options can be appended in this plugins config/plugins entry.
=head2 distance <IP Address>
Enables geodesic distance calculation. Will calculate the distance "as the
crow flies" from the remote mail server. Accepts a single argument, the IP
address to calculate the distance from. This will typically be the public
IP of your mail server.
ident/geoip [ distance 192.0.1.5 ]
Default: none. (no distance calculations)
2013-03-23 06:03:28 +01:00
=head2 too_far <distance in km>
Assign negative karma to connections further than this many km.
Default: none
=head2 db_dir </path/to/GeoIP>
The path to the GeoIP database directory.
ident/geoip [ db_dir /etc/GeoIP ]
Default: /usr/local/share/GeoIP
=head2 add_headers <true|false>
Add message headers with GeoIP data
ident/geoip [ add_headers (true|false) ]
Default: true
=head1 LIMITATIONS
The distance calculations are more concerned with being fast than accurate.
The MaxMind location data is collected from whois and is of limited accuracy.
MaxMind offers more accurate data for a fee.
For distance calculations, the earth is considered a perfect sphere. In
reality, it is not. Accuracy should be within 1%.
This plugin does not update the GeoIP databases. You may want to.
=head1 CHANGES
2014-06 - Matt Simerson - added GeoIP2 support
2012-06 - Matt Simerson - added GeoIP City support, continent, distance
2012-05 - Matt Simerson - added geoip_country_name note, added tests
=head1 SEE ALSO
MaxMind: http://www.maxmind.com/
Databases: http://geolite.maxmind.com/download/geoip/database
It may become worth adding support for Geo::IPfree, which uses another
data source: http://software77.net/geo-ip/
=head1 ACKNOWLEDGEMENTS
2014-11-05 22:50:19 +01:00
MaxMind - the packager and distributor of the free GeoIP data
Stevan Bajic, the DSPAM author, who suggested SNARE, which describes using
geodesic distance to determine spam probability. The research paper on SNARE
can be found here:
http://smartech.gatech.edu/bitstream/handle/1853/25135/GT-CSE-08-02.pdf
=cut
use strict;
use warnings;
2014-11-04 23:43:54 +01:00
use lib 'lib';
use Qpsmtpd::Constants;
2014-11-04 23:43:54 +01:00
#use GeoIP2; # eval'ed in register()
#use Geo::IP; # eval loaded if GeoIP2 doesn't
#use Math::Trig; # eval'ed in set_distance_gc
sub register {
my ($self, $qp, @args) = @_;
$self->log(LOGERROR, "Bad arguments") if @args % 2;
$self->{_args} = {@args};
$self->{_args}{db_dir} ||= '/usr/local/share/GeoIP';
$self->load_geoip() or return;
return if $self->{_args}{add_headers} !~ /false/i;
2014-11-07 23:30:54 +01:00
$self->register_hook( data_post => 'add_headers' );
}
sub load_geoip {
2014-11-07 23:22:49 +01:00
my ( $self ) = @_;
$self->load_geoip2() and return 1;
$self->load_geoip1() and return 1;
2014-11-07 23:34:51 +01:00
return 0;
2014-11-05 22:50:19 +01:00
}
2014-11-05 22:50:19 +01:00
sub load_geoip1 {
my $self = shift;
eval 'use Geo::IP';
if ($@) {
warn "could not load Geo::IP";
$self->log(LOGERROR, "could not load Geo::IP");
return;
}
$self->open_geoip_db();
# Note that opening the GeoIP DB only in register has caused problems before:
# https://github.com/smtpd/qpsmtpd/commit/29ea9516806e9a8ca6519fcf987dbd684793ebdd#plugins/ident/geoip
# Opening the DB anew for every connection is horribly inefficient.
# Instead, attempt to reopen upon connect if the DB connection fails.
$self->init_my_country_code();
2014-11-11 01:29:59 +01:00
$self->register_hook('connect', 'geoip_lookup');
return 1;
}
2014-11-05 22:50:19 +01:00
sub load_geoip2 {
my $self = shift;
eval 'use GeoIP2::Database::Reader';
if ($@) {
$self->log(LOGERROR, "could not load GeoIP2");
return;
}
$self->log(LOGINFO, "GeoIP2 loaded");
eval {
$self->{_geoip2_city} = GeoIP2::Database::Reader->new(
file => $self->{_args}{db_dir} . '/GeoLite2-City.mmdb',
);
};
if ($@) {
$self->log(LOGERROR, "unable to load GeoLite2-City.mmdb");
}
eval {
$self->{_geoip2_country} = GeoIP2::Database::Reader->new(
file => $self->{_args}{db_dir} . '/GeoLite2-Country.mmdb',
);
};
if ($@) {
$self->log(LOGERROR, "unable to load GeoLite2-Country.mmdb");
}
if ($self->{_geoip2_city} || $self->{_geoip2_country}) {
$self->register_hook('connect', 'geoip2_lookup');
return 1;
}
return;
}
2014-11-07 23:30:54 +01:00
sub add_headers {
my ( $self, $txn ) = @_;
for my $h (qw( Country Continent City ASN )) {
2014-11-07 23:30:54 +01:00
my $note = lc "geoip_$h";
next if ! $self->connection->notes($note);
$txn->header->delete("X-GeoIP-$h");
$txn->header->add( "X-GeoIP-$h", $self->connection->notes($note), 0 );
}
}
sub geoip2_lookup {
my $self = shift;
my $ip = $self->qp->connection->remote_ip;
2014-11-11 07:49:46 +01:00
return DECLINED if $self->is_localhost($ip);
if ($self->{_geoip2_city}) {
my $city_rec = $self->{_geoip2_city}->city(ip => $ip);
if ($city_rec) {
$self->qp->connection->notes('geoip_country', $city_rec->country->iso_code());
$self->qp->connection->notes('geoip_country_name', $city_rec->country->name());
$self->qp->connection->notes('geoip_continent', $city_rec->continent->code());
$self->qp->connection->notes('geoip_city', $city_rec->city->name());
$self->qp->connection->notes('geoip_asn', $city_rec->traits->autonomous_system_number());
return DECLINED;
}
}
if ($self->{_geoip2_country}) {
my $country_rec = $self->{_geoip2_country}->country(ip => $ip);
if ($country_rec) {
$self->qp->connection->notes('geoip_country', $country_rec->country->iso_code());
$self->qp->connection->notes('geoip_country_name', $country_rec->country->name());
$self->qp->connection->notes('geoip_continent', $country_rec->continent->code());
};
}
return DECLINED;
}
2014-11-04 23:43:54 +01:00
sub geoip_lookup {
my $self = shift;
2014-11-11 07:49:46 +01:00
return DECLINED if $self->is_localhost($self->qp->connection->remote_ip);
# reopen the DB if Geo::IP failed due to DB update
$self->open_geoip_db();
my $c_code = $self->set_country_code() or do {
$self->log(LOGINFO, "skip, no results");
return DECLINED;
};
2014-11-05 23:23:06 +01:00
$self->set_asn();
my $c_name = $self->set_country_name();
my ($city, $continent_code, $distance) = '';
if ($self->{_my_country_code}) {
$continent_code = $self->set_continent($c_code);
$city = $self->set_city_gc();
$distance = $self->set_distance_gc();
}
my @msg_parts;
2014-11-04 23:43:54 +01:00
if ($continent_code && $continent_code ne '--') {
push @msg_parts, $continent_code;
};
push @msg_parts, $c_code if $c_code;
#push @msg_parts, $c_name if $c_name;
push @msg_parts, $city if $city;
if ($distance) {
2013-03-23 06:03:28 +01:00
push @msg_parts, "\t$distance km";
if ($self->{_args}{too_far} && $distance > $self->{_args}{too_far}) {
$self->adjust_karma(-1);
}
}
$self->log(LOGINFO, join(", ", @msg_parts));
return DECLINED;
}
sub open_geoip_db {
my $self = shift;
# this might detect if the DB connection failed. If not, this is where
# to add more code to do it.
return if (defined $self->{_geoip_city} || defined $self->{_geoip});
# The methods for using GeoIP work differently for the City vs Country DB
# save the handles in different locations
my $db_dir = $self->{_args}{db_dir};
foreach my $db (qw/ GeoIPCity GeoLiteCity /) {
next if !-f "$db_dir/$db.dat";
2014-11-05 22:50:19 +01:00
$self->log(LOGINFO, "using db $db");
$self->{_geoip_city} = Geo::IP->open("$db_dir/$db.dat");
}
2014-11-05 22:50:19 +01:00
if (-f "$db_dir/GeoIPASNum.dat") {
$self->log(LOGINFO, "using GeoIPASNum");
$self->{GeoIPASNum} = Geo::IP->open("$db_dir/GeoIPASNum.dat");
}
if (-f "$db_dir/GeoIPASNumv6.dat") {
$self->log(LOGINFO, "using GeoIPASNumv6");
$self->{GeoIPASNumv6} = Geo::IP->open("$db_dir/GeoIPASNumv6.dat");
}
# can't think of a good reason to load country if city data is present
if (!$self->{_geoip_city}) {
$self->log(LOGDEBUG, "using default db");
$self->{_geoip} = Geo::IP->new(); # loads default Country DB
}
}
sub init_my_country_code {
my $self = shift;
my $ip = $self->{_args}{distance} or return;
$self->{_my_country_code} = $self->get_country_code($ip);
}
sub set_country_code {
my $self = shift;
my $remote_ip = $self->qp->connection->remote_ip;
2014-11-04 23:43:54 +01:00
my $code = $self->{_geoip_city}
? $self->get_country_code_gc($remote_ip)
: $self->get_country_code($remote_ip);
return if ! $code;
$self->qp->connection->notes('geoip_country', $code);
return $code;
}
sub get_country_code {
my $self = shift;
my $ip = shift || $self->qp->connection->remote_ip;
2014-11-04 23:43:54 +01:00
if ($self->{_geoip_city}) {
return $self->get_country_code_gc($ip);
};
return $self->{_geoip}->country_code_by_addr($ip);
}
sub get_country_code_gc {
my $self = shift;
my $ip = shift || $self->qp->connection->remote_ip;
$self->{_geoip_record} = $self->{_geoip_city}->record_by_addr($ip)
or return;
return $self->{_geoip_record}->country_code;
}
sub set_country_name {
my $self = shift;
my $remote_ip = $self->qp->connection->remote_ip;
2014-11-04 23:43:54 +01:00
my $name = $self->{_geoip_city}
? $self->get_country_name_gc($remote_ip)
: $self->{_geoip}->country_name_by_addr($remote_ip);
return if ! $name;
$self->qp->connection->notes('geoip_country_name', $name);
return $name;
}
2014-11-04 23:43:54 +01:00
sub get_country_name_gc {
my $self = shift;
return if !$self->{_geoip_record};
2014-11-04 23:43:54 +01:00
return $self->{_geoip_record}->country_name();
}
sub set_continent {
2014-11-04 23:43:54 +01:00
my ($self, $country_code) = @_;
$country_code or return;
my $continent = $self->{_geoip_city}
? $self->get_continent_gc()
: $self->{_geoip}->continent_code_by_country_code($country_code);
$continent or return;
$self->qp->connection->notes('geoip_continent', $continent);
return $continent;
}
2014-11-04 23:43:54 +01:00
sub get_continent_gc {
my $self = shift;
return if !$self->{_geoip_record};
2014-11-04 23:43:54 +01:00
return $self->{_geoip_record}->continent_code();
}
sub set_asn {
2014-11-05 22:50:19 +01:00
my ($self, $ip) = @_;
$ip ||= $self->qp->connection->remote_ip;
2014-11-05 22:50:19 +01:00
if (Qpsmtpd::Base->is_ipv6($ip)) {
return $self->set_asn_ipv6($ip);
}
return if ! $self->{GeoIPASNum};
2014-11-05 22:50:19 +01:00
my $asn = $self->{GeoIPASNum}->name_by_addr($ip) or return;
2014-11-05 22:52:21 +01:00
if ('AS' eq substr($asn, 0, 2)) {
$asn = substr($asn, 2);
}
$self->qp->connection->notes('geoip_asn', $asn);
return $asn;
}
2014-11-05 22:50:19 +01:00
sub set_asn_ipv6 {
my ($self, $ip) = @_;
$ip ||= $self->qp->connection->remote_ip;
return if ! $self->{GeoIPASNumv6};
my $asn = $self->{GeoIPASNumv6}->name_by_addr_v6($ip) or return;
$self->qp->connection->notes('geoip_asn', $asn);
return $asn;
}
sub set_city_gc {
my $self = shift;
return if !$self->{_geoip_record};
my $city = $self->{_geoip_record}->city() or return;
$self->qp->connection->notes('geoip_city', $city);
return $city;
}
sub set_distance_gc {
my $self = shift;
return if !$self->{_geoip_record};
my ($self_lat, $self_lon) = $self->get_my_lat_lon() or return;
my ($sender_lat, $sender_lon) = $self->get_sender_lat_lon() or return;
eval 'use Math::Trig qw(great_circle_distance deg2rad)';
if ($@) {
$self->log(LOGERROR,
"can't calculate distance, Math::Trig not installed");
return;
}
# Notice the 90 - latitude: phi zero is at the North Pole.
sub NESW { deg2rad($_[0]), deg2rad(90 - $_[1]) }
my @me = NESW($self_lon, $self_lat);
my @sender = NESW($sender_lon, $sender_lat);
my $km = great_circle_distance(@me, @sender, 6378);
$km = sprintf("%.0f", $km);
$self->qp->connection->notes('geoip_distance', $km);
#$self->log( LOGINFO, "distance $km km");
return $km;
}
sub get_my_lat_lon {
my $self = shift;
return if !$self->{_geoip_city};
if ($self->{_latitude} && $self->{_longitude}) {
return $self->{_latitude}, $self->{_longitude}; # cached
}
my $ip = $self->{_args}{distance} or return;
my $record = $self->{_geoip_city}->record_by_addr($ip) or do {
$self->log(LOGERROR, "no record for my Geo::IP location");
return;
};
$self->{_latitude} = $record->latitude();
$self->{_longitude} = $record->longitude();
if (!$self->{_latitude} || !$self->{_longitude}) {
$self->log(LOGNOTICE, "could not get my lat/lon");
}
return $self->{_latitude}, $self->{_longitude};
}
sub get_sender_lat_lon {
my $self = shift;
my $lat = $self->{_geoip_record}->latitude();
my $lon = $self->{_geoip_record}->longitude();
if (!$lat || !$lon) {
$self->log(LOGNOTICE, "could not get sender lat/lon");
return;
}
return $lat, $lon;
}