INTERNET-DRAFT Gunnar Lindberg
draft-lindberg-anti-spam-mta-00.txt Chalmers University
Expires April, 1998 of Technology
12 Nov 1997
Anti-Spam Requirements on an SMTP MTA
Abstract
This memo gives a number of technical requirement on SMTP [1] MTA:s
(Mail Transfer Agents, e.g. sendmail) to make them more capable of
reducing the impact of Spam(*).
The intent is that these requirements will help clean up the spam
situation, if applied on enough SMTP MTA:s on the Internet, and that
they should be used as guidelines for the various vendors. We are
fully aware that this is not the final solution but if these
requirements were included, and used, on all Internet SMTP MTA:s,
things would improve considerably and give time to design a more long
term solution. Some ideas are presented in the Addendum.
Status of This Memo
This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas,
and its working groups. Note that other groups may also distribute
working documents as Internet-Drafts. Comments on this draft should
be sent to <anti-spam@chalmers.se>.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
To view the entire list of current Internet-Drafts, please check the
"1id-abstracts.txt" listing contained in the Internet-Drafts Shadow
Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
ftp.isi.edu (US West Coast).
1. Introduction
This memo the result of discussions in an ad hoc group containing
Swedish universities and most Swedish ISPs. Since the group consist
of quite a number of people noone is mentioned explicitly but all
organizations are listed under Acknowledgements below.
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 1]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
1.1 Background
Mass unsolicited electronic mail, often known as Spam(*), has
increased considerably during a short period of time and has become a
serious threat to the Internet email community as a whole. Something
needs to be done fairly quickly.
The problem has several components:
o It is high volume, i.e. people get a lot of such mail in their
mailboxes.
o It is completely "blind", i.e. there is no correlation between
the receivers' areas of interest and the actual mail sent out
(at least if one assumes that not everybody on the Internet is
interested in porno pictures and spam programs...).
o It costs real money for the receivers. Since many receivers
pay for the time to transfer the mailbox from the (dialup) ISP
to their computer they in reality pay real money for this.
o It costs real money for the ISPs. Assume one 10 Kbyte message
sent to 10 000 users with their mailboxes at one ISP host;
that means an unsolicited, unexpected, storage of 100 Mbytes.
State of the art disks, 4 Gbyte, can take 40 such message
floods before they are filled. It is almost impossible to
plan ahead for such "storms".
o Several of the senders are anything but serious, e.g hide
behind false addresses or mail hosts that refuse to receive.
In fact many of the spam-programs show a pride in adding
false info that will "make the ISPs scratch their heads".
It is not uncommon that people who send in protests (often
according to the instructions in the mail) find their mail
addresses added to more lists and sold to other parties.
o It is quite common practice to make use of third party hosts
as relays to get the spam mail sent out to the receivers. This
is almost certainly illegal in all countries, but with the
original sender in the US, the (innocent) relay in Sweden and
the list of receivers back in the US the legal aspects become
almost overhelming.
This memo has no intent to be the final solution to the spam problem.
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 2]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
If, however, enough Internet MTA:s did implement enough of the rules
described below (especially the Non-Relay rules), we would get the
spammers out in the open, where they could be taken care of. Either
pure legal actions would help, or we can block them technically using
other rules described below (since the Non-Relay rules now make them
appear openly, with their own hosts and domains, we can apply various
access filters against them). In reality, a combination of legal and
technical methods is likely to give the best result.
This way, the spam problem could be reduced enough to allow the
Internet community to design and deploy a real and general solution.
1.3 Terminology
Throughout this memo we will use the following terminology:
o "MUST"
This word or the adjective "REQUIRED" means that the item is
an absolute requirement.
o "SHOULD"
This word or the adjective "RECOMMENDED" means that there may
exist valid reasons in particular circumstances to ignore
this item, but the full implications should be understood and
the case carefully weighed before choosing a different course.
o "MAY"
This word or the adjective "OPTIONAL" means that this item is
truly optional. One vendor may choose to include the item
because a particular marketplace requires it or because it
enhances the product, for example; another vendor may omit
the same item.
2 Requirements
Here we first give a brief list of requirements, followed by a more
thorough discussion of each of them.
1) MUST be able to control (deny) usage as Mail Relay.
2) MUST be able to verify "MAIL From" domain (using DNS or using
other means).
3a) MUST be able to refuse mail from a specific "MAIL From" user
(user foo@domain.com).
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 3]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
3b) MUST be able to refuse mail from an entire "MAIL From" domain
(entire domain domain.com).
4) MUST be able to refuse mail from a host / a group of hosts.
5) MUST be able to log all occurences of anti-relay service
denials.
6) SHOULD be able to verify <local-part> in outgoing mail.
7) SHOULD be able to limit ("rate control") mail flow.
8) MUST be able to configure for different Return Codes for
different rules (e.g. 451 Temp Fail vs 551 Fatal Error).
The discussion below often ends up in a need to do various forms of
pattern matching, on domain/host names and on IP addresses/subnets.
It is RECOMMENDED that the data/template for doing so may be supplied
outside of the MTA, e.g. that the pattern matching rules be included
in the MTA but that the actual patterns may be in an external file.
Of course all string matching MUST be non case sensitive.
2.1 Control use as Mail Relay
The MTA MUST be able to deny/accept to be Mail Relay based on a
combination of:
o "RCPT To" addresses (domains).
o Sending host's FQN hostname.
o Sending host's IP address.
In all cases the configuration MUST support wild cards (for FQNs) and
masks (for IP addresses), i.e. domain.com or *.domain.com; 10.0.0.0/8
or 192.168.1.0/24.
The configuration SHOULD allow for the decision data to be supplied
by an external source, e.g. text file or dbm database.
2.2 Verify "MAIL From"
The MTA MUST be able to perform a simple "sanity check" of the "MAIL
From" domain and refuse to receive mail if that domain is
nonexistent. To overcome temporary errors/problems in the DNS, Return
Codes like 4xx are recommended; however the MTA MAY allow for Return
Codes that show real DNS state - 4xx for temporary problems and 5xx
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 4]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
for NonExistent domain.
2.3 Refuse "MAIL From"
The MTA MUST be able to refuse to receive mail from a specific "MAIL
From" user (foo@domain.com) or from an entire "MAIL From" domain
(domain.com). In general this kind of rules are easily overcome by
the spammers changing "MAIL From" every so often, but the ability to
block a certain user or a certain domain is quite helpful while an
attack has just been discovered and is ongoing.
Given a combination of "domain sanity check", "non-relay" and
"<local-part> verification" the method may be more useful.
2.4 Refuse mail from a group of hosts
The MTA MUST be able to accept or refuse mail from a specific host or
from a group of hosts.
It is REQUIRED that the MTA support decisions based on FQN hostnames
(host.domain.com), on domain names with wild cards (*.domain.com), on
individual IP addresses (10.11.12.13) or on IP addresses with a
prefix length (10.0.0.0/8, 192.168.1.0/24).
It is also REQUIRED that these decision rules can be combined, e.g.
accept host.domain.com
refuse *.domain.com
accept 10.11.12.13
accept 192.168.1.0/24
refuse 10.0.0.0/8
IP-address/length is REQUIRED. However, early implementations with
wild cards, e.g. 10.11.12.*, instead are of course much better then
without.
2.5 Log service denials
The MTA MUST be able to log all service denial actions based on the
anti-spam rules. The log entries SHOULD contain at least:
o Refuse information, i.e. why the request was refused ("Mail
From", Relaying Denied, Spam User, Spam Host, etc).
o "RCPT To" addresses (domains).
o Offending host's FQN hostname.
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 5]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
o Offending host's IP address.
o Other relevant information (e.g. given during the SMTP
dialogue, before we decided to refuse the request).
2.6 Verify <local-part>
The MTA SHOULD allow that outgoing mail has its <local-part> verified
so that the sender name is a real user or an existing alias. This is
basically to protect the rest of the Internet from various "typos"
(MAIL From <fo0bar@domain.com>)
and malicious users
(MAIL From <I.am.unknown.to.you.he.he@domain.com>).
As always this can be overcome by spammers really wanting to do so,
but with more strict rules for relaying it becomes harder and harder.
In fact, catching "typos" at the initial (and official) mail relay
should in itself be enough motivation for this requirement.
2.7 Rate Control
The MTA SHOULD make it possible for the mail host to control the rate
with which mail is sent or received. The idea is twofold:
o If we happen to have a spammer legally using our mail host
(i.e. a legal mail user with an existing, legal, account that
sends out spam - be sending spam illegal or not) we may want
to reduce the speed with which he sends out mail. This is not
without controversy and must be used with extreme care, but
it may protect the rest of the Internet from him.
o If we are under a spam attack it may help us considerably just
being able to slow down the incoming mail rate for that
particular user/host.
For receiving mail the MTA SHOULD be able to do so based on "MAIL
From" user, "MAIL From" domain, sending host (name/address) or a
combination.
In both cases 4xx Return Code is recommended.
2.8 Return Codes
Our basic assumption is that refuse/accept is handled at the SMTP
layer and that an MTA that decides to refuse a message should do so
while still in the SMTP dialogue. First this means that we do not
have to store a copy of a message we later decide to refuse and
second our responsibility for that message is low or none - since we
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 6]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
have not read it we leave it to the sender to handle the error.
The MTA MUST be configurable to use different Return Codes for
different rules (e.g. 451 Temp Fail vs 551 Fatal Error). Now, this
must be used with great care:
o 5xx
is a Fatal Error and results in the mail transfer being
terminated and the mail returned to sender. For some events,
like "Denied - you're on the spammer's list", this is probably
the correct response and the right thing to say.
A mistake in configuration, however, may cause valid mail to
bounce back to the sender, which may be quite unfortunate.
o 4xx
is a Temporary Error and results in the mail transfer being
put back on queue again and a new attempt being made later.
Therefore, configuration mistakes are much less fatal, which
really helps out.
A 4xx response make the spammer's host re-queue the mail and
if it really is his own host who gets to do this, it is
probably a good thing. If, on the other hand, he is using
someone else as Relay Host, all the mail being queued is a
fairly strong evidence that something illegal is going on.
4xx, Temporary Error, is almost always the recommended Return Code.
However, 4xx Temporary Errors may have unexpected interaction with
MX-records. If the receiving domain has several MX records and the
lowest preference MX-host refuses to receive mail from a certain
"MAIL From" domain with a "451" Response Code, the sending host may
choose to use the other host on the MX list.
If that other MX host does not have the same refuse-list, it will of
course accept the mail, only to find that the final host still
refuses to receive that piece of mail ("MAIL From"). I.e. our intent
was to make the offending mail stay at the offending sender's host
and fill up his mqueue disk, but it all ended up at our friend, the
next lowest preference MX-host.
3. sendmail example
The main purpose of the memo is to define a set of requirements that
will help reduce spam. It is not intended to describe how to do that
for any particular MTA. However, many of us use and are familiar with
sendmail [2] and therefore we provide some hints; these require late
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 7]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
versions of sendmail-8.8.* (when this was written, 12 Nov 1997,
sendmail-8.8.8 was the latest). The example neither makes a claim to
solve the problem nor to be correct and you use it at your own risk.
These examples all use Return Code "451", Temp Fail. Please read that
section above once more and verify that you will not hit your
friends, your next lowest preference MX-hosts, that may have
different rules.
(NB sendmail makes a difference between <tab> and <space> used as
separators; if you use cut-and-paste from this memo you are likely
to get <space> everywhere).
##################################################################
# Deny spammers, single users or entire domains
Kspammers dbm -o /etc/mail/spammers.db
#
Scheck_mail
# check for valid domain name (name exists within DNS)
R$* $: $>3 $1
ifdef(`_NO_CANONIFY_', `
# pass to name server to make hostname canonical
# (done here if we have "nocanonify", in S3-S96 otherwise)
R$* < @ $* $~P > $* $: $1 < @ $[ $2 $3 $] > $4
')
R$* < @ $*domain.com .> $: $1<@$2domain.com> sigh
R$* < @ $+ . > $: <$1@$2> OK
R$* < @ $+ > $#error $: 451 Domain must resolve
#
#R$* $@ OK return from here if you
# have no spammers check
###
# check user@dom.ain and dom.ain versus spammers database
R<$* @ $+> $: $1<@$2> re-focus
# user@dom.ain
R$* < @ $+ > $: $1<@$2><$(spammers $1@$2 $:OK $)>
R$* < @ $+ ><OK> $: $1<@$2>
R$* < @ $+ ><$* @ $+> $#error $: 451 Denied due to spam-list
# dom.ain
R$* < @ $+ > $: $1<@$2><$(spammers $2 $:OK $)>
R$* < @ $+ ><OK> $: $1<@$2>
R$* < @ $+ ><$+> $#error $: 451 Denied due to spam-list
# You may consider more tests here, e.g. verify/check against
# $(dequote "" $&{client_name} $)
# $(dequote "" $&{client_addr} $)
R$* $@ OK
##################################################################
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 8]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
##################################################################
# In "class D" you enter domains and hosts for two purposes
# 1) You accept to relay mail TO them (MX).
# 2) You accept to relay mail FROM them (SmartHost).
# In both cases, this is "recursive", i.e. foo.se -> *.foo.se
#FD/etc/mail/sendmail.cD
CD
#
# In "class B" you enter the "exceptionally bad guys", i.e. hosts
# that you want to deny relay from regardless - this may be hosts
# that do not have enough filters or any other reason. Be careful.
# FB/etc/mail/sendmail.cB
CB
#
Scheck_rcpt
# first get rid of a%b@c type addresses
R< $+ % $+ > < $1 @ $2 >
R< $+ @ $+ @ $+ > < $1 @ $2 >
# "RCPT To" that terminates locally is OK
R< $+ @ $=w > $@ OK
R< $+ @ $=w . > $@ OK
R<$-> $@ OK
# "RCPT To" for accepted domains is OK
R< $+ @ $=D > $@ OK
R< $+ @ $=D . > $@ OK
R< $+ @ $+ . $=D > $@ OK
R< $+ @ $+ . $=D . > $@ OK
# get sender host's name
R$* $: $(dequote "" $&{client_name} $)
# if it's me it's OK
R$=w $@ OK
R$@ $@ OK
# exceptionally bad guys...
R$=B $#error $:"451 Relaying Denied, " $1
# do this in case you want "bad with recursion"
#R$+$=B $#error $:"451 Relaying Denied, " $1$2
# an accepted host is OK (with "recursion")
R$=D $@ OK
R$+$=D $@ OK
# anything else is bogus
R$* $#error $:"451 Relaying Denied, " $1
##################################################################
4. Security Considerations
This memo does not consider security in its most general form, taken
literally, but it certainly does so in a broader and more general
sense - protecting Internet users from mass mail abuse does have some
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 9]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
security implications.
5. Acknowledgements
This memo is the result of discussions in an ad hoc group of Swedish
ISPs and Universities. Without hope to mention everyone we simply
give the domain names for the ISPs here: algonet.se, global-ip.net,
pi.se, swip.net, telia.net and udac.se; the Universities stay
anonymous.
6. References
[1] Jonathan B. Postel "Simple Mail Transfer Protocol; RFC821",
August 1982. Available via anonymous ftp at
ftp://ds.internic.net/rfc/rfc821.txt
[2] sendmail Home Page. http://www.sendmail.org
* Spam (R) is a registered trademark of a meat product made by
Hormel. Use of the term Spam in the Internet community comes
from a Monty Python sketch and is almost Internet folklore.
The term Spam is usually meant negative, however this is not
in any way intended to describe the Hormel product.
7. Addendum - Future work
7.1. Impact on SMTP UAs
Despite this memo is about MTA:s and the requirements put on them,
some of what is done here falls back to the UA:s (User Agents, the
"ordinary mail programs").
An UA does two things:
1) Reads mail from a mailbox and prints on the screen.
This typically uses a protocol like POP, IMAP or NFS.
2) Reads text from the keyboard and hands that over to the
mailbox MTA for delivery as a piece of mail. This typically
uses the SMTP protocol, i.e. the same protocol that is used
between MTA:s.
When MTA:s now start to implement various anti-relay filters as
described above, a UA on a portable laptop host may get responses
like "Relaying Denied" just because it happens to use IP addresses
within the wrong range or "name". To overcome this, one should use
some other protocol to transport the piece of mail from the UA to the
(Relay) MTA.
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 10]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
This would do away with the need for SMTP between everybody and will
allow more strict authentication between SMTP MTA:s.
7.2. Anti-spam filters
Today, when an MTA with spam filter receives a message it has two
possibilities, reject or accept the message. The mechanism used by
sendmail and other MTA software is to check in a policy filter list
whether the sender, recipient and sending host is accepted. If the
message is not destined for a local user, a policy filter is checked
to see whether we relay for the sending host, whether we accept the
sender of the message and whether we accept the recipient of the
message. If the message is destined for a local user, we do the same
thing.
This way, we actually filter messages that is routed through us. We
also apply the same policy filter, not only to non-local users but to
all local users as well.
Users probably have different opinions about what addresses should be
in the policy filter list, a central filter will be considered by
some as a nuisance that should be removed. In this view it is not
unlikely that the system administrator, who is responsible for the
maintenance of the policy filter list, will hesitate to add addresses
to the list and thus give continued access to the spammers to abuse
the MTA. Less blunt tools than central filtering should be used to
fight spam.
7.2.1. Personal anti-spam filters
If users could design their own policy filter, for what should be
delivered to their mailboxes, the system manager is less likely seen
as a despotic censor. If this concept is accepted, much more freedom
is given when designing the filter and on what principles it should
be built. Some users may want a narrow filter while others want wide.
With personal filters it will be up to the user to decide how the
filter should function. The three parameters that can be used for
designing a filter should be addresses that should be rejected,
addresses that should be accepted and an indication if to accept or
reject by default.
It could function like today, a list with addresses or domains that
is rejected, default is to accept. It could function the opposite
way, a list with addresses or domains that is accepted, default is to
reject.
In all cases some sort of interface must be developed, to make the
user able to configure their personal policy filter. It may not be
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 11]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
very different from how the owner of a listserv list manages the
list, all configuration performed by email, or it could be done by
using a web browser.
If default is to reject messages, a mechanism for senders to request
to be members of the accept list needs to be developed. One thought
is to create a user-request address that will, when sent to, give the
user a hint to add the sender to the accept list.
Some redesigning of the MTA is necessary for this type of filtering
to be possible. It would then need to hear out the RCPT TO before
checking the policy filter of the recipient and deciding whether to
accept or reject the message. However, this is not a very complicated
function to implement. Sendmail already checks if users have .forward
files in their home directories, checking for a policy filter list
would not be much more trouble.
This kind of personal spam filter could be implemented today, as it
does not require global implementation to function. It will benefit
the users who does not have to accept spam, as well as the system
administrator who does not need to take the responsibility of
deciding what to accept or not. The challenge is to design the user
interface in such a way that users understand how they should operate
the filter.
7.2.2. Anti-spam filters return codes
One open question here is Return Codes - should the receiving MTA
indicate to the sending MTA that this mail was not delivered due to
some personal anti-spam filter or should it silently drop it? Should
it return 4xx or 5xx - and should that be up to the user?
First of all, it would be both rude and unfair to drop mail without
any indication, so there has to be 4xx or 5xx. Since an ordinary user
will never be able to forsee, or understand, the interactions between
several MX hosts Return Code "551 - Denied due to spam list" is
recommended.
7.3. Mandatory signing of messages
Today, the Internet is quickly becoming the groupware of the masses.
It could be argued that what characterizes the Internet, in
opposition to a traditional groupware system, is that on the Internet
users are in practice anonymous. One of the main problems with
spamming is that the sender almost always tries to achieve anonymity,
and using todays standard technology they have no problems doing
that.
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 12]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997
Spammers should not be given a chance to hide, they should be forced
to reveal their identity. With their identities in public, measures
can be taken to make them stop sending spam or to not accept email
from them.
This can be achieved if signing of messages with an electronic
signature should become mandatory. It would make spamming impossible,
or at least futile. As a side effect, assigning certificates to
Internet users would benefit the authentication procedures for other
Internet services.
It is important that this transition is made as smooth as possible,
methods to handle the intermediate stage must be developed.
There is much work going on in creating the standards necessary to
make this possible and effective.
The MTA should be given a chance to check the signature, using ESMTP,
and reject the message if incorrect, see draft-myers-smtp-auth-XX and
draft-myers-auth-sasl-XX.
Public keys and certificates needs to be signed and distributed
across the Internet, see RFC2065.
Everything should be knitted together and organized, see draft-
eastlake-muse-XX.
Editor's Address
Gunnar Lindberg
Computer Communications Group
Chalmers University of Technology
S-412 96 Gothenburg, SWEDEN,
Phone +46 31 772 5913
FAX +46 31 772 5922
lindberg@cdg.chalmers.se
Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 13]