INTERNET-DRAFT Gunnar Lindberg draft-lindberg-anti-spam-mta-00.txt Chalmers University Expires April, 1998 of Technology 12 Nov 1997 Anti-Spam Requirements on an SMTP MTA Abstract This memo gives a number of technical requirement on SMTP [1] MTA:s (Mail Transfer Agents, e.g. sendmail) to make them more capable of reducing the impact of Spam(*). The intent is that these requirements will help clean up the spam situation, if applied on enough SMTP MTA:s on the Internet, and that they should be used as guidelines for the various vendors. We are fully aware that this is not the final solution but if these requirements were included, and used, on all Internet SMTP MTA:s, things would improve considerably and give time to design a more long term solution. Some ideas are presented in the Addendum. Status of This Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Comments on this draft should be sent to <anti-spam@chalmers.se>. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). 1. Introduction This memo the result of discussions in an ad hoc group containing Swedish universities and most Swedish ISPs. Since the group consist of quite a number of people noone is mentioned explicitly but all organizations are listed under Acknowledgements below. Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 1]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 1.1 Background Mass unsolicited electronic mail, often known as Spam(*), has increased considerably during a short period of time and has become a serious threat to the Internet email community as a whole. Something needs to be done fairly quickly. The problem has several components: o It is high volume, i.e. people get a lot of such mail in their mailboxes. o It is completely "blind", i.e. there is no correlation between the receivers' areas of interest and the actual mail sent out (at least if one assumes that not everybody on the Internet is interested in porno pictures and spam programs...). o It costs real money for the receivers. Since many receivers pay for the time to transfer the mailbox from the (dialup) ISP to their computer they in reality pay real money for this. o It costs real money for the ISPs. Assume one 10 Kbyte message sent to 10 000 users with their mailboxes at one ISP host; that means an unsolicited, unexpected, storage of 100 Mbytes. State of the art disks, 4 Gbyte, can take 40 such message floods before they are filled. It is almost impossible to plan ahead for such "storms". o Several of the senders are anything but serious, e.g hide behind false addresses or mail hosts that refuse to receive. In fact many of the spam-programs show a pride in adding false info that will "make the ISPs scratch their heads". It is not uncommon that people who send in protests (often according to the instructions in the mail) find their mail addresses added to more lists and sold to other parties. o It is quite common practice to make use of third party hosts as relays to get the spam mail sent out to the receivers. This is almost certainly illegal in all countries, but with the original sender in the US, the (innocent) relay in Sweden and the list of receivers back in the US the legal aspects become almost overhelming. 1.2 Scope This memo has no intent to be the final solution to the spam problem. Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 2]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 If, however, enough Internet MTA:s did implement enough of the rules described below (especially the Non-Relay rules), we would get the spammers out in the open, where they could be taken care of. Either pure legal actions would help, or we can block them technically using other rules described below (since the Non-Relay rules now make them appear openly, with their own hosts and domains, we can apply various access filters against them). In reality, a combination of legal and technical methods is likely to give the best result. This way, the spam problem could be reduced enough to allow the Internet community to design and deploy a real and general solution. 1.3 Terminology Throughout this memo we will use the following terminology: o "MUST" This word or the adjective "REQUIRED" means that the item is an absolute requirement. o "SHOULD" This word or the adjective "RECOMMENDED" means that there may exist valid reasons in particular circumstances to ignore this item, but the full implications should be understood and the case carefully weighed before choosing a different course. o "MAY" This word or the adjective "OPTIONAL" means that this item is truly optional. One vendor may choose to include the item because a particular marketplace requires it or because it enhances the product, for example; another vendor may omit the same item. 2 Requirements Here we first give a brief list of requirements, followed by a more thorough discussion of each of them. 1) MUST be able to control (deny) usage as Mail Relay. 2) MUST be able to verify "MAIL From" domain (using DNS or using other means). 3a) MUST be able to refuse mail from a specific "MAIL From" user (user foo@domain.com). Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 3]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 3b) MUST be able to refuse mail from an entire "MAIL From" domain (entire domain domain.com). 4) MUST be able to refuse mail from a host / a group of hosts. 5) MUST be able to log all occurences of anti-relay service denials. 6) SHOULD be able to verify <local-part> in outgoing mail. 7) SHOULD be able to limit ("rate control") mail flow. 8) MUST be able to configure for different Return Codes for different rules (e.g. 451 Temp Fail vs 551 Fatal Error). The discussion below often ends up in a need to do various forms of pattern matching, on domain/host names and on IP addresses/subnets. It is RECOMMENDED that the data/template for doing so may be supplied outside of the MTA, e.g. that the pattern matching rules be included in the MTA but that the actual patterns may be in an external file. Of course all string matching MUST be non case sensitive. 2.1 Control use as Mail Relay The MTA MUST be able to deny/accept to be Mail Relay based on a combination of: o "RCPT To" addresses (domains). o Sending host's FQN hostname. o Sending host's IP address. In all cases the configuration MUST support wild cards (for FQNs) and masks (for IP addresses), i.e. domain.com or *.domain.com; 10.0.0.0/8 or 192.168.1.0/24. The configuration SHOULD allow for the decision data to be supplied by an external source, e.g. text file or dbm database. 2.2 Verify "MAIL From" The MTA MUST be able to perform a simple "sanity check" of the "MAIL From" domain and refuse to receive mail if that domain is nonexistent. To overcome temporary errors/problems in the DNS, Return Codes like 4xx are recommended; however the MTA MAY allow for Return Codes that show real DNS state - 4xx for temporary problems and 5xx Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 4]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 for NonExistent domain. 2.3 Refuse "MAIL From" The MTA MUST be able to refuse to receive mail from a specific "MAIL From" user (foo@domain.com) or from an entire "MAIL From" domain (domain.com). In general this kind of rules are easily overcome by the spammers changing "MAIL From" every so often, but the ability to block a certain user or a certain domain is quite helpful while an attack has just been discovered and is ongoing. Given a combination of "domain sanity check", "non-relay" and "<local-part> verification" the method may be more useful. 2.4 Refuse mail from a group of hosts The MTA MUST be able to accept or refuse mail from a specific host or from a group of hosts. It is REQUIRED that the MTA support decisions based on FQN hostnames (host.domain.com), on domain names with wild cards (*.domain.com), on individual IP addresses (10.11.12.13) or on IP addresses with a prefix length (10.0.0.0/8, 192.168.1.0/24). It is also REQUIRED that these decision rules can be combined, e.g. accept host.domain.com refuse *.domain.com accept 10.11.12.13 accept 192.168.1.0/24 refuse 10.0.0.0/8 IP-address/length is REQUIRED. However, early implementations with wild cards, e.g. 10.11.12.*, instead are of course much better then without. 2.5 Log service denials The MTA MUST be able to log all service denial actions based on the anti-spam rules. The log entries SHOULD contain at least: o Refuse information, i.e. why the request was refused ("Mail From", Relaying Denied, Spam User, Spam Host, etc). o "RCPT To" addresses (domains). o Offending host's FQN hostname. Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 5]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 o Offending host's IP address. o Other relevant information (e.g. given during the SMTP dialogue, before we decided to refuse the request). 2.6 Verify <local-part> The MTA SHOULD allow that outgoing mail has its <local-part> verified so that the sender name is a real user or an existing alias. This is basically to protect the rest of the Internet from various "typos" (MAIL From <fo0bar@domain.com>) and malicious users (MAIL From <I.am.unknown.to.you.he.he@domain.com>). As always this can be overcome by spammers really wanting to do so, but with more strict rules for relaying it becomes harder and harder. In fact, catching "typos" at the initial (and official) mail relay should in itself be enough motivation for this requirement. 2.7 Rate Control The MTA SHOULD make it possible for the mail host to control the rate with which mail is sent or received. The idea is twofold: o If we happen to have a spammer legally using our mail host (i.e. a legal mail user with an existing, legal, account that sends out spam - be sending spam illegal or not) we may want to reduce the speed with which he sends out mail. This is not without controversy and must be used with extreme care, but it may protect the rest of the Internet from him. o If we are under a spam attack it may help us considerably just being able to slow down the incoming mail rate for that particular user/host. For receiving mail the MTA SHOULD be able to do so based on "MAIL From" user, "MAIL From" domain, sending host (name/address) or a combination. In both cases 4xx Return Code is recommended. 2.8 Return Codes Our basic assumption is that refuse/accept is handled at the SMTP layer and that an MTA that decides to refuse a message should do so while still in the SMTP dialogue. First this means that we do not have to store a copy of a message we later decide to refuse and second our responsibility for that message is low or none - since we Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 6]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 have not read it we leave it to the sender to handle the error. The MTA MUST be configurable to use different Return Codes for different rules (e.g. 451 Temp Fail vs 551 Fatal Error). Now, this must be used with great care: o 5xx is a Fatal Error and results in the mail transfer being terminated and the mail returned to sender. For some events, like "Denied - you're on the spammer's list", this is probably the correct response and the right thing to say. A mistake in configuration, however, may cause valid mail to bounce back to the sender, which may be quite unfortunate. o 4xx is a Temporary Error and results in the mail transfer being put back on queue again and a new attempt being made later. Therefore, configuration mistakes are much less fatal, which really helps out. A 4xx response make the spammer's host re-queue the mail and if it really is his own host who gets to do this, it is probably a good thing. If, on the other hand, he is using someone else as Relay Host, all the mail being queued is a fairly strong evidence that something illegal is going on. 4xx, Temporary Error, is almost always the recommended Return Code. However, 4xx Temporary Errors may have unexpected interaction with MX-records. If the receiving domain has several MX records and the lowest preference MX-host refuses to receive mail from a certain "MAIL From" domain with a "451" Response Code, the sending host may choose to use the other host on the MX list. If that other MX host does not have the same refuse-list, it will of course accept the mail, only to find that the final host still refuses to receive that piece of mail ("MAIL From"). I.e. our intent was to make the offending mail stay at the offending sender's host and fill up his mqueue disk, but it all ended up at our friend, the next lowest preference MX-host. 3. sendmail example The main purpose of the memo is to define a set of requirements that will help reduce spam. It is not intended to describe how to do that for any particular MTA. However, many of us use and are familiar with sendmail [2] and therefore we provide some hints; these require late Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 7]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 versions of sendmail-8.8.* (when this was written, 12 Nov 1997, sendmail-8.8.8 was the latest). The example neither makes a claim to solve the problem nor to be correct and you use it at your own risk. These examples all use Return Code "451", Temp Fail. Please read that section above once more and verify that you will not hit your friends, your next lowest preference MX-hosts, that may have different rules. (NB sendmail makes a difference between <tab> and <space> used as separators; if you use cut-and-paste from this memo you are likely to get <space> everywhere). ################################################################## # Deny spammers, single users or entire domains Kspammers dbm -o /etc/mail/spammers.db # Scheck_mail # check for valid domain name (name exists within DNS) R$* $: $>3 $1 ifdef(`_NO_CANONIFY_', ` # pass to name server to make hostname canonical # (done here if we have "nocanonify", in S3-S96 otherwise) R$* < @ $* $~P > $* $: $1 < @ $[ $2 $3 $] > $4 ') R$* < @ $*domain.com .> $: $1<@$2domain.com> sigh R$* < @ $+ . > $: <$1@$2> OK R$* < @ $+ > $#error $: 451 Domain must resolve # #R$* $@ OK return from here if you # have no spammers check ### # check user@dom.ain and dom.ain versus spammers database R<$* @ $+> $: $1<@$2> re-focus # user@dom.ain R$* < @ $+ > $: $1<@$2><$(spammers $1@$2 $:OK $)> R$* < @ $+ ><OK> $: $1<@$2> R$* < @ $+ ><$* @ $+> $#error $: 451 Denied due to spam-list # dom.ain R$* < @ $+ > $: $1<@$2><$(spammers $2 $:OK $)> R$* < @ $+ ><OK> $: $1<@$2> R$* < @ $+ ><$+> $#error $: 451 Denied due to spam-list # You may consider more tests here, e.g. verify/check against # $(dequote "" $&{client_name} $) # $(dequote "" $&{client_addr} $) R$* $@ OK ################################################################## Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 8]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 ################################################################## # In "class D" you enter domains and hosts for two purposes # 1) You accept to relay mail TO them (MX). # 2) You accept to relay mail FROM them (SmartHost). # In both cases, this is "recursive", i.e. foo.se -> *.foo.se #FD/etc/mail/sendmail.cD CD # # In "class B" you enter the "exceptionally bad guys", i.e. hosts # that you want to deny relay from regardless - this may be hosts # that do not have enough filters or any other reason. Be careful. # FB/etc/mail/sendmail.cB CB # Scheck_rcpt # first get rid of a%b@c type addresses R< $+ % $+ > < $1 @ $2 > R< $+ @ $+ @ $+ > < $1 @ $2 > # "RCPT To" that terminates locally is OK R< $+ @ $=w > $@ OK R< $+ @ $=w . > $@ OK R<$-> $@ OK # "RCPT To" for accepted domains is OK R< $+ @ $=D > $@ OK R< $+ @ $=D . > $@ OK R< $+ @ $+ . $=D > $@ OK R< $+ @ $+ . $=D . > $@ OK # get sender host's name R$* $: $(dequote "" $&{client_name} $) # if it's me it's OK R$=w $@ OK R$@ $@ OK # exceptionally bad guys... R$=B $#error $:"451 Relaying Denied, " $1 # do this in case you want "bad with recursion" #R$+$=B $#error $:"451 Relaying Denied, " $1$2 # an accepted host is OK (with "recursion") R$=D $@ OK R$+$=D $@ OK # anything else is bogus R$* $#error $:"451 Relaying Denied, " $1 ################################################################## 4. Security Considerations This memo does not consider security in its most general form, taken literally, but it certainly does so in a broader and more general sense - protecting Internet users from mass mail abuse does have some Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 9]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 security implications. 5. Acknowledgements This memo is the result of discussions in an ad hoc group of Swedish ISPs and Universities. Without hope to mention everyone we simply give the domain names for the ISPs here: algonet.se, global-ip.net, pi.se, swip.net, telia.net and udac.se; the Universities stay anonymous. 6. References [1] Jonathan B. Postel "Simple Mail Transfer Protocol; RFC821", August 1982. Available via anonymous ftp at ftp://ds.internic.net/rfc/rfc821.txt [2] sendmail Home Page. http://www.sendmail.org * Spam (R) is a registered trademark of a meat product made by Hormel. Use of the term Spam in the Internet community comes from a Monty Python sketch and is almost Internet folklore. The term Spam is usually meant negative, however this is not in any way intended to describe the Hormel product. 7. Addendum - Future work 7.1. Impact on SMTP UAs Despite this memo is about MTA:s and the requirements put on them, some of what is done here falls back to the UA:s (User Agents, the "ordinary mail programs"). An UA does two things: 1) Reads mail from a mailbox and prints on the screen. This typically uses a protocol like POP, IMAP or NFS. 2) Reads text from the keyboard and hands that over to the mailbox MTA for delivery as a piece of mail. This typically uses the SMTP protocol, i.e. the same protocol that is used between MTA:s. When MTA:s now start to implement various anti-relay filters as described above, a UA on a portable laptop host may get responses like "Relaying Denied" just because it happens to use IP addresses within the wrong range or "name". To overcome this, one should use some other protocol to transport the piece of mail from the UA to the (Relay) MTA. Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 10]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 This would do away with the need for SMTP between everybody and will allow more strict authentication between SMTP MTA:s. 7.2. Anti-spam filters Today, when an MTA with spam filter receives a message it has two possibilities, reject or accept the message. The mechanism used by sendmail and other MTA software is to check in a policy filter list whether the sender, recipient and sending host is accepted. If the message is not destined for a local user, a policy filter is checked to see whether we relay for the sending host, whether we accept the sender of the message and whether we accept the recipient of the message. If the message is destined for a local user, we do the same thing. This way, we actually filter messages that is routed through us. We also apply the same policy filter, not only to non-local users but to all local users as well. Users probably have different opinions about what addresses should be in the policy filter list, a central filter will be considered by some as a nuisance that should be removed. In this view it is not unlikely that the system administrator, who is responsible for the maintenance of the policy filter list, will hesitate to add addresses to the list and thus give continued access to the spammers to abuse the MTA. Less blunt tools than central filtering should be used to fight spam. 7.2.1. Personal anti-spam filters If users could design their own policy filter, for what should be delivered to their mailboxes, the system manager is less likely seen as a despotic censor. If this concept is accepted, much more freedom is given when designing the filter and on what principles it should be built. Some users may want a narrow filter while others want wide. With personal filters it will be up to the user to decide how the filter should function. The three parameters that can be used for designing a filter should be addresses that should be rejected, addresses that should be accepted and an indication if to accept or reject by default. It could function like today, a list with addresses or domains that is rejected, default is to accept. It could function the opposite way, a list with addresses or domains that is accepted, default is to reject. In all cases some sort of interface must be developed, to make the user able to configure their personal policy filter. It may not be Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 11]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 very different from how the owner of a listserv list manages the list, all configuration performed by email, or it could be done by using a web browser. If default is to reject messages, a mechanism for senders to request to be members of the accept list needs to be developed. One thought is to create a user-request address that will, when sent to, give the user a hint to add the sender to the accept list. Some redesigning of the MTA is necessary for this type of filtering to be possible. It would then need to hear out the RCPT TO before checking the policy filter of the recipient and deciding whether to accept or reject the message. However, this is not a very complicated function to implement. Sendmail already checks if users have .forward files in their home directories, checking for a policy filter list would not be much more trouble. This kind of personal spam filter could be implemented today, as it does not require global implementation to function. It will benefit the users who does not have to accept spam, as well as the system administrator who does not need to take the responsibility of deciding what to accept or not. The challenge is to design the user interface in such a way that users understand how they should operate the filter. 7.2.2. Anti-spam filters return codes One open question here is Return Codes - should the receiving MTA indicate to the sending MTA that this mail was not delivered due to some personal anti-spam filter or should it silently drop it? Should it return 4xx or 5xx - and should that be up to the user? First of all, it would be both rude and unfair to drop mail without any indication, so there has to be 4xx or 5xx. Since an ordinary user will never be able to forsee, or understand, the interactions between several MX hosts Return Code "551 - Denied due to spam list" is recommended. 7.3. Mandatory signing of messages Today, the Internet is quickly becoming the groupware of the masses. It could be argued that what characterizes the Internet, in opposition to a traditional groupware system, is that on the Internet users are in practice anonymous. One of the main problems with spamming is that the sender almost always tries to achieve anonymity, and using todays standard technology they have no problems doing that. Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 12]
draft-lindberg-anti-spam-mta-00.txt 12 Nov 1997 Spammers should not be given a chance to hide, they should be forced to reveal their identity. With their identities in public, measures can be taken to make them stop sending spam or to not accept email from them. This can be achieved if signing of messages with an electronic signature should become mandatory. It would make spamming impossible, or at least futile. As a side effect, assigning certificates to Internet users would benefit the authentication procedures for other Internet services. It is important that this transition is made as smooth as possible, methods to handle the intermediate stage must be developed. There is much work going on in creating the standards necessary to make this possible and effective. The MTA should be given a chance to check the signature, using ESMTP, and reject the message if incorrect, see draft-myers-smtp-auth-XX and draft-myers-auth-sasl-XX. Public keys and certificates needs to be signed and distributed across the Internet, see RFC2065. Everything should be knitted together and organized, see draft- eastlake-muse-XX. Editor's Address Gunnar Lindberg Computer Communications Group Chalmers University of Technology S-412 96 Gothenburg, SWEDEN, Phone +46 31 772 5913 FAX +46 31 772 5922 lindberg@cdg.chalmers.se Lindberg et.al. Anti-Spam Requirements on an SMTP MTA [Page 13]