[Search] [txt|pdfized|bibtex] [Tracker] [Email] [Nits]
Versions: 00                                                            
Internet Draft                                          Paul Hoffman
draft-hoffman-utf8headers-00.txt            Internet Mail Consortium
December 15, 2003
Expires in six months

          SMTP Service Extensions or Transmission of Headers
                          in UTF-8 Encoding

Status of this Memo

This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups.  Note
that other groups may also distribute working documents as
Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time.  It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.


Abstract


Mailbox names often represent the names of human users. Many of these
users throughout the world have names that are not normally represented
by the users with just the ASCII repertoire of characters, and would therefore
like to use their real names in their mailbox names. These users
are also likely to use non-ASCII text in their common names and subjects
of email messages, both in what they send and what they receive.
This protocol specifies how to represent all headers
of email messages encoded in UTF-8.


1. Introduction

The format of email messages [MSGFMT] only allows ASCII characters in the
headers of messages. This prevents users from having email addresses
that contain non-ASCII characters. It further forces non-ASCII text in
common names, comments, and in free text (such as in the Subject: field)
to be in quoted-printable format [MIME3]. This specification describes a
change to the email message format, and to SMTP message transport, that
allows non-ASCII characters throughout email headers. These changes
affect SMTP clients, SMTP servers, and mail user agents (MUAs).

In this specification, the SMTP protocol [SMTP] is used to prevent the
transmission of messages with UTF-8 [UTF8] headers to systems that
cannot handle such messages. The new SMTP extension has the name
"UTF-8-HEADERS".

Using this new SMTP extension prevents the introduction of such
messages in message stores that might misrepresent or mangle such
messages. It should be noted that using an ESMTP extension does not
prevent transferring email messages with UTF-8 headers to other systems
that use the email format for messages, such as in the POP and IMAP
protocols. Those protocols will need to be changed in order to handle
messages in message stores that have UTF-8 headers.

The dual motivations of this protocol are to allow UTF-8 everywhere in
the headers and to not bounce any messages just because they originated
with UTF-8 headers. Using this protocol, messages that originated with
UTF-8 headers will only be bounced if an enabled SMTP client is speaking
to an unenabled SMTP server and some of the UTF-8 headers cannot be
downgraded to all-ASCII headers. This protocol describes how to
downgrade all headers from UTF-8 to all-ASCII, but does not guarantee
that such downgrading will always be successful.

Further, this protocol allows current users who have all-ASCII mailbox
names to step up to UTF-8 headers easily. This means that users of this
protocol should normally be able to communicate with other users of this
protocol and with users who have not yet updated.

This protocol does not require the sender or recipient of mail to have
mailbox names that do not include non-ASCII characters. For example, the
protocol might still be used if just the subject header has non-ASCII
characters, and the protocol must be used if other headers (particularly
Received headers) contain non-ASCII characters.

1.1 Terminology

The key words "MUST", "SHALL", "REQUIRED", "SHOULD", "RECOMMENDED", and
"MAY" in this document are to be interpreted as described in RFC 2119
[KEYWORDS].

Unless otherwise noted, all terms used here are defined in RFC 2821 and
RFC 2822.

In this document, an address is "all-ASCII" if every character in the
address is in the ASCII character repertoire [ASCII]; an address is
"non-ASCII" if any character is not in the ASCII character repertoire.
Similarly, a header body is "all-ASCII" if every character in the body
of the header is in the ASCII character repertoire; a header body is
"non-ASCII" if any character is not in the ASCII character repertoire.

This document is being discussed on the ietf-imaa mailing list.  See
<http://www.imc.org/ietf-imaa/> for information about subscribing and
the list's archive.


2. Changes to MUAs and to the user's mail environment

For this protocol to work well (that is, for it not to bounce mail
excessively when an enabled system encounters a non-enabled system), any
mail sender who has non-ASCII characters in the
addr-spec of their mailbox name SHOULD
have a second mailbox whose addr-spec contains only ASCII characters. This
second mailbox is used when a recipient of a message is not using this
protocol; this is the "fallback address" for the sender.

Having two mailboxes is not an absolute requirement because some mail
systems will not allow a user to be able to get mail from two addresses
(the non-ASCII and all-ASCII addresses). If a user does have two
mailboxes, they SHOULD both be on the same mail server (that is, they
should both have the same host name in the user's address).

Having two mailboxes can lead to confusion for users if the MUA does not
handle them well. MUAs that follow this specification SHOULD have
options that would make it seem like two mailboxes are one. For example,
if a user says "read my mail", the MUA SHOULD read from both the mailbox
with the non-ASCII name and the mailbox with the all-ASCII name. Note
that this feature might not be necessary: a terminating SMTP server
might have combined all incoming mail for both addresses into a single
mailbox. However, MUAs SHOULD NOT assume that combining by the SMTP
server will always be the case.

2.1 Changes to MUA administrative interfaces

The administrative interface for MUAs that use this protocol MUST have
method for a user to specify the name of their mailbox that contains
non-ASCII characters, and MUST have a method for the user to specify the
name of their mailbox that contains non-ASCII characters.

The MUA user interface SHOULD also allow users to specify the common
name associated with the non-ASCII mailbox using non-ASCII
characters; this common name MUST be encoded as UTF-8. The common name
associated with the all-ASCII mailbox MUST only contain ASCII
characters, although it can use a quoted-printable format to represent a
different encoding; this encoding SHOULD be UTF-8.

MUAs are encouraged to cache address mappings that are specified
in incoming mail. Given that mappings might change over time,
these MUAs might over-write existing mappings with new ones,
and might give the user a choice for the time-to-live for the
cached mapping.

2.2 Address-map headers

For every address in a message with a non-ASCII local-part, the mail
initiator SHOULD create a mapping in a new header, called
"Address-map:". A message SHOULD have one Address-map: header for every
non-ASCII address for which the sender knows a map. The header is only
for addresses that have a non-ASCII local-part in its addr-spec. It MUST
NOT be used for addresses that have all-ASCII addr-specs, even if those
addresses have UTF-8 domain names, and it MUST NOT be used if the
local-part of the addr-spec is all-ASCII but the display-name or the
comment is non-ASCII.

If the sender has an all-ASCII local-part associated with its non-ASCII
mailbox, the sender's MUA MUST create an Address-map header for that
association. If the sender knows (such as through caching incoming
address maps or from an address book) the mapping for any recipient that
has a non-ASCII mailbox name, the sending MUA SHOULD create an
Address-map header for it.

Both addresses in the Address-map header are full addr-specs. The body
of the Address-map header only contains addr-specs, never display-names
or comments. The format of the Address-map header is:

   Address-map: <address-with-non-ASCII-LHS>,<downgrade-address>

The encoding for address-with-non-ASCII-LHS MUST be UTF-8; the encoding
for downgrade-address MUST be ASCII. If the domain name in an
internationalized domain name [IDNA], then it MUST be encoded in UTF-8
in the address-with-non-ASCII-LHS and MUST be encoded using IDNA in the
downgrade-address.

Examples:

  Address-map: Jos<eacute>@example.com,jose@example.com

  Address-map: bj<oumlaut>n@r<aumlaut>ksm<oumlaut>rg<aring>s.se,
      bjorn-ascii@rksmrgs-5wao1o.se

Note that when receiving mail, the Address-map headers may be all in ASCII.
This would be due to an intervening SMTP server or other agent downgrading
the map. All-ASCII Address-map headers MUST be accepted.

2.3 Changes to MUA sending

Sending MUAs that follow this protocol MUST create all headers encoded
in UTF-8. No other direct encodings are allowed. MUAs MAY continue to
use quoted-printable text to specify some text in other encodings;
however this is not recommended because it is likely that this will not
interoperate well with MUAs that follow this specification.


3. Changes to SMTP

This protocol defines a new SMTP extension, UTF-8-HEADERS. (The formal
definition is in the IANA Considerations section.)

3.1 UTF-8-HEADERS extension

If an SMTP server advertises the UTF-8-HEADERS extentension, an
SMTP client that supports this protocol SHOULD send message headers
as described in this document.

The terminal SMTP server is responsible for knowing whether or not the
message store can handle UTF-8 headers. A terminal SMTP server MUST NOT
advertise the UTF-8-HEADERS extension if the message store for which it
is responsible cannot
handle UTF-8 headers.

If an SMTP client does not see the UTF-8-HEADERS extension advertised
by an SMTP server, the SMTP client MUST downgrade the
non-ASCII contents of all header bodies before continuing to send
the message. The SMTP client SHOULD send the message with the downgraded
header bodies as a normal message.
If any header body cannot be downgraded, the SMTP client
MUST bounce the message with an error code of 558.

All UTF-8 headers bodies can be downgraded to being all-ASCII.
However, any header body that contains a non-ASCII mailbox name might
not be able to be downgraded if there is no Address-map header that
gives a mapping for the downgrading.

3.2 Downgrading header bodies

This section defines how to downgrade header bodies. Note that
downgrading MUST only be done if necessary. That is, downgrading
MUST never be done on fields or bodies that are all-ASCII.

3.2.1 Mailboxes

Mailboxes appear in many standard headers, such as To:, From:, Sender:,
Reply-to:, Cc:, Bcc:, Received:, and some of the Resent-: headers.
Downgrading mailboxes is done as follows:

   1) If necessary, convert the domain using IDNA.

   2) If necessary,convert the local-parts using values from an
      Address-map: header in the message

   3) If necessary,convert any display-name or comment using
      quoted-printable with UTF-8 encoding

3.3.2 Message-ids

Downgrading message-ids is done as follows

   1) If necessary,convert the id-left using Base64

   2) If necessary,convert the id-right using Base64

3.3.3 Informational headers

If necessary, downgrading the bodies of informational headers (Subject:,
Comments:, and Keywords:) is done using quoted-printable with UTF-8
encoding.

3.3.4 Address-map headers

If necessary, the Address-map: header is downgraded using Base64 for
local-parts, and IDNA for domain names.

For example:

  Address-map: Jos<eacute>@example.com,jose@example.com

would be downgraded to:

  Address-map: Sm9zw6k=@example.com,jose@example.com

As another example:

  Address-map: bj<oumlaut>n@r<aumlaut>ksm<oumlaut>rg<aring>s.se,
      bjorn-ascii@rksmrgs-5wao1o.se

would be downgraded to:

  Address-map: YmrDtnJu@rksmrgs-5wao1o.se,
      bjorn-ascii@rksmrgs-5wao1o.se

3.3 Things not changed from RFC 2822

Note that this protocol does change the definition of header field
names. That is, only the bodies of headers are allowed to have non-ASCII
characters; the rules in RFC 2822 for header names are not changed.

Similarly, this protocol does not change the date and time specification
in RFC 2822.

3.4 Additional processing rules

In order to make mail retrieval easier, terminal SMTP servers SHOULD
write messages addressed to either the UTF-8 address or the all-ASCII
address into the same mailbox. However, given that this is quite
different than common practice today, the ramifications for doing this
should be studied carefully before this is implemented.

Intermediate SMTP servers MAY change the values in the Address-map:
header (such as to add one that is missing or to correct a mapping), but
SHOULD only do so for domains local to the intermediate SMTP server.

Terminal SMTP servers MAY look into the headers of a message to
determine whether they should upgrade a downgraded set of headers to
UTF-8. This is easy to determine: if the Address-map: header contains
only ASCII, it was downgraded earlier in the chain of SMTP server.
Upgrading is particularly useful on bounce messages caused by bad
mappings.


4. Security considerations

If a user has a non-ASCII mailbox address and a mapped all-ASCII mailbox
address, a digital certificate that identifies that user SHOULD have
both addresses in the identity. Having multiple email addresses as
identities in a single certificate is already supported in PKIX and
OpenPGP.

Internationalized local parts will cause mail addresses to become
longer, and possibly make it harder to keep lines in a header under 78
characters.  Lines that are longer than 78 characters (which is a SHOULD
specification, not a MUST specification, in RFC 2822) could possibly
cause mail user agents to fail in ways that affect security.


5. IANA considerations

IANA will assign the UTF-8-HEADERS extension for ESMTP.

The UTF-8 headers extension is defined as follows:

(1) The name of the SMTP service extension is "UTF-8 headers".

(2) The EHLO keyword value associated with the extension is
UTF-8-HEADERS.

(3) No parameter is used with the UTF-8-HEADERS EHLO keyword.

(4) No additional parameters are added to either the MAIL FROM or RCPT
TO commands.

(5) No additional SMTP verbs are defined by this extension.

(6) This document specifies how support for the extension affects the
behavior of a server and client SMTP.


6. References

6.1 Normative references

[ASCII] Cerf, V., "ASCII format for Network Interchange", RFC 20,
October 1969.

[IDNA] Faltstrom, P., Hoffman, P. and A. Costello, "Internationalizing
Domain Names in Applications (IDNA)", RFC 3490, March 2003.

[KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.

[MIME3]  Moore, K., "MIME (Multipurpose Internet Mail Extensions) Part
Three: Message Header Extensions for Non-ASCII Text", RFC 2047, November
1996.

[MSGFMT] Resnick, P., "Internet Message Format", RFC 2822, April 2001.

[SMTP] Klensin, J., "Simple Mail Transfer Protocol", RFC 2821, April
2001.

[UTF8] Yergeau, F. "UTF-8, a Transformation Format of ISO 10646", RFC
3629, November 2003.


7. Author's address

Paul Hoffman
Internet Mail Consortium
127 Segre Place
Santa Cruz, CA 95060  USA
phoffman@imc.org


A. Open issues

- POP and IMAP might be updated to allow one request to bring in two or
more mailboxes; otherwise, users will have to do two separate requests.

- It might be good to have a protocol for determining mappings, but it
is not defined here.