UTF-8, a transformation format of Unicode and ISO 10646
draft-yergeau-utf8-00

Network Working Group                                       F. Yergeau
Internet Draft                                       Alis Technologies
<draft-yergeau-utf8-00.txt>                              27 March 1996
Expires 1 October 1996

        UTF-8, a transformation format of Unicode and ISO 10646

Status of this Memo

   This document is an Internet-Draft.  Internet-Drafts are working doc-
   uments of the Internet Engineering Task Force (IETF), its areas, and
   its working groups. Note that other groups may also distribute work-
   ing documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months.  Internet-Drafts may be updated, replaced, or obsoleted by
   other documents at any time.  It is not appropriate to use Internet-
   Drafts as reference material or to cite them other than as a "working
   draft" or "work in progress".

   To learn the current status of any Internet-Draft, please check the
   1id-abstracts.txt listing contained in the Internet-Drafts Shadow
   Directories on ds.internic.net (US East Coast), nic.nordu.net
   (Europe), ftp.isi.edu (US West Coast), or munnari.oz.au (Pacific
   Rim).

   Distribution of this document is unlimited.

Abstract

   The Unicode Standard, version 1.1, and ISO/IEC 10646-1:1993 jointly
   define a 16 bit character set which encompasses most of the world's
   writing systems. 16-bit characters, however, are not compatible with
   many current applications and protocols, and this has led to the
   development of a few so-called UCS transformation formats (UTF), each
   with different characteristics.  UTF-8, the object of this memo, has
   the characteristic of preserving the full US-ASCII range: US-ASCII
   characters are encoded in one octet having the usual US-ASCII value,
   and any octet with such a value can only be an US-ASCII character.
   This provides compatibility with file systems, parsers and other
   software that rely on US-ASCII values but are transparent to other
   values.

1.  Introduction

   The Unicode Standard, version 1.1 [UNICODE], and ISO/IEC 10646-1:1993
   [ISO-10646] jointly define a 16 bit character set, UCS-2, which

                         Expires 1 October 1996         [Page 1]
Internet Draft                    UTF-8                    27 March 1996

   encompasses most of the world's writing systems.  ISO 10646 further
   defines a 32 bit character set, UCS-4, with currently no assignments
   outside of the region corresponding to UCS-2 (the Basic Multilingual
   Plane, BMP).  The UCS-2 and UCS-4 encodings, however, are hard to use
   in many current applications and protocols that assume 8 or even 7
   bit characters.  Even newer systems able to deal with 16 bit charac-
   ters cannot process UCS-4 data. This situation has led to the devel-
   opment of so-called UCS transformation formats (UTF), each with dif-
   ferent characteristics.

   UTF-1 has only historical interest, having been removed from ISO
   10646.  UTF-7 has the quality of encoding the full Unicode repertoire
   using only octets with the high-order bit clear (7 bit US-ASCII val-
   ues, [US-ASCII]), and is thus deemed a mail-safe encoding
   ([RFC1642]).  UTF-8, the object of this memo, uses all bits of an
   octet, but has the quality of preserving the full US-ASCII range: US-
   ASCII characters are encoded in one octet having the normal US-ASCII
   value, and any octet with such a value can only stand for an US-ASCII
   character, and nothing else.

   UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire
   into a pair of UCS-2 values from a reserved range.  UTF-16 impacts
   UTF-8 in that UCS-2 values from the reserved range must be treated
   specially in the UTF-8 transformation.

   UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of
   octets, where the number of octets and the value of each depend on
   the integer value assigned to the character in ISO 10646.  This
   transformation format has the following characteristics (all values
   are in hexadecimal):

   -  Character values from 0000 0000 to 0000 007E (US-ASCII repertoire)
      correspond to octets 00 to 7E (7 bit US-ASCII values).

   -  US-ASCII values do not appear otherwise in a UTF-8 encoded charac-
      ter stream.  This provides compatibility with file systems or
      other software that parse based on US-ASCII values but are trans-
      parent to other values.

   -  Round-trip conversion is easy between UTF-8 and either of UCS-4,
      UCS-2 or Unicode.

   -  The first octet of a multi-octet sequence indicates the number of
      octets in the sequence.

   -  Character boundaries are easily found from anywhere in an octet
      stream.

                         Expires 1 October 1996         [Page 2]
Internet Draft                    UTF-8                    27 March 1996

   -  The lexicographic sorting order of UCS-4 strings is preserved.

   -  The octet values FE and FF never appear.

   UTF-8 was originally a project of the X/Open Joint Internationaliza-
   tion Group XOJIG with the objective to specify a File System Safe UCS
   Transformation Format [FSS-UTF] that is compatible with UNIX systems,
   supporting multilingual text in a single encoding.  The original
   authors were Gary Miller, Greger Leijonhufvud and John Entenmann.
   Later, Ken Thompson and Rob Pike did significant work for the formal
   UTF-8.

   A description can also be found in Unicode Technical Report #4 [UNI-
   CODE].  The definitive reference, including provisions for UTF-16
   data within UTF-8, will be Annex P of ISO/IEC 10646-1 [ISO-10646].

2.  UTF-8 definition

   In UTF-8, characters are encoded using sequences of 1 to 6 octets.
   The only octet of a "sequence" of one has the higher-order bit set to
   0, the remaining 7 bits being used to encode the character value. In
   a sequence of n octets, n>1, the initial octet has the n higher-order
   bits set to 1, followed by a bit set to 0.  The remaining bit(s) of
   that octet is(are) used to encode bits from the initial character.
   The following octet(s) all have the higher-order bit set to 1 and the
   following bit set to 0, leaving 6 bits in each to encode bits from
   the initial character.

   The table below shows the format of these different octet types. The
   letter x indicates bits available for encoding bits of the initial
   character.

     Octet       Binary    Free bits   Max. character value
   1st of 1     0xxxxxxx   7           0000 007F
   1st of 2     110xxxxx   5           0000 07FF
   1st of 3     1110xxxx   4           0000 FFFF
   1st of 4     11110xxx   3           001F FFFF
   1st of 5     111110xx   2           03FF FFFF
   1st of 6     1111110x   1           7FFF FFFF
   2nd to 6th   10xxxxxx   6

   Encoding from UCS-4 to UTF-8 proceeds as follows:

   1) determine the number of octets required from the character value
      and the fourth column of the table above.

   2) Prepare the high-order bits of the octets as per the second column
      of the table.

                         Expires 1 October 1996         [Page 3]
Internet Draft                    UTF-8                    27 March 1996

   3) Fill in the bits marked x from the bits of the character value,
      starting from the lower-order bits of the character value and
      putting them first in the last octet of the sequence, then the
      next to last, etc. until all x bits are filled in.

      The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be
      obtained from the above, in principle, by simply extending each
      UCS-2 character with two zero-valued octets.  However, UCS-2 val-
      ues between D800 and DFFF, being actually UCS-4 characters trans-
      formed trough UTF-16, need special treatment: the UTF-16 transfor-
      mation must be undone, yielding a UCS-4 character that is then
      transformed as above.

      Decoding from UTF-8 to UCS-4 proceeds as follows:

   1) Initialize the 4 octets of the UCS-4 character with all bits set
      to 0.

   2) Determine which bits encode the character value from the number of
      octets in the sequence and the second column of the table above
      (the bits marked x).

   3) Distribute the bits from the sequence to the UCS-4 character,
      first the lower-order bits from the last octet of the sequence and
      proceeding to the left until no x bits are left.

      If the UTF-8 sequence is no more than three octets long, decoding
      can proceed directly to UCS-2 (or equivalently Unicode).

      A more detailed algorithm and formulae can be found in [FSS_UTF],
      [UNICODE] or the forthcoming Annex P to [ISO-10646].

3.  Examples

   The Unicode sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391,
   002E) may be encoded as follows:

   41 E2 89 A2 CE 91 2E

   The Unicode sequence "Hi Mom <WHITE SMILING FACE>!" (0048, 0069,
   0020, 004D, 006F, 004D, 0020, 263A, 0021) may be encoded as follows:

   48 69 20 4D 6F 4D 20 E2 98 BA 21

   The Unicode sequence representing the Han characters for the Japanese
   word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows:

   E6 97 A5 E6 9C AC E4 AA 9E

                         Expires 1 October 1996         [Page 4]
Internet Draft                    UTF-8                    27 March 1996

MIME registrations

   This memo is meant to serve as the basis for registration of a MIME
   character encoding (charset) as per [RFC1521].  The proposed charset
   parameter value is "UTF-8".  This string would label media types con-
   taining text consisting of characters from the repertoire of ISO
   10646-1 encoded to a sequence of octets using the encoding scheme
   outlined above.

Security Considerations

   Security issues are not discussed in this memo.

Acknowledgments

   The following have participated in the drafting and discussion of
   this memo:

   Andries Brouwer   Martin J. Drst
   David Goldsmith   Edwin F. Hart
   Markus Kuhn       Michael Kung
   Alain LaBont     Murray Sargent
   Keld Simonsen     Arnold Winkler

Bibliography

   [FSS_UTF]      X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm.
                  22p. pbk. 172g.  4/95, X/Open Company Ltd., "File Sys-
                  tem Safe UCS Transformation Format (FSS_UTF)", X/Open
                  Preleminary Specification, Document Number P316.  Also
                  published in Unicode Technical Report #4.

   [ISO-10646]    ISO/IEC 10646-1:1993. International Standard -- Infor-
                  mation technology -- Universal Multiple-Octet Coded
                  Character Set (UCS) -- Part 1: Architecture and Basic
                  Multilingual Plane.  UTF-8 is described in Annex P,
                  adopted but not yet published.  UTF-16 is described in
                  Annex O, adopted but not yet published.

   [RFC1521]      N. Borenstein and N. Freed, "MIME (Multipurpose Inter-
                  net Mail Extensions) Part One: Mechanisms for Specify-
                  ing and Describing the Format of Internet Message Bod-
                  ies", RFC 1521, Bellcore, Innosoft, September 1993.

   [RFC1641]      D. Goldsmith, M.Davis, "Using Unicode with MIME", RFC
                  1641, Taligent inc., July 1994.

                         Expires 1 October 1996         [Page 5]
Internet Draft                    UTF-8                    27 March 1996

   [RFC1642]      D. Goldsmith, M. Davis, "UTF-7: A Mail-safe Transfor-
                  mation Format of Unicode", RFC 1642, Taligent inc.,
                  July 1994.

   [UNICODE]      The Unicode Consortium, "The Unicode Standard --
                  Worldwide Character Encoding -- Version 1.0", Addison-
                  Wesley, Volume 1, 1991, Volume 2, 1992.  UTF-8 is
                  described in Unicode Technical Report #4.

   [US-ASCII]     Coded Character Set--7-bit American Standard Code for
                  Information Interchange, ANSI X3.4-1986.

Author's Address

   Franois Yergeau
   Alis Technologies
   100, boul. Alexis-Nihon
   Suite 600
   Montral  QC  H4M 2P2
   Canada

   Tel: +1 (514) 747-2547
   Fax: +1 (514) 747-2561
   EMail: fyergeau@alis.com

                         Expires 1 October 1996         [Page 6]