Network Working Group                                         J. Klensin
Internet-Draft                                              M. Padlipsky
Expires: October 31, 2006                                 April 29, 2006

                 Unicode Format for Network Interchange

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on October 31, 2006.

Copyright Notice

   Copyright (C) The Internet Society (2006).


   The Internet today is in need of a standardized form for the
   transmission of internationalized "text" information, paralleling the
   specifications for the use of ASCII that date from the early days of
   the ARPANET.  This document specifies that format, using UTF-8 with
   specification of normalization and specific line-ending sequences.

Klensin & Padlipsky     Expires October 31, 2006                [Page 1]

Internet-Draft                Network UTF-8                   April 2006

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.1.  Background . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Net-Unicode  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Normalization  . . . . . . . . . . . . . . . . . . . . . . . .  5
   4.  Versions of Unicode  . . . . . . . . . . . . . . . . . . . . .  5
   5.  Context of this Proposal . . . . . . . . . . . . . . . . . . .  6
   6.  Security Considerations  . . . . . . . . . . . . . . . . . . .  7
   7.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . .  7
   8.  References . . . . . . . . . . . . . . . . . . . . . . . . . .  7
     8.1.  Normative References . . . . . . . . . . . . . . . . . . .  7
     8.2.  Informative References . . . . . . . . . . . . . . . . . .  8
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10
   Intellectual Property and Copyright Statements . . . . . . . . . . 11

Klensin & Padlipsky     Expires October 31, 2006                [Page 2]

Internet-Draft                Network UTF-8                   April 2006

1.  Introduction

1.1.  Background

   This subsection contains a review of prior work in the ARPANET and
   Internet to establish a standard text type, work that establishes the
   context and motivation for the approach taken in this document.
   Those who are uninterested in that review and analysis can safely
   skip to the next section.

   One of the earlier application design decisions made in the
   development of ARPANET, a decision that was carried forward into the
   Internet, was the decision to standardize on a single and very
   specific coding for "text" to be passed across the network [RFC0020].
   Hosts on the network were then responsible for translating or mapping
   from whatever character coding conventions were used locally to that
   common intermediate representation, with sending hosts mapping to it
   and receiving ones mapping from it to their local forms as needed.
   It is interesting to note that at the time the ARPANET was being
   developed, participating Host operating systems used at least three
   different character coding standards: the antiquated BCD (Binary
   Coded Decimal), the then-dominant major manufacturer-backed EBCDIC
   (Extendended BCD Interchange Code), and the then-still emerging ASCII
   (American Standard Code for Information Interchange).  Since the
   ARPANET was an "open" project and EBCDIC was intimately linked to a
   particular hardware vendor, the original Network Working Group agreed
   that its standard should be ASCII.  That ASCII form was precisely
   "7-bit ASCII in an 8-bit field", which was in effect a compromise
   between Hosts that were natively 7-bit oriented (e.g., with five
   seven-bit characters in a 36 bit word), those that were 8-bit
   oriented (using eight-bit characers) and those that placed the seven-
   bit ASCII characters in 9-bit fields with two leading zero bits (four
   characters in a 36 bit word).

   More standardization was suggested in the first preliminary
   description of the Telnet protocol [RFC0097].  With the iterations of
   that protocol [RFC0137] [RFC0139] and the drawing together of an
   essentially formal definition somewhat later [RFC0318], a standard
   abstraction, the Network Virtual Terminal (NVT) was established.  NVT
   character-coding conventions (initially called "Telnet ASCII" and
   later called "NVT ASCII", or, more casually, "network ASCII")
   included the requirement that Carriage Return - Line Feed (CRLF) be
   the common representation for ending lines of text (given that some
   participating "Host" operating systems used the one natively, some
   the other, and at least one used both) and specified conventions for
   some other characters.  Also, since NVT ASCII was restricted to
   seven-bit characters, use of the high-order bit in octets was
   reserved for the transmission of control signaling information.

Klensin & Padlipsky     Expires October 31, 2006                [Page 3]

Internet-Draft                Network UTF-8                   April 2006

   At a very high level, the concept was that a system could use
   whatever character coding and line representations were appropriate
   locally, but text transmitted over the network as text must conform
   to the single "network virtual terminal" convention.

   In our more internationalized world, "text" clearly no longer equates
   unambiguously to "network ASCII".  Fortunately, however, we are
   converging on Unicode [Unicode], [ISO10646] as a single international
   interchange character coding and no longer need to deal with per-
   script standards for character sets (e.g., one standard for each of
   Arabic, Cyrillic, Devanagari, etc., or even standards keyed to
   languages that are usually considered to share a script, such as
   French, German, or Swedish).  Unfortunately, though, while it is
   certainly time to define a Unicode-based text type for use as a
   common text interchange format, "use Unicode" involves even more
   ambiguity than "use ASCII" did decades ago.  Unicode can be
   transmitted in UCS-2 (all characters 16 bit), UCS-4 or UTF-16 (all
   characters 32 bit), UTF-8 (a variable-length encoding with some
   additional properties) [RFC3629], and other forms.  Also, as with
   ASCII, any of these forms may have different line-ending conventions.

   This document proposes to establish "Net-Unicode" as a new
   standardized text transmission form for the Internet, to serve as an
   internationalized alternative for NVT ASCII when specified in new --
   and, where appropriate, updated -- protocols.  UTF-8 [RFC3629] is
   chosen for the coding because it has good compatibility properties
   with ASCII and for other reasons discussed in the existing IETF
   character set policy [RFC2277].

   In circumstances in which there is a choice, use of Unicode and the
   text encoding specified here is preferred to the double-byte encoding
   of "extended ASCII" [RFC0698] and SHOULD be used.

1.2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in [RFC2119].

2.  Net-Unicode

   The Network Unicode (Net-Unicode or Net-UTF-8) format is defined as

   1.  Characters will be coded in UTF-8 as defined in [RFC3629]

Klensin & Padlipsky     Expires October 31, 2006                [Page 4]

Internet-Draft                Network UTF-8                   April 2006

   2.  Line-endings will be indicated by the sequence Carriage-Return
       (U+000D) followed by Line-Feed (U+000A)
   3.  Before transmission, all character sequences will be normalized
       according to Unicode method "NFC" (see Section 3).
   4.  As suggested in Section 6 of RFC 3629, the Byte Order Mark
       ("BOM") signature MUST NOT appear at the beginning of these text

   The NVT specification contained a number of additional provisions,
   e.g., for the optional use of backspacing and "bare CR" (sent as CR
   NUL) to generate overstruck character sequences.  The much greater
   number of precomposed characters in Unicode, the availability of
   combining characters, and the growing use of markup conventions of
   various types (rather than character-coding) to show, e.g., emphasis,
   should make such sequences largely unnecessary.  Because they were
   optional in NVT applications, they SHOULD be avoided if at all
   possible; if they are used, this specification does not change the
   NVT rules and conventions of RFC 318 and RFC 854 [RFC0854] including
   the prohibition on CR without NUL (note that NUL, X'00' is hostile to
   programming languages that use that character as a string delimiter).

3.  Normalization

   There are a number of characters in Unicode that can be represented
   in two ways, either as a single precomposed character or as the
   combination of a base character and one or more combining characters.
   If any combination of ways to represent characters is permitted in
   the data stream, comparison and use of those strings becomes
   difficult or impossible.  The Unicode Consortium specifies a
   normalization method, known as NFC [NFC], which provides the
   necessary mappings to convert sequences of combining characters into
   a precomposed character when that is feasible.

4.  Versions of Unicode

   In retrospect, one of the advantages of ASCII [X3.4-1978] when it was
   chosen was that the code space was full when the Standard was first
   published.  There was no practical way to add characters or change
   code point assignments without being obviously incompatible.  Unicode
   does not have that property: there are large blocks of space reserved
   for future expansion and new versions, with new characters and code
   point assignments, appear at regular intervals.

   While there are some security issues if people deliberately try to
   trick the system (see Section 6), Unicode version changes should not
   have a significant impact on the text stream specification of this

Klensin & Padlipsky     Expires October 31, 2006                [Page 5]

Internet-Draft                Network UTF-8                   April 2006

   document for the following reasons:

   o  The transformation between Unicode code table positions and the
      corresponding UTF-8 code is algorithmic; it does not depend on
      whether a code point has been assigned or not.

   o  The normalization specified here, NFC (see Section 3), performs a
      very limited set of mappings, much more limited than those of the
      more extensive NFKC used in, e.g., nameprep [RFC3491].  Assuming
      that the Unicode Consortium is consistent with its stated rules
      and does not add any more precomposed characters whose equivalents
      can be built up from composing characters, the NFC tables should
      be completely forward and backward-stable, with no additions or
      changes for characters added in future versions of the Standard.

   Were Unicode to be changed in a way that violated these assumptions,
   i.e., that either invalidated the string order of RFC 3629 or that
   required additions or changes to the mappings of NFC, this
   specification would not apply.  Put differently, this specification
   applies only to versions of Unicode starting with version 3.2 and
   extending to, but not including, any version for which those
   conditions do not apply.  This specification therefore applies to
   versions of Unicode through and including the version current when it
   is written, version 4.1.0 [Unicode410].  Subsequent versions would
   require changes to this specification, presumably including string-
   type labeling in all cases.  Where this specification is referenced
   in a specification or implementation, otherwise unidentified UTF-8
   strings are to be treated as conforming to it.

5.  Context of this Proposal

   [[anchor5: RFC Editor: This section to be removed before publication
   if it lasts even that long]]

   There has been some small amount of confusion about the motivation
   for this proposal given that, e.g., MIME and HTTP have their own
   rules about UTF-8 character types.  The answer is that we have
   several protocols are dependent on either Telnet or other
   arrangements requiring a standard, interoperable, string definition
   without specific content-labels of one sort or another.  In
   particular, if this proposal is approved, or even appears to be
   getting significant traction, it will be immediately followed by a
   Telnet option to specify this type of stream (requiring some special
   provisions for Telnet control codes, of course) and an FTP extension
   to permit a new "Unicode text" data TYPE.

Klensin & Padlipsky     Expires October 31, 2006                [Page 6]

Internet-Draft                Network UTF-8                   April 2006

6.  Security Considerations

   This specification provides a standard form for the use of Unicode as
   "network text".  The same security issues that apply to UTF-8, and
   discussed in [RFC3629] could be argued for it, although it should be
   slightly less subject to some risks by virtue of requiring NFC
   normalization and generally being somewhat more restrictive.

   While not specifically a security issue, the requirement in NVT, and
   hence here, that the CR character never appear alone but only when
   followed by ASCII NUL (an octet with all bits zero) may be
   problematic for some programming languages, and hence a trap for the
   unwary, unless caution is used.  This may be an additional reason to
   avoid the use of CR entirely, as suggested above.

   The discussion about Unicode versions above (Section 4) makes several
   assumptions about future versions of Unicode, about NFC normalization
   being applied properly, and about UTF-8 being processed and
   transmitted exactly as specified [RFC3629].  If any of those
   assumptions are not correct, then there are cases in which strings
   that would be considered equivalent do not compare equal.  Robust
   code should be prepared for those possibilities.

7.  Acknowledgments

   Many thanks to Mark Davis, Martin Duerst, and Michel Suignard for
   suggestions about Unicode normalization that led to the format
   described here.

8.  References

8.1.  Normative References

              International Organization for Standardization,
              "Information Technology - Universal Multiple- Octet Coded
              Character Set (UCS) - Part 1: Architecture and Basic
              Multilingual Plane"", ISO/IEC 10646-1:2000, October 2000.

   [NFC]      Davis, M. and M. Duerst, "Unicode Standard Annex #15:
              Unicode Normalization Forms", March 2005,

   [RFC0137]  O'Sullivan, T., "Telnet Protocol - a proposed document",
              RFC 137, April 1971.

Klensin & Padlipsky     Expires October 31, 2006                [Page 7]

Internet-Draft                Network UTF-8                   April 2006

   [RFC0139]  O'Sullivan, T., "Discussion of Telnet Protocol", RFC 139,
              May 1971.

   [RFC0318]  Postel, J., "Telnet Protocols", RFC 318, April 1972.

   [RFC0854]  Postel, J. and J. Reynolds, "Telnet Protocol
              Specification", STD 8, RFC 854, May 1983.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC3629]  Yergeau, F., "UTF-8, a transformation format of ISO
              10646", STD 63, RFC 3629, November 2003.

   [Unicode]  The Unicode Consortium, "The Unicode Standard, Version
              3.0", 2000.

              (Reading, MA, Addison-Wesley, 2000.  ISBN 0-201-61633-5).
              Version 3.2 consists of the definition in that book as
              amended by the Unicode Standard Annex #27: Unicode 3.1
              ( and by the Unicode
              Standard Annex #28: Unicode 3.2

              The Unicode Consortium, "The Unicode Standard, Version
              4.1.0", March 2005.

              Defined by: The Unicode Standard, Version 4.0 (Boston, MA,
              Addison-Wesley, 2003.  ISBN 0-321-18578-1), as amended by
              Unicode 4.0.1
              ( and by
              Unicode 4.1.0

8.2.  Informative References

              International Organization for Standardization,
              "Information Processing: ISO 7-bit and 8-bit coded
              character sets: Code extension techniques", ISO Standard
              2022, 1986.

              International Organization for Standardization,
              "Information technology - ISO 7-bit coded character set
              for information interchange", ISO Standard 646, 1991.

Klensin & Padlipsky     Expires October 31, 2006                [Page 8]

Internet-Draft                Network UTF-8                   April 2006

              International Organization for Standardization,
              "Information processing - 8-bit single-byte coded graphic
              character sets - Part 1: Latin alphabet No. 1 (1998) -
              Part 2: Latin alphabet No. 2 (1999) - Part 3: Latin
              alphabet No. 3 (1999) - Part 4: Latin alphabet No. 4
              (1998) - Part 5: Latin/Cyrillic alphabet (1999) - Part 6:
              Latin/Arabic alphabet (1999) - Part 7: Latin/Greek
              alphabet (2003) - Part 8: Latin/Hebrew alphabet (1999) -
              Part 9: Latin alphabet No. 5 (1999) - Part 10: Latin
              alphabet No. 6 (1998) - Part 11: Latin/Thai alphabet
              (2001) - Part 13: Latin alphabet No. 7 (1998) - Part 14:
              Latin alphabet No. 8 (Celtic) (1998) - Part 15: Latin
              alphabet No. 9 (1999) - Part 16: Part 16: Latin alphabet
              No. 10 (2001)", ISO Standard 8859, 2003.

   [RFC0020]  Cerf, V., "ASCII format for network interchange", RFC 20,
              October 1969.

   [RFC0097]  Melvin, J. and R. Watson, "First Cut at a Proposed Telnet
              Protocol", RFC 97, February 1971.

   [RFC0698]  Mock, T., "Telnet extended ASCII option", RFC 698,
              July 1975.

   [RFC2277]  Alvestrand, H., "IETF Policy on Character Sets and
              Languages", BCP 18, RFC 2277, January 1998.

   [RFC3491]  Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
              Profile for Internationalized Domain Names (IDN)",
              RFC 3491, March 2003.

              American National Standards Institute (formerly United
              States of America Standards Institute), "USA Code for
              Information Interchange", ANSI X3.4-1968, 1968.

              ANSI X3.4-1968 has been replaced by newer versions with
              slight modifications, but the 1968 version remains
              definitive for the Internet.

Klensin & Padlipsky     Expires October 31, 2006                [Page 9]

Internet-Draft                Network UTF-8                   April 2006

Authors' Addresses

   John C Klensin
   1770 Massachusetts Ave, #322
   Cambridge, MA  02140

   Phone: +1 617 491 5735

   Michael A. Padlipsky
   8011 Stewart Ave.
   Los Angeles, CA  90045

   Phone: +1 310-670-4288

Klensin & Padlipsky     Expires October 31, 2006               [Page 10]

Internet-Draft                Network UTF-8                   April 2006

Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at

Disclaimer of Validity

   This document and the information contained herein are provided on an

Copyright Statement

   Copyright (C) The Internet Society (2006).  This document is subject
   to the rights, licenses and restrictions contained in BCP 78, and
   except as set forth therein, the authors retain all their rights.


   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Klensin & Padlipsky     Expires October 31, 2006               [Page 11]