Network Working Group J. Klensin
Internet-Draft M. Padlipsky
Expires: November 23, 2006 May 22, 2006
Unicode Format for Network Interchange
draft-klensin-net-utf8-01.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on November 23, 2006.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
The Internet today is in need of a standardized form for the
transmission of internationalized "text" information, paralleling the
specifications for the use of ASCII that date from the early days of
the ARPANET. This document specifies that format, using UTF-8 with
specification of normalization and specific line-ending sequences.
Klensin & Padlipsky Expires November 23, 2006 [Page 1]
Internet-Draft Network UTF-8 May 2006
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Background . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4
2. Net-Unicode . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Normalization . . . . . . . . . . . . . . . . . . . . . . . . 5
4. Versions of Unicode . . . . . . . . . . . . . . . . . . . . . 5
5. Context of this Proposal . . . . . . . . . . . . . . . . . . . 6
6. Security Considerations . . . . . . . . . . . . . . . . . . . 7
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 7
8. Change log . . . . . . . . . . . . . . . . . . . . . . . . . . 7
8.1. Changes from -00 to -01 . . . . . . . . . . . . . . . . . 8
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8
9.1. Normative References . . . . . . . . . . . . . . . . . . . 8
9.2. Informative References . . . . . . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 11
Intellectual Property and Copyright Statements . . . . . . . . . . 12
Klensin & Padlipsky Expires November 23, 2006 [Page 2]
Internet-Draft Network UTF-8 May 2006
1. Introduction
1.1. Background
This subsection contains a review of prior work in the ARPANET and
Internet to establish a standard text type, work that establishes the
context and motivation for the approach taken in this document.
Those who are uninterested in that review and analysis can safely
skip to the next section.
One of the earlier application design decisions made in the
development of ARPANET, a decision that was carried forward into the
Internet, was the decision to standardize on a single and very
specific coding for "text" to be passed across the network [RFC0020].
Hosts on the network were then responsible for translating or mapping
from whatever character coding conventions were used locally to that
common intermediate representation, with sending hosts mapping to it
and receiving ones mapping from it to their local forms as needed.
It is interesting to note that at the time the ARPANET was being
developed, participating Host operating systems used at least three
different character coding standards: the antiquated BCD (Binary
Coded Decimal), the then-dominant major manufacturer-backed EBCDIC
(Extendended BCD Interchange Code), and the then-still emerging ASCII
(American Standard Code for Information Interchange). Since the
ARPANET was an "open" project and EBCDIC was intimately linked to a
particular hardware vendor, the original Network Working Group agreed
that its standard should be ASCII. That ASCII form was precisely
"7-bit ASCII in an 8-bit field", which was in effect a compromise
between Hosts that were natively 7-bit oriented (e.g., with five
seven-bit characters in a 36 bit word), those that were 8-bit
oriented (using eight-bit characers) and those that placed the seven-
bit ASCII characters in 9-bit fields with two leading zero bits (four
characters in a 36 bit word).
More standardization was suggested in the first preliminary
description of the Telnet protocol [RFC0097]. With the iterations of
that protocol [RFC0137] [RFC0139] and the drawing together of an
essentially formal definition somewhat later [RFC0318], a standard
abstraction, the Network Virtual Terminal (NVT) was established. NVT
character-coding conventions (initially called "Telnet ASCII" and
later called "NVT ASCII", or, more casually, "network ASCII")
included the requirement that Carriage Return - Line Feed (CRLF) be
the common representation for ending lines of text (given that some
participating "Host" operating systems used the one natively, some
the other, and at least one used both) and specified conventions for
some other characters. Also, since NVT ASCII was restricted to
seven-bit characters, use of the high-order bit in octets was
reserved for the transmission of control signaling information.
Klensin & Padlipsky Expires November 23, 2006 [Page 3]
Internet-Draft Network UTF-8 May 2006
At a very high level, the concept was that a system could use
whatever character coding and line representations were appropriate
locally, but text transmitted over the network as text must conform
to the single "network virtual terminal" convention.
In our more internationalized world, "text" clearly no longer equates
unambiguously to "network ASCII". Fortunately, however, we are
converging on Unicode [Unicode], [ISO10646] as a single international
interchange character coding and no longer need to deal with per-
script standards for character sets (e.g., one standard for each of
Arabic, Cyrillic, Devanagari, etc., or even standards keyed to
languages that are usually considered to share a script, such as
French, German, or Swedish). Unfortunately, though, while it is
certainly time to define a Unicode-based text type for use as a
common text interchange format, "use Unicode" involves even more
ambiguity than "use ASCII" did decades ago. Unicode can be
transmitted in UCS-2 (all characters 16 bit), UCS-4 or UTF-16 (all
characters 32 bit), UTF-8 (a variable-length encoding with some
additional properties) [RFC3629], and other forms. Also, as with
ASCII, any of these forms may have different line-ending conventions.
This document proposes to establish "Net-Unicode" as a new
standardized text transmission form for the Internet, to serve as an
internationalized alternative for NVT ASCII when specified in new --
and, where appropriate, updated -- protocols. UTF-8 [RFC3629] is
chosen for the coding because it has good compatibility properties
with ASCII and for other reasons discussed in the existing IETF
character set policy [RFC2277].
In circumstances in which there is a choice, use of Unicode and the
text encoding specified here is preferred to the double-byte encoding
of "extended ASCII" [RFC0698] and SHOULD be used.
1.2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
2. Net-Unicode
The Network Unicode (Net-Unicode or Net-UTF-8) format is defined as
follows:
1. Characters will be coded in UTF-8 as defined in [RFC3629]
Klensin & Padlipsky Expires November 23, 2006 [Page 4]
Internet-Draft Network UTF-8 May 2006
2. Line-endings will be indicated by the sequence Carriage-Return
(U+000D) followed by Line-Feed (U+000A)
3. Before transmission, all character sequences will be normalized
according to Unicode method "NFC" (see Section 3).
4. As suggested in Section 6 of RFC 3629, the Byte Order Mark
("BOM") signature MUST NOT appear at the beginning of these text
strings.
The NVT specification contained a number of additional provisions,
e.g., for the optional use of backspacing and "bare CR" (sent as CR
NUL) to generate overstruck character sequences. The much greater
number of precomposed characters in Unicode, the availability of
combining characters, and the growing use of markup conventions of
various types (rather than character-coding) to show, e.g., emphasis,
should make such sequences largely unnecessary. Because they were
optional in NVT applications, they SHOULD be avoided if at all
possible; if they are used, this specification does not change the
NVT rules and conventions of RFC 318 and RFC 854 [RFC0854] including
the prohibition on CR without NUL (note that NUL, X'00' is hostile to
programming languages that use that character as a string delimiter).
3. Normalization
There are cases where strings of Unicode are fundamentally
equivalent, essentially representing the same text. These are called
"canonical equivalents" in the Unicode Standard. For example, the
following pairs of strings are canonically equivalent:
U+2126 OHM SIGN U+03A9 GREEK CAPITAL LETTER OMEGA
U+0061 LATIN SMALL LETTER A, U+0300 COMBINING GRAVE ACCENT
U+00E0 LATIN SMALL LETTER A WITH GRAVE
Comparison of strings becomes much easier if any such cases are
always represented by a single unique form. The Unicode Consortium
specifies a normalization method, known as NFC [NFC], which provides
the necessary mappings and mechanisms to convert all canonically
equivalent sequences a single unique form. Typically, this form
produces precomposed characters for any sequences that can be
represented in that fashion. It also reorders other combining marks
so that they have a unique order.
4. Versions of Unicode
In retrospect, one of the advantages of ASCII [X3.4-1978] when it was
chosen was that the code space was full when the Standard was first
Klensin & Padlipsky Expires November 23, 2006 [Page 5]
Internet-Draft Network UTF-8 May 2006
published. There was no practical way to add characters or change
code point assignments without being obviously incompatible. Unicode
does not have that property: there are large blocks of space reserved
for future expansion and new versions, with new characters and code
point assignments, appear at regular intervals.
While there are some security issues if people deliberately try to
trick the system (see Section 6), Unicode version changes should not
have a significant impact on the text stream specification of this
document for the following reasons:
o The transformation between Unicode code table positions and the
corresponding UTF-8 code is algorithmic; it does not depend on
whether a code point has been assigned or not.
o The normalization specified here, NFC (see Section 3), performs a
very limited set of mappings, much more limited than those of the
more extensive NFKC used in, e.g., nameprep [RFC3491]. Assuming
that the Unicode Consortium is consistent with its stated rules
and does not add any more precomposed characters whose equivalents
can be built up from composing characters, the NFC tables should
be completely forward and backward-stable, with no additions or
changes for characters added in future versions of the Standard.
Were Unicode to be changed in a way that violated these assumptions,
i.e., that either invalidated the string order of RFC 3629 or that
required additions or changes to the mappings of NFC, this
specification would not apply. Put differently, this specification
applies only to versions of Unicode starting with version 3.2 and
extending to, but not including, any version for which those
conditions do not apply. This specification therefore applies to
versions of Unicode through and including the version current when it
is written, version 4.1.0 [Unicode410]. Subsequent versions would
require changes to this specification, presumably including string-
type labeling in all cases. Where this specification is referenced
in a specification or implementation, otherwise unidentified UTF-8
strings are to be treated as conforming to it.
5. Context of this Proposal
[[anchor5: RFC Editor: This section to be removed before publication
if it lasts even that long]]
There has been some small amount of confusion about the motivation
for this proposal given that, e.g., MIME and HTTP have their own
rules about UTF-8 character types. The answer is that we have
several protocols are dependent on either Telnet or other
Klensin & Padlipsky Expires November 23, 2006 [Page 6]
Internet-Draft Network UTF-8 May 2006
arrangements requiring a standard, interoperable, string definition
without specific content-labels of one sort or another. In
particular, if this proposal is approved, or even appears to be
getting significant traction, it will be immediately followed by a
Telnet option to specify this type of stream (requiring some special
provisions for Telnet control codes, of course) and an FTP extension
to permit a new "Unicode text" data TYPE.
6. Security Considerations
This specification provides a standard form for the use of Unicode as
"network text". The same security issues that apply to UTF-8, and
discussed in [RFC3629] could be argued for it, although it should be
slightly less subject to some risks by virtue of requiring NFC
normalization and generally being somewhat more restrictive.
While not specifically a security issue, the requirement in NVT, and
hence here, that the CR character never appear alone but only when
followed by ASCII NUL (an octet with all bits zero) may be
problematic for some programming languages, and hence a trap for the
unwary, unless caution is used. This may be an additional reason to
avoid the use of CR entirely, as suggested above.
The discussion about Unicode versions above (Section 4) makes several
assumptions about future versions of Unicode, about NFC normalization
being applied properly, and about UTF-8 being processed and
transmitted exactly as specified [RFC3629]. If any of those
assumptions are not correct, then there are cases in which strings
that would be considered equivalent do not compare equal. Robust
code should be prepared for those possibilities.
7. Acknowledgments
Many thanks to Mark Davis, Martin Duerst, and Michel Suignard for
suggestions about Unicode normalization that led to the format
described here and especially to Mark for providing the paragraphs
that describe the role of NFC.
8. Change log
[[anchor7: RFC Editor: Please remove this section before
publication.]]
Klensin & Padlipsky Expires November 23, 2006 [Page 7]
Internet-Draft Network UTF-8 May 2006
8.1. Changes from -00 to -01
o Replaced the section on Normalization with text provided by Mark
Davis
o Several small editorial changes and corrections.
9. References
9.1. Normative References
[ISO10646]
International Organization for Standardization,
"Information Technology - Universal Multiple- Octet Coded
Character Set (UCS) - Part 1: Architecture and Basic
Multilingual Plane"", ISO/IEC 10646-1:2000, October 2000.
[NFC] Davis, M. and M. Duerst, "Unicode Standard Annex #15:
Unicode Normalization Forms", March 2005,
<http://www.unicode.org/reports/tr15/>.
[RFC0137] O'Sullivan, T., "Telnet Protocol - a proposed document",
RFC 137, April 1971.
[RFC0139] O'Sullivan, T., "Discussion of Telnet Protocol", RFC 139,
May 1971.
[RFC0318] Postel, J., "Telnet Protocols", RFC 318, April 1972.
[RFC0854] Postel, J. and J. Reynolds, "Telnet Protocol
Specification", STD 8, RFC 854, May 1983.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, November 2003.
[Unicode] The Unicode Consortium, "The Unicode Standard, Version
3.0", 2000.
(Reading, MA, Addison-Wesley, 2000. ISBN 0-201-61633-5).
Version 3.2 consists of the definition in that book as
amended by the Unicode Standard Annex #27: Unicode 3.1
(http://www.unicode.org/reports/tr27/) and by the Unicode
Standard Annex #28: Unicode 3.2
(http://www.unicode.org/reports/tr28/).
Klensin & Padlipsky Expires November 23, 2006 [Page 8]
Internet-Draft Network UTF-8 May 2006
[Unicode410]
The Unicode Consortium, "The Unicode Standard, Version
4.1.0", March 2005.
Defined by: The Unicode Standard, Version 4.0 (Boston, MA,
Addison-Wesley, 2003. ISBN 0-321-18578-1), as amended by
Unicode 4.0.1
(http://www.unicode.org/versions/Unicode4.0.1) and by
Unicode 4.1.0
(http://www.unicode.org/versions/Unicode4.1.0).
9.2. Informative References
[ISO.2022.1986]
International Organization for Standardization,
"Information Processing: ISO 7-bit and 8-bit coded
character sets: Code extension techniques", ISO Standard
2022, 1986.
[ISO.646.1991]
International Organization for Standardization,
"Information technology - ISO 7-bit coded character set
for information interchange", ISO Standard 646, 1991.
[ISO.8859.2003]
International Organization for Standardization,
"Information processing - 8-bit single-byte coded graphic
character sets - Part 1: Latin alphabet No. 1 (1998) -
Part 2: Latin alphabet No. 2 (1999) - Part 3: Latin
alphabet No. 3 (1999) - Part 4: Latin alphabet No. 4
(1998) - Part 5: Latin/Cyrillic alphabet (1999) - Part 6:
Latin/Arabic alphabet (1999) - Part 7: Latin/Greek
alphabet (2003) - Part 8: Latin/Hebrew alphabet (1999) -
Part 9: Latin alphabet No. 5 (1999) - Part 10: Latin
alphabet No. 6 (1998) - Part 11: Latin/Thai alphabet
(2001) - Part 13: Latin alphabet No. 7 (1998) - Part 14:
Latin alphabet No. 8 (Celtic) (1998) - Part 15: Latin
alphabet No. 9 (1999) - Part 16: Part 16: Latin alphabet
No. 10 (2001)", ISO Standard 8859, 2003.
[RFC0020] Cerf, V., "ASCII format for network interchange", RFC 20,
October 1969.
[RFC0097] Melvin, J. and R. Watson, "First Cut at a Proposed Telnet
Protocol", RFC 97, February 1971.
[RFC0698] Mock, T., "Telnet extended ASCII option", RFC 698,
July 1975.
Klensin & Padlipsky Expires November 23, 2006 [Page 9]
Internet-Draft Network UTF-8 May 2006
[RFC2277] Alvestrand, H., "IETF Policy on Character Sets and
Languages", BCP 18, RFC 2277, January 1998.
[RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
Profile for Internationalized Domain Names (IDN)",
RFC 3491, March 2003.
[X3.4-1978]
American National Standards Institute (formerly United
States of America Standards Institute), "USA Code for
Information Interchange", ANSI X3.4-1968, 1968.
ANSI X3.4-1968 has been replaced by newer versions with
slight modifications, but the 1968 version remains
definitive for the Internet.
Klensin & Padlipsky Expires November 23, 2006 [Page 10]
Internet-Draft Network UTF-8 May 2006
Authors' Addresses
John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA 02140
USA
Phone: +1 617 491 5735
Email: john-ietf@jck.com
Michael A. Padlipsky
8011 Stewart Ave.
Los Angeles, CA 90045
USA
Phone: +1 310-670-4288
Email: the.map@alum.mit.edu
Klensin & Padlipsky Expires November 23, 2006 [Page 11]
Internet-Draft Network UTF-8 May 2006
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2006). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Klensin & Padlipsky Expires November 23, 2006 [Page 12]
