ASCII Escaping of Unicode Characters
RFC 5137

Network Working Group                                         J. Klensin
Request for Comments: 5137                                 February 2008
BCP: 137
Category: Best Current Practice

                  ASCII Escaping of Unicode Characters

Status of This Memo

   This document specifies an Internet Best Current Practices for the
   Internet Community, and requests discussion and suggestions for
   improvements.  Distribution of this memo is unlimited.

Abstract

   There are a number of circumstances in which an escape mechanism is
   needed in conjunction with a protocol to encode characters that
   cannot be represented or transmitted directly.  With ASCII coding,
   the traditional escape has been either the decimal or hexadecimal
   numeric value of the character, written in a variety of different
   ways.  The move to Unicode, where characters occupy two or more
   octets and may be coded in several different forms, has further
   complicated the question of escapes.  This document discusses some
   options now in use and discusses considerations for selecting one for
   use in new IETF protocols, and protocols that are now being
   internationalized.

Klensin                  Best Current Practice                  [Page 1]
RFC 5137                    Unicode Escapes                February 2008

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.1.  Context and Background . . . . . . . . . . . . . . . . . .  3
     1.2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  4
     1.3.  Discussion List  . . . . . . . . . . . . . . . . . . . . .  4
   2.  Encodings that Represent Unicode Code Points: Code
       Position versus UTF-8 or UTF-16 Octets . . . . . . . . . . . .  4
   3.  Referring to Unicode Characters  . . . . . . . . . . . . . . .  5
   4.  Syntax for Code Point Escapes  . . . . . . . . . . . . . . . .  6
   5.  Recommended Presentation Variants for Unicode Code Point
       Escapes  . . . . . . . . . . . . . . . . . . . . . . . . . . .  7
     5.1.  Backslash-U with Delimiters  . . . . . . . . . . . . . . .  7
     5.2.  XML and HTML . . . . . . . . . . . . . . . . . . . . . . .  7
   6.  Forms that Are Normally Not Recommended  . . . . . . . . . . .  8
     6.1.  The C Programming Language: Backslash-U  . . . . . . . . .  8
     6.2.  Perl: A Hexadecimal String . . . . . . . . . . . . . . . .  8
     6.3.  Java: Escaped UTF-16 . . . . . . . . . . . . . . . . . . .  9
   7.  Security Considerations  . . . . . . . . . . . . . . . . . . .  9
   8.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . .  9
   9.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 10
     9.1.  Normative References . . . . . . . . . . . . . . . . . . . 10
     9.2.  Informative References . . . . . . . . . . . . . . . . . . 10
   Appendix A.  Formal Syntax for Forms Not Recommended . . . . . . . 12
     A.1.  The C Programming Language Form  . . . . . . . . . . . . . 12
     A.2.  Perl Form  . . . . . . . . . . . . . . . . . . . . . . . . 12
     A.3.  Java Form  . . . . . . . . . . . . . . . . . . . . . . . . 12

Klensin                  Best Current Practice                  [Page 2]
RFC 5137                    Unicode Escapes                February 2008

1.  Introduction

1.1.  Context and Background

   There are a number of circumstances in which an escape mechanism is
   needed in conjunction with a protocol to encode characters that
   cannot be represented or transmitted directly.  With ASCII [ASCII]
   coding, the traditional escape has been either the decimal or
   hexadecimal numeric value of the character, written in a variety of
   different ways.  For example, in different contexts, we have seen
   %dNN or %NN for the decimal form, %NN, %xNN, X'nn', and %X'NN' for
   the hexadecimal form. "%NN" has become popular in recent years to
   represent a hexadecimal value without further qualification, perhaps
   as a consequence of its use in URLs and their prevalence.  There are
   even some applications around in which octal forms are used and,
   while they do not generalize well, the MIME Quoted-Printable and
   Encoded-word forms can be thought of as yet another set of escapes.
   So, even for the fairly simple cases of ASCII and standard built by
   extending ASCII, such as the ISO 8859 family, we have been living
   with several different escaping forms, each the result of some
   history.

   When one moves to Unicode [Unicode] [ISO10646], where characters
   occupy two or more octets and may be coded in several different
   forms, the question of escapes becomes even more complicated.
   Unicode represents characters as code points: numeric values from 0
   to hex 10FFFF.  When referencing code points in flowing text, they