FTPEXT Working Group                                         G. Lundberg
Internet-Draft                                 WU-FTPD Development Group
Expiration Date: November 28, 2002                              May 2002

                          UTF-8 Option for FTP

Status of this Memo

   This document is an Internet-Draft and is subject to all provisions
   of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   To view the list of IETF Internet-Draft Mirror Sites, see

Copyright Notice

   Copyright (C) The Internet Society (2002).  All Rights Reserved.


   This document specifies an extension to the File Transfer Protocol
   (FTP) which provides for inter-operation between existing
   implementations and those supporting the exchange of UTF-8 encoded
   pathnames, and clarifies certain issues involved with UTF-8 encoding.

   It introduces a new option, UTF-8, negotiated by use of the OPTS
   command.  Through use of this option, the user informs the server of
   its willingness to accept UTF-8 encoded pathnames.  The proposed
   extension requires that neither party transmit UTF-8 encoded
   pathnames without having first successfully negotiated this option.

   Implementation of this extension is RECOMMENDED.

Lundberg                Expires November 28, 2002               [Page 1]

Internet-Draft            UTF-8 Option for FTP                  May 2002

Table of Contents

      Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . .   1
   1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . .   3
   2. UTF-8 Option . . . . . . . . . . . . . . . . . . . . . . . . .   5
   3. UTF-8 Encoding Issues  . . . . . . . . . . . . . . . . . . . .   7
   4. Misuse CR NUL in Pathnames . . . . . . . . . . . . . . . . . .   7
   5. ABNF for Pathnames . . . . . . . . . . . . . . . . . . . . . .   7
   6. IANA Considerations  . . . . . . . . . . . . . . . . . . . . .   8
   7. Security Considerations  . . . . . . . . . . . . . . . . . . .   8
      Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . .   8
      Normative References . . . . . . . . . . . . . . . . . . . . .   8
      Author's Address . . . . . . . . . . . . . . . . . . . . . . .   9
      Annex A - Specific Changes to Existing Specifications  . . . .  10
      Full Copyright Statement . . . . . . . . . . . . . . . . . . .  11

Lundberg                Expires November 28, 2002               [Page 2]

Internet-Draft            UTF-8 Option for FTP                  May 2002

1. Introduction

   Internationalization of the File Transfer Protocol [RFC2640] enhances
   the capabilities of the FTP by removing the 7-bit restrictions on
   pathnames used in commands and responses.  It defines a single
   character set, in addition to NVT ASCII and EBCDIC, which is to be
   understandable by all systems, and specifies that character set to be
   ISO/IEC 10646:1993 (UCS) using UTF-8 encoding when exchanging

   UTF-8 encoding is a file-safe encoding which may avoid the use of
   byte values that have special significance during the parsing of
   pathname character strings.  It represents each UCS character as a
   sequence of 1 to 6 bytes in length, using a modified Huffman encoding
   scheme.  For all sequences of one byte the most significant bit is
   ZERO.  For all sequences of more than one byte the number of ONE bits
   in the first byte, starting from the most significant bit position,
   indicates the number of bytes in the UTF-8 encoded sequence, followed
   by a ZERO bit.

   A property of UTF-8 encoding is that its single byte sequence is
   consistent with the 7-bit ASCII character set.  [RFC2640] incorrectly
   asserts that this feature of UTF-8 encoding will allow existing
   implementations to inter-operate with implementations which support
   UTF-8 encoding.

   [RFC2640] further ignores problems of inter-operation by only
   requiring that conforming implementations must support UTF-8 encoding
   for the transfer and receipt of pathnames.  It contains a great deal
   of discussion of how a compliant implementation should treat UTF-8
   encoding under various conditions.  Unfortunately, other than to
   allow existing implementations to continue to use local character set
   encodings where pathnames encoded in those character sets are not
   UTF-8 encoded (and thus not ASCII), [RFC2640] gives no thought to the
   effect of UTF-8 encoding upon existing implementations.

   At a strictly protocol level, [RFC2640] is generally correct; the use
   of UTF-8 encoding should not inherently prevent most existing
   implementations from correctly reading the character sequence as a
   pathname.  In the best case, existing implementations, when presented
   with UTF-8 encoded pathname, will treat it as an error and recover.
   The possibility exists, however, that existing implementations will
   not detect the error, and either fail directly, pass the information
   to the host system causing a failure at that level, or treat the
   characters as invocations of special functions (such as end-of-line
   markers or command-line editing).  Experience has shown that
   presenting hosts and applications with unexpected character sequences
   may result in serious security issues [RR, EL, AD].

Lundberg                Expires November 28, 2002               [Page 3]

Internet-Draft            UTF-8 Option for FTP                  May 2002

   The specifications of [RFC2640] provide the means for the server to
   indicate its willingness to accept UTF-8 encoded pathnames.  To
   restore inter-operation with existing implementations, the FTP should
   provide a means for the user to express its willingness to accept
   UTF-8 encoded pathnames, and servers should not transmit UTF-8
   encoded pathnames without prior authorization from the user.

   [RFC2640] also incorrectly requires interpreting the Telnet end-of-
   line sequence CR NUL as a pathname character.  [RFC1123] attempted to
   address this issue with inter-operation of the Telnet protocol, but
   its effect upon the FTP has been largely ignored.  The intention of
   [RFC1123] was to clarify that a server implemenation must transmit
   the Telnet end-of-line sequence as CR LF, but that both user and
   server implementations must be prepared to accept either CR LF or CR
   NUL as representing Telnet end-of-line when received, and that the
   choice of whether to send CR LF or CR NUL is up to the user and
   should be configurable.

   The requirement in [RFC2640] that CR NUL be an allowed pathname
   character, when considered in the context of the [RFC1123]
   requirements, should cause existing implementations to incorrectly
   interpret the sequence as Telnet end-of-line, causing loss of
   synchronization.  Experience shows implementations often fail in a
   number of ways once they have lost synchronization.  While server
   implementations usually cope fairly well with the problem, user
   implementations often lock up.  It is possible, however, that an
   implementation will fail in some critical manner that may cause
   serious security issues.

   The receipt of unexpected UTF-8 encoded information alone raises the
   possibility of serious security problems.  When taken together with
   the misuse of the Telnet end-of-line sequence, the security
   implications increase dramatically since not only is unexpected
   information being received, but the sender has control of the
   protocol's primary sequence point.

   This document addresses the deficiencies of [RFC2640] by adding a new
   option which must be successully negotiated prior to transmission of
   UTF-8 encoded pathnames, making specific clarifications in the use
   UTF-8 encoding, and clarifying the proper interpretation of the
   sequence CR NUL as being Telnet end-of-line and thus not a usable
   pathname character.

   In the development of the protocol specified in this document, an
   alternative was considered: the server could interpret the LANG
   command as an indication that the user is willing to accept UTF-8
   encoded pathnames.  In this case, the server should provide the
   language EN (English) as an alternative so that the user can

Lundberg                Expires November 28, 2002               [Page 4]

Internet-Draft            UTF-8 Option for FTP                  May 2002

   negotiate the LANG command without any change to the message text
   included with replies.  This approach unreasonably assumes that
   implementations will either adopt [RFC2640] in its entirety, or not
   at all.  While that may often be a safe assumption, the concept of
   UTF-8 encoded pathnames is logically distinct from the concept of the
   language used for free-form response text.  It unnecessarily limits
   implementers, forcing them to support the internationalization of
   response text when they desire only to allow internationalized

   When reading the following specifications, the key words "MUST",
   "RECOMMENDED",  "MAY", and "OPTIONAL" are to be interpreted as
   described in RFC 2119 [RFC2119].

2. UTF-8 Option

   The user issues the OPTS UTF-8 command to indicate its willingness to
   send and receive UTF-8 encoded pathnames over the control connection.
   Prior to sending this command, the user should not transmit UTF-8
   encoded pathnames.

   The specifications of [RFC2640] apply only to pathnames sent over the
   control connection.  The OPTS UTF-8 command provides an optional
   parameter which modifies the behavior of the NLST command.  When this
   option is specified, pathnames transmitted over the data connection
   in response to the NLST command must be UTF-8 encoded, and the data
   transmission assumes TYPE L 8 character framing.  By using this
   optional parameter, responsibility for conversion to the local
   character set of the pathnames contained in the NLST data transfer
   shifts from the server implementation to the user.

   Before sending the UTF-8 option, the user should issue the FEAT
   command and examine the response to that command.  If the response
   contains the UTF-8 option, the user should take that option to mean
   the server is willing to transmit UTF-8 encoded pathnames, and may
   support the OPTS UTF-8 command to enable their use.  Note that the
   specification of the OPTS command, and the OPTS UTF-8 variant,
   provide a reliable means to determine support for UTF-8 encoded
   pathnames; no harmful effect occurs if the user does not issue the
   FEAT command.

   The format of the OPTS UTF-8 command is:

      OPTS <SP> UTF-8 [ <SP> NLST ] <CRLF>

   The text of the command line is not case sensitive, but should be
   transmitted in upper case as shown.

Lundberg                Expires November 28, 2002               [Page 5]

Internet-Draft            UTF-8 Option for FTP                  May 2002

   The UTF-8 option allows one optional parameter: NLST.  When present,
   NLST must be separated from the UTF-8 option by exactly one space

   Possible replies to the OPTS UTF-8 command, and their meanings,

      200 Command okay.
      421 Service not available, closing control connection.
      500 Syntax error, command unrecognized.
      501 Syntax error in parameters or arguments.
      502 Command not implemented.

   The User-FTP process must not depend upon the actual text (if any)
   included with the reply.

   A Server-FTP process which does not implement the OPTS command will
   reply with either response code 500 or 502.  For compatibility with
   existing implementations, the User-FTP process must be prepared for
   this reply and must not transmit UTF-8 encoded pathnames.

   The response code 501 indicates the Server-FTP process does not
   implement the OPTS UTF-8 command or was unable to recognize the
   parameters given with the command.  For compatibility with existing
   implementations, the User-FTP process must be prepared for this reply
   and must not transmit UTF-8 encoded pathnames.

   Prior to transmitting response code 200 in response to the OPTS UTF-8
   command, the Server-FTP must not transmit UTF-8 encoded pathnames and
   should not accept them on commands: the Server-FTP should transmit
   either response code 501 or 553 in reply to any command which
   includes a pathname outside the range of 7-bit ASCII; and the Server-
   FTP should transmit response code 550 in reply to any command to
   which the server would otherwise have sent a UTF-8 encoded pathname.

   Upon receiving response code 200, the user should transmit only UTF-8
   encoded pathnames, and should expect to receive only UTF-8 encoded
   pathnames from the server.

   The user may issue the OPTS UTF-8 command without the NLST parameter
   to restore the operation of the NLST command.

   To terminate the use of UTF-8 encoding on the control connection, the
   user must either issue the REIN command or terminate the FTP session
   and begin anew.

Lundberg                Expires November 28, 2002               [Page 6]

Internet-Draft            UTF-8 Option for FTP                  May 2002

3. UTF-8 Encoding Issues

   The UTF-8 encoding scheme allows the possibility of multiple
   encodings for a single character.

   For each character, there is a single, shortest form of the UTF-8
   encoding.  When transmitting UTF-8 encoded characters, the shortest
   form should be used.

   When interpreting received UTF-8 encoded information, the
   implementation should accept the non-shortest form as meaning the
   same character as the preferred, shortest form.  (One method would be
   for the implementation to actually replace the character with the
   shortest form encoding.)

   Implementations, however, must attach no special significance to any
   non-shortest form encodings.  In particular, the non-shortest form
   encodings for CR, LF and NUL are not to be treated as potential
   Telnet end-of-line characters.

4. Misuse of CR NUL in Pathnames

   The assertion in [RFC2640] that CR NUL is not a Telnet end-of-line
   sequence is incorrect.

   The Telnet protocol requires the character CR to always be followed
   by either the character LF or NUL.  The design of the FTP requires
   that the sequences CR LF and CR NUL be treated as a Telnet end-of-
   line and all existing implementations properly recognize them as

   Pathnames must not include the character CR.  This applies all uses
   of the character CR whether alone or followed by any other character.

5. ABNF for Pathnames

   The ABNF for pathnames presented in [RFC2640] is incorrect.

   When UTF-8 encoding is not present, the correct syntax is

      PATHNAME = *( %x20-7E )

   When UTF-8 encoding is in use, the correct syntax is

      PATHNAME = *( %x20-7E / %x80-FF )

   Note that both cases render moot all discussion about the use of the
   characters CR, LF, and NUL, in pathnames.

Lundberg                Expires November 28, 2002               [Page 7]

Internet-Draft            UTF-8 Option for FTP                  May 2002

6. IANA Considerations

   The list of valid option names for the FTP OPTS command is believed
   to be first-come first-served, and managed outside the control of the
   Internet Assigned Numbers Authority (IANA).

7. Security Issues

   While it should improve inter-operation, and therefore may improve
   security, the addition of the UTF-8 option itself should have no
   effect upon the security of the FTP, networks or hosts.

   The intention of this document is to address inter-operational issues
   in the existing protocol specifications.

   Some of those issues can lead to unexpected data appearing on the
   communications channel.  Experience shows this can lead to serious
   security issues, potentially including the compromising hosts on the
   network.  While one would hope that implementations were hardened
   against such occurances, some implementations may not be.  The
   importance of such hardening cannot be emphasized strongly enough.


   The following people provided significant assistance with the
   analysis of the problem, the proposed solution, and the preparation
   of this document:

      The members of vulnerability handling team at the CERT
      Coordination Center.

Normative References

   [RFC854]    J. Postel and J. Reynolds, "TELNET PROTOCOL
               SPECIFICATION", STD 8, RFC 854, May 1983.

   [RFC959]    J. Postel and J. Reynolds, "FILE TRANSFER PROTOCOL
               (FTP)", STD 9, RFC 959, October 1985.

   [RFC1123]   IETF, "Requirements for Internet Hosts -- Application and
               Support", STD 3, RFC 1123, October 1989.

   [RFC2119]   S. Bradner, "Key words for use in RFCs to Indicate
               Requirements Levels", RFC 2119, BCP 14, March 1997.

   [RFC2234]   D. Crocker and P. Overell, "Augmented BNF for Syntax
               Specifications: ABNF", RFC 2234, November 1997.

Lundberg                Expires November 28, 2002               [Page 8]

Internet-Draft            UTF-8 Option for FTP                  May 2002

   [RFC2389]   P. Hethmon and R. Elz, "Feature negotiation mechanism for
               the File Transfer Protocol", RFC 2389, August 1998.

   [RFC2640]   B. Curtin, "Internationalization of the File Transfer
               Protocol", RFC 2640, July 1999.

Informative References

   [RR]        R. Russell and S. Cunningham, "Hack Proofing Your
               Network: Internet Tradecraft", ISBN 1-928994-15-6,
               January 2000.

   [EL]        E. Labbate, "Vulnerability as a Function of Software
               Quality", http://rr.sans.org/code/quality.php, March

   [AD]        A. Davis, et al, "Understanding the Risks of SNMP
               March 2002.

Author's Address

   Gregory A. Lundberg
   WU-FTPD Development Group
   1441 Elmdale Drive
   Dayton, OH 45409-1615

   Phone: +1 937 299 7653
   Email: lundberg@vr.net

Lundberg                Expires November 28, 2002               [Page 9]

Internet-Draft            UTF-8 Option for FTP                  May 2002

Annex A - Specific Changes to Existing Specifications

   In summary, the specifications presented in this memo make the
   following specific changes with respect to the requirements of

    - add the option UTF-8, and require that implementations must not
      transmit UTF-8 encoded pathnames until after successful
      negotiation of the UTF-8 option;

    - clarify that UTF-8 encoding applies only to pathnames transmitted
      over the control connection, and provide a means to specify UTF-8
      encoding in the data transfer sent in response to the NLST

    - clarify the use of UTF-8 encoding with respect to non-shortest

    - clarify that the character sequence CR NUL is a Telnet end-of-line
      sequence and must not be treated as a pathname character;

    - specify the correct ABNF syntax for pathnames when UTF-8 encoding
      is, and is not, in use.

Lundberg                Expires November 28, 2002              [Page 10]

Internet-Draft            UTF-8 Option for FTP                  May 2002

Full Copyright Statement

   Copyright (C) The Internet Society (2002).  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to The Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than

   The limited permissions granted above are perpetual and will not be
   revoked by The Internet Society or its successors or assigns.

   This document and the information contained herein is provided on an


   Funding for the RFC Editor function is currently provided by The
   Internet Society.

Lundberg                Expires November 28, 2002              [Page 11]