IPS Working Group               M. Rajagopal, R. Bhagwat, R. A. Helland,
INTERNET-DRAFT                                           LightSand Comm.
<draft-ietf-ips-fcovertcpip-03.txt>           E. Rodriguez, Lucent Tech.
(Expires December, 2001)                              C. Carlson, QLogic
Category: standards-track                              D. Fraser, Compaq
                                                      D. Peterson, Cisco
                                                   L. Lamers, SAN Valley
                                     V. Chau, G. Hecht, Gadzoox Networks
                          S. Wilson, B. Snively, R. Weber, Brocade Comm.
                                   M. O'Donnell, A. Rijhsinghani, McDATA
                                            S. Rupanagunta, Aarohi Comm.
                                            V. Rangan, Rhapsody Networks
                                             J. Nelson, K. Hirata, Vixel
                                               M. Merhar, Pirus Networks
                                                     N. Wanamaker, Akara


                    Fibre Channel Over TCP/IP (FCIP)

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC 2026 [1].

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that
   other groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time. It is inappropriate to use Internet-Drafts as Reference
   material or to cite them other than as "work in progress".

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/lid-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

Abstract

   Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the
   interconnection of islands of Fibre Channel storage area networks
   over IP-based networks to form a unified storage area network in a
   single Fibre Channel fabric. FCIP relies on IP-based network
   services to provide the connectivity between the storage area
   network islands over local area networks, metropolitan area
   networks, or wide area networks.


Rajagopal, et al.               Standards Track                 [Page 1]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [2].

Table Of Contents

   1. Purpose, Motivation and Objectives . . . . . . . . . . . . . . . 3
   2. Relationship to Fibre Channel Standards  . . . . . . . . . . . . 4
   2.1 Relevant Fibre Channel Standards  . . . . . . . . . . . . . . . 4
   2.2 This Specification and Fibre Channel Standards  . . . . . . . . 4
   3. Terminology  . . . . . . . . . . . . . . . . . . . . . . . . . . 5
   4. Open Issues  . . . . . . . . . . . . . . . . . . . . . . . . . . 6
   5. Protocol Summary . . . . . . . . . . . . . . . . . . . . . . . . 7
   6. The FCIP Model . . . . . . . . . . . . . . . . . . . . . . . . . 9
   6.1 FCIP Protocol Model . . . . . . . . . . . . . . . . . . . . . . 9
   6.2 FCIP Link . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
   6.3 FC Entity  . . . . . . . . . . . . . . . . . . . . . . . . . . 10
   6.4 FCIP Entity  . . . . . . . . . . . . . . . . . . . . . . . . . 11
   6.5 FCIP Link Endpoint (FCIP_LEP)  . . . . . . . . . . . . . . . . 12
   6.6 FCIP Data Engine (FCIP_DE) . . . . . . . . . . . . . . . . . . 13
   6.6.1 FCIP Encapsulation of FC Frames  . . . . . . . . . . . . . . 15
   6.6.2 FCIP Data Engine Error Detection and Recover . . . . . . . . 16
   6.6.2.1 TCP Assistance With Error Detection and Recovery . . . . . 16
   6.6.2.2 Errors in FCIP Headers and Discarding FCIP Frames  . . . . 16
   6.6.2.3 IP Network Transit Time Validation . . . . . . . . . . . . 17
   6.6.2.4 Synchronization Failures . . . . . . . . . . . . . . . . . 17
   7. TCP Connection Management . . . . . . . . . . . . . . . . . . . 18
   7.1 TCP Connection Establishment . . . . . . . . . . . . . . . . . 18
   7.1.1 Creating a New TCP Connection  . . . . . . . . . . . . . . . 18
   7.1.2 Processing TCP Connect Requests  . . . . . . . . . . . . . . 19
   7.2 TCP Connection Parameters  . . . . . . . . . . . . . . . . . . 19
   7.2.1 TCP Selective Acknowledgement Option . . . . . . . . . . . . 20
   7.2.2 TCP Window Scale Option  . . . . . . . . . . . . . . . . . . 20
   7.2.3 IP DSCP Option . . . . . . . . . . . . . . . . . . . . . . . 20
   7.2.4 Protection against sequence number wrap  . . . . . . . . . . 20
   7.2.5 TCP No Delay Option  . . . . . . . . . . . . . . . . . . . . 20
   7.2.6 TCP Acknowledgement Timeout  . . . . . . . . . . . . . . . . 20
   7.3 TCP Connection Considerations  . . . . . . . . . . . . . . . . 20
   7.4 Flow Control Mapping between TCP and FC  . . . . . . . . . . . 21
   8. Security  . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
   8.1 Considerations . . . . . . . . . . . . . . . . . . . . . . . . 22
   8.2 IP Network Security Requirements . . . . . . . . . . . . . . . 22
   8.3 Integrated Security  . . . . . . . . . . . . . . . . . . . . . 23
   8.4 External Security Gateway  . . . . . . . . . . . . . . . . . . 24
   8.5 Security Information Exchanged Between FC and FCIP Entities  . 24
   9. Performance . . . . . . . . . . . . . . . . . . . . . . . . . . 24


Rajagopal, et al.               Standards Track                 [Page 2]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   9.1 Considerations . . . . . . . . . . . . . . . . . . . . . . . . 24
   9.2 QoS Support  . . . . . . . . . . . . . . . . . . . . . . . . . 25
   9.3 QoS Information Exchanged Between FC and FCIP Entities . . . . 26
   10. Dynamic Discovery of Participating FCIP Entities . . . . . . . 26
   10.1 Requirements  . . . . . . . . . . . . . . . . . . . . . . . . 26
   10.2 Discovery Information Exchanged Between FC and FCIP Entities  26
   11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27
   12. Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . 28
   13. Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 28
   14. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 29
   15. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 31

   Annex
   A  Example of synchronization recovery algorithm . . . . . . . . . 31
   B  Relationship between FCIP and IP over FC (IPFC) . . . . . . . . 36
   C  FC Frame Format . . . . . . . . . . . . . . . . . . . . . . . . 36
   D  FCIP Requirements on an FC Entity . . . . . . . . . . . . . . . 38
   E  FC-BB-2 Inputs  . . . . . . . . . . . . . . . . . . . . . . . . 39

1. Purpose, Motivation and Objectives

   Fibre Channel (FC) is a gigabit speed networking technology
   primarily used to implement Storage Area Networks (SANs). See
   section 2 for information about how Fibre Channel is standardized
   and the relationship of this specification to Fibre Channel standards.

   This specification describes mechanisms that allow the
   interconnection of islands of Fibre Channel SANs over IP Networks to
   form a unified SAN in a single Fibre Channel fabric. The motivation
   behind defining these interconnection mechanisms is a desire to
   connect physically remote FC sites allowing remote disk access, tape
   backup, and live mirroring.

   Fibre Channel standards have chosen nominal distances between switch
   elements that are less than the distances available in an IP
   Network. Since Fibre Channel and IP Networking technologies are
   compatible, it is logical to turn IP Networking for extending the
   allowable distances between Fibre Channel switch elements.

   The fundamental assumption made in this specification is that the
   Fibre Channel traffic is carried over the IP Network in such a
   manner that the Fibre Channel Fabric and all Fibre Channel devices
   on the Fabric are unaware of the presence of the IP Network. This
   means that the FC datagrams MUST be delivered in such time as to
   comply with existing Fibre Channel specifications. The FC traffic
   MAY span LANs, MANs and WANs, so long as this fundamental assumption
   is adhered to.



Rajagopal, et al.               Standards Track                 [Page 3]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   The objectives of this document are to:

   1)  specify the encapsulation and mapping of Fibre Channel (FC)
       frames employing FC Frame Encapsulation [23].

   2)  apply the mechanism described in 1) to an FC Fabric using an IP
       network as an interconnect for two or more islands in an FC
       Fabric.

   3)  address any FC concerns arising from tunneling FC traffic over
       an IP-based network, including security, data integrity (loss),
       congestion, and performance. This will be accomplished by
       utilizing the existing IETF-specified suite of protocols.

   4)  be compatible with the referenced FC standards. While new work
       may be undertaken in T11 [8] to optimize and enhance FC Fabrics,
       this specification requires conformance only to the referenced
       FC standards.

   5)  be compatible with all applicable IETF standards so that the IP
       Network used to extend an FC Fabric can be used concurrently for
       other reasonable purposes.

2. Relationship to Fibre Channel Standards

2.1 Relevant Fibre Channel Standards

   FC is standardized under American National Standard for Information
   Systems of the National Committee for Information Technology
   Standards (ANSI-NCITS) in its T11 technical committee. T11 has
   specified a number of documents describing FC protocols, operations,
   and services. T11 documents of interest to readers of this
   specification include:

    - FC-BB   - Fibre Channel Backbone [3]
    - FC-BB-2 - Fibre Channel Backbone -2 [4]
    - FC-SW-2 - Fibre Channel Switch Fabric -2 [5]
    - FC-FS   - Fibre Channel Framing and Signaling [6]
    - FC-GS-3 - Fibre Channel Generic Services -3 (FC-GS-3) [7]

   Additional information regarding T11 activities is available on the
   committee's web site [8].

2.2 This Specification and Fibre Channel Standards

   Building a high performance device that successfully extends a FC
   Fabric over an IP Network requires tight integration of FC and TCP/
   IP technologies. Since these two technologies are standardized by


Rajagopal, et al.               Standards Track                 [Page 4]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   multiple organizations, specifying all the requirements in one
   document and getting high quality review of that document has proven
   to be impossible.

   Therefore, this specification addresses only the requirements
   necessary to properly utilize an IP Network as a conduit for an FC
   Fabric. The result is a specification for an FCIP Entity (see
   section 6.4).

   A product that tunnels an FC Fabric through an IP Network must
   combine the FCIP Entity with an FC Entity (see section 6.3) using an
   implementation specific interface. Although the requirements placed
   on an FC Entity by this specification are listed in annex D, the
   list here is not exhaustive. More information about FC Entities can
   be found in the Fibre Channel standards and an example of an FC
   Entity can be found in FC-BB-2 [4].

   No attempt is being made to define the interface between an FCIP
   Entity and an FC Entity at this time because doing so risks
   compromising the performance and efficacy of the resulting products.
   Current experience in this area is simply insufficient to guide
   definition of the interface appropriately.

   The objectives and motivations of this specification are not
   impacted by the decision not to standardize the interface between
   FCIP Entities and FC Entities because fully functional and compliant
   products can be built provided they contain both an FCIP Entity and
   an FC Entity. The only products that cannot be built are those that
   contain only one or the other and there is no urgent need for such
   products at this time.

3. Terminology

   Terms needed to clarify the concepts presented in FCIP are defined
   here.

   FC End Node - A FC device that uses the connection services provided
   by the FC Fabric.

   FC Entity - The Fibre Channel specific element that combines with an
   FCIP Entity to form an interface between an FC Fabric and an IP
   Network (see section 6.3).

   FC Fabric - An entity that interconnects various Nx_Ports (see [6])
   attached to it, and is capable of routing frames using only the
   destination ID information in a frame header (see annex C).




Rajagopal, et al.               Standards Track                 [Page 5]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   FC Frame - The basic unit of Fibre Channel data transfer (see annex
   C).

   FC Receiver Portal - The access point through which an FC Frame
   enters an FCIP Data Engine from the FC Entity.

   FC Transmitter Portal - The access point through which a
   reconstituted FC frame leaves an FCIP Data Engine to the FC Entity.

   FCIP Data Engine (FCIP_DE) - The component of an FCIP Entity that
   handles FC Frame encapsulation, de-encapsulation, and transmission
   through a single TCP connection (see section 6.6).

   FCIP Entity - The principle FCIP interface point to the IP Network
   (see section 6.4).

   FCIP Link - One or more TCP connections that connect one FCIP_LEP to
   another (see section 6.2).

   FCIP Link Endpoint (FCIP_LEP) - The component of an FCIP Entity that
   handles FC Frame transmission through a single FCIP Link (see
   section 6.5).

   Encapsulated Frame Receiver Portal - The TCP access point through
   which an FCIP encapsulated frame is received from the IP Network by
   an FCIP Data Engine.

   Encapsulated Frame Transmitter Portal - The TCP access point through
   which an FCIP encapsulated frame is transmitted to the IP Network by
   an FCIP Data Engine.

4. Open Issues

   This draft is a work in progress and this section identifies areas
   where the work is known to be incomplete and discusses the current
   status of these efforts. This section will be removed before this
   draft is considered for standardization.

    - FCIP Entity Discovery - The basic principles of FCIP Entity
      discovery are agreed and represented in section 5. Work on the
      details of dynamic FCIP Entity discovery are incomplete (see
      section 10). Work on FCIP Entity Discovery may change the way an
      FCIP Entity is identified from the currently specified IP
      Address usage.

    - Security - In general, FCIP will follow or subset the security
      mechanisms agreed for iSCSI. The basic principles of FCIP
      security requirements are agreed and described in section 5.


Rajagopal, et al.               Standards Track                 [Page 6]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


      Section 8 contains the latest information on the details of FCIP
      security. It must be noted that the association between IP
      Addresses and FCIP Entities is open to changes based on yet to
      be finalized decisions about security. The point at which a TCP
      connection is authorized to carry data is still being debated.

    - Timeout Coordination with Fibre Channel - In this revision, the
      only timeout Fiber Channel timeout consideration enforced by the
      FCIP Entity is R_A_TOV. All other timeout issues are the
      responsibility of the FC Entity. Section 7.2.6 contains a
      discussion of the TCP Acknowledge Timeout that needs to be
      reviewed to determine if it is still needed.

    - Performance - The discussion of performance considerations in
      section 9 and particularly the quality of service discussion in
      section 9.2 are known to require additional work. A particular
      concern is that quality of service not be limited to using
      diffserv.

5. Protocol Summary

   The FCIP protocol is summarized as follows:

   1)  The primary function of an FCIP Entity is forwarding FC frames,
       employing FC Frame Encapsulation described in [23].

   2)  Viewed from the IP Network perspective, all FCIP Entities are
       peers and communicate using TCP/IP. Each FCIP Entity is a TCP
       endpoint in the IP-based network.

   3)  Viewed from the FC Fabric perspective, each pair of FCIP
       Entities, in combination with their associated FC Entities,
       serves as a frame transmission component of the FC Fabric. The
       FC End Nodes are unaware of the existence of the FCIP Link.

   4)  FC Primitive Signals, Primitive Sequences, and Class 1 FC Frames
       are not transmitted across an FCIP Link because they cannot be
       encoded using FC Frame Encapsulation [23].

   5)  The path (route) taken by an encapsulated FC Frame follows the
       normal routing procedures of the IP Network.

   6)  An FCIP Entity SHALL have exactly one IP Address.

   7)  An FCIP Entity may contain multiple FCIP Link Endpoints, but
       each FCIP Link Endpoint (FCIP_LEP) communicates with exactly one
       other FCIP_LEP, possibly with multiple FCIP Data Engines.
       (FCIP_DEs) and multiple TCP connections.


Rajagopal, et al.               Standards Track                 [Page 7]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   8)  When multiple FCIP_LEPs or multiple FCIP_DEs are in use,
       selection of which FCIP_DE to use for encapsulating and
       transmitting a given FC Frame outside the scope of this
       document. FCIP Entities do not actively participate in FC Frame
       routing.

   9)  The FCIP Control & Services function MAY use TCP/IP quality of
       service features (see section 9.2) to support Fibre Channel
       capabilities.

   10) Each FCIP Entity is statically or dynamically configured with a
       list of IP addresses and port numbers corresponding to
       participating FCIP Entities. If dynamic discovery of
       participating FCIP Entities is supported, the function SHALL be
       performed using the Service Location Protocol (SLPv2) [21]. It
       is outside the scope of this specification to describe any
       static configuration method for participating FCIP Entity
       discovery. Refer to section 10 for a detailed description of
       dynamic discovery of participating FCIP Entities using SLPv2.

   11) FCIP Entities do not actively participate in the discovery of FC
       source and destination identifiers. Discovery of FC addresses
       (accessible via the FCIP Entity) is provided by techniques and
       protocols within the FC architecture as described in FC-FS [6],
       FC-SW-2 [5], and FC-GS-3 [7].

   12) To support IP Network security, FCIP Entities MUST:
       a)  implement cryptographically protected authentication and
           cryptographic data integrity keyed to the authentication
           process, or
       b)  be capable of operating with external IP security mechanisms
           that provide cryptographically protected authentication and
           cryptographic data integrity keyed to the authentication
           process.
       FCIP entities MAY implement data privacy security features.
       Security features and requirements are detailed in section 8.

   13) FCIP relies on TCP to recover from re-ordering in the IP network.

   14) FCIP relies on both TCP and FC error recovery mechanisms to
       detect and recover from data loss and corruption within the IP
       Network.







Rajagopal, et al.               Standards Track                 [Page 8]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


6. The FCIP Model

6.1 FCIP Protocol Model

   The relationship between FCIP and other protocols is illustrated in
   figure 1.

       +------------------------+ FCIP Link +------------------------+
       |          FCIP          |===========|          FCIP          |
       +--------+------+--------+           +--------+------+--------+
       |  FC-2  |      |  TCP   |           |  TCP   |      |  FC-2  |
       +--------+      +--------+           +--------+      +--------+
       |  FC-1  |      |   IP   |           |   IP   |      |  FC-1  |
       +--------+      +--------+           +--------+      +--------+
       |  FC-0  |      |  LINK  |           |  LINK  |      |  FC-0  |
       +--------+      +--------+           +--------+      +--------+
            |          |   PHY  |           |   PHY  |           |
            |          +--------+           +--------+           |
            |               |                    |               |
            |               |                    |               |
            V               +--------------------+               V
         to Fibre                                             to Fibre
         Channel                                              Channel
       Environment                                          Environment

       Fig. 1  FCIP Protocol Stack Model

   Note that the objective of the FCIP Protocol is creation and
   maintenance of one or more FCIP Links.

6.2 FCIP Link

   The FCIP Link is the basic unit of service provided by the FCIP
   Protocol to a FC Fabric. As shown in figure 2, an FCIP Link connects
   two portions of an FC Fabric using an IP Network as a transport to
   form a single FC Fabric.

       /\/\/\/\/\/\         /\/\/\/\/\/\         /\/\/\/\/\/\
       \    FC    /   FCIP  \    IP    /   Link  \    FC    /
       /  Fabric  \=========/  Network \=========/  Fabric  \
       \/\/\/\/\/\/         \/\/\/\/\/\/         \/\/\/\/\/\/

       Fig. 2  FCIP Link Model

   At the points where the ends of the FCIP Link meet portions of the
   FC Fabric, an FCIP Entity (see section 6.4) combines with an FC
   Entity as described in section 6.3 to serve as the interface between
   FC and IP.


Rajagopal, et al.               Standards Track                 [Page 9]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   An FCIP Link SHALL contain at least one TCP connection and MAY
   contain more than one TCP connection. The endpoints of a single TCP
   connection are FCIP Data Engines (see section 6.6). The endpoints of
   a single FCIP Link are FCIP Link Endpoints (see section 6.5).

6.3 FC Entity

   A product that tunnels an FC Fabric through an IP Network must
   combine an FC Entity with an FCIP Entity (see section 6.4) to form a
   complete interface between the FC Fabric and IP Network as shown in
   figure 3.

       +----------+         /\/\/\/\/\/\         +----------+
       |   FCIP   |   FCIP  \    IP    /   Link  |   FCIP   |
       |  Entity  |=========/  Network \=========|  Entity  |
       +----------+         \/\/\/\/\/\/         +----------+
       |    FC    |                              |    FC    |
       |  Entity  |                              |  Entity  |
       +----------+                              +----------+
            |                                         |
       /\/\/\/\/\/\                              /\/\/\/\/\/\
       \    FC    /                              \    FC    /
       /  Fabric  \                              /  Fabric  \
       \/\/\/\/\/\/                              \/\/\/\/\/\/

       Fig. 3  FC Entity and FCIP Entity Model

   The interface between the FC and FCIP Entities is implementation
   specific. The minimum requirements placed on an FC Entity by this
   specification are listed in annex D. More information about FC
   Entities can be found in the Fibre Channel standards and an example
   of an FC Entity can be found in FC-BB-2 [4].

















Rajagopal, et al.               Standards Track                [Page 10]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


6.4 FCIP Entity

   The model for an FCIP Entity is shown in figure 4.

       .......................................................
       : FCIP Entity                                         :
       :                                                     :
       :  +-----------+                                      :
       :  | FCIP      |                                      :
       :  | Control & |------------------------------------+ :
       :  | Services  |                                    | :
       :  | Module    |                                    | :
       :  +-----------+                                    | :
       :        |            +--------------------+        | :
       :        |   +-------+--------------------+|----+   | :
       :        |   |+-----+--------------------+|----+|   | :
       :        |   ||+----| FCIP Link Endpoint |----+||   | :
       :        |   |||    +--------------------+    |||   | :
       :.............................................|||.....:
                |   |||                              |||   |
                |   |||                              |||   o<--+
                |   |||                unique TCP    |||   |   |
                |   |||                connections-->|||   |   |
                |   |||                              |||   |   |
             +----------+                         /\/\/\/\/\/\ |
             |    FC    |                         \    IP    / |
             |  Entity  |                         /  Network \ |
             +----------+                         \/\/\/\/\/\/ |
                  |                                            |
             /\/\/\/\/\/\                   +------------------+
             \    FC    /                   +->IP Address &
             /  Fabric  \                   +->Well Known Port
             \/\/\/\/\/\/

       Fig. 4  FCIP Entity Model

   The FCIP Entity is the connection interface point for the IP Network
   and is the owner of the IP Address and Well Known Port used to form
   TCP connections. An FC Fabric to IP Network interface product SHALL
   contain one FCIP Entity for each IP Address assigned to the product.

   An FCIP Entity contains an FCIP Control & Services Module to provide
   the FC Entity with an interface to key IP Network features. The
   interfaces to the IP Network features is implementation specific,
   however, to maintain interoperability, the TCP/IP mechanisms used
   are specified in this document as follows:

    - TCP Connections - see section 7


Rajagopal, et al.               Standards Track                [Page 11]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


    - Security - see section 8
    - Performance - see section 9
    - Discovery - see section 10

   The FCIP Link Endpoints in an FCIP Entity provide the FC Frame
   transmission features of FCIP.

6.5 FCIP Link Endpoint (FCIP_LEP)

   Each time a TCP connection is formed to an IP Address for which no
   TCP connection already exists, the FCIP Entity SHALL create a new
   FCIP Link Endpoint containing one FCIP Data Engine.

   An FCIP_LEP is a transparent data translation point between an FC
   Entity and an IP Network. A pair of FCIP_LEPs communicating over one
   or more TCP connections create an FCIP Link to join two islands of a
   FC Fabric, producing a single FC Fabric.

   The IP Network over which the two FCIP_LEPs communicate is not aware
   of the FC payloads that it is carrying. Likewise, the FC End Nodes
   connected to the FC Fabric are unaware of the TCP/IP based transport
   employed in the structure of the FC Fabric.

   As shown in figure 5, the FCIP Link Endpoint contains one FCIP Data
   Engine for each TCP connection in the FCIP Link.

        ................................................
        : FCIP Link Endpoint                           :
        :                   +------------------+       :
        :          +-------+------------------+|----+  :
        :          |+-----+------------------+|----+|  :
        :          ||+----| FCIP Data Engine |----+||  :
        :          |||    +------------------+    |||  :
        :..............................................:
                   |||                            |||
              +----------+                    /\/\/\/\/\/\
              |    FC    |                    \    IP    /
              |  Entity  |                    /  Network \
              +----------+                    \/\/\/\/\/\/
                    |
              /\/\/\/\/\/\
              \    FC    /
              /  Fabric  \
              \/\/\/\/\/\/

       Fig. 5  FCIP Link Endpoint Model




Rajagopal, et al.               Standards Track                [Page 12]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   An FCIP_LEP uses normal TCP based flow control mechanisms for
   managing its internal resources and matching them with the
   advertised TCP Receiver Window Size. An FCIP_LEP MAY communicate
   with its FC Entity counterpart to coordinate flow control.

6.6 FCIP Data Engine (FCIP_DE)

   The model for one of the multiple FCIP_DEs that may be present in an
   FCIP_LEP is shown in figure 6.

              +--------------------------------+
              |                                |
              |-+    +------------------+    +-|
         C    |p|    |  Encapsulation   |    |p|    N
       F h -->|1|--->|     Engine       |--->|2|--> e
       i a    |-+    +------------------+    +-|    t
       b n    |                                |  I w
       r n    |-+    +------------------+    +-|  P o
       e e    |p|    | De-Encapsulation |    |p|    r
         l <--|4|<---|     Engine       |<---|3|<-- k
              |-+    +------------------+    +-|
              |                                |
              +--------------------------------+

       Fig. 6  FCIP Data Engine Model

   Data enters and leaves the FCIP_DE through four portals (p1 - p4).
   The portals do not process or examine the data that passes through
   them. They are only the named access points where the FCIP_DE
   interfaces with external world. The names of the portals are as
   follows:

   p1) FC Receiver Portal - The interface through which an FC Frame
       enters an FCIP_DE from the FC Entity.

   p2) Encapsulated Frame Transmitter Portal - The TCP interface
       through which an FCIP encapsulated frame is transmitted to the
       IP Network by an FCIP_DE.

   p3) Encapsulated Frame Receiver Portal - The TCP interface through
       which an FCIP encapsulated frame is received from the IP Network
       by an FCIP_DE.

   p4) FC Transmitter Portal - The interface through which a
       reconstituted FC frame exits an FCIP_DE to the FC Entity.

   The work of the FCIP_DE is done by the Encapsulation and De-
   Encapsulation Engines. The Engines have two functions:


Rajagopal, et al.               Standards Track                [Page 13]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   1)  Encapsulating and de-encapsulating FC Frames using the
       encapsulation format described in FC Frame Encapsulation [23]
       and in section 6.6.1 of this document, and

   2)  Detecting some data transmission errors and performing minimal
       error recovery as described in section 6.6.2.

   Data flows through the FCIP_DE as follows:

   1)  An FC Frame arrives at the FC Receiver Portal and is passed to
       the Encapsulation Engine. The FC Frame is assumed to have been
       processed by the FC Entity according to the applicable FC rules
       and is not validated by the FCIP_DE.

   2)  In the Encapsulation Engine the encapsulation format described
       in FC Frame Encapsulation [23] and in section 6.6.1 of this
       document SHALL be applied to prepare the FC Frame for
       transmission over the IP Network.

   3)  The entire encapsulated frame SHALL be passed to the
       Encapsulated Frame Transmitter Portal where it SHALL be inserted
       in the TCP byte stream.

   4)  Transmission of the encapsulated frame over the IP Network
       follows all the TCP rules of operation. This includes but is not
       limited to the in-order delivery of bytes in the stream, as
       specified by TCP [9].

   5)  The encapsulated FC Frame arrives at the partner FCIP Entity
       where it enters the FCIP_DE through the Encapsulated Frame
       Receiver Portal and is passed to the De-Encapsulation Engine for
       processing.

   6)  The De-Encapsulation Engine SHALL validate the incoming TCP byte
       stream as described in section 6.6.2 and SHALL de-encapsulate
       the FC Frame according to the encapsulation format described in
       FC Frame Encapsulation [23] and in section 6.6.1 of this document.

   7)  In the absence of errors, the de-encapsulated frame SHALL be
       passed to the FC Transmitter Portal for delivery to the FC Entity.

   Every FC Frame that arrives at the FC Receiver Portal SHALL be
   transmitted on the IP Network as described in steps 1 through 4
   above. Data bytes arriving at the Encapsulated Frame Receiver Portal
   SHOULD be transmitted to the FC Transmitter Portal as described in
   steps 5 through 7, but this MAY NOT always be the case.



Rajagopal, et al.               Standards Track                [Page 14]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


6.6.1 FCIP Encapsulation of FC Frames

   The FCIP encapsulation of FC frames employs FC Frame Encapsulation
   [23].

   The features from FC Frame Encapsulation that are unique to
   individual protocols SHALL be applied as follows for the FCIP
   encapsulation of FC frames.

   The Protocol# field SHALL contain 1 in accordance with the IANA
   Considerations annex of FC Frame Encapsulation [23].

   The Protocol Specific field SHALL have the format shown in figure 7.
   Note: the word numbers in figure 7 are relative to the complete FC
   frame encapsulation header, not to the Protocol Specific field.

    W|------------------------------Bit------------------------------|
    o|                                                               |
    r|3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1                    |
    d|1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
     +---------------------------------------------------------------+
    1|               replication of encapsulation word 0             |
     +-------------------------------+-------------------------------+
    2|            reserved           |           -reserved           |
     +-------------------------------+-------------------------------+

       Fig. 7  FCIP Usage of FC Frame Encapsulation Protocol Specific
           field

   Word 1 of the Protocol Specific field SHALL contain an exact copy of
   word 0 in FC Frame Encapsulation [23].

   Word 2 of the Protocol Specific field is reserved for future
   enhancements to the FCIP protocol.

   The reserved field (bits 31-16 in word 2): SHALL contain 0.

   The -reserved field (bits 15-0 in word 2): SHALL contain 65535 (or
   0xFFFF).

   The CRCV (CRC Valid) Flag SHALL be set to 0.

   The CRC field SHALL be set to 0.







Rajagopal, et al.               Standards Track                [Page 15]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


6.6.2 FCIP Data Engine Error Detection and Recover

6.6.2.1 TCP Assistance With Error Detection and Recovery

   The FCIP_LEP assumes that, if TCP determines that there are TCP
   checksum errors, TCP applies the appropriate TCP retransmission and
   error recovery procedures. So the FCIP_DE gets an ordered delivery
   of FCIP frames with the TCP detected errors being transparent to the
   FCIP_DE.

6.6.2.2 Errors in FCIP Headers and Discarding FCIP Frames

   Bytes delivered through the Encapsulated Frame Receiver Portal that
   are not correctly delimited as defined by the FC Frame Encapsulation
   [23] SHOULD NOT be forwarded on to the FC Entity.

   Synchronization of the FCIP_DE to the FCIP Frames in the data stream
   entering the Encapsulated Frame Receiver Portal is maintained using
   the FC Frame Encapsulation header's frame length field to determine
   where in the data stream the next FC Encapsulation header is
   located. Synchronization SHALL be verified by checking the validity
   and positioning of any combination of the following FC Frame
   Encapsulation information:

   a)  Protocol # field and its ones complement;
   b)  Version field and its ones complement;
   c)  Replication of encapsulation word 0 in word 1;
   d)  Reserved field and its ones complement;
   e)  Flags field and its ones complement;
   f)  Length field and its ones complement;
   g)  Time stamp [integer] and time stamp [fraction] fields;
   h)  CRC field is equal to zero;
   i)  SOF fields and ones complement fields;
   j)  Format and values of FC header;
   k)  CRC of FC frame;
   l)  EOF fields and ones complement fields; and/or
   m)  FC Encapsulation header information of next encapsulated frame.

   For FCIP Frames with header errors, the FCIP_DE SHALL discard the
   frame. Such errors should be considered carefully, since some may be
   synchronization errors. Errors in encapsulated FCIP Frames detected
   by the FCIP_DE that affect synchronization with the Encapsulated
   Frame Receiver Portal byte stream SHALL be handled as defined by
   section 6.6.2.4.

   An error in an encapsulated FCIP Frame that effects the
   synchronization may require the FCIP Entity to notify the FC Entity
   that the previously delivered FC Frame was invalid.


Rajagopal, et al.               Standards Track                [Page 16]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   Whenever an FCIP_DE discards bytes delivered through the
   Encapsulated Frame Receiver Portal, it SHALL cause the FC Entity to
   be notified and provided with a suitable description of the reason
   bytes were discarded.

6.6.2.3 IP Network Transit Time Validation

   The FCIP_DE SHALL use the valid time stamp information in the FC
   Frame Encapsulation [23] header to determine if received FCIP Frames
   have been delayed by more than R_A_TOV in the IP Network. If an FCIP
   Frame has been delayed by more than R_A_TOV in the IP network, the
   FCIP_DE SHALL discard the FCIP Frame as described in section
   6.6.2.2. The discarding of delayed FCIP frames SHALL continue until
   a FCIP Frame is processed whose life in the IP Network is smaller
   than R_A_TOV.

   Note that unlike a physical Fibre Channel link, an FCIP Link MAY
   involve IP routing dynamics that produce reliable, ordered delivery
   at the TCP layer, with the result that some FC Fabric operating
   constraints may be violated. The FCIP_DE is responsible for
   detecting violations R_A_TOV FC Fabric constraint and discarding
   affected frames.

6.6.2.4 Synchronization Failures

   If an FCIP_DE determines that it cannot find the next FCIP header in
   the byte stream entering through the Encapsulated Frame Receiver
   Portal, the FCIP_DE SHALL either:

   a)  close the TCP connection [9] [11];
   b)  recover synchronization by searching the bytes delivered by the
       Encapsulated Frame Receiver Portal for a valid FCIP Frame header
       having the correct properties, and discarding bytes delivered by
       the Encapsulated Frame Receiver Portal until a valid FCIP Frame
       header is found; or
   c)  attempt to recover synchronization as described in b) and if
       synchronization cannot be recovered close the TCP connection as
       described in a).


   If the FCIP_DE attempts to recover synchronization, the
   resynchronization algorithm used SHALL meet the following
   requirements:

   a)  discard or identify with an EOFa (see FC-FS [6] and FC Frame
       Encapsulation [23])those FC frames and fragments of frames
       identified before synchronization has again been completely


Rajagopal, et al.               Standards Track                [Page 17]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


       verified. The number of FC frames not forwarded may vary based
       on the algorithm used;

   b)  return to sending valid FC frames only after synchronization has
       been verified; and

   c)  close the TCP/IP connection if the algorithm ends without
       verifying successful synchronization. The probability of failing
       to synchronize successfully and the time necessary to determine
       whether or not synchronization was successful may vary with the
       algorithm used.

   An example algorithm meeting these requirements can be found in
   annex A.

7. TCP Connection Management

7.1 TCP Connection Establishment

7.1.1 Creating a New TCP Connection

   The FC Entity SHALL request creation of a new TCP Connection by
   transmitting at least the following information to the FCIP Entity:

    - IP Address
    - R_A_TOV for the FCIP_Link
    - TCP Connection Parameters (see section 7.2)
    - Security Parameters (see section 8)
    - Quality of Service Parameters (see section 9)

   In response to a request from the FC Entity the FCIP Entity shall
   generate a TCP connect request [9] to the FCIP Well-Known Port at
   the specified IP Address. If the TCP connect request is rejected,
   the FCIP Entity SHALL so inform the FC Entity.

   If the TCP connect request is accepted, and the IP Address is one to
   which no other TCP connections exist, the FCIP Entity SHALL:

   1)  Create a new FCIP_LEP for the new FCIP Link,

   2)  Create a new FCIP_DE within the newly created FCIP_LEP to
       service the new TCP connection, and

   3)  Inform the FC Entity of the new FCIP_LEP and FCIP_DE.






Rajagopal, et al.               Standards Track                [Page 18]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   If the TCP connect request is accepted, and the IP Address is one
   for which a TCP connection already exists, the FCIP Entity SHALL:

   1)  Create a new FCIP_DE within the existing FCIP_LEP to service the
       new TCP connection, and

   2)  Inform the FC Entity of the FCIP_LEP and new FCIP_DE.

7.1.2 Processing TCP Connect Requests

   The FCIP Entity SHALL listen for new TCP connection requests [9] on
   the FCIP Well-Known Port. An FCIP Entity MAY also accept and
   establish TCP connections to a TCP port number other than the FCIP
   Well-Known Port, as configured by the network administrator.

   Upon receipt of a TCP connect request, the FCIP Entity SHALL
   determine if a TCP connection already exists for the IP Address
   making the TCP connect request. The FCIP Entity SHALL notify the FC
   Entity of the TCP connect request, transmitting at least the
   following information:

    - IP Address
    - R_A_TOV for the FCIP_Link (zero for a new FCIP_LEP)
    - Information about the FCIP_LEP, new or existing
    - Information about the FCIP_DE for the new TCP connection
    - TCP Connection Parameters (see section 7.2)
    - Security Parameters (see section 8)
    - Quality of Service Parameters (see section 9)

   In response to the information provided by the FCIP Entity, the FC
   Entity MUST either accept or reject the TCP connect request. If the
   FC Entity rejects the TCP connect request, the FCIP Entity SHALL
   terminate the TCP connect request [9]. If the FC Entity accepts the
   TCP connect request, the FCIP Entity SHALL:

   1)  Accept the TCP connect request,

   2)  Finalize creation of the new FCIP_DE for the new TCP connection,
       and

   3)  If the new TCP connection is to an IP Address for which no other
       TCP connection exists, finalize the creation of the FCIP_LEP.

7.2 TCP Connection Parameters

   In order to provide efficient management of FCIP_LEP resources as
   well as FCIP Link resources, coordination of certain TCP connection
   parameters between the FC Entity and FCIP Entity is RECOMMENDED.


Rajagopal, et al.               Standards Track                [Page 19]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



7.2.1 TCP Selective Acknowledgement Option

   The Selective Acknowledgement option RFC 2883 [22] allows the
   receiver to acknowledge multiple lost packets in a single ACK,
   enabling faster recovery. If authorized by the FC Entity, an FCIP
   Entity MAY negotiate use of TCP SACK and use it for faster recovery
   from lost packets and holes in TCP sequence number space.

7.2.2 TCP Window Scale Option

   This option allows TCP window sizes larger than 16-bit limits to be
   advertised by the receiver. It is necessary to allow data in long
   fat networks to fill the available pipe. This also implies buffering
   on the TCP sender that matches the (bandwidth*RTT) product of the
   TCP connection. An FCIP_LEP SHALL use locally available mechanisms
   to set a window size that matches the available local buffer
   resources and the desired throughput.

7.2.3 IP DSCP Option

   The RECOMMENDED IP DSCP field setting is 101110 corresponding to the
   EF service.

   <Need better wording to fit current Diffserv specifications.>

7.2.4 Protection against sequence number wrap

   It is RECOMMENDED that FCIP Entities implement protection against
   sequence number wrap. It is quite possible that within a single
   connection, TCP sequence numbers wrap within a timeout window.

7.2.5 TCP No Delay Option

   FCIP Entities SHALL disable the Nagle TCP No Delay option. This
   option is designed for usage in a telnet environment.

7.2.6 TCP Acknowledgement Timeout

   TCP has a TCP acknowledgement timeout. This is a variable timeout.

   <Need to elaborate on TCP timeouts and define how Fibre Channel
   timeouts map to TCP timeouts.>

7.3 TCP Connection Considerations

   An FCIP_LEP SHALL implement established TCP mechanisms as defined in
   RFC 2581 [18] for congestion control on its connections.


Rajagopal, et al.               Standards Track                [Page 20]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   It is RECOMMENDED that FCIP_LEPs use the TCP mechanisms for Long Fat
   Networks (LFNs) (i.e. an IP network with large (bandwidth*delay)
   product), as defined in RFC 1072 [10].

   In idle mode, a TCP connection "keep alive" option of TCP is
   normally used to keep a connection alive. However, this timeout is
   fairly large and may prevent early detection of loss of
   connectivity. In order to facilitate faster detection of loss of
   connectivity, FC Entities SHOULD implement some form of Fibre
   Channel connection failure detection.

7.4 Flow Control Mapping between TCP and FC

   The FCIP Entity and FC Entity are connected the IP Network and FC
   Fabric, respectively, and they need to follow the flow control
   mechanisms of both TCP and FC, which work independent of each other.

   This section provides guidelines as to how the FCIP Entity can map
   TCP flow control to status notifications to the FC Entity.

   There are two scenarios when the flow control management becomes
   crucial:

   1)  When there is line speed mismatch between the FC and IP
       interfaces.

       Even though it is RECOMMENDED that both the FC and IP interfaces
       to the FC Entity and FCIP Entity, respectively, be of comparable
       speeds, it is possible to carry FC traffic over an IP Network
       that has a different line speed and bit error rate.

   2)  When the FC Fabric or IP Network encounters congestion.

       Even when both the FC Fabric or IP network are of comparable
       speeds, during the course of operation the FC Fabric or the IP
       Network could encounter congestion due to transient conditions.

   The FCIP Entity and FC Entity need to work cooperatively to use the
   available flow control mechanisms in the TCP and FC protocols to
   handle these situations. This specification does not specify any
   particular mechanism to handle the flow control but leaves this to
   implementation's choice.

   If the Encapsulation Frame Transmitter Portal is unable to transmit
   encapsulated FCIP Frames at the experienced data rate, the FCIP
   Entity MUST request that the FC Entity reduce the rate at which new
   FC Frames arrive at the FC Receiver Portal.


Rajagopal, et al.               Standards Track                [Page 21]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   If the FC Receiver Portal is unable to accept de-encapsulated FC
   Frames at the experienced rate, the FC Entity MAY request the FCIP
   Entity to reduce the rate at which new FC Frames are delivered. The
   FCIP_DE MAY use TCP windowing techniques to control the packet
   arrival rate from the IP Network. This MAY involve advertising zero-
   window on TCP connection(s) occasionally so that the TCP
   connection(s) are flow controlled while the FC Fabric is
   encountering congestion.

8. Security

8.1 Considerations

   Using a wide-area, general purpose network such as an IP Network in
   a position normally occupied by physical cabling introduces some
   security problems not normally encountered in Fibre Channel Fabrics.
   FC transport media are typically protected physically from outside
   access; IP Networks typically invite outside access.

   The general effect is that the security of the entire FC Fabric is
   only as good as the security of the entire IP Network through which
   it tunnels. The following broad classes of attacks are possible:

   1)  Unauthorized Fibre Channel elements can gain access to resources
       through normal Fibre Channel processes.

   2)  Unauthorized agents can monitor and manipulate Fibre Channel
       traffic flowing over physical media used by the IP Network and
       under control of the agent.

   To a large extent, these security risks are typical of the risks
   facing any other application using an IP Network. They are mentioned
   here only because Fibre Channel storage networks are not normally
   suspicious of the media. Fibre Channel Fabric administrators will
   need to be aware of these additional security risks.


8.2 IP Network Security Requirements

   Security protocols and procedures used in other IP applications MAY
   be used for FCIP. FCIP Entities MUST ensure secure operation of FCIP
   Links by implementing one of the following two methods:

   1)  by using ESP [13] from the IPSec Security Protocol Suite with
       NULL encryption [14] for cryptographic data integrity and
       integrity of authentication. Authentication is performed using
       SRP [RFC2945]. This method is discussed in section 8.3; or


Rajagopal, et al.               Standards Track                [Page 22]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   2)  by appropriate configuration of an external entity that
       implements IP security using mechanisms such as IPSec and
       Virtual Private Networks. This method is discussed in section 8.4.

   The mechanism for configuring whether a particular deployment uses
   1) or 2) is outside the scope of this document.

   Note: Two overviews of the IPSec Security Protocol Suite are
   available in RFC 2401 [12] and RFC 2411 [15].

8.3 Integrated Security

   When both FCIP Entity and IP Security implementations are integrated
   into a single device, IPSec ESP (in transport mode) MUST be
   implemented.

   Upon receiving a TCP connection request, the receiving FCIP Entity
   SHALL identify the FCIP Link per the IP address pair of the
   connection. It SHALL then verify that the FCIP Link has been
   previously authenticated. If not, the FCIP Entity SHALL authenticate
   a new peer using a separate TCP connection. This TCP connection is
   used for negotiation of SRP related parameters.

   <SRP message negotiation will be per iSCSI discussions>

   If authentication fails, the original TCP connection that initiated
   the authentication exchange is terminated and the FC Entity is not
   informed that a TCP connect request was received.

   If the authentication is successful, a new FCIP_LEP is created, with
   the authenticated FCIP Link as described in section 7.1.2.

   The FCIP Entity remembers the IP pair and the key material for
   authentication, so that any future TCP connections for that IP
   address pair bypasses this authentication step. The key material is
   then used as part of the ESP Security Parameters.

   <association of SRP key material with ESP header will be per iSCSI
   discussions>










Rajagopal, et al.               Standards Track                [Page 23]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


8.4 External Security Gateway

   Figure 8 illustrates the use of an externally supplied security
   gateway for securing the FCIP Link.

   +--------+ Insecure +--------+ Secure  +--------+ Insecure +-------+
   | FCIP   | Network  | IPSec  | Network | IPSec  | Network  |FCIP   |
   | Entity |----------| Device |---------| Device |----------|Entity |
   +--------+          +--------+         +--------+          +-------+

       Fig. 8  External Security Gateway Model

   In this deployment, only certain parts of the FCIP Link are exposed
   to security threats and so only these specific parts of the FCIP
   Link need to be secured. The part of the network between the two
   security gateways is secured using devices implementing IPSec.

   The IPSec Device or any other equivalent gateway is required to
   operate in tunnel mode, so that the IP addresses of the two FCIP
   Entities are visible through the security devices that are
   implemented.

8.5 Security Information Exchanged Between FC and FCIP Entities

   TBD

9. Performance

9.1 Considerations

   The FCIP_DE does not interpret the contents of an FC Frame (except
   for attaching the correct byte-encoded SOF and EOF) nor does it do
   any FC payload processing. This allows any FC traffic to be tunneled
   across at high throughput rates.

   If fragmentation at the data link and IP layers is avoided by the
   use of path MTU discovery, throughput performance is enhanced.

   The Flow Control Protocol (discussed in section 7.4) provides the
   ability to stream gigabit FC data when using a large window size.

   It is RECOMMENDED that FCIP Entities use the TCP mechanisms for Long
   Fat Networks (LFNs) when they are used in IP networks with a large
   (bandwidth*delay) product. These mechanisms include TCP window scale
   option, Selective Acknowledgement, among others. See section 7.2.

   In order to achieve better TCP aggregate throughput in the face of
   packet losses, a pair of peer FC Entities MAY use multiple TCP


Rajagopal, et al.               Standards Track                [Page 24]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   connections, and use appropriate policies for mapping FC Frames to
   these connections. Section 7.1 describes the steps an FCIP Entity
   takes in support of multiple TCP connection usage. All other aspects
   of using multiple TCP connections are outside the scope of this
   document.

   The reason for using multiple TCP connections is the TCP's slow-
   start algorithm, which reduces TCP's window whenever it detects
   congestion in the network. If, on the other hand, the traffic is
   distributed across multiple connections, all the connections will
   not be affected at the same time, resulting in a better aggregate
   throughput.

   Note that even though multiple connections provide better aggregate
   throughput (when packet losses occur on IP Networks), their use is
   not a requirement. A pair of FC Entities MAY choose to use single
   TCP connection to tunnel the FC traffic.

9.2 QoS Support

   The Differentiated Services Architecture (diffserv) provides a
   "Class of Service" to a flow aggregate [16], [17]. At so-called
   diffserv boundaries, IP packets are classified and marked. Within
   the diffserv domain, resources - bandwidth and buffers - are
   allocated for each classification.

   Packets with the same classification use the resources allocated for
   the classification. IP packets with the same destination and class
   marking exit a diffserv capable router in the same order they
   arrived. Packets with the same destination but different class
   markings exit according to priorities assigned to the different
   class markings.

   The Diffserv has renamed the Ipv4 TOS field as Differentiated
   Services Code Point (DSCP). The DSCP indicates the particular
   behavior a packet is to receive at each router.

   How a packet gets marked is based on a policy administered and
   configured into the network. [20] and [19] provide various encodings
   of the DSCP field to achieve a specific behavior from the routers.
   There may be several ways to administer the policies and the policy
   definition is up to the network provider. That is one network
   provider MAY choose to mark all packets going from one source IP
   address to a specific destination as "high priority", while another
   might mark just a specific traffic type (e.g., HTTP) as "high
   priority". Thus packets carry the desired class information and each
   diffserv-capable router treats the packet according to the
   information in its DSCP field. This is referred to as Per Hop


Rajagopal, et al.               Standards Track                [Page 25]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   Behavior (PHB).

   Currently, the IETF standards define essentially 3 types of
   services: Expedited Forwarding (EF) [20], Assured Forwarding (AF)
   [19], or Default Forwarding (DF) [16], [17] - that corresponds to
   its DSCP.

   [17] specifies the AF service AF PHB provides a way to prioritize
   best-effort traffic. Currently, 4 AF classes and 3 drop precedence
   levels are specified providing 12 different levels of forwarding
   assurances. The DSCP value specifies a drop-order in the event that
   a packet experiences congestion at a subsequent diffserv router.

   [20] specifies the DSCP code point equal to 101110 EF service which
   is also sometimes refereed to as "Premium" service. When supported,
   this class behavior has the lowest levels of latency, packet loss,
   and delay variation. This service behavior most closely matches the
   Fibre Channel characteristics. This is therefore the RECOMMENDED
   DSCP setting in the IP DSCP field.

   What resources are not used for EF and AF are left for the DF
   services which is really a best-effort service.

   Note that if a packet is being forwarded over an underlying network
   without diffserv support, then the packet would simply receive best-
   effort service regardless of its DSCP field setting.

9.3 QoS Information Exchanged Between FC and FCIP Entities

   TBD

10. Dynamic Discovery of Participating FCIP Entities

10.1 Requirements

   If dynamic discovery of participating FCIP Entities is supported the
   function SHALL be performed using the Service Location Protocol
   (SLPv2) [21].

   Additional details TBD.

10.2  Discovery Information Exchanged Between FC and FCIP Entities

   TBD






Rajagopal, et al.               Standards Track                [Page 26]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


11. References

   [1] Bradner, S., "The Internet Standards Process -- Revision 3", BCP
       9, RFC 2026, October 1996.

   [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement
       Levels", BCP 14, RFC 2119, March 1997.

   [3] Fibre Channel Backbone (FC-BB), T11 Project 1238-D, Rev 4.8,
       March 5, 2001 (www.t11.org).

   [4] Fibre Channel Backbone -2 (FC-BB-2), T11 Project 1466-D,
       (www.t11.org).

   [5] Fibre Channel Switch Fabric -2 (FC-SW-2), T11 Project 1305-D,
       Rev. 5.2, May 23, 2001 (www.t11.org).

   [6] Fibre Channel Framing and Signaling (FC-FS), T11 Project 1331-D,
       Rev 1.2, February 16, 2001 (www.t11.org).

   [7] Fibre Channel Generic Services -3, ANSI NCITS.348-200x, November
       28, 2000.

   [8] http://www.t11.org

   [9] "Transmission Control Protocol", RFC 793, Sept. 1981.

   [10] Jacobson & Braden, "TCP Extensions for Long-Delay Paths", RFC
       1072, October 1988.

   [11] Braden, "Requirements for Internet Hosts -- Communication
       Layers", RFC 1122, October 1989

   [12] Kent, S. and Atkinson, R., "Security Architecture for the
       Internet Protocol", RFC 2401, Nov. 1998.

   [13] Kent, S. and Atkinson, R., "IP Encapsulating Security Payload
       (ESP)", RFC 2406, Nov. 1998.

   [14] Glenn, R., Kent, S., "The NULL Encryption Algorithm and Its Use
       With IPsec", RFC 2410, Nov. 1998

   [15] Thayer, R., Glenn, R., and Doraswamy, N., "IP Security Document
       Roadmap", RFC 2411, Nov. 1998.

   [16] Nichols, K., Blake, S., Baker, F. and D. Black, "Definition of
       the Differentiated Services Field (DS Field) in the IPv4 and
       Ipv6 Headers", RFC 2474, December 1998.


Rajagopal, et al.               Standards Track                [Page 27]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001



   [17] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss,
       W., "An Architecture for Differentiated Services", RFC 2475,
       Dec. 1998.

   [18] Allman, et. al., "TCP Congestion Control", RFC 2581, April 1999.

   [19] Heinanen, J., Baker, F., Weiss, W., Wroclawski, J., "An Assured
       Forwarding PHB", RFC 2597, June 1999.

   [20] Jacobson, V., Nichols, K., Poduri, K., "An Expedited Forwarding
       PHB Group", RFC 2598, June 1999.

   [21] E.Guttman, C. Perkins, J. Veizades, M. Day. Service Location
       Protocol, version 2, RFC 2608, July, 1999.

   [22] Floyd, et al, "SACK Extension", RFC 2883, July 2000.

   [23] Weber, Rajagopal, Travostino, Chau, O'Donnell, Monia Merhar,
       "FC Frame Encapsulation", draft-ietf-ips-fcencapsulation-__.txt
       (RFC reference and date to be added during standards action).

   The following reference concerns SLP, see [21]. It is not referenced
   in this revision of this draft but may be referenced in future
   revisions.

   [24] E.Guttman, C. Perkins, J. Kempf. Service Templates and Service:
       Schemes, RFC 2609, July 1999.

12. Bibliography

   The following references may prove informative to readers unfamiliar
   with Fibre Channel.

Kembel, R., "The Fibre Channel Consultant: A Comprehensive
Introduction", Northwest Learning Associates, 1998

13. Acknowledgments

   Funding for the RFC Editor function is currently provided by the
   Internet Society.









Rajagopal, et al.               Standards Track                [Page 28]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


14. Authors' Addresses

Murali Rajagopal                   Raj Bhagwat
LightSand Communications, Inc.     LightSand Communications, Inc.
24411 Ridge Route Dr.              24411 Ridge Route Dr.
Suite 135                          Suite 135
Laguna Hills, CA 92653             Laguna Hills, CA 92653
USA                                USA
Phone: +1 949 837 1733 x101        Phone: +1 949 837 1733 x104
Email: muralir@lightsand.com       Email: rajb@lightsand.com

R. Andy Helland                    Elizabeth G. Rodriguez
LightSand Communications, Inc.     Lucent Technologies
375 Los Coches Street              1202 Richardson Drive, Suite 104
Milpitas, CA 95035                 Richardson, TX 75080
USA                                USA
Phone: +1 408 404 3119             Phone: +1 972 231 0672
Fax: +1 408 941 2166               Fax: +1 972 644 5857
Email: andyh@lightsand.com         Email: egrodriguez@lucent.com

Sriram Rupanagunta                 Neil Wanamaker
Aarohi Communications              Akara
3200 Montelena Drive               10624 Icarus Court
San Jose, CA 95135                 Austin, TX 78726
USA                                USA
Phone: +1 408 966 8309             Phone: +1 512 257 7633
Email: sriramr@aarohi-inc.com      Fax: +1 512 257 7877
                                   Email: nwanamaker@akara.com

Steve Wilson                       Bob Snively
Brocade Comm. Systems, Inc.        Brocade Comm. Systems, Inc.
1745 Technology Drive              1745 Technology Drive
San Jose, CA. 95110                San Jose, CA 95110
USA                                USA
Phone: +1 408 487 8128             Phone: +1 408 487 8135
Fax: +1 408 487 8101               Email: rsnively@brocade.com
email: swilson@brocade.com

Ralph Weber
ENDL Texas, representing Brocade
Suite 102 PMB 178
18484 Preston Road
Dallas, TX 75252
USA
Phone: +1 214 912 1373
Email: roweber@acm.org




Rajagopal, et al.               Standards Track                [Page 29]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


David Peterson                     Donald R. Fraser
Cisco Systems - SRBU               Compaq Computer Corporation
6450 Wedgwood Road                 301 Rockrimmon Blvd., Bldg. 5
Maple Grove, MN 55311              Colorado Springs, CO 80919
USA                                USA
Phone: +1 763 398 1007             Phone: +1 719 548 3272
Cell: +1 612 802 3299              Email: don.fraser@compaq.com
Email: dap@cisco.com

Vi Chau                            Gaby Hecht
Gadzoox Networks, Inc.             Gadzoox Network, Inc.
16241 Laguna Canyon Road           16241 Laguna Canyon Road
Suite 100                          Suite 100
Irvine, CA 92618                   Irvine, CA 92618
USA                                USA
Phone: +1 949 789 4639             Phone: +1 949 789 4642
Fax: +1 949 453 1271               Fax: +1 949 453 1271
Email: vchau@gadzoox.com           Email: ghecht@Gadzoox.com

Ken Hirata                         Jim Nelson
Vixel Corporation                  Vixel Corporation
15245 Alton Parkway, Suite 100     15245 Alton Parkway, Suite 100
Irvine, CA 92618                   Irvine, CA 92618
USA                                USA
Phone: +1 949 788 6368             Phone: +1 949 450 6159
Fax: +1 949 753 9500               Fax: +1 949 753 9500
Email: ken.hirata@vixel.com        Email: Jim.Nelson@vixel.com

Michael E. O'Donnell               Anil Rijhsinghani
McDATA Corporation                 McDATA Corporation
310 Interlocken Parkway            5 Brickyard lane
Broomfield, Co. 80021              Westboro, MA 01581
USA                                USA
Phone: +1 303 460 4142             Phone: +1 508 870 6593
Fax: +1 303 465 4996               Email:
Email: modonnell@mcdata.com             anil.rijhsinghani@mcdata.com

Milan J. Merhar                    Craig W. Carlson
43 Nagog Park                      QLogic Corporation
Pirus Networks                     6321 Bury Drive
Acton, MA 01720                    Eden Prairie, MN 55346
USA                                USA
Phone: +1 978 206 9124             Phone: +1 952 932 4064
Email: Milan@pirus.com             Email: craig.carlson@qlogic.com






Rajagopal, et al.               Standards Track                [Page 30]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


Venkat Rangan                      Larwrence J. Lamers
Rhapsody Networks Inc.             SAN Valley
3450 W. Warren Ave.                4611 Park Norton Place
Fremont, CA 94538                  San Jose, CA 95136-2523
USA                                USA
Phone: +1 510 743 3018             Phone: +1 408 626 1285
Fax: +1 510 687 0136               Email: ljlamers@ieee.org
Email: venkat@rhapsodynetworks.com

15. Full Copyright Statement

   Copyright (C) The Internet Society (2001). All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph
   are included on all such copies and derivative works. However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

ANNEX A - Example of synchronization recovery algorithm

   Synchronization may be recovered as specified in section 6.6.2.4. An
   example of an algorithm for searching the bytes delivered to the
   Encapsulated Frame Receiver Portal for a valid FCIP Frame header is
   provided in this annex.

   This resynchronization uses the principle that a valid FCIP data
   stream must contain at least one valid header every 2148 bytes (the
   maximum length of an encapsulated frame). Although other data
   patterns containing apparently valid headers may be contained in the


Rajagopal, et al.               Standards Track                [Page 31]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   stream, the FC CRC or frame validity of the data patterns contained
   in the data stream will always be either interrupted by or
   resynchronized with the valid FCIP Frame encapsulation headers.

   Consider the case shown in figure 9. A series of short FCIP Frames,
   perhaps from a trace, are embedded in larger FCIP Frames, say as a
   result of a trace file being transferred from one disk to another.
   The headers for the short frames are denoted SFH and the long frame
   headers are marked as LFH.

       +-+--+-+----+-+----+-+----+-+-+-+---+-+---
       |L|  |S|    |S|    |S|    |S| |L|   |S|
       |F|  |F|    |F|    |F|    |F| |F|   |F|...
       |H|  |H|    |H|    |H|    |H| |H|   |H|
       +-+--+-+----+-+----+-+----+-+-+-+---+-+---
       |                             |
       |<---------2148 bytes-------->|

       Fig. 9  Example of resynchronization data stream

   A resynchronization attempt that starts just to the right of an LFH
   will find several SFH frames before discovering that they do not
   represent the transmitted stream of frames. Within 2148 bytes plus
   or minus, however, the resynchronization attempt will encounter an
   SFH whose length does not match up with the next SFH because the LFH
   will fall in the middle of the short frame pushing the next header
   farther out in the byte stream.

   Note that the resynchronization algorithm cannot forward any
   prospective FC Frames to the FC Transmitter Portal because until
   synchronization is completely established there is no certainty that
   anything that looked like an FCIP Frame really was one. For example,
   an SFH might fortuitously contain a length that points exactly to
   the beginning of an LFH. The LFH would identify the correct
   beginning of a transmitted frame, but that in no way guarantees that
   the SFH was also a correct FCIP Frame header.

   There exist some data streams that cannot be resynchronized by this
   algorithm. If such a data stream is encountered, the algorithm
   causes the TCP connection to be closed.

   The resynchronization assumes that security and authentication
   procedures outside the FCIP Entity are protecting the valid data
   stream from being replaced by an intruding data stream containing
   valid FCIP data.





Rajagopal, et al.               Standards Track                [Page 32]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   The following steps are one example of how an FCIP_DE might
   resynchronize with the data stream entering the Encapsulated Frame
   Receiver Portal.

   1)  Search for candidate and strong headers:

       The data stream entering the Encapsulated Frame Receiver Portal
       is searched for 12 bytes in a row containing the required values
       for:
       a)  Protocol field,
       b)  Version field,
       c)  ones complement of the Protocol field,
       d)  ones complement of the Version field,
       e)  replication of encapsulation word 0 in word 1, and
       f)  Reserved field and its ones complement.

       If such a 12-byte grouping is found, the FCIP_DE assumes that it
       has identified bytes 0-2 of a candidate FCIP encapsulation header.

       All bytes up to and including the candidate header byte are
       discarded.

       If no candidate header has been found after searching a
       specified number of bytes greater than some multiple of 2148
       (the maximum length of an encapsulated frame), resynchronization
       has failed and the TCP/IP connection is closed.

       Word 3 of the candidate header contains the Frame Length and
       Flags fields and their ones complements. If the fields are
       consistent with their ones complements, the candidate header is
       considered a strong candidate header. The Frame Length field is
       used to determine where in byte stream the next strong candidate
       header should be and processing continues at step 2).

   2)  Use multiple strong candidate headers to locate a verified
       candidate header:

       The Frame Length in one strong candidate header is used to skip
       incoming bytes until the expected location of the next strong
       candidate header is reached. Then the tests described in step 1)
       are applied to see if another strong candidate header has
       successfully been located.

       All bytes skipped and all bytes in all strong candidate headers
       processed are discarded.

       Strong candidate headers continue to be verified in this way for
       at least 4296 bytes (twice the maximum length of an encapsulated


Rajagopal, et al.               Standards Track                [Page 33]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


       frame). If at anytime a verification test fails, processing
       restarts at step 1 and a retry counter is incremented. If the
       retry counter exceeds 3 retries, resynchronization has failed
       and the TCP connection is closed.

       After strong candidate headers haves been verified for at least
       4296 bytes, the next header identified is a verified candidate
       header and processing continues at step 3).

       Note: If a strong candidate header was part of the data content
       of an FC frame, the encapsulated frame defined by that or a
       subsequent strong candidate header will eventually cross an
       actual header in the byte stream. As a result it will either
       identify the actual header as a strong candidate header or it
       will lose synchronization again because of the extra 28 bytes in
       the length, returning to step 1 as described above.

   3)  Use multiple strong candidate headers to locate a verified
       candidate header:

       Incoming bytes are skipped and discarded until the next verified
       candidate header is reached. Each verified candidate header is
       tested against the full collection of tests listed in section
       6.6.2.2 as would normally be the case.

       Verified candidate headers continue to be located and tested in
       this way for a minimum of 4296 bytes (twice the maximum length
       of an encapsulated frame). If all are verified candidate headers
       encountered are valid, the last verified candidate header is a
       valid header. At this point the FCIP_DE stops discarding bytes
       and begins normal FCIP de-encapsulation begins, including for
       the first time since synchronization was lost, delivery of FC
       frames through the FC Transmitter Portal according to normal
       FCIP rules.

       If any verified candidate headers are invalid but meet all the
       requirements of a strong candidate header, increment the retry
       counter and return to step 2). If any verified candidate headers
       are invalid and fail to meet the tests for a strong candidate
       header, increment the retry counter and return to step 1. If the
       retry counter exceeds 4 retries, resynchronization has failed
       and the TCP/IP connection is closed.

   A flowchart for this algorithm can be found in figure 10.






Rajagopal, et al.               Standards Track                [Page 34]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


                         Synchronization is lost
                                  |
                     _____________v_______________
                    |                             |
                    | Search for candidate header |
       +----------->|                             |
       |            |   Found           Not Found |
       |            | (Strong candidate)          |
       |            |_____________________________|
       |                    |              |
       |                    |              + --------->Close TCP/IP
       |             _______v_____________________     Connection
       |            |                             |
       |            |   Enough strong candidate   |
       |      +---->|     headers identified?     |
       |      |     |                             |
       |      |     |     No               Yes    |
       |      |     |        (Verified candidate) |
       |      |     |_____________________________|
       |___________________|                |
       ^      |                             |
       |      |                             |
       |      |      _______________________v_____
       |      |     |                             |
       |      |     | Enough verified candidate   |
       |      |     |   headers validated?        |
       |      |     |                             |
       |      |     |     No               Yes    |
       |      |     |            (Resynchronized) |
       |      |     |_____________________________|
       |      |            |                |
       |      |      ______v__________      |      Resume
       |      |     |                 |     + ---> Normal
       |      |     | Synchronization |            De-encapsulation
       |      |     |      Lost?      |
       |      |     |                 |
       |      |     | No          Yes |
       |      |     |_________________|
       |      |        |           |
       |      |________|           |
       |___________________________|

       Fig. 10  Flow diagram of simple synchronization example







Rajagopal, et al.               Standards Track                [Page 35]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


ANNEX B - Relationship between FCIP and IP over FC (IPFC)

   IPFC (RFC 2625) describes the encapsulation of IP packets in FC
   frames. It is intended to facilitate IP communication over an FC
   network.

   FCIP describes the encapsulation of FC frames in TCP segments which
   in turn are encapsulated inside IP packets for transporting over an
   IP network. It gives no consideration to the type of FC frame that
   is being encapsulated. Therefore, the FC frame may actually contain
   an IP packet as described in the IP over FC specification (RFC
   2625). In such a case, the data packet would have:

    - Data Link Header
    - IP Header
    - TCP Header
    - FCIP Header
    - FC Header
    - IP Header

   Note:   The two IP headers would not be identical to each other. One
   would have information pertaining to the final destination while the
   other would have information pertaining to the FCIP Entity.

   The two documents focus on different objectives. As mentioned above,
   implementation of FCIP will lead to IP encapsulation within IP.
   While perhaps inefficient, this should not lead to issues with IP
   communication. One caveat: if a Fibre Channel device is
   encapsulating IP packets in an FC frame (e.g. an IPFC device), and
   that device is communicating with a device running IP over a non-FC
   medium, a second IPFC device may need to act as a gateway between
   the two networks. This scenario is not specifically addressed by FCIP.

   There is nothing in either of the specifications to prevent a single
   device from implementing both FCIP and IP-over-FC (IPFC), but this
   is implementation specific, and is beyond the scope of this document.

ANNEX C - FC Frame Format

   All FC frames have a standard format much like LAN's 802.x
   protocols. However, the exact size of each frame varies depending on
   the size of the variable fields. The size of the variable field
   ranges from 0 to 2112-bytes as shown in the FC Frame Format in
   figure 11 resulting in the minimum size FC Frame of 36 bytes and the
   maximum size FC frame of 2148 bytes. Valid Fibre Channel frame
   lengths are always a multiple of four bytes.




Rajagopal, et al.               Standards Track                [Page 36]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


       +------+--------+-----------+----//-------+------+------+
       | SOF  |Frame   |Optional   |  Frame      | CRC  |  EOF |
       | (4B) |Header  |Header     | Payload     | (4B) | (4B) |
       |      |(24B)   |<----------------------->|      |      |
       |      |        | Data Field = (0-2112B)  |      |      |
       +------+--------+-----------+----//-------+------+------+

       Fig. 11  FC Frame Format

SOF and EOF Delimiters

   On an FC link, Start-of-Frame (SOF) and End-Of-Frame (EOF) are
   called Ordered Sets and are sent as special words constructed from
   the 8B/10B comma character (K28.5) followed by three additional 8B/
   10B data characters making them uniquely identifiable in the data
   stream.

   On an FC link the SOF delimiter serves to identify the beginning of
   a frame and prepares the receiver for frame reception. The SOF
   contains information about the frame's Class of Service, position
   within a sequence, and in some cases, connection status.

   The EOF delimiter identifies the end of the frame and the final
   frame of a sequence. In addition, it serves to force the running
   disparity to negative. The EOF is used to end the connection in
   connection-oriented classes of service.

   It is therefore important to preserve the information conveyed by
   the delimiters across the IP-based network, so that the receiving
   FCIP Entity can correctly reconstruct the FC frame in its original
   SOF and EOF format before forwarding it to its ultimate FC
   destination on the FC link.

   When an FC frame is encapsulated and sent over a byte-oriented
   interface, the SOF and EOF delimiters are represented as sequences
   of four consecutive bytes, which carry the equivalent Class of
   Service and frame termination information as the FC ordered sets.
   The representation of SOF and EOF in an encapsulation FC frame is
   described in FC Frame Encapsulation [23].

Frame Header

   The FC Frame Header is transparent to the FCIP Entity. The FC Frame
   Header is 24 bytes long and has several fields that are associated
   with the identification and control of the payload. Current FC
   Standards allow up to 3 Optional Header fields [6]:

    - Network_Header (16-bytes)


Rajagopal, et al.               Standards Track                [Page 37]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


    - Association_Header (32-bytes)
    - Device_Header (up to 64-bytes).

Frame Payload

   The FC Frame Payload is transparent to the FCIP Entity. An FC
   application level payload is called an Information Unit at the FC-4
   Level. This is mapped into the Frame Payload of the FC Frame. A
   large Information Unit is segmented using a structure consisting of
   FC Sequences. Typically, a Sequence consists of more than one FC
   frame. FCIP does not maintain any state information regarding the
   relationship of frames within a FC Sequence.

CRC

   The FC CRC is 4 bytes long and uses the same 32-bit polynomial used
   in FDDI and is specified in ANSI X3.139 Fiber Distributed Data
   Interface. This CRC value is calculated over the entire FC header
   and the FC payload; it does not include the SOF and EOF delimiters.

   Note: When FC frames are encapsulated into FCIP frames, the FC frame
   CRC is untouched by the FCIP Entity.

ANNEX D - FCIP Requirements on an FC Entity

   The capabilities that FCIP requires of an FC Entity include:

   1)  The FC Entity must deliver FC frames to the correct FCIP Data
       Engine (in the correct FCIP Link Endpoint) and forward FC Frames
       from FCIP Data Engine(s) to the FC Fabric.

   2)  The only delivery ordering guarantee provided by FCIP is
       correctly ordered delivery of FC Frames between a pair of FCIP
       Data Engines. FCIP expects the FC Entity to implement all other
       FC Frame delivery ordering requirements.

   3)  The FC Entity must support the FCIP Entity in the processing of
       incoming connect requests by deciding to accept a connect request.

   4)  The FC Entity may generate connect and terminate requests.

   5)  The FC Entity may instruct the FCIP Entity regarding TCP
       connection parameter settings and the R_A_TOV to be applied to
       an FCIP Link.

   6)  The FC Entity may recover from connection failures.




Rajagopal, et al.               Standards Track                [Page 38]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   7)  The FC Entity must recover from events that the FCIP Entity
       cannot handle, such as:
       a)  loss of synchronization with FCIP Frame headers from the
           Encapsulated Frame Receiver Portal requiring resetting the
           TCP connection,
       b)  additional examples, TBD

   8)  The FC Entity must work cooperatively with the FCIP Entity to
       manage flow control problems in either the IP Network or FC
       Fabric.

   9)  The FC Entity may test for failed TCP connections.

   10) TBD support for dynamic discovery

   11) TBD support for security

   12) TBD support for connection QoS features

   13) TBD support for monitoring

   Note that the Fibre Channel standards MUST be consulted for a
   complete understanding of the requirements placed on an FC Entity.

ANNEX E - FC-BB-2 Inputs

   This annex contains text from previous FCIP drafts that, because of
   the new model structure, probably belongs in FC-BB-2 [4]. As soon as
   the correctness of this annex is agreed, its contents will be
   transferred to a T11 document do be used in the development of FC-BB-
   2.

   No attempt has been made to rewrite this text for inclusion in an
   T11 standards, so it should be considered a guide to T11 content no
   a specification.

   All section references in this annex come from draft-ietf-ips-
   fcovertcpip-02.txt. When only parts of a section are included here,
   "{partial}" is appended to the section title.

4.3 FCIP's Interaction with FC and TCP {partial}

   Since FC Primitive Signals and Primitive Sequences are not exchanged
   between FCIP devices, there may be times when an FC frame is lost
   within the IP network. When this event occurs it is the
   responsibility of the communicating FC devices to detect and correct
   the errors based on the features defined in FC-FS [6]. The FCIP
   devices MAY choose not to generate Fibre Channel's F_BSY or F_RJT


Rajagopal, et al.               Standards Track                [Page 39]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   frames or otherwise participate in FC frame recovery.

   Note that the order of the FC Frames sent by the Encapsulated Frame
   Transmitter MAY not be the same as the order sent by the source FC
   End Node. This is due to the fact that some types of FC login allow
   FC Frames to be re-ordered in the FC Fabric before reaching the FC
   Receiver Port.

5.3 TCP Connection Management

   In order to realize a Virtual ISL between two FC end-points, an FCIP
   Device establishes TCP connection(s) with its peer FCIP Device. In
   order to achieve better TCP aggregate throughput properties in the
   face of packet losses, a pair of peer FCIP devices MAY use multiple
   TCP connections between them, and use appropriate policies for
   mapping FC frames to these connections. It may also be useful to
   assign a pool of connections for transmission of high priority and
   control messages (e.g., Class F messages) on connections so they do
   not encounter "head of line" blocking behind Class 2 or Class 3
   traffic. The use of multiple connections and policies for
   distributing frames on these connections are described in section 5.5.

   A Virtual ISL and the two FCIP Device endpoints that are involved
   are operational only after the first TCP connection is established.
   The sequence of operations performed in order to establish a Virtual
   ISL is as follows.

   1)  The FCIP device initializes its local resources to enable it to
       listen to TCP connection requests.

   2)  The FCIP device discovers the FCIP device endpoints that it can
       establish a virtual ISL. The result of the discovery SHALL be,
       at the minimum, the IP address and the TCP port of the peer
       endpoint. The discovery process MAY rely on administrative
       configuration or on services such as SLP or iSNS (TBD). (Needs
       to have its own section eventually).

   3)  The FCIP device endpoint SHALL exchange security context and
       authenticate itself to the peer endpoint. The use of security
       context is explained in section 8.

   4)  After connection establishment, FCIP devices use the FCIP frame
       encapsulation defined in FC Frame Encapsulation [23] and in
       section 6.6.1 of this document.

   5)  At this point the FCIP device endpoint SHALL exchange Fibre
       Channel port initialization frames (SW_ILS) to enable and
       identify port operation. Port state machine and initialization


Rajagopal, et al.               Standards Track                [Page 40]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


       are described in FC-SW-2 [5].

   6)  An FCIP device operates in E-port or B-Port mode. When operating
       in E-Port mode, normal FC-SW2 FSPF messages are exchanged and
       the switch port becomes operational.

   7)  For computing the link cost of the ISL, the following formula
       SHALL be used: <TBD>.

   In certain deployments, a single FCIP device endpoint MAY establish
   virtual ISLs with multiple FCIP device endpoints. In this situation,
   the FCIP device endpoint SHALL manage TCP operational parameters
   independently for each ISL. Also, the FCIP Device Endpoint SHALL
   perform the E_Port or B_Port initialization independently, for each
   connection.

5.4.1 Determining loss of connectivity {partial}

   In order to facilitate faster detection of loss of connectivity, FC
   Switches MAY process the Hello (HLO) SW_ILS request through a pair
   of FCIP devices.

   The relationship between the HLO SW_ILS and the paired FCIP devices
   is TBD.

   Upon detecting a loss of connectivity, an FCIP Device SHALL
   establish a new connection, or SHALL use an existing TCP connection
   to the same FCIP Device endpoint. An FCIP Device SHALL NOT
   retransmit an FCIP frame on the new connection. This is to ensure
   exactly-once delivery semantics to the Fibre Channel endpoint.

5.5 Multiple Connection Management

   A pair of FCIP device endpoints MAY establish a certain number of
   TCP connections between them. Since a Virtual ISL potentially maps a
   fairly large number of FC flows (where a flow is a pair of Fibre
   Channel S_ID, D_ID addresses), it may not be practical to establish
   a separate TCP connection for each Fibre Channel flow. In order to
   address this, an implementation MAY choose to manage a pool of TCP
   connections for a single Virtual ISL and map Fibre Channel flows to
   TCP connections of that ISL. However, while assigning Fibre Channel
   flows to TCP connections, an implementation SHALL follow the
   following rules:

   1)  Once a Fibre Channel flow is assigned to a TCP connection within
       the virtual ISL, it SHALL send all Fibre Channel frames of that
       flow on that connection.



Rajagopal, et al.               Standards Track                [Page 41]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   2)  When an FCIP endpoint processes any response traffic from a
       particular target, the Endpoint SHALL send the response on the
       same connection on which the request was sent.

   3)  Any class 2 ACK frames SHALL be sent on the same connection in
       which the original frame was sent.

   These rules are in place to honor any in-order delivery guarantees
   that may have been made between the two end points of the Fibre
   Channel flow.

5.6 Multi Virtual ISL Management

   It is quite likely that a single switch MAY provide multiple Virtual
   ISLs, all providing alternate connectivity paths between two
   switches. In this situation, a switch SHALL select any of the
   available ISLs for mapping a FCIP flow. In doing so, a switch MUST
   follow a flow allegiance model, where a pair of Fibre Channel [S_ID,
   D_ID] end points are always mapped to the same Virtual ISL.
   Furthermore, switches SHALL implement a connection allegiance
   policy, which ensures that the responses to particular [S_ID, D_ID]
   pair is always sent back on the same Virtual ISL.

8.3 Corruption {partial}

   Data corruption is detected at two different levels: TCP checksum
   and Fibre Channel frame encapsulation errors. Data corruption
   detected at the TCP level SHALL be recovered via TCP data recovery
   mechanisms. The recovery for Fibre Channel frame errors is described
   below. The TCP and Fibre Channel frame recovery operations are
   performed independently.

   Fibre Channel frame errors and the expected resolution of those
   errors are described below:

   a)  Incoming frames on the FC Receiver Port SHALL be verified for
       correct header, proper format, valid length and valid CRC.
       Frames having incorrect headers or CRC SHALL be discarded or
       processed in accordance with the rules for the particular type
       of FC Port.

   b)  All frames transmitted by the Encapsulated Frame Transmitter
       shall be valid FC Encapsulations of valid FC frames with correct
       TCP check sums on the correct TCP/IP connection.

   e)  The FC frames contained in incoming encapsulated frames on the
       FC Receiver Port SHALL be verified for a valid header, proper
       content, proper SOF and EOF values, and valid length. FC frames


Rajagopal, et al.               Standards Track                [Page 42]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


       that are not valid according to those checks SHALL be managed
       according to the following rules.

       1)  The frame may be discarded before being transmitted by the
           FC Transmitter Port.

       2)  The frame may be transmitted in whole or in part by the FC
           Transmitter Port and ended with an EOF indicating that the
           content of the frame is invalid.

   f)  Any encapsulated frame received by the Encapsulated Frame
       Receiver that has an invalid Fibre Channel CRC shall be managed
       according to the following rules.

       1)  The frame may be transmitted unchanged by the FC Transmitter
           Port. The frame will be discarded by the receiving FC Port
           because of invalid CRC.

       2)  The frame may be discarded before being transmitted by the
           FC Transmitter Port.

       3)  The frame may be transmitted in whole or in part by the FC
           Transmitter Port and ended with an EOF indicating that the
           content of the frame is invalid. The FC encapsulation header
           Frame Length field MUST correctly specify the transmitted
           frame length.

8.4 Timeouts {partial}

   Fibre Channel has two important timeouts to consider in FCIP. These
   are: E_D_TOV, and R_A_TOV.

   E_D_TOV determines the life of an individual Fibre Channel frame in
   any particular fabric element. The effects of E_D_TOV on the fabric
   as a whole are typically cumulative since each fabric element
   contains it's own E_D_TOV timers for any frame received.

   R_A_TOV determines the life of an individual Fibre Channel frame in
   the fabric as a whole.   For a fabric, R_A_TOV implies that no
   particular frame will remain in (and thus be emitted from) the
   fabric after the timer expires.

10.1 Flow control on FC network

   When the Fibre channel traffic is encapsulated over TCP
   connection(s), the FCIP device needs to ensure that the TCP
   connections can handle the frame arrival rate from FC Fabric. This
   MAY require FCIP device to use Buffer-to-Buffer flow control (see FC-


Rajagopal, et al.               Standards Track                [Page 43]


Internet-Draft        Fibre Channel Over TCP/IP (FCIP)        June, 2001


   FS [6]) on its Fibre Channel port(s) to control the frame arrival
   rate.




















































Rajagopal, et al.               Standards Track                [Page 44]