IPS Working Group   M. Rajagopal, R. Bhagwat, LightSand Communications
INTERNET-DRAFT                       E. Rodriguez, Lucent Technologies
<draft-ietf-ips-fcovertcpip-00.txt>          V. Chau, Gadzoox Networks
(Expires March 31, 2001)                              S. Berman, Vixel
                                     S. Wilson, Brocade Communications
                                                  M. O'Donnell, McDATA
                                                    C. Carlson, QLogic

                    Fibre Channel Over TCP/IP (FCIP)

Status of this Memo

   This document is an  Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC 2026 [1].

   Internet-Drafts  are working documents of the  Internet Engineering
   Task Force (IETF),  its areas,  and its  working groups.  Note that
   other groups  may also distribute  working  documents  as Internet-

   Internet-Drafts  are draft  documents  valid for a  maximum  of six
   monthsx  and  may  be updated,  replaced,  or  obsoleted  by  other
   documents at any time.  It is inappropriate  to use Internet-Drafts
   as  Reference  material or to cite  them  other  than as  ``work in

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

1. Abstract

   Fibre Channel (FC) is a dominant technology  used  in  Storage Area
   Networks (SAN).  The purpose of this draft is to specify a standard
   standard way of encapsulating FC frames over TCP/IP and to describe
   mechanisms that allow islands of FC SANs to be interconnected  over
   IP-based  networks.  FC  over  TCP/IP  relies  on IP-based  network
   services to provide the connectivity  between the SAN  islands over
   LANs, MANs, or WANs.  The FC  over TCP/IP specification relies upon
   TCP for congestion  control and management and upon both TCP and FC
   for data error  and data loss recovery.  FC over TCP/IP  treats all
   classes of FC frames the same -- as datagrams.

2. Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   this document are to be interpreted as described in RFC 2119 [2].

Rajagopal, et al.                                               [Page 1]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

3. Motivation and Objectives

   Fibre  Channel  (FC)  is  a  gigabit  speed  networking  technology
   primarily used for Storage Area Networking (SAN).  Fibre Channel is
   standardized  under  American  National  Standard  for  Information
   Systems  of  the  National  Committee  for  Information  Technology
   Standards  (ANSI-NCITS)  and  has  specified a  number of documents
   describing  its protocols, operations, and services [13].

   The motivation behind connecting  remote sites include disk or tape
   backup and live mirroring, or simply distance extension between two
   FC devices or FC Switch clusters (SAN islands).

   A  fundamental assumption made  in this  specification  is that the
   Fibre  Channel  traffic is  carried  over the  IP network in such a
   manner that the Fibre Channel  fabric and all Fibre Channel devices
   on  the  fabric are  unaware of the fact.   This means that  the FC
   datagrams  must be  delivered  in  such  time  as  to  comply  with
   existing  Fibre Channel  specifications.   The FC traffic  may span
   LANs,  MANs  and  WANs, so long as  this fundamental  assumption is
   adhered to.

   Fibre  Channel  operates  at  Gigabit  speeds.  This  specification
   is written such that Fibre Channel traffic may be  transported over
   an IP  backbone  that has  been  engineered to  have  equivalent or
   better  bit-error-rate (BER)  and line  speed as the Fibre  Channel
   environments  being  bridged.  While the tunneling of Fibre Channel
   traffic over other IP networks not  so engineered is not precluded,
   the above environment is an  important one,  and this specification
   has been written  so that to optimize  for such  traffic, while not
   over-burdening other configured IP networks.

   The main objectives of this document are to:

        1) specify the TCP encapsulation and mapping of FC frames
        2) apply the mechanism described in (1) to an FC network using
           IP  as a  backbone, or  more generally,  between any two FC
        3) address  any FC concerns  arising from tunneling FC traffic
           over  an IP  network, including  security,  data  integrity
           (loss),   congestion,  and   performance.   This  will   be
           accomplished, where appropriate,  by utilizing the existing
           IETF-specified suite of protocols.
        4) be  backwards  compatible  with existing  FC specifications.
           While  new work may  be  undertaken  in T11[13]  to optimize
           and  enhance  the bridging of FC  networks and fabrics, this
           specification  will not  require  adherence  to such  future

Rajagopal, et al.                                               [Page 2]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

4. FCIP Protocol

4.1 FCIP Device

   In this  specification,  the term  FCIP device generally refers  to
   any  device that encapsulates  FC frames into TCP  segments and re-
   assembles TCP segments to regenerate FC frames.

   Note: In an actual implementation,  the FCIP device may be a stand-
         alone box or integrated with an FC device such as an FC back-
         bone switch (BBW)  or integrated with any  TCP/IP device such
         as an IP switch or an IP router.

   The FCIP device is a transparent translation point.  The IP network
   is not  aware of the FC  payload that it is carrying. Likewise, the
   FC  fabric  and the  FC end  nodes  are  unaware  of  the  IP-based

4.2 Protocol

   The FCIP protocol specifies  the TCP/IP encapsulation,  mapping and
   routing of FC  frames and applies these mechanisms to an FC network
   utilizing IP for  its backbone, or more generally, between  any two
   FC devices. The FCIP protocol is summarized below:

     1. All  FCIP  protocol  devices are peers  and  communicate using
        TCP/IP.  Each FCIP device behaves like a TCP end-node from the
        perspective of the IP-based network. That is, these devices do
        not  perform IP routing or IP switching but simply  forward FC

     2. There is  no  requirement for an  FCIP device to  establish  a
        login with a peer  before communication begins.  However, FCIP
        devices  may authenticate the  IP  packet  before accepting it
        using  the  IPSec  protocols.  Each  IP  datagram  is  treated
        independently and an FCIP device  receiver  simply  listens to
        the appropriate socket value contained in the TCP header.

     3. Each FCIP  device may be statically  or dynamically configured
        with  a  list  of  IP  addresses   corresponding  to  all  the
        participating FCIP devices. Dynamic discovery of participating
        FCIP devices may be performed using Internet protocols such as
        LDAP, DHCP  or other  discovery  protocols.  It is outside the
        scope of this specification  to describe any static or dynamic
        scheme for  participating FCIP device IP address discovery.

     4. Discovery  of FC addresses  (accessible via  the FCIP  device)
        is  provided   by  techniques  and  protocols  within  the  FC
        architecture.  These techniques and protocols are described in
        Fibre Channel ANSI standards ([3], [7], [15]). The FCIP device
        does not participate in the discovery of FC addresses. Routing
        in the IP plane and the FC plane are largely independent.

Rajagopal, et al.                                               [Page 3]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     5. The exact path (route) taken by an FC over TCP/IP encapsulated
        packet follows the normal procedures of routing any IP packet.
        From the perspective of the FCIP devices this communication is
        between only two FCIP devices for any given packet.

     6. An FCIP device may send FC encapsulated TCP/IP packets to more
        than one FCIP device.  However, these encapsulated packets are
        treated as separate instances  and are not  correlated  in any
        way  by the FCIP  protocol  devices.  The source  FCIP  device
        routes  its packets based on the 3-byte FC destination Address
        Identifier (D_ID) contained in each FC frame.

     7. An IP packet  may  make use of  the  IPSec protocol to provide
        secure communications across the IP-based network.

     8. Any re-ordering  of data  link frames due to MTU fragmentation
        will be recovered  in accordance  with  a normal  TCP reliable
        delivery behavior.

        Any re-ordering of FC frames due to IP packet re-ordering will
        be resolved via the standard TCP reliable delivery behavior.

     9. FCIP relies on both TCP error recovery mechanism and normal FC
        recovery  mechanisms to detect and recover  from data loss due
        to any loss of IP packets.

     10. FC over TCP/IP encapsulated IP packets shall indicate the use
         of the Premium Service in the DSCP bits in the IP header.

     11. The TCP layer in  the sending FCIP device shall  package each
         FC frame handed  down by the FC  layer into a TCP segment and
         set the PSH control flag in the TCP header to ensure that the
         entire FC frame  is sent in one TCP segment.  If the FC frame
         cannot be packaged in one TCP segment (e.g. the FC frame size
         is greater than TCP MSS),  the last part of the FC frame must
         occupy  one TCP segment  and the PSH of that segment  must be

5. FCIP Encapsulation

5.1 FC Frame Format

     All  FC  frames  have  a  standard  format  much like LAN's 802.x
     protocols. However, the exact size of each frame varies depending
     on the  sizes  of the variable fields.  The sizes of the variable
     fields range from 0 to 2112-bytes as shown in the FC Frame Format
     in Fig. 1  resulting in the minimum size FC Frame of 36 bytes and
     the maximum size  FC frame of 2148 bytes.

Rajagopal, et al.                                               [Page 4]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

          | SOF  |Frame   |Optional   |  Frame      | CRC  |  EOF |
          | (4B) |Header  |Header     | Payload     | (4B) | (4B) |
          |      |(24B)   |<----------------------->|      |      |
          |      |        | Data Field = (0-2112B)  |      |      |
                           Fig. 1 FC Frame Format

     Start Of Frame (SOF) and End Of Frame (EOF) Delimiters:

     On an FC link,  Start-of-Frame (SOF)  and End-Of-Frame (EOF)  are
     called  Ordered Sets  and are sent as special  words  constructed
     from  the  10-bit  comma  character (K28.5)  followed  by   three
     additional 10-bit  data characters.  On a non-Fibre Channel link,
     the  SOF  and  EOF delimiters  are both byte-encoded and  4-bytes

     On an FC link the SOF delimiter  serves to identify the beginning
     of a frame and prepares the receiver for frame reception. The SOF
     contains information about the frame's Class of Service, position
     within a sequence, and in some cases, connection status.

     The EOF delimiter  identifies the  end of the frame and the final
     frame of a sequence.  In addition, it serves to force the running
     disparity to negative.  The EOF is used to  end the connection in
     connection-oriented classes of service.

     It is therefore important to preserve the information conveyed by
     the delimiters across the IP-based network, so that the receiving
     FCIP  device  can  correctly  reconstruct  the  FC  frame  in its
     original  SOF and EOF format before forwarding it to its ultimate
     FC destination on the FC link.

     Start of Frame  (SOF) and End of Frame  (EOF)  byte-encodings are
     defined in Annex A.  Although the SOF and EOF codes are  32-bits,
     the  format  makes  use of a single-byte  to  represent  each  FC
     Ordered Set.   The remaining  three bytes  associated  with these
     delimiters in  the TCP  encapsulated FC frames do not necessarily
     carry any meaningful information.

     Frame Header:

     The Frame Header is 24-bytes long and has several fields that are
     associated  with the  identification  and control of the payload.
     Current  FC  Standards  allow up  to three Optional Header fields

          - Network_Header (16-bytes)
          - Association_Header (32-bytes)
          - Device_Header (up to 64-bytes).

Rajagopal, et al.                                               [Page 5]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     Frame Payload:

     The FC  Frame  Payload is  transparent  to the  FCIP  device.  An
     FC application level payload is called an Information Unit at the
     FC-4  Level.  This  is  mapped into the Frame  Payload  of the FC
     Frame. A large Information  Unit is segmented  using a  structure
     consisting  of FC  Sequences.  Typically, a Sequence  consists of
     more  than one  FC  frames.  FCIP  does  not  maintain any  state
     information   regarding  the relationship of frames  within an FC


     The CRC is 4-bytes long and uses the same 32-bit polynomial  used
     in FDDI and is  specified  in ANSI  X3.139 Fiber Distributed Data

     Note: When FC frames are encapsulated into TCP segments,  the CRC
           is untouched.

5.2 FC Frame Mapping to TCP/IP packets

     Fig.2  shows the mapping of the FC frame into an TCP/IPv4 Packet.
     The FC to TCP/IP  mapping (and the reverse) mapping is one-to-one
     since  the maximum size  of the encapsulated FC  Frame along with
     the header fields does not exceed 2148 bytes.

     The  minimum size FC  Frame is 36 bytes resulting in a  maximally
     minimum IP MTU size of 156 bytes. (The maximally minimum MTU size
     is  the IP packet  with the minimum size payload and  the maximum
     size IP headers and TCP headers).

     The maximum size FC frame is 2148 bytes resulting in a nominal IP
     packet size of 2188 bytes.  Fig. 2  shows the  format of the IPv4
     packet  with  the  standard  20-byte  IP fixed header,  a 40-byte
     optional IP header, the standard 20-byte TCP fixed header and TCP
     options.  For the case of the maximum size payload of 2148 bytes,
     the maximum TCP/IPv4 packet size is 2268 bytes.

     The  maximum  size  FC  frame  can  cause  the  IP  packet  to be
     fragmented  when  the  data  link  cannot  support this MTU size.
     Furthermore, TCP may also segment the data if the maximum segment
     size (MSS) is less than the size of the frame.  When an IP packet
     is fragmented, the required parts of the header must be copied by
     all fragments and the option field may or may not be copied. When
     a TCP packet is segmented, normal TCP  operations are responsible
     for receiving  and re-sequencing the data prior  to passing it on
     to the FC processing portion of the device.

Rajagopal, et al.                                               [Page 6]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     | IP Header |IP Opt. Header|TCP Header|TCP Options| FC Frame  |
     |(20 bytes) |  (40 bytes   |(20 bytes)| (40 bytes |(2148 bytes|
     |           |     Max)     |          |    Max)   |   Max)    |

                Fig. 2 Format of an IPv4 Payload carrying FC

     <Placeholder for IPv6 format>

     If IPSec is used for security  it introduces  its own headers and
     the IP packet size  increase depends  on the exact mode of  IPSec
     usage  (AH versus ESP,  Tunnel versus  Transport).  However, this
     additional increase in the IP packet size due to IPSec headers is
     relatively small  (see [8], [9], [10]),  and the  maximum size IP
     packet will remain within its maximum size of 65535 bytes. Adding
     IPSec header  may,  in some cases,  lead to  fragmentation if  it
     exceeds the data link MTU.

     IP Header Field Setting:

     DSCP (6 bits):  The Differentiated Service Code Points (DSCP) [6]
     shall be set to correspond to the Premium Service.  This  service
     provides "Expedited Forwarding" at each  IP hop (Per Hop Behavior

     Protocol (8 bits):  This  8-bit  field  defines  the higher level
     protocol that  uses the  service of  the IP  layer. In this case,
     this is set to the TCP Protocol Value 6 as defined in [12].

     Source  IP  Address (32 bits):  This  is  the  IP  address of the
     ingress FCIP device that is transmitting the IP packet containing
     the FC frame.

     Destination IP Address (32 bits):   This  is  the  IP address  of
     the egress FCIP device that is receiving the IP packet containing
     the FC frame.

     The FCIP specification treats all classes of FC frames the same -
     as datagrams. Since FC Primitives and Primitive Sequences are not
     exchanged  between FCIP devices  there may  be times when a frame
     is  lost in the  IP  network.  When  this event occurs  it is the
     responsibility of  the  communicating  Fibre  Channel  devices to
     detect  and  correct  the  errors.  The  FCIP  devices shall  not
     generate  Fibre Channel  F_BSY and/or  F_RJT frames  or otherwise
     participate  in FC frame recovery.  FCIP may  not be suitable for
     the transport of Class 1 traffic  since these frames are  treated
     the same way as any Class 2 or 3 frame by the FCIP devices.

Rajagopal, et al.                                               [Page 7]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     5.3 FC frame mapping to TCP Segment

     The FC-to-TCP mapping  (and reverse mapping) may not  necessarily
     be one-to-one  as the FC frame  size may exceed the  TCP  Maximum
     Segment  Size (MSS).  In this case,  the TCP layer in the sending
     FCIP device may segment the  FC frame into multiple TCP segments.
     The sending device must ensure that the last TCP segment contains
     only the last part of the encapsulated FC frame and that last TCP
     segment does not contain the  beginning of another FC frame.  The
     last  TCP  segment  must also have  the PSH flag  set so that the
     receiving FCIP  device  knows to send the  frame to  the FC layer
     above  it.  The fields in  the TCP  header are  explained  in the
     following paragraphs:

     Source Port:

     The port number selected by the sending FCIP device.

     Destination Port:

     The  well-known  port number assigned for FC encapsulation.  This
     number is assigned by ... <to be added>

     Sequence Number:

     The sequence number of the SOF byte-code  of the FC frame or  the
     first byte of the current TCP segment payload.

     Acknowledge Number:

     If the ACK flag is set, this  field contains  the sequence number
     of the next byte that the receiving TCP expects to receive.

     Data Offset:

     The  number of 32-bit  words in  the TCP  header.  The use of TCP
     options are not precluded for FCIP devices.

     Control (flag) bits:

         URG:  Urgent  Pointer field significant -  must be 0; the use
               of urgent  data is not allowed  for segments containing
               encapsulated FC frames.

         ACK:  Acknowledgment  field  significant  -  FCIP devices use
               this  bit  to indicate that  the  Acknowledge number is

         PSH:  Push Function - The sending  FCIP device shall set this
               bit  to 1  if the  entire  FC frame  fits  inside a TCP

Rajagopal, et al.                                               [Page 8]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

               segment or the last portion of an FC frame  (containing
               the EOF) is  sent in its own TCP segment.
         RST:  Reset the  connection - the FCIP device shall  set this
               flag  according to the state  of the TCP connection and
               in compliance to the TCP specifications.

         SYN:  Synchronize sequence  numbers -  the FCIP  device shall
               set  this  flag  according  to the  state  of  the  TCP
               connection and in compliance to the TCP specifications.

         FIN:  No more data  from  sender - the FCIP  device shall set
               this flag according to the state of the TCP  connection
               and in compliance to the TCP specifications.


     The  FCIP device  shall operate this field according to the state
     of  the   TCP   connection   and  in  compliance   to   the   TCP


     The FCIP device  shall calculate the  checksum over the entire FC
     frame or the  portion  of the  FC frame  encapsulated  inside the
     current segment.  The checksum shall  also include the TCP header
     and the pseudo IP header as specified in [16].

     Urgent Pointer:

     The FCIP devices shall  not use this field as the urgent function
     is not allowed in tunneling FC frames over the IP network.

     Options and Padding:

     FCIP  devices  are  allowed  to  use TCP options  and  padding as
     specified by the TCP specifications [16].

5.4 Fibre Channel Bit and Byte Ordering

     Fibre Channel's FC-1 Level defines the method used to encode data
     prior to  transmission  and  subsequently  decode  the data  upon
     reception.  The method encodes 8-bit bytes into 10-bit characters
     to  improve the transmission  characteristics of  the serial data
     stream.  In Fibre  Channel,  data  fields  are  aligned  on  word
     boundaries.  A word in FC is defined as  4 bytes or 32 bits.  The
     resulting  transmission  word after the 8-bit  to 10-bit encoding
     consists of 40 bits.

     Data words  or Ordered Sets  (special FC-2  Level  control words)
     from the FC-2 Level map to the FC-1 Level with no change in order

Rajagopal, et al.                                               [Page 9]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     and the bytes in the word are transmitted in the Most Significant
     Byte first to  Least Significant  Byte  order.  The  transmission
     order of  bits in each byte is  from the Least Significant Bit to
     the Most Significant Bit.

6. FCIP Network

6.1 FC Backbone Switches

     Fibre  Channel  Standards  [3]  describe  the  operation  of  and
     interactio betweenn FC  Switches.  Two distinct levels of  switch
     interconnections  are  specified.  Autonomous   Regions  (AR) are
     defined to allow clusters of FC Switches to be connected across a
     backbone network called an FSPF-backbone. An Autonomous Region is
     administratively defined with each AR encompassing one or more FC
     Domains.  The  FSPF-backbone network  is formed  from one or more
     Backbone  Switches  (BSW)  that  run  the  FSPF-backbone  routing
     protocol. The FSPF-backbone routing protocol is based on OSPF and
     the FSPF backbone  may consist of  an arbitrary  mesh network.  A
     BSW may communicate with multiple neighbors. As specified in [3],
     native FC frames traverse the backbone between BSW neighbors. The
     FSPF-backbone  Routing  Protocol  messages are exchanged  between
     BSWs on the FSPF backbone.

     An  example  network  consisting  of  4 ARs and  an FSPF backbone
     consisting of 3 links is given in Fig. 3. There is no restriction
     in  adding other links to this network as needed.  The connection
     between BSWs below may in fact form a fully connected mesh.

      _______                                           _______
     |       |                                         |       |
     | AR #1 |_____                               _____| AR #4 |
     |_______|     |                             |     |_______|
                ___|___                       ___|___
               | BSW 1 |---------------------| BSW 4 |
               |_______|                     |_______|
                ___|___                       _______
               | BSW 2 |---------------------| BSW 3 |
               |_______|                     |_______|
      ___ ___      |                             |      _______
     |       |     |                             |     |       |
     | AR #2 |-----                               -----| AR #3 |
     |_______|                                         |_______|

     Fig. 3 Example Network showing FSPF-Backbone Switching

     BSW 1 knows it is connected to BSWs 2 and 4;
     BSW 2 knows it is connected to BSWs 1 and 3;
     BSW 4 knows it is connected to BSWs 1.

Rajagopal, et al.                                              [Page 10]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     An FCIP device  provides a single, logical interface to the FSPF-
     backbone protocol connecting  multiple  BSW neighbors  on the IP-
     network.  From the FSPF-backbone  routing's  point of view,  the
     connection to  each neighbor  on the IP-network  is treated as a
     separate logical FC link.

     In  FCIP,  the native  FC  frames are  first encapsulated  in TCP
     segments which then traverse the IP-based network. The IP network
     provides a  new transport  path for  each  emulated FSPF-backbone

     The IP  network itself may consist of any number  of hops between
     two FCIP devices.  Also, the route taken by the IP packet between
     any two FCIP devices is dictated by normal IP routing.

     A functional and logical diagram of an IP-based FSPF-backbone for
     the example network given in  Fig. 3 is shown in Fig. 4.  In this
     figure, each BSW is logically connected to other BSWs.

      _______                                              _______
     |       |                                            |       |
     | AR #1 |---                                         | AR #4 |
     |_______|   |         ______    ________    ______   |_______|
               __|_ __    |      |  |        |  |      |   ___|___
              | BSW 1 |---| FCIP |--|   IP   |--| FCIP |--| BSW 4 |
              |_______|   |______|  | Network|  |______|  |_______|
                                    |        |
                          ______      |   |    ______
               ______    |      |     |   |   |      |    _______
              | BSW 2|---| FCIP |-----|   |---| FCIP |---| BSW 3 |
              |______|   |______|             |______|   |_______|
      ________   |                                       ___|___
     |        |  |                                      |       |
     |  AR #2 |__|                                      | AR #3 |
     |________|                                         |_______|

     Fig. 4 Example Network showing an IP-based FC Backbone Switching

     The  IP-based  network  has transformed  the FSPF-backbone into a
     fully connected  network.  From the  perspective of each  BSW all
     remote BSWs  therefore appear to be neighbors.  The FSPF-backbone
     routing protocol  computations would  make  the IP-based  network
     topology appear as a fully connected mesh.

     The  FSPF-backbone routing protocol exchanges  between BSWs occur
     transparently to the  FCIP  devices.  Encapsulated FC  frames are
     routed  on  the  IP  network according to  the  normal IP routing

Rajagopal, et al.                                              [Page 11]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     procedures. In this mode, the FSPF-backbone routing protocol lays
     over  the  IP network and has  no  knowledge of the underlying IP
     protocol and IP routing or the underlying technology that carries
     the  IP datagram. This concept is shown in Fig.5

      ________                                             _______
     |  AR #1 |                                           | AR #2 |
     |        |--                                         |       |
     |________|  |        ______    ________    _____     |_______|
               __|___    |      |  |        |  |     |   ____|__
              | BSW 1|---| FCIP |--|   IP   |--|FCIP |--| BSW 2 |
              |      |   |      |  | Network|  |     |  |       |
              |______|   |______|  |________|  |_____|  |_______|
                                   IP Routing
                            FSPF-backbone Routing Plane

     Fig. 5 FC packet routing over IP based backbone network

     Note: IP Network routing may consist of multiple paths

6.2 FC Device

     The protocol encapsulation  and mapping of the FC frame described
     in earlier  sections  applies  equally  to any pair of FC devices
     (e.g.  switch to switch or host to storage subsystem)  wishing to
     tunnel FC frames  across  an IP-based  network.  Any  FC  routing
     protocol  exchanges  may still  occur  transparent  to  the  FCIP
     devices.   It  should  be  noted  that  Fibre  Channel  Primitive
     Sequences and Primitives are not exchanged between FCIP devices.

7. Security Considerations

     Using a wide-area, general purpose network such as an IP internet
     in a position normally occupied by  physical  cabling  introduces
     some security problems  not normally encountered in Fibre Channel
     storage networks. Normal media are typically protected physically
     from  outside  access;  IP  internets  typically  invite  outside

     The  general  effect is that  the security  of the  entire  Fibre
     Channel  internetwork  is  only  as  good  as the security of the
     entire IP internet through which it tunnels.  The following broad
     classes of attacks are possible:

Rajagopal, et al.                                              [Page 12]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     1.  Unauthorized  Fibre  Channel  controllers  can gain access to
         resources through normal Fibre Channel processes.

     2.  Unauthorized  agents can monitor and manipulate Fibre Channel
         traffic flowing  over physical media used  by the IP internet
         and under control of the agent.

     To a large extent, these security risks are  typical of the risks
     facing  any other application  using  an  IP internet.  They  are
     mentioned here  only because Fibre Channel  storage  networks are
     not  normally  suspicious  of the media.  Fibre  Channel  storage
     network administrators will need to  be aware of these additional
     security risks.

     Security protocols  and procedures used in  other IP applications
     may also be used for FC over TCP/IP.

     For Virtual Private Networks,  both authentication and encryption
     are generally desired, because it is important both to (1) assure
     that unauthorized users do not penetrate the virtual private net-
     network  and (2) assure that eavesdroppers on the  network cannot
     read messages sent over the network.

     IPSec provides 3 main facilities: an authentication-only function
     referred  to  as   Authentication    Header   (AH) ,  a  combined
     authentication/encryption function called  Encapsulating Security
     Payload (ESP), and a key exchange function.

     Because both features are generally  desirable,  ESP  may be more
     suitable  than AH.  The key  exchange  function allows for manual
     exchange of  keys  as  well  as an  automated  scheme.  The IPSec
     specifications described in [8], [9], [10], and [11] covers these
     topics.  It  is beyond  the scope  of this  document  to  discuss
     specific uses of the IPSec protocols.

     Note: Use of IPSec protocol is optional.

8. Data Integrity Considerations

     8.1 Loss

     Recovery from data loss due to  IP datagram  loss is provided via
     the  TCP reliable  delivery mechanism.   Note: Due to varying TCP
     timeouts, competing FC and TCP recovery schemes is a possibility.
     This issue is addressed in section 8.4.

     8.2 Corruption

     Data corruption is detected at two different levels: TCP checksum
     and Fibre Channel CRC.  Data corruption detected at the TCP level
     shall be recovered via TCP reliable data recovery mechanisms.

Rajagopal, et al.                                              [Page 13]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     Data  corruption  detected  at the  Fibre  Channel level shall be
     handled within the Fibre Channel end nodes.

     8.3 Timeouts

     Fibre Channel  has two important timeouts to be consider in FCIP.
     These are: ED_TOV, and R_A_TOV.

     ED_TOV determines the  life of an individual  Fibre Channel frame
     in any particular  fabric element.  The effects of  ED_TOV on the
     fabric  as a  whole  are typically  cumulative  since each fabric
     element contains it's own ED_TOV timers for any frame received.

     R_A_TOV determines the life of an individual Fibre Channel  frame
     in the fabric as a whole.  For a fabric,  R_A_TOV implies that no
     particular  frame will  remain in (and thus be  emitted from) the
     fabric after the timer expires.

     TCP  has  the TCP  acknowledgement  timeout.  This is a  variable
     timeout value.
     <TBD: Need to elaborate on TCP timeouts and define how Fibre
     Channel timeouts map to TCP timeouts.>

     8.4 Recovery Mechanisms

     When an FC over TCP/IP  encapsulated frame is transported over an
     IP network, there is a possibility  of the frame getting dropped.
     This can  happen if there is congestion along the path within the
     IP network, if there are no empty buffers available on one of the
     incoming ports, due to  bit errors, etc.  When  this happens, the
     TCP  acknowledgement  will  not be received  by the  source,  and
     normal TCP retry mechanisms will be activated.

     An  issue may  arise during these  recovery mechanisms, since TCP
     timeout is  variable,  and may  exceed  Fibre Channel  FC ED_TOV/
     R_A_TOV timeouts.

     <Placeholder: Need to add details on how to handle this here.>

8.5 IP Fragmentation:

     The Fibre Channel maximum transmittable unit (MTU) is 2148 bytes.
     A  maximum of 60 bytes of TCP header  increases the MTU of the IP
     payload  to 2208  bytes.  It  is  preferable  that  FCIP  packets
     encompass the TCP + FC MTU to avoid fragmentation of FCIP packets.
     The  resulting  packet size exceeds  the MTU of  some IP physical
     layers  (e.g.  Ethernet  MTU = 1518 bytes).  FCIP  devices should
     handle fragmentation and must handle re-assembly of FCIP packets.
     An  FCIP  device  may use  Path MTU  Discovery  (RFC 1191) or  an
     equivalent   mechanism  to  adjust  FCIP  packet  size  to  avoid

Rajagopal, et al.                                              [Page 14]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     fragmentation.   Alternatively,  the MTUs  of all FC nodes may be
     manually set to match the path MTU of the IP network.

     8.6 TCP Segmentation:

     8.7 Ordering:

     In an  IP network, packets are not guaranteed to be  delivered in
     the same  order in  which they were  transmitted.  But,  TCP does
     handle ordering.  This means that frames will be delivered in the
     order that they were  transmitted by the FCIP device.  Note, this
     does  not necessarily  mean that  the frames  will be transmitted
     by the FCIP  device in the order  transmitted from the source end
     FC device, or received by the  destination  end FC device  in the
     order received by the  FCIP device.  This is due to the fact that
     FC frames may be reordered in the FC network/fabric itself.

9. Performance Considerations

     Mapping the IP header DSCP bits to  correspond to Premium Service
     provides a preferred  service at each IP Router Per Hop  Behavior
     (PHB) [6].

     The FCIP  protocol does not necessitate the "cracking"  of the FC
     Frame (except for attaching the correct byte-encoded SOF and EOF)
     nor does it call for any  FC payload processing.  This allows any
     FC traffic to be tunneled across at high throughput rates.

     In the case where  there  is no data link fragmentation,  each FC
     frame has a one-to-one mapping to a TCP segment;  the TCP segment
     has a one-to-one  mapping to an IP dategram;  and the IP datagram
     has  a one-to-one  mapping to  a data link  frame.  This has  the
     tendency to further improve the throughput performance.

     Note:  Class 1  FC  traffic  expects  dedicated  bandwidth.  This
            specification does not address this requirement.

     The  Flow  Control  Protocol  (discussed  in  the  next  section)
     provides the ability to stream gigabit FC data when using a large
     window size.

10. Flow Control and Congestion Management

     <Text to be added>

11. References:

     [1] Bradner, S., "The Internet Standards Process -- Revision 3",
         BCP 9, RFC 2026, October 1996.

Rajagopal, et al.                                              [Page 15]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     [2] Bradner, S., "Key words for use in RFCs to Indicate
         RequirementLevels", BCP 14, RFC 2119, March 1997

     [3] NCITS 321-200x (ANSI) T11/Project 1305-D/Rev 4.7 "Fibre Channel
         Switch-Fabric-2", (FC-SW-2) September 29, 2000 (www.t11.org)

     [4] Fibre Channel Physical and Signaling Interface-3 (FC-PH-3),
         Rev.  9.4, ANSI X3.303-1998

     [5] The Fibre Channel Consultant: A Comprehensive Introduction,
         "Robert W. Kembel", Northwest Learning Associates, 1998

     [6] Nichols, K., Blake, S., Baker, F.  and D. Black, " Definition
         of the Differentiated Services Field (DS Field) in the IPv4 and
         Ipv6 Headers", RFC 2474, December 1998.

     [7] NCITS T11/Project 1238-D/Rev4.7 "Fibre Channel Backbone", (FC-
         BB) June 8, 2000 (www.t11.org)

     [8] Kent, S. and Atkinson, R., "Security Architecture for the
         Internet Protocol", RFC 2401, Nov 1998

     [9] Kent, S. and Atkinson, R., "IP Authentication Header",
         RFC 2402, Nov 1998

     [10] Kent, S. and Atkinson, R., "IP Encapsulating Security
          Payload (ESP)", RFC 2406, Nov 1998

     [11] Maughan, D. et all, "Internet Security Association and Key
          Management Protocol (ISAKMP)", RFC 2408, Nov 1998

     [12] http://www.isi.edu/in-notes/iana/assignments/protocol-numbers

     [13] http://www.t11.org

     [14] Fibre Channel Physical and Signaling Interphace (FC-PH), Rev
          4.3, ANSI X3.230-1994.

     [15] Fibre  Channel  NCITS  321-200x (ANSI) T11/Project  1356-
          D/Rev4.3 " Fibre Channel - Generic  Services 3", June 2000

     [16] ISI, "Transmission Control Protocol", RFC 793, Sep 1981

12. Acknowledgments

Rajagopal, et al.                                              [Page 16]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

13. Authors' Addresses

     Murali Rajagopal
     LightSand Communications, Inc.
     375 Los Coches St.
     Milpitas, CA 95035

     Phone: +1 408 404 3164
     Email: muralir@lightsand.com

     Raj Bhagwat
     LightSand Communications, Inc.
     375 Los Coches St.
     Milpitas, CA 95035

     Phone: +1 408 404 3194
     Email: rajb@lightsand.com

     Elizabeth G. Rodriguez
     Lucent Technologies
     1202 Richardson Drive, Suite 210
     Richardson, TX 75080

     Phone: +1 972 231 0672
     Fax: +1 972 671 5476
     Email: egrodriguez@lucent.com

     Vi Chau
     Gadzoox Networks, Inc.
     16241 Laguna Canyon Road, Suite 100
     Irvine, CA 92618

     Phone: +1 949 789 4639
     Fax: +1 949 453 1271
     Email: vchau@gadzoox.com

     Stuart Berman
     Vixel Corporation
     15245 Alton Parkway Suite 100
     Irvine, CA 92618

     Phone:  +1 949 450 6100
     Fax:      +1 949 753 9500
     Email: sberman@vixel.com

     Steve Wilson
     Brocade Communications Systems, Inc.
     1745 Technology Drive
     San Jose, CA. 95110

Rajagopal, et al.                                              [Page 17]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     Phone: 408-487-8128
     Fax: 408-487-8101
     email: swilson@brocade.com

     Craig W. Carlson
     QLogic Corporation
     6321 Bury Drive
     Eden Prairie, MN 55346

     Phone:  +1 952 932 4064
     Email: craig.carlson@qlogic.com

     Michael E. O'Donnell
     McDATA Corporation
     310 Interlocken Parkway
     Broomfield, Co. 80021

     Phone: +1 303 460 4142
     Fax: +1 303 465 4996
     Email: modonnell@mcdata.com

              ANNEX A: Fibre Channel EOF and SOF Encodings

A.1 Ordered Sets

     On a Fibre  Channel link,  Ordered Sets  (OS) are sent as special
     out-of-band  words constructed  from a combination of the  10-bit
     comma  character  (K28.5)  followed by 3  additional 10-bit  data
     characters. The Ordered Sets defined by Fibre Channel include the
     Frame Delimiters,  Start of Frame  (SOF) and  End of Frame (EOF),
     other  Primitive  Signals  (IDLE, R_RDY, ARB, OPN, CLS, MRK), and
     Primitive Sequences (OLS,NOS, LR, LRR, LIP, LPB, LPE).

     When  Fibre  Chanel frames are  encapsulated in an IP packet, the
     Byte-encoded frame format is used.  The Byte-encoded frame format
     uses 32-bit OS  Code Words to represent valid Fibre channel frame
     delimiters.  This format uses a single-byte OS Code to  represent
     each FC Ordered Set.

     FC Over IP  makes use  of the OS Codes defined  in Annex A of [7]
     for  the frame  delimiters.  SOF  and  EOF  codes  defined in the
     figures (see below) in this Annex are inserted into the FC frame.

     Primitive  Signals and  Primitive  Sequences are  stripped at the
     FCIP  boundary  and not sent  between  FCIP  devices  over the IP

Rajagopal, et al.                                              [Page 18]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     The  frame  delimiters  are  identified  by  their  position.  An
     encapsulated Byte-encoded frame must use the corresponding 32-bit
     OS Code Word as the first and last words in the encapsulated PDU.

     Fibre Channel  frame delimiters  shall be  encoded  in the format
     shown in Table below.

                       Table 1. Frame Delimiter Format

|Wrd|    <31:24>     |    <23:16>     |    <15:08>     |   <07:00>   |
|0  |    OS Code     |                    Reserved                   |

     A.2 Encoded FC Frame Delimiters

     The SOF OS-codes are a single  byte  encoding of the SOF  Ordered
     Set.  The first  word in an  encapsulated  Byte-encoded  FC frame
     shall map the SOF Ordered Set to the corresponding 32-bit OS Code

     The EOF OS-codes are a single  byte  encoding of the EOF  Ordered
     Set. The last word in an encapsulated Byte-encoded FC frame shall
     map the the EOF  Ordered Set to the  corresponding 32-bit OS Code

                    |     OS-Code     | Delimiter Name |
                    |      (hex)      |                |
                    |      0x28       |     SOFf       |
                    |      0x3F       |     SOFc1      |
                    |      0x2F       |     SOFi1      |
                    |      0x37       |     SOFn1      |
                    |      0x3D       |     SOFc2      |
                    |      0x2D       |     SOFi2      |
                    |      0x35       |     SOFn2      |
                    |      0x3E       |     SOFc3      |
                    |      0x2E       |     SOFi3      |
                    |      0x36       |     SOFn3      |
                    |      0x39       |     SOFc4      |
                    |      0x29       |     SOFi4      |

Rajagopal, et al.                                              [Page 19]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

                    |      0x31       |     SOFn4      |
                    |      0x38       |     SOFcf      |
                    |      0x30       |     SOFnf      |
                    |      0x41       |     EOFn       |
                    |      0x42       |     EOFt       |
                    |      0x46       |     EOFdt      |
                    |      0x44       |     EOFrt      |
                    |      0x49       |     EOFni      |
                    |      0x4E       |     EOFdti     |
                    |      0x4F       |     EOFrti     |
                    |      0x50       |     EOFa       |

     ANNEX B: Relationship between FC over IP  (FCIP, FCoverIP) and IP
              over FC (IPFC)

     IPFC describes the encapsulation  of IP packets in  FC frames. It
     is intended to facilitate IP communication over an FC network.

     FC  over IP describes  the  encapsulation  of FC  frames  in  TCP
     segments which  in  turn are encapsulated  inside IP  packets for
     transporting over an IP network. It gives no consideration to the
     type of  FC frame that is being encapsulated.  Therefore,  the FC
     frame may  actually  contain an IP  packet as described in the IP
     over FC specification (RFC 2625). In such a case, the encapsulated
     IP packet would have:

            IP Header
            FC Header
            IP Header
            TCP header

     Note:   The two IP headers  would not be identical to each other.
             One  would  have  information  pertaining  to  the  final
             destination  while  the  other  would  have  information
             pertaining to the FCIP device.

     The two documents  focus on different  objectives.  As  mentioned
     above, implementation of FC over IP will lead to IP encapsulation

Rajagopal, et al.                                              [Page 20]

Internet-Draft         Fibre Channel over TCP/IP            October 2000

     within IP.  While  perhaps  inefficient,  this should not lead to
     issues  with IP  communication.  One  caveat: if a  Fibre Channel
     device is encapsulating IP  packets in an FC  frame (e.g. an IPFC
     device),  and that  device is communicating with a device running
     IP over a non-FC medium, a second IPFC device will need to act as
     a  gateway  between  the  two  networks.  This  scenario  is  not
     specifically addressed by FC over IP.

     There  is nothing  in  either  specification to  prevent a single
     device from  implementing  both  FC-over-IP  and  IP-over-FC, but
     this  is implementation specific, and is beyond the scope of this

     Full Copyright Statement

     Copyright (C) The Internet Society (1999).  All Rights Reserved.

     This document and translations of it may be copied  and furnished
     to  others,  and  derivative  works that comment  on or otherwise
     explain  it or  assist  in its  implementation  may be  prepared,
     copied, published and distributed, in whole  or in  part, without
     restriction of any kind, provided that the above copyright notice
     and this paragraph are included on all such copies and derivative
     works.  However,  this document itself may not be modified in any
     way,  such as by removing  the copyright  notice or references to
     the Internet  Society or other Internet organizations,  except as
     needed  for the purpose of developing Internet standards in which
     case  the  procedures  for  copyrights  defined  in the  Internet
     Standards process  must be  followed, or as required to translate
     it into languages other than English.

     The limited permissions granted above are perpetual and  will not
     be revoked by the Internet Society or its successors or assigns.

     This document and the information contained herein is provided on


     Funding for the RFC Editor function  is currently provided by the
     Internet Society.

     [draft-ietf-ips-fcovertcpip-00.txt]  [This INTERNET DRAFT expires
      on March 31, 2001]

Rajagopal, et al.                                              [Page 21]