Internet Engineering Task Force                         D. Joachimpillai
Internet-Draft                                                   Verizon
Intended status: Informational                             J. Hadi Salim
Expires: June 7, 2015                                  Mojatatu Networks
                                                        December 4, 2014


                          ForCES Inter-FE LFB
                draft-joachimpillai-forces-interfelfb-05

Abstract

   Forwarding and Control Element Separation (ForCES) defines an
   architectural framework and associated protocols to standardize
   information exchange between the control plane and the forwarding
   plane in a ForCES Network Element (ForCES NE).  RFC5812 has defined
   the ForCES Model which provides a formal way to represent the
   capabilities, state, and configuration of forwarding elements(FEs)
   within the context of the ForCES protocol.  More specifically, the
   model describes the logical functions that are present in an FE, what
   capabilities these functions support, and how these functions are or
   can be interconnected.  The control elements (CEs) can control the
   FEs using the ForCES model definition.

   The ForCES WG charter has been extended to allow the LFB topology to
   be across FEs.  This documents describes a non-intrusive way to
   extend the LFB topology across FEs.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on June 7, 2015.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the



Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 1]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.


Table of Contents

   1.  Terminology and Conventions  . . . . . . . . . . . . . . . . .  3
     1.1.  Requirements Language  . . . . . . . . . . . . . . . . . .  3
     1.2.  Definitions  . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   3.  Problem Scope And Use Cases  . . . . . . . . . . . . . . . . .  4
     3.1.  Basic Router . . . . . . . . . . . . . . . . . . . . . . .  4
       3.1.1.  Distributing The LFB Topology  . . . . . . . . . . . .  6
     3.2.  Arbitray Network Function  . . . . . . . . . . . . . . . .  7
       3.2.1.  Distributing The Arbitray Network Function . . . . . .  8
   4.  Proposal Overview  . . . . . . . . . . . . . . . . . . . . . .  9
     4.1.  Inserting The Inter-FE LFB . . . . . . . . . . . . . . . .  9
   5.  Inter-FE connectivity  . . . . . . . . . . . . . . . . . . . . 11
     5.1.  Inter-FE Ethernet connectivity . . . . . . . . . . . . . . 13
       5.1.1.  Inter-FE Ethernet Connectivity Issues  . . . . . . . . 15
   6.  Detailed Description of the Ethernet inter-FE LFB  . . . . . . 15
     6.1.  Data Handling  . . . . . . . . . . . . . . . . . . . . . . 16
       6.1.1.  Egress Processing  . . . . . . . . . . . . . . . . . . 16
       6.1.2.  Ingress Processing . . . . . . . . . . . . . . . . . . 18
     6.2.  Metadata . . . . . . . . . . . . . . . . . . . . . . . . . 19
     6.3.  Components . . . . . . . . . . . . . . . . . . . . . . . . 19
     6.4.  Inter-FE LFB XML Model . . . . . . . . . . . . . . . . . . 19
   7.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21
   8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 21
   9.  Security Considerations  . . . . . . . . . . . . . . . . . . . 22
   10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
     10.1. Normative References . . . . . . . . . . . . . . . . . . . 22
     10.2. Informative References . . . . . . . . . . . . . . . . . . 22
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22








Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 2]


Internet-Draft             ForCES Inter-FE LFB             December 2014


1.  Terminology and Conventions

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

1.2.  Definitions

   This document reiterates the terminology defined in several ForCES
   documents [RFC3746], [RFC5810], [RFC5811], and [RFC5812] for the sake
   of contextual clarity.

      Control Engine (CE)

      Forwarding Engine (FE)

      FE Model

      LFB (Logical Functional Block) Class (or type)

      LFB Instance

      LFB Model

      LFB Metadata

      ForCES Component

      LFB Component

      ForCES Protocol Layer (ForCES PL)

      ForCES Protocol Transport Mapping Layer (ForCES TML)


2.  Introduction

   In the ForCES architecture, a packet service can be modelled by
   composing a graph of one or more LFB instances.  The reader is
   refered to the details in the ForCES Model [RFC5812].

   The FEObject LFB capabilities in the ForCES Model [RFC5812] define
   component ModifiableLFBTopology which, when advertised with by the
   FE, implies that the advertising FE is capable of allowing creation
   and modification the LFB graph by the control plane.  Details on how
   a graph of LFB class instances can be created can be derived by the



Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 3]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   control plane by looking at the FE's FEObject LFB table
   SupportedLFBs.  That table contains information about each LFB class
   that the FE supports.  For each LFB class supported, details are
   provided on how that LFB class may be connected to other LFB classes.
   The SupportedLFBs table describes which LFB class a specified LFB
   class may succeed or precede in an LFB class instance topology.  Each
   link connecting two LFB class instances is described in the
   LFBLinkType dataTypeDef and has sufficient details to identify
   precisely the end points of a link of a service graph.

   The CE may therefore create a packet service by describing an LFB
   instance graph connections; achieved by updating the FEOBject
   LFBTopology table.

   Often there are requirements for the packet service graph to cross FE
   boundaries.  This could be from a desire to scale the service or need
   to interact with LFBs which reside in a separate FE (eg lookaside
   interface to a shared TCAM, an interconnected chip, or as coarse
   grained functionality as an external NAT FE box being part of the
   service graph etc).

   Given that the ForCES inter-LFB architecture calls out for ability to
   pass metadata between LFBs, it is imperative therefore to define
   mechanisms to extend that existing feature and allow passing the
   metadata between LFBs across FEs.

   The new ForCES charter allows the LFB links in a topology to be
   across multiple FE (inter-FE connectivity).

   This document describes extending the LFB topology across FEs i.e
   inter-FE connectivity without needing any changes to the ForCES
   definitions.  It focusses on using Ethernet as the interconnection as
   a starting point while leaving room for other protocols (such as
   directly on top of IP, UDP, VXLAN, etc) for different documents.


3.  Problem Scope And Use Cases

   The scope of this document is to solve the challenge of passing
   ForCES defined metadata and exceptions across FEs (be they physical
   or virtual).  To illustrate the problem scope we present two use
   cases where we start with a single FE running all the functionality
   then split it into multiple FEs.

3.1.  Basic Router

   A sample LFB topology Figure 1 demonstrates a service graph for
   delivering basic IPV4 forwarding service within one FE.  Note:



Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 4]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   although the diagram shows LFB classes connecting in the graph in
   reality it is a graph of LFB class instances that are inter-
   connected.

   Since the illustration is meant only as an exercise to showcase how
   data and metadata is sent down or upstream on a graph of LFBs, it
   abstracts out any ports in both directions and talks about a generic
   ingress and egress LFB.  Again, for illustration purposes, the
   diagram does not show expection or error paths.  Also left out are
   details on Reverse Path Filtering, ECMP, multicast handling etc.  In
   other words, this is not meant to be a complete description of an
   IPV4 forwarding application; for a more complete example, please
   refer to the LFBlib document [RFC6956] .

   The output of the ingress LFB(s) coming into the IPv4 Validator LFB
   will have both the IPV4 packets and, depending on the implementation,
   a variety of ingress metadata such as offsets into the different
   headers, any classification metadata, physical and virtual ports
   encountered, tunnelling information etc.  These metadata are lumped
   together as "ingress metadata".

   Once the IPV4 validator vets the packet (example ensures that no
   expired TTL etc), it feeds the packet and inherited metadata into the
   IPV4 unicast LPM LFB.



                      +----+
                      |    |
           IPV4 pkt   |    | IPV4 pkt     +-----+             +---+
       +------------->|    |------------->|     |             |   |
       |  + ingress   |    | + ingress    |IPv4 |   IPV4 pkt  |   |
       |   metadata   |    | metadata     |Ucast|------------>|   |--+
       |              +----+              |LPM  |  + ingress  |   |  |
     +-+-+             IPv4               +-----+  + NHinfo   +---+  |
     |   |             Validator                   metadata   IPv4   |
     |   |             LFB                                    NextHop|
     |   |                                                     LFB   |
     |   |                                                           |
     |   |                                                  IPV4 pkt
     |   |                                               + {ingress
     +---+                                                  + NHdetails}
     Ingress                                                metadata |
      LFB                                +-------+                   |
                                         |Egress |                   |
                                      <--|LFB    |<------------------+
                                         +-------+




Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 5]


Internet-Draft             ForCES Inter-FE LFB             December 2014


             Figure 1: Basic IPV4 packet service LFB topology

   The IPV4 unicast LPM LFB does a longest prefix match lookup on the
   IPV4 FIB using the destination IP address as a search key.  The
   result is typically a next hop selector which is passed downstream as
   metadata.

   The Nexthop LFB receives the IPv4 packet with an associated next hop
   info metadata.  The NextHop LFB consumes the NH info metadata and
   derives from it a table index to look up the next hop table in order
   to find the appropriate egress information.  The lookup result is
   used to build the next hop details to be used downstream on the
   egress.  This information may include any source and destination
   information (MAC address to use, if ethernet;) as well egress ports.
   [Note: It is also at this LFB where typically the forwarding TTL
   decrement and IP checksum recalculation occurs.]

   The details of the egress LFB are considered out of scope for this
   discussion.  Suffice it is to say that somewhere within or beyond the
   Egress LFB the IPV4 packet will be sent out a port (ethernet, virtual
   or physical etc).

3.1.1.  Distributing The LFB Topology

   Figure 2 demonstrates one way the router LFB topology in Figure 1 may
   be split across two FEs (eg two ASICs).  Figure 2 shows the LFB
   topology split across FEs after the IPV4 unicast LPM LFB.
























Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 6]


Internet-Draft             ForCES Inter-FE LFB             December 2014


      FE1
    +-------------------------------------------------------------+
    |                            +----+                           |
    | +----------+               |    |                           |
    | | Ingress  |    IPV4 pkt   |    | IPV4 pkt     +-----+      |
    | |  LFB     |+------------->|    |------------->|     |      |
    | |          |  + ingress    |    | + ingress    |IPv4 |      |
    | +----------+    metadata   |    |   metadata   |Ucast|      |
    |      ^                     +----+              |LPM  |      |
    |      |                      IPv4               +-----+      |
    |      |                     Validator              |         |
    |                             LFB                   |         |
    +---------------------------------------------------|---------+
                                                        |
                                                   IPv4 packet +
                                                 {ingress + NHinfo}
                                                     metadata
      FE2                                               |
    +---------------------------------------------------|---------+
    |                                                   V         |
    |             +--------+                       +--------+     |
    |             | Egress |     IPV4 packet       | IPV4   |     |
    |       <-----|  LFB   |<----------------------|NextHop |     |
    |             |        |{ingress + NHdetails}  | LFB    |     |
    |             +--------+      metadata         +--------+     |
    +-------------------------------------------------------------+

             Figure 2: Split IPV4 packet service LFB topology

   Some proprietary inter-connect (example Broadcom Higig over XAUI
   (XXX: ref needed)) maybe exist to carry both the IPV4 packet and the
   related metadata between the IPV4 Unicast LFB and IPV4 NextHop LFB
   across the two FEs.

   The purpose of the inter-FE LFB is to define standard mechanisms for
   interconnecting FEs and for that reason we are not going to touch
   anymore on proprietary chip-chip interconnects other than state the
   fact they exist and that it is feasible to have translation to and
   from proprietary approaches.  The focus is going to stick to FE-FE
   interconnect where the FE could be physical or virtual and the
   interconnecting technology runs a standard protocol such as ethernet,
   IP or other protocols on top of IP.

3.2.  Arbitray Network Function

   In this section we show an example of an arbitrary network function
   which is more coarse grained in terms of functionality.  Each Network
   function may constitute more than one LFB.



Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 7]


Internet-Draft             ForCES Inter-FE LFB             December 2014


      FE1
    +-------------------------------------------------------------+
    |                            +----+                           |
    | +----------+               |    |                           |
    | | Network  |   pkt         |NF2 |    pkt       +-----+      |
    | | Function |+------------->|    |------------->|     |      |
    | |    1     |  + NF1        |    | + NF1/2      |NF3  |      |
    | +----------+    metadata   |    |   metadata   |     |      |
    |      ^                     +----+              |     |      |
    |      |                                         +-----+      |
    |      |                                            |         |
    |                                                   |         |
    +---------------------------------------------------|---------+
                                                        V

         Figure 3: A Network Function Service Chain within one FE

   The setup in Figure 3 is atypical of most packet processing boxes
   where we have functions like DPI, NAT, Routing, etc connected in such
   a topology to deliver a packet processing service to flows.

3.2.1.  Distributing The Arbitray Network Function

   The setup in Figure 3 can be split out across 3 FEs instead as
   demonstrated in Figure 4.  This could be motivated by scale out
   reasons or because different vendors provide different functionality
   which is plugged-in to provide such functionality.  The end result is
   to have the same packet service delivered to the different flows
   passing through.



      FE1                        FE2
      +----------+               +----+               FE3
      | Network  |   pkt         |NF2 |    pkt       +-----+
      | Function |+------------->|    |------------->|     |
      |    1     |  + NF1        |    | + NF1/2      |NF3  |
      +----------+    metadata   |    |   metadata   |     |
           ^                     +----+              |     |
           |                                         +-----+
                                                        |
                                                        V

       Figure 4: A Network Function Service Chain Distributed Across
                               Multiple FEs






Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 8]


Internet-Draft             ForCES Inter-FE LFB             December 2014


4.  Proposal Overview

   We address the inter-FE connectivity by proposing an inter-FE LFB.
   Using an LFB implies no change to the basic ForCES architecture in
   the form of the core LFBs (FE Protocol or Object LFBs).  This design
   choice was made after considering an alternative approach that would
   have required changes to both the FE Object capabilities
   (SupportedLFBs) as well LFBTopology component to describe the
   inter-FE connectivity capabilities as well as runtime topology of the
   LFB instances.

4.1.  Inserting The Inter-FE LFB

   The distributed LFB topology described in Figure 2 is re-illustrated
   in Figure 5 to show the topology location where the inter-FE LFB
   would fit in.



































Joachimpillai & Hadi Salim  Expires June 7, 2015                [Page 9]


Internet-Draft             ForCES Inter-FE LFB             December 2014


      FE1
    +-------------------------------------------------------------+
    | +----------+               +----+                           |
    | | Ingress  |    IPV4 pkt   |    | IPV4 pkt     +-----+      |
    | |  LFB     |+------------->|    |------------->|     |      |
    | |          |  + ingress    |    | + ingress    |IPv4 |      |
    | +----------+    metadata   |    |   metadata   |Ucast|      |
    |      ^                     +----+              |LPM  |      |
    |      |                      IPv4               +-----+      |
    |      |                     Validator              |         |
    |      |                      LFB                   |         |
    |      |                                  IPv4 pkt + metadata |
    |      |                        {ingress + NHinfo + InterFEid}|
    |      |                                            |         |
    |                                              +----V----+    |
    |                                              | InterFE |    |
    |                                              |   LFB   |    |
    |                                              +---------+    |
    +---------------------------------------------------|---------+
                                                        |
                                         IPv4 packet and metadata
                                {ingress + NHinfo + Inter FE info}
     FE2                                                |
    +---------------------------------------------------|---------+
    |                                              +----V----+    |
    |                                              | InterFE |    |
    |                                              |   LFB   |    |
    |                                              +---------+    |
    |                                                   |         |
    |                                         IPv4 pkt + metadata |
    |                                          {ingress + NHinfo} |
    |                                                   |         |
    |             +--------+                       +----V---+     |
    |             | Egress |     IPV4 packet       | IPV4   |     |
    |       <-----|  LFB   |<----------------------|NextHop |     |
    |             |        |{ingress + NHdetails}  | LFB    |     |
    |             +--------+      metadata         +--------+     |
    +-------------------------------------------------------------+

         Figure 5: Split IPV4 forwarding service with Inter-FE LFB

   As can be observed in Figure 5, the same details passed between IPV4
   unicast LPM LFB and the IPV4 NH LFB are passed to the egress side of
   the Inter-FE LFB.  In addition an index for the inter-FE LFB
   (interFEid) is passed as metadata.

   The egress of the inter-FE LFB uses the received Inter-FE index
   (InterFEid metadata) to select details for encapsulation towards the



Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 10]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   neighboring FE.  These details will include what the source and
   destination FEID to be communicated to the neighboring FE.  In
   addition the original metadata, any exception IDs may be passed along
   with the original IPV4 packet.

   On the ingress side of the inter-FE LFB the received packet and its
   associated details are used to decide the graph continuation i.e
   which FE instance is to be passed the packet and what of the original
   metadata and exception IDs.  In the illustrated case above, an IPV4
   Nexthop LFB instance metadata is passed.

   The ingress side of the inter-FE LFB consumes some of the information
   passed (eg the destination FEID) and passes on the IPV4 packet
   alongside with the ingress + NHinfo metadata to the IPV4 NextHop LFB
   as was done earlier in both Figure 1 and Figure 2.


5.  Inter-FE connectivity

   We describe the generic encapsulation format in Figure 6 extended
   from the ForCES redirect packet format.  We intend for this
   encapsulation to be a generic guideline of the different needed
   fields to be made available by any used transport for inter-FE LFB
   connectivity.  We expect that for any transport mechanism used, that
   a description of how the different fields will be encapsulated to be
   correlated to the information described in Figure 6.  The goal of
   this document is to provide ethernet encapsulation, and to that end
   in Section 5.1 we illustrate how we use the guidelines provided in
   this section to describe the fit for inter-FE LFB interfacing over
   ethernet.





















Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 11]


Internet-Draft             ForCES Inter-FE LFB             December 2014


            +-- Main ForCES header
            |   |
            |   +---- msg type = REDIRECT
            |   +---- Destination FEID
            |   +---- Source FEID
            |   +---- NEID (first word of Correlator)
            |
            +-- T = ExceptionID-TLV
               |  |
               |  +-- +-Exception Data ILV (I = exceptionID , L= length)
               |  |   |  |
               |  |   |  +----- V= Metadata value
               |  .   |
               |  .   |
               |  .   +-Exception Data ILV
               .
               |
               +- T = METADATA-TLV
               |  |
               |  +-- +-Meta Data ILV (I = metaid, L= length)
               |  |   |  |
               |  |   |  +----- V= Metadata value
               |  .   |
               |  .   |
               |  .   +-Meta Data ILV
               .
               +- T = REDIRECTDATA-TLV
                  |
                  +--  Redirected packet Data

                    Figure 6: Packet format suggestion

   o  The ForCES main header as described in RFC5810 is used as a fixed
      header to describe the Inter-FE encapsulation.

      *  The Source ID field is mapped to the originating FE and the
         destination ID is mapped to the destination FEID.

      *  The first 32 bits of the correlator field are used to carry the
         NEID.  The 32-bit NEID defaults to 0.

   o  The ExceptionID TLV carries one or more exception IDs within ILVs.
      The I in the ILV carries a globally defined exceptionID as per-
      ForCES specification defined by IANA.  This TLV is new to ForCES
      and sits in the global ForCES TLV namespace.

   o  The METADATA and REDIRECTDATA TLV encapsulations are taken
      directly from [RFC5810] section 7.9.



Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 12]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   It is expected that a variety of transport encapsulations would be
   applicable to carry the format described in Figure 6.  In such a
   case, a description of a mapping to intepret the inter-FE details and
   translate into proprietary or legacy formatting would need to be
   defined.  For any mapping towards these definitions a different
   document to describe the mapping, one per transport, is expected to
   be defined.

5.1.  Inter-FE Ethernet connectivity

   In this specific document, we describe a format that is to be used
   over Ethernet.

   The following describes the mapping from Figure 6 to ethernet wire
   encapsulation illustrated in Figure 7.

   o  When an NE tag is needed, a VLAN tag will be used.  Note: that the
      NEID as per Figure 6 is described as being 32 bits while a vlan
      tag is 12 bits.  It is however thought to be sufficient to use 12
      bits within the scope of a LAN NE cluster.

   o  An ethernet type will be used to imply that a wire format is
      carrying an inter-FE LFB packet.  The ethernet type will be
      requested from the appropriate IEEE Standards Association.  We
      feel that given that a ForCES NE may end up being owned by a
      single organization, the CE could program all participating FEs
      via the inter-FE LFB (described in this document) to recognize a
      private ethernet type used for inter-LFB traffic.

   o  The destination FEID will be mapped to the destination MAC address
      of the target FEID.

   o  The source FEID will be mapped to the source MAC address of the
      originating FEID.

   o  In this version of the specification, we only focus on data and
      metadata.  Therefore we are not going to describe how to carry the
      ExceptionID information (future versions may).  We are also not
      going to use METADATA-TLV or REDIRECTDATA-TLV in order to save
      shave off some overhead bytes.  Figure 7 describes the payload
      when










Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 13]


Internet-Draft             ForCES Inter-FE LFB             December 2014


       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Outer Destination MAC Address  (Destination FEID)            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Outer Destination MAC Address | Outer Source MAC Address      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Outer Source MAC Address  (Source FEID)                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Optional 802.1Q info (NEID)   | Inter-FE ethertype            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Metadata length               | TLV encoded Metadata          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | TLV encoded Metadata ~~~..............~~                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Original Ethernet payload ~~................~~                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


                    Figure 7: Packet format suggestion

   An outer Ethernet header is introduced to carry the information on
   Destination FEID, Source FEID and optional NEID.

   o  The Outer Destination MAC Address carries the Destination FEID
      identification.

   o  Outer Source MAC Address carries the Source FEID identification.

   o  When an NEID is needed, an optional 802.1Q is carried with 12-bit
      VLANid representing the NEID.

   o  The ethernet type is used to identify the frame as inter-FE LFB
      type.

   o  The 16-bit metadata length is used to described the total encoded
      metadata length (including the 16 bits used to encode the metadata
      length).

   o  One or more TLV encoded metadatum follows the metadata length
      field.  The TLV type identifies the Metadata id.  We recognize
      that this restricts the metadata id to 16 bits instead of ForCES
      define space of 32 bits.  However, at the time of publication we
      believe this is sufficient to carry all the info we need and this
      would save us 4 bytes per Metadata transferred.  XXX: If there is
      objection from the we could convert this to an ILV.





Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 14]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   o  The original ethernet payload is appended at the end.

5.1.1.  Inter-FE Ethernet Connectivity Issues

   There are several issues that may arise due to using direct ethernet
   encapsulation.

   o  Because we are adding data to existing ethernet frames, MTU issues
      may arise.  We recommend:

      *  To use large MTUs when possible (example with jumbo frames).

      *  Limit the amount of metadata that could be transmitted; our
         definition allows for filtering of which metadata is to be
         encapsulated in the frame.  We recommend complementing this by
         setting the egress port MTU to allow space for maximum size of
         the metadata total size you wish to allow between FEs.  MTU
         setting can be achieved by configuration or ForCES control of
         the port LFB.  An extra 64 bytes reserved on the MTU will
         account for 5 32-bit metadatum or 3 64-bit metadatum.  In
         essence, the control plane making a decision for the MTU size
         of the egress port is implicitly deciding how much metadata
         will be allowed.

   o  The frame may be dropped if there is congestion on the receiving
      FE side.  One approach to mitigate this issue is to make sure that
      inter-FE LFB frames receive the highest priority treatment when
      scheduled on the wire.  Typically protocols that tunnel in the
      middle box do not care and depend on the packet originator to
      resend if the originator cares about reliability.  We do not
      expect to be any different.


6.  Detailed Description of the Ethernet inter-FE LFB

   The ethernet inter-FE LFB has two LFB input ports and three LFB
   output ports.














Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 15]


Internet-Draft             ForCES Inter-FE LFB             December 2014


                   +-----------------+
    Inter-FE LFB   |                 |
    Encapsulated   |             OUT2+--> decapsulated Packet + metadata
    -------------->|IN2              |
    Packet         |                 |
                   |                 |
    raw Packet +   |             OUT1+--> encapsulated Packet
    -------------->|IN1              |
    Metadata       |                 |
                   |    EXCEPTIONOUT +--> Errorid, packet + metadata
                   |                 |
                   +-----------------+


                          Figure 8: Inter-FE LFB

6.1.  Data Handling

   The Inter-FE LFB will be positioned at the egress of an FE at the
   source FE.  In such a case an Inter-FE LFB instance receives via port
   IN1, raw packet and metadata IDs from the preceeding LFB instance.
   The InterFEid metadatum MAY be present on the incoming raw data.  The
   processed encapsulated packet will go out on either LFB port OUT1 to
   a downstream LFB or EXCEPTIONOUT port in the case of a failure.

   The Inter-FE LFB will be positioned at the ingress of a receiving FE.
   In such a case an Inter-FE LFB receives, via port IN2, an
   encapsulated packet.  Successful processing of the packet will result
   in a raw packet with associated metadata IDs going downstream to an
   LFB connected on OUT2.  On failure the data is sent out EXCEPTIONOUT.

   Depending on the implementation, the Inter-FE LFB may use the
   InterFEid metadatum on egress of an FE to lookup the NextFE table.
   The interFEid in such a case will be generated by an upstream LFB
   instance (i.e one preceeding the Inter-FE LFB).  The output result
   constitutes a matched table row which has the InterFEinfo details
   i.e. the tuple {NEID,Destination FEID,Source FEID, inter FE type,
   metafilters}.  The metafilters lists define which Metadatum are to be
   passed to the neighboring FE.  XXX: alternative implementations may
   preprogram the Inter-FE LFB details to be used (and therefore do not
   need presence of InterFEid metadatum).  We look at the InterFEid
   metadatum as lowest common denominator for inter-operability but
   leave it up to the implementation to make the call.

6.1.1.  Egress Processing

   The egress Inter-FE LFB will receive an ethernet frame and
   accompanying metadatum (including optionally the InterFEid



Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 16]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   metadatum).  The ethernet frame may be 802.1Q tagged.

   The InterFEid may be used to lookup NextFE table.  If lookup is
   successful, the inter-FE LFB will perform the following actions:

   o  create the outer ethernet header which is a duplicate of the
      incoming frame's ethernet header.  The outer ethernet header may
      have an optional 802.1q header.

   o  If the NEID field is present (not 0) and the original header had a
      vlan tag, replace the vlan tag on the outer header with the value
      from the matched NEID field.  If the NEID field is present (not 0)
      and the original header did not have a vlan tag, create one that
      matches the NEID field and appropriately add it to the outer
      header.  If the NEID field is absent or 0, do nothing.

   o  Set the Destination MAC address of the outer header with value
      found in the DSTFE field.

   o  If the optional SRCFE is present, set the Source MAC address of
      the outer header with value found in the SRCFE field.  If SRCFE is
      absent then the inner source MAC address is used (at this point
      already copied).

   o  If the optional IFETYPE is present, set the outer ethernet type to
      the value found in IFETYPE.  If IFETYPE is absent then the
      standard ethernet type is used (XXX: to be requested from IEEE).

   o  Walk the passed metadatum, apply against the MetaFilterList and
      encapsulate each allowed metadatum in a TLV.  Use the Metaid as
      the "type" field in the TLV header.  The TLV should be aligned to
      32 bits.  This means you may need to add padding of zeroes to
      ensure alignment.

   o  Update the Metadata length to the sum of each TLV's space + 2
      bytes (for the Metadata length field 16 bit space).

   The resulting packet is sent to the LFB instance connected to the
   OUT1; typicall a port LFB.

   In the case of a failed lookup or a zero-value InterFEid, or absence
   of InterFEid, the default inter-FE LFB processing will:

   o  create the outer ethernet header which is a duplicate of the
      incoming frame's ethernet header.  The outer ethernet header may
      have an optional 802.1q header.





Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 17]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   o  If the DefaultNextFE NEID component is not 0 and the original
      header had a vlan tag, replace the vlan tag on the outer header
      with the value from the matched DefaultNextFE NEID field.  If the
      DefaultNextFE NEID field is present (not 0) and the original
      header did not have a vlan tag, create one that matches the
      DefaultNextFE NEID field and appropriately add it to the outer
      header.  If the DefaultNextFE NEID field is absent or 0, do
      nothing.

   o  Set the Destination MAC address of the outer header with value
      found in the DefaultNextFE DSTFE component.

   o  If the DefaultNextFE SRCFE is non-zero, set the Source MAC address
      of the outer header with value found in the DefaultNextFE SRCFE
      field.  If DefaultNextFE SRCFE is zero then the inner source MAC
      address is used (at this point already copied).

   o  If the optional DefaultNextFE IFETYPE is non-zero, set the outer
      ethernet type to the value found in DefaultNextFE IFETYPE.  If
      DefaultNextFE IFETYPE is zero then the standard ethernet type is
      used (XXX: to be requested from IEEE).

   o  Walk all the passed metadatum, apply against the DefaultNextFE
      MetaFilterList and encapsulate each allowed metadatum in a TLV.
      Use the Metaid as the "type" field in the TLV header.  The TLV
      should be aligned to 32 bits.  This means you may need to add
      padding of zeroes to ensure alignment.

   o  Update the Metadata length to the sum of each TLV's space + 2
      bytes (for the Metadata length field 16 bit space).

   The resulting packet is sent to the LFB instance connected to the
   OUT1 LFB port (typicall a Port LFB).

6.1.2.  Ingress Processing

   An inter-FE LFB packet is recognized by looking at the etherype.

   o  The inter-FE LFB instance looks at the metadata length field and
      walks the packet data extracting from the TLVs the metadata
      values.  For each metadata extracted, the corresponding
      implementation metadatum field is set.

   o  Upon completion of processing all the metadata, the inter-FE LFB
      instance resets the header to point to the original (inner)
      ethernet header i.e skips the metadata information.  At this point
      the the original ethernet frame that was passed to the egress
      Inter-FE LFB at the source FE is reconstructed.  This data is then



Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 18]


Internet-Draft             ForCES Inter-FE LFB             December 2014


      passed alongside the reconstructed metadata downstream to the next
      programmed LFB instance.

   In the case of processing failure of either ingress or egress
   positioning of the LFB, the packet and metadata are sent out the
   EXCEPTIONOUT LFB port with proper error id (XXX: More description to
   be added).

6.2.  Metadata

   A single (to be define from IANA space) metadatum, InterFEid, is
   defined.

6.3.  Components

   There are two LFB component populated by the CE.  Each components
   information is of type IFEInfo.  The IFEInfo datatype constitutes:
   NEID, optional IFETYPE, Destination FEID(DSTFE), optional Source FEID
   (SRCFE), array of allowed Metaids (MetaFilterList).

   The CE optionally programs LFB instances in a service graph that
   require inter-FE connectivity with InterFEid values to correspond to
   the inter-FE LFB NextFE table entries to use.

   The first component is an array known as the NextFE table.  The array
   rows are made up of IFEInfo structure.  The table is looked up by a
   32 bit index passed from an upstream LFB class instance in the form
   of InterFEid metadatum.

   The second component(ID 2) is an IFEInfo structure known as
   DefaultNextFE.  The DefaultNextFE component carries similar
   information to any one table row in the NextFE table and is used as
   the default source of Inter-FE encapsulation information if there is
   failure to use any entry in the NextFE table.

6.4.  Inter-FE LFB XML Model

   XXX: add ports and metadata definition

   XXX: add stats as in implementation (including error stats)


   <LFBLibrary xmlns="urn:ietf:params:xml:ns:forces:lfbmodel:1.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          provides="IFE">
     <dataTypeDefs>

        <dataTypeDef>



Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 19]


Internet-Draft             ForCES Inter-FE LFB             December 2014


           <name>IFEInfo</name>
           <synopsis>Describing IFE table row Information</synopsis>
           <struct>
              <component componentID="1">
                <name>NEID</name>
                <synopsis>
                     The VLAN Id 12 bits part of the 802.1q TCI field.
                </synopsis>
                <typeRef>uint16</typeRef>
              </component>
              <component componentID="2">
                <name>IFETYPE</name>
                <synopsis>
                    the ethernet type to be used for outgoing IFE frame
                </synopsis>
                <optional/>
                <typeRef>uint16</typeRef>
              </component>
              <component componentID="3">
                <name>DSTFE</name>
                <synopsis>
                        the destination MAC address of destination FE
                </synopsis>
                <typeRef>byte[6]</typeRef>
              </component>
              <component componentID="4">
                <name>SRCFE</name>
                <synopsis>
                        the source MAC address used for the source FE
                </synopsis>
                <optional/>
                <typeRef>byte[6]</typeRef>
              </component>
              <component componentID="5">
                <name>MetaFilterList</name>
                <synopsis>
                        the metadata filter table
                </synopsis>
                <array type="variable-size">
                  <typeRef>uint32</typeRef>
                </array>
               </component>
           </struct>
        </dataTypeDef>

     </dataTypeDefs>

     <LFBClassDefs>



Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 20]


Internet-Draft             ForCES Inter-FE LFB             December 2014


       <LFBClassDef LFBClassID="6612">
         <name>IFE</name>
         <synopsis>
            This LFB describes IFE connectivity parametrization
         </synopsis>
         <version>1.0</version>

         <components>

            <component componentID="1" access="read-write">
               <name>NextFE</name>
               <synopsis>
                  the table of all InterFE relations
               </synopsis>
               <array type="variable-size">
                  <typeRef>IFEInfo</typeRef>
               </array>
            </component>

            <component componentID="2" access="read-write">
               <name>DefaultNextFE</name>
               <synopsis>
                 the Default Next FE info. Used when we are not able
                 to determine what to use from NextFE
               </synopsis>
               <typeRef>IFEInfo</typeRef>
            </component>

        </components>

       </LFBClassDef>
      </LFBClassDefs>
   </LFBLibrary>

                        Figure 9: Inter-FE LFB XML


7.  Acknowledgements

   The authors would like to thank Joel Halpern and Dave Hood for the
   stimulating discussions.


8.  IANA Considerations

   This memo includes two IANA requests within the registry
   https://www.iana.org/assignments/forces




Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 21]


Internet-Draft             ForCES Inter-FE LFB             December 2014


   The first request is for the sub-registry "Logical Functional Block
   (LFB) Class Names and Class Identifiers" to request for the
   reservation of LFB class name IFE with LFB classid 6112 with version
   1.0.

   The second request is for the sub-registry "Metadata ID" to request
   for the InterFEid metadata the value 0x00000010.


9.  Security Considerations

   TBD


10.  References

10.1.  Normative References

   [RFC3746]  Yang, L., Dantu, R., Anderson, T., and R. Gopal,
              "Forwarding and Control Element Separation (ForCES)
              Framework", RFC 3746, April 2004.

   [RFC5810]  Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang,
              W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and
              Control Element Separation (ForCES) Protocol
              Specification", RFC 5810, March 2010.

   [RFC5811]  Hadi Salim, J. and K. Ogawa, "SCTP-Based Transport Mapping
              Layer (TML) for the Forwarding and Control Element
              Separation (ForCES) Protocol", RFC 5811, March 2010.

   [RFC5812]  Halpern, J. and J. Hadi Salim, "Forwarding and Control
              Element Separation (ForCES) Forwarding Element Model",
              RFC 5812, March 2010.

10.2.  Informative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.












Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 22]


Internet-Draft             ForCES Inter-FE LFB             December 2014


Authors' Addresses

   Damascane M. Joachimpillai
   Verizon
   60 Sylvan Rd
   Waltham, Mass.  02451
   USA

   Email: damascene.joachimpillai@verizon.com


   Jamal Hadi Salim
   Mojatatu Networks
   Suite 400, 303 Moodie Dr.
   Ottawa, Ontario  K2H 9R4
   Canada

   Email: hadi@mojatatu.com

































Joachimpillai & Hadi Salim  Expires June 7, 2015               [Page 23]