Internet Engineering Task Force                            E. Haleplidis
Internet-Draft                                      University of Patras
Intended status: Informational                                  K. Ogawa
Expires: February 8, 2010                                NTT Corporation
                                                                 X. Wang
                                           Huawei Technologies Co., Ltd.
                                                                   C. Li
                                           Zhejiang Gongshang University
                                                          August 7, 2009


                     ForCES Interoperability Draft
                 draft-ietf-forces-interoperability-03

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on February 8, 2010.

Copyright Notice

   Copyright (c) 2009 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents in effect on the date of
   publication of this document (http://trustee.ietf.org/license-info).
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.




Haleplidis, et al.      Expires February 8, 2010                [Page 1]


Internet-Draft        ForCES Interoperability Draft          August 2009


Abstract

   This document describes the details of the interoperability test of
   the Forward and Control Element Separation (ForCES) protocol that
   took place in the University of Patras in Rio, Greece, 15 and 16 July
   2009.  This informational draft provided necessary information, for
   all parties who wish to participate in the interoperability test.

   This update also includes the results of the test.


Table of Contents

   1.  Terminology and Conventions  . . . . . . . . . . . . . . . . .  4
     1.1.  Requirements Language  . . . . . . . . . . . . . . . . . .  4
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  5
     2.1.  ForCES Protocol  . . . . . . . . . . . . . . . . . . . . .  5
     2.2.  ForCES Model . . . . . . . . . . . . . . . . . . . . . . .  5
     2.3.  Transport mapping layer  . . . . . . . . . . . . . . . . .  5
   3.  Definitions  . . . . . . . . . . . . . . . . . . . . . . . . .  6
   4.  Date, Location and Access  . . . . . . . . . . . . . . . . . .  9
     4.1.  Date . . . . . . . . . . . . . . . . . . . . . . . . . . .  9
     4.2.  Location . . . . . . . . . . . . . . . . . . . . . . . . .  9
     4.3.  Access . . . . . . . . . . . . . . . . . . . . . . . . . .  9
   5.  Testbed architecture . . . . . . . . . . . . . . . . . . . . . 10
     5.1.  Local configuration  . . . . . . . . . . . . . . . . . . . 10
     5.2.  Distributed configuration  . . . . . . . . . . . . . . . . 11
   6.  Scenarios  . . . . . . . . . . . . . . . . . . . . . . . . . . 12
     6.1.  Scenario 1 - Pre-association Setup . . . . . . . . . . . . 12
     6.2.  Scenario 2 - TML priority channels connection  . . . . . . 13
     6.3.  Scenario 3 - Association Setup - Association Complete  . . 13
     6.4.  Scenario 4 - CE query  . . . . . . . . . . . . . . . . . . 13
     6.5.  Scenario 5 - Heartbeat monitoring  . . . . . . . . . . . . 14
     6.6.  Scenario 6 - Simple Config Command . . . . . . . . . . . . 14
     6.7.  Scenario 7 - Association Teardown  . . . . . . . . . . . . 14
   7.  Tested Features  . . . . . . . . . . . . . . . . . . . . . . . 16
     7.1.  ForCES Protocol Features . . . . . . . . . . . . . . . . . 16
       7.1.1.  Protocol Messages  . . . . . . . . . . . . . . . . . . 16
       7.1.2.  MainHeader Handling  . . . . . . . . . . . . . . . . . 17
       7.1.3.  TLV Handling . . . . . . . . . . . . . . . . . . . . . 17
       7.1.4.  Operation Types Supported  . . . . . . . . . . . . . . 18
     7.2.  ForCES Model Features  . . . . . . . . . . . . . . . . . . 18
       7.2.1.  Basic Atomic Types Supported . . . . . . . . . . . . . 18
       7.2.2.  Compound Types Supported . . . . . . . . . . . . . . . 18
       7.2.3.  LFBs Supported . . . . . . . . . . . . . . . . . . . . 19
         7.2.3.1.  FE Protocol LFB  . . . . . . . . . . . . . . . . . 19
         7.2.3.2.  FE Object LFB  . . . . . . . . . . . . . . . . . . 19
     7.3.  ForCES SCTP-TML Features . . . . . . . . . . . . . . . . . 20



Haleplidis, et al.      Expires February 8, 2010                [Page 2]


Internet-Draft        ForCES Interoperability Draft          August 2009


       7.3.1.  TML Priority Ports . . . . . . . . . . . . . . . . . . 20
       7.3.2.  Message Handling at specific priorities  . . . . . . . 20
   8.  Test details . . . . . . . . . . . . . . . . . . . . . . . . . 22
   9.  Results  . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
   10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 29
   11. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 30
   12. Security Considerations  . . . . . . . . . . . . . . . . . . . 31
   13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32
     13.1. Normative References . . . . . . . . . . . . . . . . . . . 32
     13.2. Informative References . . . . . . . . . . . . . . . . . . 32
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 34








































Haleplidis, et al.      Expires February 8, 2010                [Page 3]


Internet-Draft        ForCES Interoperability Draft          August 2009


1.  Terminology and Conventions

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].












































Haleplidis, et al.      Expires February 8, 2010                [Page 4]


Internet-Draft        ForCES Interoperability Draft          August 2009


2.  Introduction

   Forwarding and Control Element Separation (ForCES) defines an
   architectural framework and associated protocols to standardize
   information exchange between the control plane and the forwarding
   plane in a ForCES Network Element (ForCES NE).  [RFC3654] has defined
   the ForCES requirements, and [RFC3746] has defined the ForCES
   framework.

2.1.  ForCES Protocol

   The ForCES protocol works in a master-slave mode in which FEs are
   slaves and CEs are masters.  The protocol includes commands for
   transport of Logical Function Block (LFB) configuration information,
   association setup, status, and event notifications, etc.  The reader
   is encouraged to read FE-protocol [I-D.ietf-forces-protocol] for
   further information.

2.2.  ForCES Model

   The FE-MODEL [I-D.ietf-forces-model] presents a formal way to define
   FE Logical Function Blocks (LFBs) using XML.  LFB configuration
   components, capabilities, and associated events are defined when the
   LFB is formally created.  The LFBs within the FE are accordingly
   controlled in a standardized way by the ForCES protocol.

2.3.  Transport mapping layer

   The TML transports the PL messages.  The TML is where the issues of
   how to achieve transport level reliability, congestion control,
   multicast, ordering, etc. are handled.  It is expected that more than
   one TML will be standardized.  The various possible TMLs could vary
   their implementations based on the capabilities of underlying media
   and transport.  However, since each TML is standardized,
   interoperability is guaranteed as long as both endpoints support the
   same TML.  All ForCES Protocol Layer implementations MUST be portable
   across all TMLs.  Although more than one TML may be standardized for
   the ForCES Protocol, for the purposes of the interoperability test,
   the mandated MUST IMPLEMENT SCTP TML [I-D.ietf-forces-sctptml] will
   be used.











Haleplidis, et al.      Expires February 8, 2010                [Page 5]


Internet-Draft        ForCES Interoperability Draft          August 2009


3.  Definitions

   This document follows the terminology defined by the ForCES
   Requirements in [RFC3654] and by the ForCES framework in [RFC3746].
   The definitions below are repeated below for clarity.

   o  Control Element (CE) - A logical entity that implements the ForCES
      protocol and uses it to instruct one or more FEs on how to process
      packets.  CEs handle functionality such as the execution of
      control and signaling protocols.

   o  CE Manager (CEM) - A logical entity responsible for generic CE
      management tasks.  It is particularly used during the pre-
      association phase to determine with which FE(s) a CE should
      communicate.  This process is called FE discovery and may involve
      the CE manager learning the capabilities of available FEs.

   o  Forwarding Element (FE) - A logical entity that implements the
      ForCES protocol.  FEs use the underlying hardware to provide per-
      packet processing and handling as directed/controlled by one or
      more CEs via the ForCES protocol.

   o  FE Manager (FEM) - A logical entity responsible for generic FE
      management tasks.  It is used during pre-association phase to
      determine with which CE(s) an FE should communicate.  This process
      is called CE discovery and may involve the FE manager learning the
      capabilities of available CEs.  An FE manager may use anything
      from a static configuration to a pre-association phase protocol
      (see below) to determine which CE(s) to use.  Being a logical
      entity, an FE manager might be physically combined with any of the
      other logical entities such as FEs.

   o  ForCES Network Element (NE) - An entity composed of one or more
      CEs and one or more FEs.  To entities outside an NE, the NE
      represents a single point of management.  Similarly, an NE usually
      hides its internal organization from external entities.

   o  LFB (Logical Function Block) - The basic building block that is
      operated on by the ForCES protocol.  The LFB is a well defined,
      logically separable functional block that resides in an FE and is
      controlled by the CE via ForCES protocol.  The LFB may reside at
      the FE's datapath and process packets or may be purely an FE
      control or configuration entity that is operated on by the CE.
      Note that the LFB is a functionally accurate abstraction of the
      FE's processing capabilities, but not a hardware-accurate
      representation of the FE implementation.





Haleplidis, et al.      Expires February 8, 2010                [Page 6]


Internet-Draft        ForCES Interoperability Draft          August 2009


   o  FE Topology - A representation of how the multiple FEs within a
      single NE are interconnected.  Sometimes this is called inter-FE
      topology, to be distinguished from intra-FE topology (i.e., LFB
      topology).

   o  LFB Class and LFB Instance - LFBs are categorized by LFB Classes.
      An LFB Instance represents an LFB Class (or Type) existence.
      There may be multiple instances of the same LFB Class (or Type) in
      an FE.  An LFB Class is represented by an LFB Class ID, and an LFB
      Instance is represented by an LFB Instance ID.  As a result, an
      LFB Class ID associated with an LFB Instance ID uniquely specifies
      an LFB existence.

   o  LFB Metadata - Metadata is used to communicate per-packet state
      from one LFB to another, but is not sent across the network.  The
      FE model defines how such metadata is identified, produced and
      consumed by the LFBs.  It defines the functionality but not how
      metadata is encoded within an implementation.

   o  LFB Component - Operational parameters of the LFBs that must be
      visible to the CEs are conceptualized in the FE model as the LFB
      components.  The LFB components include, for example, flags,
      single parameter arguments, complex arguments, and tables that the
      CE can read and/or write via the ForCES protocol (see below).

   o  LFB Topology - Representation of how the LFB instances are
      logically interconnected and placed along the datapath within one
      FE.  Sometimes it is also called intra-FE topology, to be
      distinguished from inter-FE topology.

   o  Pre-association Phase - The period of time during which an FE
      Manager and a CE Manager are determining which FE(s) and CE(s)
      should be part of the same network element.

   o  Post-association Phase - The period of time during which an FE
      knows which CE is to control it and vice versa.  This includes the
      time during which the CE and FE are establishing communication
      with one another.

   o  ForCES Protocol - While there may be multiple protocols used
      within the overall ForCES architecture, the term "ForCES protocol"
      and "protocol" refer to the Fp reference points in the ForCES
      Framework in [RFC3746].  This protocol does not apply to CE-to-CE
      communication, FE-to-FE communication, or to communication between
      FE and CE managers.  Basically, the ForCES protocol works in a
      master- slave mode in which FEs are slaves and CEs are masters.
      This document defines the specifications for this ForCES protocol.




Haleplidis, et al.      Expires February 8, 2010                [Page 7]


Internet-Draft        ForCES Interoperability Draft          August 2009


   o  ForCES Protocol Transport Mapping Layer (ForCES TML) - A layer in
      ForCES protocol architecture that uses the capabilities of
      existing transport protocols to specifically address protocol
      message transportation issues, such as how the protocol messages
      are mapped to different transport media (like TCP, IP, ATM,
      Ethernet, etc), and how to achieve and implement reliability,
      multicast, ordering, etc.  The ForCES TML specifications are
      detailed in separate ForCES documents, one for each TML.











































Haleplidis, et al.      Expires February 8, 2010                [Page 8]


Internet-Draft        ForCES Interoperability Draft          August 2009


4.  Date, Location and Access

4.1.  Date

   The date that the Interoperability test took place was 15-16/07/2009,
   one and a half week before IETF 75, in Stockholm.

4.2.  Location

   Patras is a major harbor of Greece connecting it with Italy.

   The University of Patras is located in Rio, 10km east out of Patras.

   The following coordinates mark the Electrical and Computer
   Engineering building in the University.

   o  North: 38o17'15.99"

   o  East: 21o47'19.28"

4.3.  Access

   The best way to come to Greece is by plane to the Athens
   International Airport.

   From there there are three ways to arrive in the University of
   Patras.

   1.  Renting a car and driving to the University.  It is a maximum
       2:30 hours drive from the aiport.

   2.  Via coach station.  Get from the airport to the coach station via
       X93 bus towards the Kifissos Coach Station.  At the Coach Station
       there are buses to Patras every 30 minutes.  The Bus to Patras
       may take about 2:30 - 3:00 hours, and the ride of the X93 bus may
       take about 30 mins - 1hour depending on the traffic, so it's
       about 3:30 - 4:30 hours away with the wait at the Coach Station.

   3.  Via Train.  It is recommended you already have booked your ticket
       beforehand as there are not many trains going to Patras, and
       mostly are booked in advanced.  It is not recommended that you
       take the train to Patras, as you have to change at least 2
       trains.  In order to reach Patras from the Athens International
       Airport you need to take the Suburban Rail to Kiato.  From Kiato
       you can catch a train to Patras.  It will take you at least 5
       hours to reach Patras.





Haleplidis, et al.      Expires February 8, 2010                [Page 9]


Internet-Draft        ForCES Interoperability Draft          August 2009


5.  Testbed architecture

   Most FEs and CEs were located locally at the University of Patras
   premises.  One party participated connecting over the internet.

   The test took place between FEs and CEs of different implementors
   with different permutations.

   All protocol messages of each scenario were monitored using a
   protocol network analyzer that tested validity.  Two tools were used:

   o  A modified tcpdump [tcpdump].

   o  A modified Ethereal [ethereal].

   All NE's in all the scenarios were comprised of one CE and one FE
   from different implementors.

5.1.  Local configuration

   Hardware/Software (CEs and FEs) that were located within the
   University of Patras premises, were connected together using
   switches.

   The scenarios were tested with only one CE associated with one or
   multiple FEs from different implementors.  The CE and the FE(s) were
   connected in one LAN as shown in the following figure.

                                  +-----+
                                  | CE1 |
                                  |Impl1|
                                  +-----+
                                     |
                                     |
                   +------------------------------------+
                   |                LAN                 |
                   +------------------------------------+
                      |       |         |          |
                      |       |   ...   |          |
                   +-----+ +-----+   +-----+   +--------+
                   | FE1 | | FE2 |   | FEn |   |Protocol|
                   |Impl1| |Impl2|   |Impln|   |Analyzer|
                   +-----+ +-----+   +-----+   +--------+

   All scenarios were tested more than once with permutation of the CE
   from different implementors.  In the next permutation, the setup were
   as shown in the following figure.




Haleplidis, et al.      Expires February 8, 2010               [Page 10]


Internet-Draft        ForCES Interoperability Draft          August 2009


                                  +-----+
                                  | CE2 |
                                  |Impl2|
                                  +-----+
                                     |
                                     |
                   +------------------------------------+
                   |                LAN                 |
                   +------------------------------------+
                      |       |         |          |
                      |       |   ...   |          |
                   +-----+ +-----+   +-----+   +--------+
                   | FE1 | | FE2 |   | FEn |   |Protocol|
                   |Impl1| |Impl2|   |Impln|   |Analyzer|
                   +-----+ +-----+   +-----+   +--------+

5.2.  Distributed configuration

   For parties that cannot participate, public IPs can be provided and
   associations can be achieved over the internet as seen in the
   following figure.

       +-----+   +------------+   /\/\/\/\/\   +----------+   +-----+
       |FE/CE|   |Implementor |   \Internet/   |University|   |FE/CE|
       |ImplX|---|   Router   |---/        \---|  Router  |---|ImplY|
       +-----+   +------------+   \/\/\/\/\/   +----------+   +-----+

   For interoperability issues, all CEs and FEs MUST implement no
   security even in the TML.  For security, firewalls MUST be used that
   will allow only the specific IPs and the SCTP ports defined in the
   SCTP-TML draft [I-D.ietf-forces-sctptml].




















Haleplidis, et al.      Expires February 8, 2010               [Page 11]


Internet-Draft        ForCES Interoperability Draft          August 2009


6.  Scenarios

   Since the main goal of this interoperability test is to test the
   basic protocol functionality, we will limit the test parameters.
   Therefore:

   1.  In the Association Setup Message, all report messages will be
       ignored.

   2.  In the Association Setup Phase, the messages, FEO OperEnable
       Event (FE to CE), Config FEO Adminup (CE to FE) and FEO Config-
       Resp (FE to CE) will be ignored.  The CE will assume that the FE
       is enabled once the LFBSelectors has been queried.

   3.  Only FullDataTLVs are going to be used and not SparseData TLV's.

   4.  There will be no transaction operations.

   5.  Each message shall have only one LFBSelector TLV, one Operation
       TLV and one PathDataTLV per message when these are used.

6.1.  Scenario 1 - Pre-association Setup

   While the Pre-association setup is not in the ForCES current scope it
   is an essential step before CEs and FEs communicate.  As the first
   part in a successfull CE-FE connection the participating CEs and FEs
   should be able to be configured.

   In the Pre-association Phase the following configuration items MUST
   be setup regarding the CEs:

   o  The CE ID.

   o  The FE IDs that will be connected to this CE

   o  The IP of the FEs that will connect

   o  The TML priority ports.

   In the Pre-association Phase the following configuration items MUST
   be setup regarding the FEs:

   o  The FE ID.

   o  The CE ID that this FE will be connecting to.

   o  The IP of the CE that will connect to




Haleplidis, et al.      Expires February 8, 2010               [Page 12]


Internet-Draft        ForCES Interoperability Draft          August 2009


   o  The TML priority ports.

   Once each element is setup and configured, Scenario 1 is successful.

6.2.  Scenario 2 - TML priority channels connection

   For the current interoperability test, the SCTP will be used as TML.
   The TML connection with the associating element is needed for the
   scenario 2 to be successful.

   The SCTP-TML draft [I-D.ietf-forces-sctptml] defines 3 priority
   channels, with specific ports:

   o  High priority - Port number: 6700

   o  Medium priority - Port number: 6701

   o  Lower priority - Port number: 6702

   Once these channels have been established with each associated
   element, will the Scenario 2 be successful.

6.3.  Scenario 3 - Association Setup - Association Complete

   Once the Pre-association phase has been complete in the previous 2
   scenarios, CEs and FEs are ready to communicate using the ForCES
   protocol, and enter the Association Setup stage.  In this stage the
   FEs attempts to join the NE.  The following ForCES protocol messages
   will be exchanged for each CE-FE pair in the specified order:

   o  Association Setup Message (from FE to CE)

   o  Association Setup Response Message (from CE to FE)

   o  Query Message: FEO LFBSelectors(from CE to FE)

   o  Query Response: FEO LFBSelectors response (from FE to CE)

   Once the associations has been initialized scenario 3 will have been
   successful.

6.4.  Scenario 4 - CE query

   Once the Association Phase stage has been complete, the FEs and CEs
   will enter the Established stage.  In this stage the FE is
   continuously updated or queried.  The CE should query the FE a
   specific value from the FE Object LFB and from the FE Protocol LFB.
   An example from the FE Protocol LFB is the HeartBeat Timer (FEHI) and



Haleplidis, et al.      Expires February 8, 2010               [Page 13]


Internet-Draft        ForCES Interoperability Draft          August 2009


   from the FE Object LFB is the State of the LFB (FEState)

   The following ForCES protocol messages will be exchanged:

   o  Query Message

   o  Query Response Message

6.5.  Scenario 5 - Heartbeat monitoring

   The Heartbeat (HB) Message is used for one ForCES element (FE or CE)
   to asynchronously notify one or more other ForCES elements in the
   same ForCES NE on its liveness.  The default configuration of the
   Heartbeat Policy of the FE is set to 0 which means, that the FE
   should not generate any Heartbeat messages. the CE is responsible for
   checking FE liveness by setting the PL header ACK flag of the message
   it sends to AlwaysACK.  In this Scenario the CE should send a
   Heartbeat message with the ACK flag set to AlwaysACK and the FE
   should respond.

   The following ForCES protocol messages will be exchanged:

   o  Heartbeat Message

6.6.  Scenario 6 - Simple Config Command

   A config message is sent by the CE to the FE to configure LFB
   components in the FE.  A simple config command easily visible and
   metered would be to change the Heartbeat configuration.  This will be
   done in two steps:

   1.  Change the FE Heartbeat Policy (FEHBPolicy) to value 1, to force
       the FE to send heartbeats.

   2.  After some heartbeats from the FE, the FE Heartbeat Interval
       (FEHI) will be changed.

   The following ForCES protocol messages will be exchanged:

   o  Config Message

   o  Config Response Message

6.7.  Scenario 7 - Association Teardown

   In the end, the association must be terminated.  There are two
   scenarios by which the association maybe terminated:




Haleplidis, et al.      Expires February 8, 2010               [Page 14]


Internet-Draft        ForCES Interoperability Draft          August 2009


   1.  Normal tear down by exchanging Association Teardown Message

   2.  Irregular tear down by stopping heartbeats from a FE or a CE.

   3.  Irregular tear down by externally shutting down/rebooting a FE or
       a CE.

   All scenarios may be tested in the interoperability test.

   The following ForCES protocol messages will be exchanged:

   o  Association Teardown Message







































Haleplidis, et al.      Expires February 8, 2010               [Page 15]


Internet-Draft        ForCES Interoperability Draft          August 2009


7.  Tested Features

   The features that were tested are:

7.1.  ForCES Protocol Features

                              +------------+
                              |   Feature  |
                              +------------+
                              |  Batching  |
                              |            |
                              | HeartBeats |
                              +------------+

                         ForCES Protocol Features

   Although Batching was not initially designed to be tested, it was
   tested during the interoperability test.

7.1.1.  Protocol Messages

                      +----------------------------+
                      |      Protocol Message      |
                      +----------------------------+
                      |      Association Setup     |
                      |                            |
                      | Association Setup Response |
                      |                            |
                      |    Association TearDown    |
                      |                            |
                      |        Configuration       |
                      |                            |
                      |   Configuration Response   |
                      |                            |
                      |            Query           |
                      |                            |
                      |       Query Response       |
                      |                            |
                      |          HeartBeat         |
                      +----------------------------+

                          ForCES Protocol Message









Haleplidis, et al.      Expires February 8, 2010               [Page 16]


Internet-Draft        ForCES Interoperability Draft          August 2009


7.1.2.  MainHeader Handling

                           +------------------+
                           |   Header Field   |
                           +------------------+
                           |    Correlator    |
                           |                  |
                           | Acknowledge Flag |
                           |                  |
                           |   Priority Flag  |
                           +------------------+

                            MainHeader Handling

7.1.3.  TLV Handling

                    +---------------------------------+
                    |               TLV               |
                    +---------------------------------+
                    |   Association Setup Result TLV  |
                    |                                 |
                    | Association TearDown Reason TLV |
                    |                                 |
                    |         LFBSelector TLV         |
                    |                                 |
                    |          Operation TLV          |
                    |                                 |
                    |           PathData TLV          |
                    |                                 |
                    |           FullData TLV          |
                    |                                 |
                    |            Result TLV           |
                    +---------------------------------+

                              TLVs Supported
















Haleplidis, et al.      Expires February 8, 2010               [Page 17]


Internet-Draft        ForCES Interoperability Draft          August 2009


7.1.4.  Operation Types Supported

                             +--------------+
                             |   Operation  |
                             +--------------+
                             |      Set     |
                             |              |
                             | Set Response |
                             |              |
                             |      Get     |
                             |              |
                             | Get Response |
                             |              |
                             |    Report    |
                             +--------------+

                         Operation Type Supported

7.2.  ForCES Model Features

7.2.1.  Basic Atomic Types Supported

                              +-------------+
                              | Atomic Type |
                              +-------------+
                              |    uchar    |
                              |             |
                              |    uint32   |
                              +-------------+

                       Basic Atomic Types Supported

7.2.2.  Compound Types Supported

                             +---------------+
                             | Compound Type |
                             +---------------+
                             |    structs    |
                             |               |
                             |     arrays    |
                             +---------------+

                         Compound Types Supported








Haleplidis, et al.      Expires February 8, 2010               [Page 18]


Internet-Draft        ForCES Interoperability Draft          August 2009


7.2.3.  LFBs Supported

7.2.3.1.  FE Protocol LFB

                          +--------------------+
                          | Protocol DataTypes |
                          +--------------------+
                          |     CEHBPolicy     |
                          |                    |
                          |     FEHIBPolicy    |
                          +--------------------+

                         FE Protocol LFB Datatypes

                          +---------------------+
                          | Protocol Components |
                          +---------------------+
                          |         FEID        |
                          |                     |
                          |      CEHBPolicy     |
                          |                     |
                          |        CEHDI        |
                          |                     |
                          |      FEHBPolicy     |
                          |                     |
                          |         FEHI        |
                          |                     |
                          |         CEID        |
                          +---------------------+

                        FE Protocol LFB Components

7.2.3.2.  FE Object LFB

                           +------------------+
                           | Object DataTypes |
                           +------------------+
                           |   FEStateValues  |
                           |                  |
                           |  LFBSelectorType |
                           +------------------+

                          FE Object LFB Datatypes








Haleplidis, et al.      Expires February 8, 2010               [Page 19]


Internet-Draft        ForCES Interoperability Draft          August 2009


                           +-------------------+
                           | Object Components |
                           +-------------------+
                           |    LFBSelectors   |
                           |                   |
                           |      FEState      |
                           +-------------------+

                         FE Object LFB Components

7.3.  ForCES SCTP-TML Features

7.3.1.  TML Priority Ports

                        +------------------------+
                        |          Port          |
                        +------------------------+
                        |  High priority (6700)  |
                        |                        |
                        | Medium priority (6701) |
                        |                        |
                        |   Low priority (6702)  |
                        +------------------------+

                              Priority Ports

7.3.2.  Message Handling at specific priorities

                      +----------------------------+
                      |       ForCES Message       |
                      +----------------------------+
                      |      Association Setup     |
                      |                            |
                      | Association Setup Response |
                      |                            |
                      |    Association Teardown    |
                      |                            |
                      |           Config           |
                      |                            |
                      |       Config Response      |
                      |                            |
                      |            Query           |
                      |                            |
                      |       Query Response       |
                      +----------------------------+

               Message Handling at High priority (6700) Port




Haleplidis, et al.      Expires February 8, 2010               [Page 20]


Internet-Draft        ForCES Interoperability Draft          August 2009


                            +----------------+
                            | ForCES Message |
                            +----------------+
                            |   Heartbeats   |
                            +----------------+

               Message Handling at Low priority (6702) Port












































Haleplidis, et al.      Expires February 8, 2010               [Page 21]


Internet-Draft        ForCES Interoperability Draft          August 2009


8.  Test details

   The following tests occured:

   +------+----------+----------+-----------+-----------+--------------+
   | Test |    CE    |   FE(s)  |  Teardown |   Result  |    Comment   |
   | #    |          |          |   Option  |           |              |
   +------+----------+----------+-----------+-----------+--------------+
   |   1  | Zhejiang |    NTT   |  Teardown |  Success  |              |
   |      | Gongshan |          |  from FE  |           |              |
   |      | g        |          |           |           |              |
   |      |  Univers |          |           |           |              |
   |      | it    y  |          |           |           |              |
   |      |          |          |           |           |              |
   |   2  | Zhejiang |    NTT   |  Teardown |  Success  |              |
   |      | Gongshan |          |  from CE  |           |              |
   |      | g        |          |           |           |              |
   |      |  Univers |          |           |           |              |
   |      | it    y  |          |           |           |              |
   |      |          |          |           |           |              |
   |   3  | Zhejiang |    NTT   |   Cable   |  Success  |  Nobody saw  |
   |      | Gongshan |          | disconnec |           |  the loss of |
   |      | g        |          | t         |           |    cable.    |
   |      |  Univers |          |           |           |   Everybody  |
   |      | it    y  |          |           |           |   found out  |
   |      |          |          |           |           | from loss of |
   |      |          |          |           |           | PL-heartbeat |
   |      |          |          |           |           | s            |
   |      |          |          |           |           |              |
   |   4  | Zhejiang |    NTT   |  Loss of  |  Success  |   FE didn't  |
   |      | Gongshan |          |     CE    |           |     send     |
   |      | g        |          | Heartbeat |           | Teardown and |
   |      |  Univers |          | s         |           |    closed    |
   |      | it    y  |          |           |           |  connection  |
   |      |          |          |           |           |              |
   |   5  | Zhejiang |    NTT   |  Loss of  | Untestabl |              |
   |      | Gongshan |          |     FE    | e         |              |
   |      | g        |          | Heartbeat |           |              |
   |      |  Univers |          | s         |           |              |
   |      | it    y  |          |           |           |              |
   |      |          |          |           |           |              |
   |   6  |    NTT   | Zhejiang |  Teardown |  Initial  |  CE couldn't |
   |      |          | Gongshan |  from CE  |  Failure  | handle Query |
   |      |          | g        |           |           |  Result for  |
   |      |          |  Univers |           |           |    unknown   |
   |      |          | it    y  |           |           |  LFBSelects. |
   |      |          |          |           |           |              |




Haleplidis, et al.      Expires February 8, 2010               [Page 22]


Internet-Draft        ForCES Interoperability Draft          August 2009


   |   7  | Zhejiang | Universi |  Teardown |  Success  |   Problems   |
   |      | Gongshan | t  y of  |  from FE  |           |     with     |
   |      | g        |   Patras |           |           | retransmitti |
   |      |  Univers |          |           |           | o      n     |
   |      | it    y  |          |           |           |              |
   |      |          |          |           |           |              |
   |   8  | Zhejiang | Universi |  Teardown |  Success  |   Problems   |
   |      | Gongshan | t  y of  |  from CE  |           |     with     |
   |      | g        |   Patras |           |           | retransmitti |
   |      |  Univers |          |           |           | o      n     |
   |      | it    y  |          |           |           |              |
   |      |          |          |           |           |              |
   |   9  | Zhejiang | Universi |   Cable   |  Success  |  Nobody saw  |
   |      | Gongshan | t  y of  | disconnec |           |  the loss of |
   |      | g        |   Patras | t         |           |    cable.    |
   |      |  Univers |          |           |           |   Everybody  |
   |      | it    y  |          |           |           |   found out  |
   |      |          |          |           |           | from loss of |
   |      |          |          |           |           | PL-heartbeat |
   |      |          |          |           |           | s            |
   |      |          |          |           |           |              |
   |  10  | Zhejiang | Universi |  Loss of  |  Success  |              |
   |      | Gongshan | t  y of  |     CE    |           |              |
   |      | g        |   Patras | Heartbeat |           |              |
   |      |  Univers |          | s         |           |              |
   |      | it    y  |          |           |           |              |
   |      |          |          |           |           |              |
   |  11  |    NTT   | Zhejiang |  Teardown |  Success  |   Test# 6.   |
   |      |          | Gongshan |  from CE  | on Repeat |   Problems   |
   |      |          | g        |           |           |     fixed    |
   |      |          |  Univers |           |           |              |
   |      |          | it    y  |           |           |              |
   |      |          |          |           |           |              |
   |  12  |    NTT   | Zhejiang |  Teardown |  Success  |              |
   |      |          | Gongshan |  from FE  |           |              |
   |      |          | g        |           |           |              |
   |      |          |  Univers |           |           |              |
   |      |          | it    y  |           |           |              |
   |      |          |          |           |           |              |
   |  13  |    NTT   | Zhejiang |   Cable   |  Success  |  Nobody saw  |
   |      |          | Gongshan | disconnec |           |  the loss of |
   |      |          | g        | t         |           |    cable.    |
   |      |          |  Univers |           |           |   Everybody  |
   |      |          | it    y  |           |           |   found out  |
   |      |          |          |           |           | from loss of |
   |      |          |          |           |           | PL-heartbeat |
   |      |          |          |           |           | s      .     |
   |      |          |          |           |           |              |



Haleplidis, et al.      Expires February 8, 2010               [Page 23]


Internet-Draft        ForCES Interoperability Draft          August 2009


   |  14  |    NTT   | Zhejiang |  Loss of  |  Success  |   Problems   |
   |      |          | Gongshan |     CE    |           |     with     |
   |      |          | g        | Heartbeat |           | retransmitti |
   |      |          |  Univers | s         |           | o      n     |
   |      |          | it    y  |           |           |              |
   |      |          |          |           |           |              |
   |  15  | Universi | Zhejiang |  Teardown |  Success  |   CE didn't  |
   |      | t  y of  | Gongshan |  from FE  |           |   terminat   |
   |      |   Patras | g        |           |           |     after    |
   |      |          |  Univers |           |           |    sending   |
   |      |          | it    y  |           |           |  Teardown.   |
   |      |          |          |           |           |    FE did    |
   |      |          |          |           |           |              |
   |  16  | Universi | Zhejiang |  Teardown |  Success  |   Problems   |
   |      | t  y of  | Gongshan |  from CE  |           |     with     |
   |      |   Patras | g        |           |           | retransmitti |
   |      |          |  Univers |           |           | o      n     |
   |      |          | it    y  |           |           |              |
   |      |          |          |           |           |              |
   |  17  | Universi | Zhejiang |  Loss of  |  Success  |   FE didn't  |
   |      | t  y of  | Gongshan |     CE    |           |     send     |
   |      |   Patras | g        | Heartbeat |           | Teardown and |
   |      |          |  Univers | s         |           |    closed    |
   |      |          | it    y  |           |           |  connection  |
   |      |          |          |           |           |              |
   |  18  | Zhejiang |   NTT &  |  Teardown |  Success  |              |
   |      | Gongshan | Universi |  from CE  |           |              |
   |      | g        | t  y of  |           |           |              |
   |      |  Univers |  Patrasx |           |           |              |
   |      | it    y  | 2        |           |           |              |
   |      |          |          |           |           |              |
   |  19  |    NTT   | Zhejiang |  Teardown |  Success  |              |
   |      |          | Gongshan |  from CE  |           |              |
   |      |          | g        |           |           |              |
   |      |          |  Univers |           |           |              |
   |      |          | it   y & |           |           |              |
   |      |          |   Univer |           |           |              |
   |      |          | sit  y o |           |           |              |
   |      |          | f  Patra |           |           |              |
   |      |          | sx2      |           |           |              |
   |      |          |          |           |           |              |










Haleplidis, et al.      Expires February 8, 2010               [Page 24]


Internet-Draft        ForCES Interoperability Draft          August 2009


   |  20  | Universi |   NTT &  |  Teardown |  Success  |              |
   |      | t  y of  | Zhejiang |  from CE  |           |              |
   |      |   Patras | Gongshan |           |           |              |
   |      |          | g        |           |           |              |
   |      |          |  Univers |           |           |              |
   |      |          | it   y & |           |           |              |
   |      |          |   Univer |           |           |              |
   |      |          | sit  y o |           |           |              |
   |      |          | f  Patra |           |           |              |
   |      |          | sx2      |           |           |              |
   |      |          |          |           |           |              |
   |  21  | Universi | Zhejiang |  Batching |  Success  |              |
   |      | t  y of  | Gongshan | Query and |           |              |
   |      |   Patras | g        |   Config  |           |              |
   |      |          |  Univers |           |           |              |
   |      |          | it    y  |           |           |              |
   |      |          |          |           |           |              |
   |  22  | Universi |    NTT   |  Teardown |  Success  |              |
   |      | t  y of  |          |  from FE  |           |              |
   |      |   Patras |          |           |           |              |
   |      |          |          |           |           |              |
   |  23  | Universi |    NTT   |  Teardown |  Success  |              |
   |      | t  y of  |          |  from CE  |           |              |
   |      |   Patras |          |           |           |              |
   |      |          |          |           |           |              |
   |  24  | Universi |    NTT   |  Loss of  |  Success  |   FE didn't  |
   |      | t  y of  |          |     CE    |           |     send     |
   |      |   Patras |          | Heartbeat |           | Teardown and |
   |      |          |          | s         |           |    closed    |
   |      |          |          |           |           |  connection  |
   |      |          |          |           |           |              |
   |  25  | Universi |    NTT   |   Cable   |  Success  |  Nobody saw  |
   |      | t  y of  |          | disconnec |           |  the loss of |
   |      |   Patras |          | t         |           |    cable.    |
   |      |          |          |           |           |   Everybody  |
   |      |          |          |           |           |   found out  |
   |      |          |          |           |           | from loss of |
   |      |          |          |           |           | PL-heartbeat |
   |      |          |          |           |           | s            |
   |      |          |          |           |           |              |
   |  26  |    NTT   | Universi |  Teardown |  Success  |              |
   |      |          | t  y of  |  from FE  |           |              |
   |      |          |   Patras |           |           |              |
   |      |          |          |           |           |              |
   |  27  |    NTT   | Universi |  Teardown |  Success  |              |
   |      |          | t  y of  |  from CE  |           |              |
   |      |          |   Patras |           |           |              |
   |      |          |          |           |           |              |



Haleplidis, et al.      Expires February 8, 2010               [Page 25]


Internet-Draft        ForCES Interoperability Draft          August 2009


   |  28  |    NTT   | Universi |  Loss of  |  Success  |   FE didn't  |
   |      |          | t  y of  |     CE    |           |     send     |
   |      |          |   Patras | Heartbeat |           | Teardown and |
   |      |          |          | s         |           |    closed    |
   |      |          |          |           |           |  connection  |
   |      |          |          |           |           |              |
   |  29  |    NTT   | Universi |   Cable   |  Success  |  Nobody saw  |
   |      |          | t  y of  | disconnec |           |  the loss of |
   |      |          |   Patras | t         |           |    cable.    |
   |      |          |          |           |           |   Everybody  |
   |      |          |          |           |           |   found out  |
   |      |          |          |           |           | from loss of |
   |      |          |          |           |           | PL-heartbeat |
   |      |          |          |           |           | s            |
   +------+----------+----------+-----------+-----------+--------------+

                          Interoperability Tests


































Haleplidis, et al.      Expires February 8, 2010               [Page 26]


Internet-Draft        ForCES Interoperability Draft          August 2009


9.  Results

   All implementations were found to be interoperable with each other.

   All scenarios were tested successfully.

   The following issues were found and dealt with.

   1.   Some messages were sent to the wrong priority channels.  There
        was some ambiguities on the SCTP-TML draft that have been
        corrected.

   2.   At some point, a CE sent a TearDown message to the FE.  The CE
        expected the FE to shut down the connection, and the FE waited
        the CE to shut down the connection and were caught in a
        deadlock.  This was a code bug and was fixed.

   3.   Sometimes the association setup message, only on the remote
        connection test, although sent, was not received by the other
        end and made impossible the association.  This was caused by
        network problems.

   4.   An implementation did not take into account that the padding in
        TLVs MUST NOT be included in the lenght of the TLV.  This was a
        code bug and was fixed.

   5.   EM Flag was set to reserved by a CE and was not ignored by the
        FE.  This was a code bug and was fixed.

   6.   After the FEHBPolicy was set to 1 the FE didn't send any
        HeartBeats.  This was a code bug and was fixed.

   7.   Some FE's sent HeartBeats with the ACK flag with a value other
        than NoACK.  The CE responded.  This was a code bug and was
        fixed.

   8.   When a cable was disconnected, the TML didn't realize that.  The
        association was dropped due to heartbeats, this was a success,
        but this is an implementation issue implementers should keep in
        mind.  This is a SCTP options issue.  Nothing was needed to be
        done.

   9.   A CE crashed due to unknown LFBSelector values.  This was a code
        bug and was fixed.

   10.  With the remote connection there were a lot of forces packet
        retransmittion.  The problem is that packets like Heartbeats
        were retransmitted.  This is a SCTP issue.  Perhaps SCTP-PR is



Haleplidis, et al.      Expires February 8, 2010               [Page 27]


Internet-Draft        ForCES Interoperability Draft          August 2009


        needed to be used.

   The implementers went beyond the call of duty.  The test was extended
   with another test for batching messages.  This test was also done
   successfully.














































Haleplidis, et al.      Expires February 8, 2010               [Page 28]


Internet-Draft        ForCES Interoperability Draft          August 2009


10.  Acknowledgements

   The authors of this draft would like to acknowledge and thank the
   chair of the ForCES working group Jamal Hadi Salim.

   Also, the authors would like to acknowledge Professors Odysseas
   Koufopavlou and Spyros Denazis, as well as the Department of
   Electrical and Computer Engineering of the University of Patras for
   hosting the event.










































Haleplidis, et al.      Expires February 8, 2010               [Page 29]


Internet-Draft        ForCES Interoperability Draft          August 2009


11.  IANA Considerations

   This memo includes no request to IANA.
















































Haleplidis, et al.      Expires February 8, 2010               [Page 30]


Internet-Draft        ForCES Interoperability Draft          August 2009


12.  Security Considerations

   Section 9 of the FE-protocol [I-D.ietf-forces-protocol] specifies
   security considerations of the ForCES protocol.  For this
   interoperability test, no security MUST be chosen even for the
   distributed architecture.













































Haleplidis, et al.      Expires February 8, 2010               [Page 31]


Internet-Draft        ForCES Interoperability Draft          August 2009


13.  References

13.1.  Normative References

   [I-D.ietf-forces-model]
              Halpern, J. and J. Salim, "ForCES Forwarding Element
              Model", draft-ietf-forces-model-16 (work in progress),
              October 2008.

   [I-D.ietf-forces-protocol]
              Dong, L., Doria, A., Gopal, R., HAAS, R., Salim, J.,
              Khosravi, H., and W. Wang, "ForCES Protocol
              Specification", draft-ietf-forces-protocol-22 (work in
              progress), March 2009.

   [I-D.ietf-forces-sctptml]
              Salim, J. and K. Ogawa, "SCTP based TML (Transport Mapping
              Layer) for ForCES protocol", draft-ietf-forces-sctptml-02
              (work in progress), January 2009.

13.2.  Informative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2629]  Rose, M., "Writing I-Ds and RFCs using XML", RFC 2629,
              June 1999.

   [RFC3552]  Rescorla, E. and B. Korver, "Guidelines for Writing RFC
              Text on Security Considerations", BCP 72, RFC 3552,
              July 2003.

   [RFC3654]  Khosravi, H. and T. Anderson, "Requirements for Separation
              of IP Control and Forwarding", RFC 3654, November 2003.

   [RFC3746]  Yang, L., Dantu, R., Anderson, T., and R. Gopal,
              "Forwarding and Control Element Separation (ForCES)
              Framework", RFC 3746, April 2004.

   [RFC5226]  Narten, T. and H. Alvestrand, "Guidelines for Writing an
              IANA Considerations Section in RFCs", BCP 26, RFC 5226,
              May 2008.

   [ethereal]
              "Ethereal is a protocol analyzer. The specific ethereal
              that will be used is an updated Ethereal, by Fenggen Jia,
              that can analyze and decode the ForCES protocol
              messages.", <http://peach.ease.lsoft.com/scripts/



Haleplidis, et al.      Expires February 8, 2010               [Page 32]


Internet-Draft        ForCES Interoperability Draft          August 2009


              wa.exe?A2=ind0906&L=FORCES&T=0&F=&S=&P=1048>.

   [tcpdump]  "Tcpdump is a linux protocol analyzer. The specific
              tcpdump that will be used is a modified tcpdump, by Jamal
              Hadi Salim, that can analyze and decode the ForCES
              protocol messages.", <http://peach.ease.lsoft.com/scripts/
              wa.exe?A2=ind0906&L=FORCES&T=0&F=&S=&P=2262>.












































Haleplidis, et al.      Expires February 8, 2010               [Page 33]


Internet-Draft        ForCES Interoperability Draft          August 2009


Authors' Addresses

   Evangelos Haleplidis
   University of Patras
   Patras,
   Greece

   Email: ehalep@ece.upatras.gr


   Kentaro Ogawa
   NTT Corporation
   Tokyo,
   Japan

   Email: ogawa.kentaro@lab.ntt.co.jp


   Xin-ping Wang
   Huawei Technologies Co., Ltd.
   China

   Email: carly.wang@huawei.com


   Chuanhuang Li
   Zhejiang Gongshang University
   18, Xuezheng Str., Xiasha University Town
   Hangzhou,   310018
   P.R.China

   Phone: +86-571-28877751
   Email: chuanhuang_li@pop.zjgsu.edu.cn


















Haleplidis, et al.      Expires February 8, 2010               [Page 34]