[Search] [pdf|bibtex] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 02 rfc6053                                 Informational
Internet Engineering Task Force                            E. Haleplidis
Internet-Draft                                      University of Patras
Intended status: Informational                                  K. Ogawa
Expires: December 29, 2010                               NTT Corporation
                                                                 W. Wang
                                           Zhejiang Gongshang University
                                                           J. Hadi Salim
                                                       Mojatatu Networks
                                                           June 27, 2010


                    Implementation Report for ForCES
               draft-ietf-forces-implementation-report-02

Abstract

   Forwarding and Control Element Separation (ForCES) defines an
   architectural framework and associated protocols to standardize
   information exchange between the control plane and the forwarding
   plane in a ForCES Network Element (ForCES NE).  RFC3654 has defined
   the ForCES requirements, and RFC3746 has defined the ForCES
   framework.

   This document is an implementation report of the ForCES Protocol,
   Model and SCTP-TML, including the report on interoperability testing
   and the current state of ForCES implementations.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on December 29, 2010.

Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors.  All rights reserved.



Haleplidis, et al.      Expires December 29, 2010               [Page 1]


Internet-Draft      Implementation Report for ForCES           June 2010


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.


Table of Contents

   1.  Terminology and Conventions  . . . . . . . . . . . . . . . . .  4
     1.1.  Requirements Language  . . . . . . . . . . . . . . . . . .  4
     1.2.  Definitions  . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  6
     2.1.  ForCES Protocol  . . . . . . . . . . . . . . . . . . . . .  6
     2.2.  ForCES Model . . . . . . . . . . . . . . . . . . . . . . .  6
     2.3.  Transport mapping layer  . . . . . . . . . . . . . . . . .  6
   3.  Summary  . . . . . . . . . . . . . . . . . . . . . . . . . . .  7
   4.  Methodology  . . . . . . . . . . . . . . . . . . . . . . . . .  8
   5.  Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . .  9
   6.  Detail Section . . . . . . . . . . . . . . . . . . . . . . . . 10
     6.1.  Implementation Experience  . . . . . . . . . . . . . . . . 10
       6.1.1.  ForCES Protocol Features . . . . . . . . . . . . . . . 10
         6.1.1.1.  Protocol Messages  . . . . . . . . . . . . . . . . 11
         6.1.1.2.  MainHeader Handling  . . . . . . . . . . . . . . . 12
         6.1.1.3.  TLV Handling . . . . . . . . . . . . . . . . . . . 13
         6.1.1.4.  Operation Types Supported  . . . . . . . . . . . . 14
         6.1.1.5.  ForCES Protocol Advanced Features  . . . . . . . . 15
       6.1.2.  ForCES Model Features  . . . . . . . . . . . . . . . . 15
         6.1.2.1.  Basic Atomic Types Supported . . . . . . . . . . . 16
         6.1.2.2.  Compound Types Supported . . . . . . . . . . . . . 17
         6.1.2.3.  LFBs Supported . . . . . . . . . . . . . . . . . . 17
       6.1.3.  ForCES SCTP-TML Features . . . . . . . . . . . . . . . 20
         6.1.3.1.  TML Priority Ports . . . . . . . . . . . . . . . . 20
         6.1.3.2.  Message Handling at specific priorities  . . . . . 21
         6.1.3.3.  TML Security Feature . . . . . . . . . . . . . . . 22
     6.2.  Interoperability Report  . . . . . . . . . . . . . . . . . 22
       6.2.1.  Scenarios  . . . . . . . . . . . . . . . . . . . . . . 23
         6.2.1.1.  Scenario 1 - Pre-association Setup . . . . . . . . 23
         6.2.1.2.  Scenario 2 - TML priority channels connection  . . 24
         6.2.1.3.  Scenario 3 - Association Setup - Association
                   Complete . . . . . . . . . . . . . . . . . . . . . 24
         6.2.1.4.  Scenario 4 - CE query  . . . . . . . . . . . . . . 25
         6.2.1.5.  Scenario 5 - Heartbeat monitoring  . . . . . . . . 25
         6.2.1.6.  Scenario 6 - Simple Config Command . . . . . . . . 25



Haleplidis, et al.      Expires December 29, 2010               [Page 2]


Internet-Draft      Implementation Report for ForCES           June 2010


         6.2.1.7.  Scenario 7 - Association Teardown  . . . . . . . . 26
       6.2.2.  Tested Features  . . . . . . . . . . . . . . . . . . . 26
         6.2.2.1.  ForCES Protocol Features . . . . . . . . . . . . . 26
         6.2.2.2.  ForCES Model Features  . . . . . . . . . . . . . . 28
         6.2.2.3.  ForCES SCTP-TML Features . . . . . . . . . . . . . 31
       6.2.3.  Interoperability Results . . . . . . . . . . . . . . . 32
   7.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 34
   8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 35
   9.  Security Considerations  . . . . . . . . . . . . . . . . . . . 36
   10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 37
     10.1. Normative References . . . . . . . . . . . . . . . . . . . 37
     10.2. Informative References . . . . . . . . . . . . . . . . . . 37
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38






































Haleplidis, et al.      Expires December 29, 2010               [Page 3]


Internet-Draft      Implementation Report for ForCES           June 2010


1.  Terminology and Conventions

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

1.2.  Definitions

   This document follows the terminology defined by the ForCES
   Requirements in [RFC3654] and by the ForCES framework in [RFC3746].
   The definitions below are repeated below for clarity.

      Control Element (CE) - A logical entity that implements the ForCES
      protocol and uses it to instruct one or more FEs on how to process
      packets.  CEs handle functionality such as the execution of
      control and signaling protocols.

      Forwarding Element (FE) - A logical entity that implements the
      ForCES protocol.  FEs use the underlying hardware to provide per-
      packet processing and handling as directed/controlled by one or
      more CEs via the ForCES protocol.

      LFB (Logical Function Block) - The basic building block that is
      operated on by the ForCES protocol.  The LFB is a well defined,
      logically separable functional block that resides in an FE and is
      controlled by the CE via ForCES protocol.  The LFB may reside at
      the FE's datapath and process packets or may be purely an FE
      control or configuration entity that is operated on by the CE.
      Note that the LFB is a functionally accurate abstraction of the
      FE's processing capabilities, but not a hardware-accurate
      representation of the FE implementation.

      LFB Class and LFB Instance - LFBs are categorized by LFB Classes.
      An LFB Instance represents an LFB Class (or Type) existence.
      There may be multiple instances of the same LFB Class (or Type) in
      an FE.  An LFB Class is represented by an LFB Class ID, and an LFB
      Instance is represented by an LFB Instance ID.  As a result, an
      LFB Class ID associated with an LFB Instance ID uniquely specifies
      an LFB existence.

      LFB Metadata - Metadata is used to communicate per-packet state
      from one LFB to another, but is not sent across the network.  The
      FE model defines how such metadata is identified, produced and
      consumed by the LFBs.  It defines the functionality but not how
      metadata is encoded within an implementation.




Haleplidis, et al.      Expires December 29, 2010               [Page 4]


Internet-Draft      Implementation Report for ForCES           June 2010


      LFB Components - Operational parameters of the LFBs that must be
      visible to the CEs are conceptualized in the FE model as the LFB
      components.  The LFB components include, for example, flags,
      single parameter arguments, complex arguments, and tables that the
      CE can read and/or Components write via the ForCES protocol (see
      below).

      ForCES Protocol - While there may be multiple protocols used
      within the overall ForCES architecture, the term "ForCES protocol"
      and "protocol" refer to the Fp reference points in the ForCES
      Framework in [RFC3746].  This protocol does not apply to CE-to-CE
      communication, FE-to-FE communication, or to communication between
      FE and CE managers.  Basically, the ForCES protocol works in a
      master- slave mode in which FEs are slaves and CEs are masters.
      This document defines the specifications for this ForCES protocol.

      ForCES Protocol Transport Mapping Layer (ForCES TML) - A layer in
      ForCES protocol architecture that uses the capabilities of
      existing transport protocols to specifically address protocol
      message transportation issues, such as how the protocol messages
      are mapped to different transport media (like TCP, IP, ATM,
      Ethernet, etc), and how to achieve and implement reliability,
      multicast, ordering, etc.  The ForCES TML specifications are
      detailed in separate ForCES documents, one for each TML.



























Haleplidis, et al.      Expires December 29, 2010               [Page 5]


Internet-Draft      Implementation Report for ForCES           June 2010


2.  Introduction

   This is an implementation report for the ForCES protocol, model and
   SCTP-TML documents and includes an interoperability report.

   It follows the outline suggested by [RFC5657].

   Forwarding and Control Element Separation (ForCES) defines an
   architectural framework and associated protocols to standardize
   information exchange between the control plane and the forwarding
   plane in a ForCES Network Element (ForCES NE).  [RFC3654] has defined
   the ForCES requirements, and [RFC3746] has defined the ForCES
   framework.

2.1.  ForCES Protocol

   The ForCES protocol works in a master-slave mode in which FEs are
   slaves and CEs are masters.  The protocol includes commands for
   transport of Logical Function Block (LFB) configuration information,
   association setup, status, and event notifications, etc.  The reader
   is encouraged to read the ForCES Protocol [RFC5810] for further
   information.

2.2.  ForCES Model

   The ForCES Model [RFC5812] presents a formal way to define FE Logical
   Function Blocks (LFBs) using XML.  LFB configuration components,
   capabilities, and associated events are defined when the LFB is
   formally created.  The LFBs within the FE are accordingly controlled
   in a standardized way by the ForCES protocol.

2.3.  Transport mapping layer

   The TML transports the PL messages.  The TML is where the issues of
   how to achieve transport level reliability, congestion control,
   multicast, ordering, etc. are handled.  All ForCES Protocol Layer
   implementations MUST be portable across all TMLs.  Although more than
   one TML may be standardized for the ForCES Protocol, all
   implementations MUST implement the SCTP-TML [RFC5811].












Haleplidis, et al.      Expires December 29, 2010               [Page 6]


Internet-Draft      Implementation Report for ForCES           June 2010


3.  Summary

   The authors attest that the ForCES Protocol, Model and SCTP-TML meet
   the requirements for Draft Standard.

   Three independent implementations, NTT Japan, University of Patras
   and Zhejiang Gongshang University, were surveyed and found to already
   implement all the major features.  All implementors mentioned they
   will be implementing all missing features in the future.

   An interop test was conducted in July, 2009 for all three
   implementations.  Two other organizations, Mojatatu Networks and
   Hangzhou Baud Information and Networks Technology Corporation, which
   independently extended two different well known public domain
   protocol analyzers, Ethereal/Wireshark and tcpdump, also participated
   in the interop for a total of five independent organizations
   implementing.  The two protocol analyzers were used to verify
   validity of ForCEs protocol messages (and in some cases semantics).

   There were no notable difficulties in the interoperability test and
   almost all issues were code bugs that were dealt with mostly on site
   and tests repeated successfully as stated in Section 6.2.3.





























Haleplidis, et al.      Expires December 29, 2010               [Page 7]


Internet-Draft      Implementation Report for ForCES           June 2010


4.  Methodology

   This report has both an implementation experience survey as well as
   the results of the interoperability test.

   The survey information was gathered after implementors answered a
   brief questionnaire with all ForCES Protocol, Model and SCTP-TML
   features.  The results can be seen in Section 6.1

   The interoperability results were part of the interoperability test.
   Extended Ethereal and extended Tcpdump were used to verify the
   results.  The results can be seen in Section 6.2







































Haleplidis, et al.      Expires December 29, 2010               [Page 8]


Internet-Draft      Implementation Report for ForCES           June 2010


5.  Exceptions

   The core features of the ForCES Protocol, Model and SCTP-TML have
   been implemented and tested in an interop in July, 2009.  The
   intention of the interop testing was to validate that all the main
   features of the 3 core documents were inter-operable amongst
   different implementations.  The tested features can be seen in
   Section 6.2.2.

   Different organizations surveyed have implemented certain features
   but not others.  This approach is driven by presence of different
   LFBs the different organizations have currently implemented.  All
   organizations surveyed have indicated intention to implement all
   outstanding features in due time.  The implemented features can be
   seen in Section 6.1.

   The mandated TML security requirement, IPSec, was not validated
   during the interop and is not discussed in this document.  Since
   IPSec is well known and widely deployed not testing in presence of
   IPSec does not invalidate the tests done.  Note that Section 6.1.3.3
   indicates that none of the implementations reporting included support
   for IPSec, but all indicated their intention to implement.

   Although the SCTP priority ports have been changed since the
   interoperability test with the latest SCTP-TML draft, the change has
   no impact in the validity of the interoperability test.

























Haleplidis, et al.      Expires December 29, 2010               [Page 9]


Internet-Draft      Implementation Report for ForCES           June 2010


6.  Detail Section

6.1.  Implementation Experience

   Three different organizations have implemented the ForCES Protocol,
   Model and SCTP-TML and answered a questionnaire.  These are:

   o  NTT Japan.

   o  University of Patras.

   o  Zhejiang Gongshang University.

   Also, not actual implementations, but extensions on protocol
   analyzers capable of understanding ForCES protocol messages, also are
   considered part of an implementation as they can offer validation of
   exchanged protocol messages.  Two such extensions have been created:

   o  Extension to Ethereal/Wireshark [ethereal].

   o  Extension to Tcpdump [tcpdump].

   All implementors were asked regarding the ForCES features they have
   implemented.  For every item listed the respondents indicated whether
   they had implemented, will implement, or won't implement at all.

6.1.1.  ForCES Protocol Features
























Haleplidis, et al.      Expires December 29, 2010              [Page 10]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.1.1.  Protocol Messages

   +------------------+-------------+---------------+------------------+
   | Protocol Message |  NTT Japan  | University of |     Zhejiang     |
   |                  |             |     Patras    |     Gongshang    |
   |                  |             |               |    University    |
   +------------------+-------------+---------------+------------------+
   |    Association   | Implemented |  Implemented  |    Implemented   |
   |       Setup      |             |               |                  |
   |                  |             |               |                  |
   |    Association   | Implemented |  Implemented  |    Implemented   |
   |  Setup Response  |             |               |                  |
   |                  |             |               |                  |
   |    Association   | Implemented |  Implemented  |    Implemented   |
   |     TearDown     |             |               |                  |
   |                  |             |               |                  |
   |   Configuration  | Implemented |  Implemented  |    Implemented   |
   |                  |             |               |                  |
   |   Configuration  | Implemented |  Implemented  |    Implemented   |
   |     Response     |             |               |                  |
   |                  |             |               |                  |
   |       Query      | Implemented |  Implemented  |    Implemented   |
   |                  |             |               |                  |
   |  Query Response  | Implemented |  Implemented  |    Implemented   |
   |                  |             |               |                  |
   |       Event      | Implemented |      Will     |    Implemented   |
   |   Notification   |             |   Implement   |                  |
   |                  |             |               |                  |
   |  Packet Redirect | Implemented |      Will     |    Implemented   |
   |                  |             |   Implement   |                  |
   |                  |             |               |                  |
   |     HeartBeat    | Implemented |  Implemented  |    Implemented   |
   +------------------+-------------+---------------+------------------+

                          ForCES Protocol Message
















Haleplidis, et al.      Expires December 29, 2010              [Page 11]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.1.2.  MainHeader Handling

   +---------------+-------------+----------------+--------------------+
   |  Header Field |  NTT Japan  |  University of | Zhejiang Gongshang |
   |               |             |     Patras     |     University     |
   +---------------+-------------+----------------+--------------------+
   |   Correlator  | Implemented |   Implemented  |     Implemented    |
   |               |             |                |                    |
   |  Acknowledge  | Implemented |   Implemented  |     Implemented    |
   |      Flag     |             |                |                    |
   |               |             |                |                    |
   | Priority Flag |     Will    |   Implemented  |     Implemented    |
   |               |  Implement  |                |                    |
   |               |             |                |                    |
   |   Execution   |     Will    | Will Implement |     Implemented    |
   |   Mode Flag   |  Implement  |                |                    |
   |               |             |                |                    |
   |  Atomic Flag  |     Will    | Will Implement |     Implemented    |
   |               |  Implement  |                |                    |
   |               |             |                |                    |
   |  Transaction  |     Will    | Will Implement |     Implemented    |
   |      Flag     |  Implement  |                |                    |
   +---------------+-------------+----------------+--------------------+

                            MainHeader Handling


























Haleplidis, et al.      Expires December 29, 2010              [Page 12]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.1.3.  TLV Handling

   +------------------+-------------+--------------+-------------------+
   |        TLV       |  NTT Japan  |  University  |      Zhejiang     |
   |                  |             |   of Patras  |     Gongshang     |
   |                  |             |              |     University    |
   +------------------+-------------+--------------+-------------------+
   |   Redirect TLV   | Implemented |     Will     |    Implemented    |
   |                  |             |   Implement  |                   |
   |                  |             |              |                   |
   |    Association   | Implemented |  Implemented |    Implemented    |
   | Setup Result TLV |             |              |                   |
   |                  |             |              |                   |
   |    Association   | Implemented |  Implemented |    Implemented    |
   |  TearDown Reason |             |              |                   |
   |        TLV       |             |              |                   |
   |                  |             |              |                   |
   |  LFBSelector TLV | Implemented |  Implemented |    Implemented    |
   |                  |             |              |                   |
   |   Operation TLV  | Implemented |  Implemented |    Implemented    |
   |                  |             |              |                   |
   |   PathData TLV   | Implemented |  Implemented |    Implemented    |
   |                  |             |              |                   |
   |    KeyInfo TLV   |     Will    |     Will     |    Implemented    |
   |                  |  Implement  |   Implement  |                   |
   |                  |             |              |                   |
   |   FullData TLV   | Implemented |  Implemented |    Implemented    |
   |                  |             |              |                   |
   |  SparseData TLV  |     Will    |     Will     |    Implemented    |
   |                  |  Implement  |   Implement  |                   |
   |                  |             |              |                   |
   |        ILV       |     Will    |     Will     |    Implemented    |
   |                  |  Implement  |   Implement  |                   |
   |                  |             |              |                   |
   |   Metadata TLV   |     Will    |     Will     |    Implemented    |
   |                  |  Implement  |   Implement  |                   |
   |                  |             |              |                   |
   |    Result TLV    | Implemented |  Implemented |    Implemented    |
   |                  |             |              |                   |
   |   Redirect Data  | Implemented |     Will     |    Implemented    |
   |        TLV       |             |   Implement  |                   |
   +------------------+-------------+--------------+-------------------+

                              TLVs Supported







Haleplidis, et al.      Expires December 29, 2010              [Page 13]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.1.4.  Operation Types Supported

   +--------------+-------------+-----------------+--------------------+
   |   Operation  |  NTT Japan  |  University of  | Zhejiang Gongshang |
   |              |             |      Patras     |     University     |
   +--------------+-------------+-----------------+--------------------+
   |      Set     | Implemented |   Implemented   |     Implemented    |
   |              |             |                 |                    |
   |   Set Prop   |     Will    |  Will Implement |     Implemented    |
   |              |  Implement  |                 |                    |
   |              |             |                 |                    |
   | Set Response | Implemented |   Implemented   |     Implemented    |
   |              |             |                 |                    |
   |   Set Prop   |     Will    |  Will Implement |     Implemented    |
   |   Response   |  Implement  |                 |                    |
   |              |             |                 |                    |
   |      Del     | Implemented |  Will Implement |     Implemented    |
   |              |             |                 |                    |
   | Del Response | Implemented |  Will Implement |     Implemented    |
   |              |             |                 |                    |
   |      Get     | Implemented |   Implemented   |     Implemented    |
   |              |             |                 |                    |
   |   Get Prop   |     Will    |  Will Implement |     Implemented    |
   |              |  Implement  |                 |                    |
   |              |             |                 |                    |
   | Get Response | Implemented |   Implemented   |     Implemented    |
   |              |             |                 |                    |
   |   Get Prop   |     Will    |  Will Implement |     Implemented    |
   |   Response   |  Implement  |                 |                    |
   |              |             |                 |                    |
   |    Report    | Implemented |   Implemented   |     Implemented    |
   |              |             |                 |                    |
   |    Commit    |     Will    |  Will Implement |     Implemented    |
   |              |  Implement  |                 |                    |
   |              |             |                 |                    |
   |    Commit    |     Will    |  Will Implement |     Implemented    |
   |   Response   |  Implement  |                 |                    |
   |              |             |                 |                    |
   |    TRComp    |     Will    |  Will Implement |     Implemented    |
   |              |  Implement  |                 |                    |
   +--------------+-------------+-----------------+--------------------+

                         Operation Type Supported








Haleplidis, et al.      Expires December 29, 2010              [Page 14]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.1.5.  ForCES Protocol Advanced Features

   +---------------+-------------+----------------+--------------------+
   |    Feature    |  NTT Japan  |  University of | Zhejiang Gongshang |
   |               |             |     Patras     |     University     |
   +---------------+-------------+----------------+--------------------+
   |  Execute Mode |     Will    | Will Implement |     Implemented    |
   |               |  Implement  |                |                    |
   |               |             |                |                    |
   |  Transaction  |     Will    | Will Implement |     Implemented    |
   |               |  Implement  |                |                    |
   |               |             |                |                    |
   |    Batching   |     Will    |   Implemented  |     Implemented    |
   |               |  Implement  |                |                    |
   |               |             |                |                    |
   |    Command    |     Will    | Will Implement |   Will Implement   |
   |   Pipelining  |  Implement  |                |                    |
   |               |             |                |                    |
   |   HeartBeats  | Implemented |   Implemented  |     Implemented    |
   +---------------+-------------+----------------+--------------------+

                     ForCES Protocol Advanced Features

6.1.2.  ForCES Model Features



























Haleplidis, et al.      Expires December 29, 2010              [Page 15]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.2.1.  Basic Atomic Types Supported

   +----------------+-------------+---------------+--------------------+
   |   Atomic Type  |  NTT Japan  | University of | Zhejiang Gongshang |
   |                |             |     Patras    |     University     |
   +----------------+-------------+---------------+--------------------+
   |      char      | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |      uchar     | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   |      int16     | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |     uint16     | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |      int32     | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   |     uint32     | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   |      int16     | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |     uint64     | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |     boolean    | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   |    string[N]   | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   |     string     | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   |     byte[N]    | Implemented |  Implemented  |     Implemented    |
   |                |             |               |                    |
   | octetstring[N] | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |     float32    | Implemented |  Implemented  |   Will Implement   |
   |                |             |               |                    |
   |     float64    | Implemented |  Implemented  |   Will Implement   |
   +----------------+-------------+---------------+--------------------+

                       Basic Atomic Types Supported













Haleplidis, et al.      Expires December 29, 2010              [Page 16]


Internet-Draft      Implementation Report for ForCES           June 2010


6.1.2.2.  Compound Types Supported

   +------------+-------------+-----------------+----------------------+
   |  Compound  |  NTT Japan  |  University of  |  Zhejiang Gongshang  |
   |    Type    |             |      Patras     |      University      |
   +------------+-------------+-----------------+----------------------+
   |   structs  | Implemented |   Implemented   |      Implemented     |
   |            |             |                 |                      |
   |   arrays   | Implemented |   Implemented   |      Implemented     |
   +------------+-------------+-----------------+----------------------+

                         Compound Types Supported

6.1.2.3.  LFBs Supported

6.1.2.3.1.  FE Protocol LFB

   +------------------+-------------+----------------+-----------------+
   |     Protocol     |  NTT Japan  |  University of |     Zhejiang    |
   |     DataTypes    |             |     Patras     |    Gongshang    |
   |                  |             |                |    University   |
   +------------------+-------------+----------------+-----------------+
   |    CEHBPolicy    | Implemented |   Implemented  |   Implemented   |
   |                  |             |                |                 |
   |    FEHIBPolicy   | Implemented |   Implemented  |   Implemented   |
   |                  |             |                |                 |
   |  FERestarPolicy  | Implemented |   Implemented  |   Implemented   |
   |                  |             |                |                 |
   | CEFailoverPolicy | Implemented |   Implemented  |   Implemented   |
   |                  |             |                |                 |
   |     FEHACapab    | Implemented |   Implemented  |  Will Implement |
   +------------------+-------------+----------------+-----------------+

                         FE Protocol LFB Datatypes

















Haleplidis, et al.      Expires December 29, 2010              [Page 17]


Internet-Draft      Implementation Report for ForCES           June 2010


   +-----------------------+-------------+-------------+---------------+
   |  Protocol Components  |  NTT Japan  |  University |    Zhejiang   |
   |                       |             |  of Patras  |   Gongshang   |
   |                       |             |             |   University  |
   +-----------------------+-------------+-------------+---------------+
   | CurrentRunningVersion | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |          FEID         | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |     MulticastFEIDs    | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |       CEHBPolicy      | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |         CEHDI         | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |       FEHBPolicy      | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |          FEHI         | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |          CEID         | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |       BackupCEs       | Implemented |     Will    |      Will     |
   |                       |             |  Implement  |   Implement   |
   |                       |             |             |               |
   |    CEFailoverPolicy   | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |         CEFTI         | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |    FERestartPolicy    | Implemented | Implemented |      Will     |
   |                       |             |             |   Implement   |
   |                       |             |             |               |
   |        LastCEID       | Implemented | Implemented |      Will     |
   |                       |             |             |   Implement   |
   +-----------------------+-------------+-------------+---------------+

                        FE Protocol LFB Components

   +---------------------+-------------+-------------+-----------------+
   |     Capabilities    |  NTT Japan  |  University |     Zhejiang    |
   |                     |             |  of Patras  |    Gongshang    |
   |                     |             |             |    University   |
   +---------------------+-------------+-------------+-----------------+
   | SupportableVersions | Implemented | Implemented |   Implemented   |
   |                     |             |             |                 |
   |    HACapabilities   | Implemented | Implemented |  Will Implement |
   +---------------------+-------------+-------------+-----------------+

                          Capabilities Supported



Haleplidis, et al.      Expires December 29, 2010              [Page 18]


Internet-Draft      Implementation Report for ForCES           June 2010


   +---------------+------------+----------------+---------------------+
   |     Events    |  NTT Japan |  University of |  Zhejiang Gongshang |
   |               |            |     Patras     |      University     |
   +---------------+------------+----------------+---------------------+
   | PrimaryCEDown |    Will    | Will Implement |    Will Implement   |
   |               |  Implement |                |                     |
   +---------------+------------+----------------+---------------------+

                             Events Supported

6.1.2.3.2.  FE Object LFB

   +-------------------------+-------------+-------------+-------------+
   |     Object DataTypes    |  NTT Japan  |  University |   Zhejiang  |
   |                         |             |  of Patras  |  Gongshang  |
   |                         |             |             |  University |
   +-------------------------+-------------+-------------+-------------+
   |    LFBAdjacencyLimit    | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   |    PortGroupLimitType   | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   |     SupportedLFBType    | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   |      FEStateValues      | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   | FEConfiguredeighborType | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   | FEConfiguredeighborType | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   |     LFBSelectorType     | Implemented | Implemented | Implemented |
   |                         |             |             |             |
   |       LFBLinkType       | Implemented | Implemented | Implemented |
   +-------------------------+-------------+-------------+-------------+

                          FE Object LFB Datatypes
















Haleplidis, et al.      Expires December 29, 2010              [Page 19]


Internet-Draft      Implementation Report for ForCES           June 2010


   +--------------+-------------+----------------+---------------------+
   |    Object    |  NTT Japan  |  University of |  Zhejiang Gongshang |
   |  Components  |             |     Patras     |      University     |
   +--------------+-------------+----------------+---------------------+
   |  LFBTopology | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   | LFBSelectors | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   |    FEName    | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   |     FEID     | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   |   FEVendor   | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   |    FEModel   | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   |    FEState   | Implemented |   Implemented  |     Implemented     |
   |              |             |                |                     |
   |  FENeighbors | Implemented |   Implemented  |     Implemented     |
   +--------------+-------------+----------------+---------------------+

                         FE Object LFB Components

   +-----------------------+-------------+-------------+---------------+
   |      Capabilities     |  NTT Japan  |  University |    Zhejiang   |
   |                       |             |  of Patras  |   Gongshang   |
   |                       |             |             |   University  |
   +-----------------------+-------------+-------------+---------------+
   | ModifiableLFBTopology | Implemented | Implemented |  Implemented  |
   |                       |             |             |               |
   |     SupportedLFBs     | Implemented | Implemented |  Implemented  |
   +-----------------------+-------------+-------------+---------------+

                          Capabilities Supported

6.1.3.  ForCES SCTP-TML Features

6.1.3.1.  TML Priority Ports

   +----------------+-------------+---------------+--------------------+
   |      Port      |  NTT Japan  | University of | Zhejiang Gongshang |
   |                |             |     Patras    |     University     |
   +----------------+-------------+---------------+--------------------+
   |  High priority | Implemented |  Implemented  |     Implemented    |
   |     (6700)     |             |               |                    |
   |                |             |               |                    |





Haleplidis, et al.      Expires December 29, 2010              [Page 20]


Internet-Draft      Implementation Report for ForCES           June 2010


   |     Medium     | Implemented |  Implemented  |     Implemented    |
   |    priority    |             |               |                    |
   |     (6701)     |             |               |                    |
   |                |             |               |                    |
   |  Low priority  | Implemented |  Implemented  |     Implemented    |
   |     (6702)     |             |               |                    |
   +----------------+-------------+---------------+--------------------+

                              Priority Ports

6.1.3.2.  Message Handling at specific priorities

   +------------------+-------------+---------------+------------------+
   |  ForCES Message  |  NTT Japan  | University of |     Zhejiang     |
   |                  |             |     Patras    |     Gongshang    |
   |                  |             |               |    University    |
   +------------------+-------------+---------------+------------------+
   |    Association   | Implemented |  Implemented  |    Implemented   |
   |       Setup      |             |               |                  |
   |                  |             |               |                  |
   |    Association   | Implemented |  Implemented  |    Implemented   |
   |  Setup Response  |             |               |                  |
   |                  |             |               |                  |
   |    Association   | Implemented |  Implemented  |    Implemented   |
   |     Teardown     |             |               |                  |
   |                  |             |               |                  |
   |      Config      | Implemented |  Implemented  |    Implemented   |
   |                  |             |               |                  |
   |  Config Response | Implemented |  Implemented  |    Implemented   |
   |                  |             |               |                  |
   |       Query      | Implemented |  Implemented  |    Implemented   |
   |                  |             |               |                  |
   |  Query Response  | Implemented |  Implemented  |    Implemented   |
   +------------------+-------------+---------------+------------------+

               Message Handling at High priority (6700) Port

   +---------------+-------------+----------------+--------------------+
   |     ForCES    |  NTT Japan  |  University of | Zhejiang Gongshang |
   |    Message    |             |     Patras     |     University     |
   +---------------+-------------+----------------+--------------------+
   |     Event     | Implemented |   Implemented  |     Implemented    |
   |  Notification |             |                |                    |
   +---------------+-------------+----------------+--------------------+

              Message Handling at Medium priority (6701) Port





Haleplidis, et al.      Expires December 29, 2010              [Page 21]


Internet-Draft      Implementation Report for ForCES           June 2010


   +-------------+-------------+-----------------+---------------------+
   |    ForCES   |  NTT Japan  |  University of  |  Zhejiang Gongshang |
   |   Message   |             |      Patras     |      University     |
   +-------------+-------------+-----------------+---------------------+
   |    Packet   | Implemented |   Implemented   |     Implemented     |
   |   Redirect  |             |                 |                     |
   |             |             |                 |                     |
   |  Heartbeats | Implemented |   Implemented   |     Implemented     |
   +-------------+-------------+-----------------+---------------------+

               Message Handling at Low priority (6702) Port

6.1.3.3.  TML Security Feature

   +--------------+------------+-----------------+---------------------+
   |   Security   |  NTT Japan |  University of  |  Zhejiang Gongshang |
   |    Feature   |            |      Patras     |      University     |
   +--------------+------------+-----------------+---------------------+
   |     IPSec    |    Will    |  Will Implement |    Will Implement   |
   |              |  Implement |                 |                     |
   +--------------+------------+-----------------+---------------------+

                         Security Feature Support

6.2.  Interoperability Report

   The interoperability test took place at the University of Patras, in
   the Department of Electrical and Computer Engineering.

   There were two options to participate in the interoperability test.

   1.  Locally at the University of Patras premises.

   2.  Remotely via internet.

   Implementations from NTT and University of Patras, were present
   locally at the University of Patras premises in Greece, while the
   implementation from Zhejiang Gongshang University, which was behind a
   NAT, connected remotely from China.

   The interoperability test, tested the basic functionality of the
   ForCES protocol, mainly message exchanging and handling.

   The following scenarios were tested.







Haleplidis, et al.      Expires December 29, 2010              [Page 22]


Internet-Draft      Implementation Report for ForCES           June 2010


6.2.1.  Scenarios

   The main goal of the interoperability test was to test the basic
   protocol functionality, the test parameters were limited.

   1.  In the Association Setup Message, all report messages were
       ignored.

   2.  In the Association Setup Phase, the messages, FEO OperEnable
       Event (FE to CE), Config FEO Adminup (CE to FE) and FEO Config-
       Resp (FE to CE) were ignored.  The CEs assumed that the FEs were
       enabled once the LFBSelectors had been queried.

   3.  Only FullDataTLVs were used and not SparseData TLVs.

   4.  There were no transaction operations.

   5.  Each message had only one LFBSelector TLV, one Operation TLV and
       one PathDataTLV per message when these were used.

6.2.1.1.  Scenario 1 - Pre-association Setup

   While the Pre-association setup is not in the ForCES current scope it
   is an essential step before CEs and FEs communicate.  As the first
   part in a successful CE-FE connection the participating CEs and FEs
   had to be able to be configured.

   In the Pre-association Phase the following configuration items were
   setup regarding the CEs:

   o  The CE ID.

   o  The FE IDs that were connected to this CE

   o  The IP of the FEs that connected

   o  The TML priority ports.

   In the Pre-association Phase the following configuration items were
   setup regarding the FEs:

   o  The FE ID.

   o  The CE ID that this FE were connecting to.

   o  The IP of the CE that connected to





Haleplidis, et al.      Expires December 29, 2010              [Page 23]


Internet-Draft      Implementation Report for ForCES           June 2010


   o  The TML priority ports.

6.2.1.2.  Scenario 2 - TML priority channels connection

   For the interoperability test, the SCTP was used as TML.  The TML
   connection with the associating element was needed for the scenario 2
   to be successful.

   Although SCTP-TML [RFC5811] defines 3 priority channels, with
   specific ports:

   o  High priority - Port number: 6704

   o  Medium priority - Port number: 6705

   o  Lower priority - Port number: 6706

   At the time of the interoperability test, the sctp ports of the three
   priority channels were the following:

   o  High priority - Port number: 6700

   o  Medium priority - Port number: 6701

   o  Lower priority - Port number: 6702

   As specified in the exceptions section, this does not invalidate the
   results of the interoperability test.

6.2.1.3.  Scenario 3 - Association Setup - Association Complete

   Once the Pre-association phase had been complete in the previous 2
   scenarios, CEs and FEs would be ready to communicate using the ForCES
   protocol, and enter the Association Setup stage.  In this stage the
   FEs would attempt to join the NE.  The following ForCES protocol
   messages would be exchanged for each CE-FE pair in the specified
   order:

   o  Association Setup Message (from FE to CE)

   o  Association Setup Response Message (from CE to FE)

   o  Query Message: FEO LFBSelectors(from CE to FE)

   o  Query Response: FEO LFBSelectors response (from FE to CE)






Haleplidis, et al.      Expires December 29, 2010              [Page 24]


Internet-Draft      Implementation Report for ForCES           June 2010


6.2.1.4.  Scenario 4 - CE query

   Once the Association Phase stage has been complete, the FEs and CEs
   would enter the Established stage.  In this stage the FE will be
   continuously updated or queried.  The CE should query the FE a
   specific value from the FE Object LFB and from the FE Protocol LFB.
   An example from the FE Protocol LFB is the HeartBeat Timer (FEHI) and
   from the FE Object LFB is the State of the LFB (FEState)

   The following ForCES protocol messages were exchanged:

   o  Query Message

   o  Query Response Message

6.2.1.5.  Scenario 5 - Heartbeat monitoring

   The Heartbeat (HB) Message is used for one ForCES element (FE or CE)
   to asynchronously notify one or more other ForCES elements in the
   same ForCES NE on its liveness.  The default configuration of the
   Heartbeat Policy of the FE is set to 0 which means, that the FE
   should not generate any Heartbeat messages. the CE is responsible for
   checking FE liveness by setting the PL header ACK flag of the message
   it sends to AlwaysACK.  In this Scenario the CE will send a Heartbeat
   message with the ACK flag set to AlwaysACK and the FE should respond.

   The following ForCES protocol messages were exchanged:

   o  Heartbeat Message

6.2.1.6.  Scenario 6 - Simple Config Command

   A config message is sent by the CE to the FE to configure LFB
   components in the FE.  A simple config command easily visible and
   metered would be to change the Heartbeat configuration.  This was
   done in two steps:

   1.  Change the FE Heartbeat Policy (FEHBPolicy) to value 1, to force
       the FE to send heartbeats.

   2.  After some heartbeats from the FE, the FE Heartbeat Interval
       (FEHI) was changed.

   The following ForCES protocol messages were exchanged:

   o  Config Message





Haleplidis, et al.      Expires December 29, 2010              [Page 25]


Internet-Draft      Implementation Report for ForCES           June 2010


   o  Config Response Message

6.2.1.7.  Scenario 7 - Association Teardown

   In the end, the association must be terminated.  There were three
   scenarios by which the association was terminated:

   1.  Normal tear down by exchanging Association Teardown Message

   2.  Irregular tear down by stopping heartbeats from a FE or a CE.

   3.  Irregular tear down by externally shutting down/rebooting a FE or
       a CE.

   All scenarios were tested in the interoperability test.

   The following ForCES protocol messages were exchanged:

   o  Association Teardown Message

6.2.2.  Tested Features

   The features that were tested are:

6.2.2.1.  ForCES Protocol Features

6.2.2.1.1.  Protocol Messages

                      +----------------------------+
                      |      Protocol Message      |
                      +----------------------------+
                      |      Association Setup     |
                      |                            |
                      | Association Setup Response |
                      |                            |
                      |    Association TearDown    |
                      |                            |
                      |        Configuration       |
                      |                            |
                      |   Configuration Response   |
                      |                            |
                      |            Query           |
                      |                            |
                      |       Query Response       |
                      |                            |
                      |          HeartBeat         |
                      +----------------------------+




Haleplidis, et al.      Expires December 29, 2010              [Page 26]


Internet-Draft      Implementation Report for ForCES           June 2010


                          ForCES Protocol Message

   o  PASS: All implementations handled the protocol messages and all
      protocol analyzers captured them.

6.2.2.1.2.  MainHeader Handling

                           +------------------+
                           |   Header Field   |
                           +------------------+
                           |    Correlator    |
                           |                  |
                           | Acknowledge Flag |
                           |                  |
                           |   Priority Flag  |
                           +------------------+

                            MainHeader Handling

   o  PASS: All implementations handled these main header flags and all
      protocol analyzers captured them.

6.2.2.1.3.  TLV Handling

                    +---------------------------------+
                    |               TLV               |
                    +---------------------------------+
                    |   Association Setup Result TLV  |
                    |                                 |
                    | Association TearDown Reason TLV |
                    |                                 |
                    |         LFBSelector TLV         |
                    |                                 |
                    |          Operation TLV          |
                    |                                 |
                    |           PathData TLV          |
                    |                                 |
                    |           FullData TLV          |
                    |                                 |
                    |            Result TLV           |
                    +---------------------------------+

                              TLVs Supported

   o  PASS: All implementations handled these TLVs and all protocol
      analyzers captured them.





Haleplidis, et al.      Expires December 29, 2010              [Page 27]


Internet-Draft      Implementation Report for ForCES           June 2010


6.2.2.1.4.  Operation Types Supported

                             +--------------+
                             |   Operation  |
                             +--------------+
                             |      Set     |
                             |              |
                             | Set Response |
                             |              |
                             |      Get     |
                             |              |
                             | Get Response |
                             |              |
                             |    Report    |
                             +--------------+

                         Operation Type Supported

   o  PASS: All implementations handled these Operations and all
      protocol analyzers captured them.

6.2.2.1.5.  ForCES Protocol Advanced Features

                              +------------+
                              |   Feature  |
                              +------------+
                              |  Batching  |
                              |            |
                              | HeartBeats |
                              +------------+

                     ForCES Protocol Advanced Features

   Although Batching was not initially designed to be tested, it was
   tested during the interoperability test.

   o  PASS: Two implementations handled batching and all handled
      Heartbeats.  The protocol analyzers captured both.

6.2.2.2.  ForCES Model Features











Haleplidis, et al.      Expires December 29, 2010              [Page 28]


Internet-Draft      Implementation Report for ForCES           June 2010


6.2.2.2.1.  Basic Atomic Types Supported

                              +-------------+
                              | Atomic Type |
                              +-------------+
                              |    uchar    |
                              |             |
                              |    uint32   |
                              +-------------+

                       Basic Atomic Types Supported

   o  PASS: All implementations handled these basic atomic types.

6.2.2.2.2.  Compound Types Supported

                             +---------------+
                             | Compound Type |
                             +---------------+
                             |    structs    |
                             |               |
                             |     arrays    |
                             +---------------+

                         Compound Types Supported

   o  PASS: All implementations handled these compound types.

6.2.2.2.3.  LFBs Supported

6.2.2.2.3.1.  FE Protocol LFB

                          +--------------------+
                          | Protocol DataTypes |
                          +--------------------+
                          |     CEHBPolicy     |
                          |                    |
                          |     FEHIBPolicy    |
                          +--------------------+

                         FE Protocol LFB Datatypes

   o  PASS: All implementations handled these FE Protocol LFB Datatypes.








Haleplidis, et al.      Expires December 29, 2010              [Page 29]


Internet-Draft      Implementation Report for ForCES           June 2010


                          +---------------------+
                          | Protocol Components |
                          +---------------------+
                          |         FEID        |
                          |                     |
                          |      CEHBPolicy     |
                          |                     |
                          |        CEHDI        |
                          |                     |
                          |      FEHBPolicy     |
                          |                     |
                          |         FEHI        |
                          |                     |
                          |         CEID        |
                          +---------------------+

                        FE Protocol LFB Components

   o  PASS: All implementations handled these FE Protocol LFB
      Components.

6.2.2.2.3.2.  FE Object LFB

                           +------------------+
                           | Object DataTypes |
                           +------------------+
                           |   FEStateValues  |
                           |                  |
                           |  LFBSelectorType |
                           +------------------+

                          FE Object LFB Datatypes

   o  PASS: All implementations handled these FE Object LFB Datatypes.

                           +-------------------+
                           | Object Components |
                           +-------------------+
                           |    LFBSelectors   |
                           |                   |
                           |      FEState      |
                           +-------------------+

                         FE Object LFB Components

   o  PASS: All implementations handled these FE Object LFB Components.





Haleplidis, et al.      Expires December 29, 2010              [Page 30]


Internet-Draft      Implementation Report for ForCES           June 2010


6.2.2.3.  ForCES SCTP-TML Features

6.2.2.3.1.  TML Priority Ports

                        +------------------------+
                        |          Port          |
                        +------------------------+
                        |  High priority (6700)  |
                        |                        |
                        | Medium priority (6701) |
                        |                        |
                        |   Low priority (6702)  |
                        +------------------------+

                              Priority Ports

   o  PASS: All implementations opened and connected to all the SCTP
      priority ports.  The protocol analyzers captured all ports and
      corresponding priority.

6.2.2.3.2.  Message Handling at specific priorities

                      +----------------------------+
                      |       ForCES Message       |
                      +----------------------------+
                      |      Association Setup     |
                      |                            |
                      | Association Setup Response |
                      |                            |
                      |    Association Teardown    |
                      |                            |
                      |           Config           |
                      |                            |
                      |       Config Response      |
                      |                            |
                      |            Query           |
                      |                            |
                      |       Query Response       |
                      +----------------------------+

               Message Handling at High priority (6700) Port

   o  PASS: All implementations handled these messages at this SCTP
      priority port.  The protocol analyzers captured these messages at
      these priority ports.






Haleplidis, et al.      Expires December 29, 2010              [Page 31]


Internet-Draft      Implementation Report for ForCES           June 2010


                            +----------------+
                            | ForCES Message |
                            +----------------+
                            |   Heartbeats   |
                            +----------------+

               Message Handling at Low priority (6702) Port

   o  PASS: All implementations handled these messages at this SCTP
      priority port.  The protocol analyzers captured these messages at
      these priority ports.

6.2.3.  Interoperability Results

   All implementations were found to be interoperable with each other.

   All scenarios were tested successfully.

   The following issues were found and dealt with.

   1.   Some messages were sent on the wrong priority channels.  There
        were some ambiguities on the SCTP-TML document on how to deal
        with such a situation.  The possibilities were: an FE response
        on the same (wrong) channel as a CE query; on the correctly
        documented channel for the message; or to simply drop the
        packet.  This has been corrected by mandating the message to
        channel mapping to be a MUST in the SCTP-TML document [RFC5811]
        before it was published as an RFC.

   2.   At some point, a CE sent a TearDown message to the FE.  The CE
        expected the FE to shut down the connection, and the FE waited
        the CE to shut down the connection and were caught in a
        deadlock.  This was a code bug and was fixed.

   3.   Sometimes, only when the CE and FE were remote to each other
        (one being in China and another in Greece), the association
        setup message was not received by the CE side and therefore an
        association never completed.  This was not an implementation
        issue, rather it was a network issue.  This issue is solved with
        the retransmission of the non delivered messages.

   4.   An implementation did not take into account that the padding in
        TLVs MUST NOT be included in the length of the TLV.  This was a
        code bug and was fixed.

   5.   EM Flag was set to reserved by a CE and was not ignored by the
        FE.  This was a code bug and was fixed.




Haleplidis, et al.      Expires December 29, 2010              [Page 32]


Internet-Draft      Implementation Report for ForCES           June 2010


   6.   After the FEHBPolicy was set to 1 the FE didn't send any
        HeartBeats.  This was a code bug and was fixed.

   7.   Some FEs sent HeartBeats with the ACK flag with a value other
        than NoACK.  The CE responded.  This was a code bug and was
        fixed.

   8.   When a cable was disconnected, all TML implementation didn't
        detect it.  The association was eventually dropped due to
        heartbeats, this was a success, but this is an implementation
        issue implementers should keep in mind.  This is a SCTP options
        issue.  Nothing was needed to be done.

   9.   A CE crashed due to unknown LFBSelector values.  This was a code
        bug and was fixed.

   10.  With the remote connection from China, which was behind a NAT,
        to Greece there were a lot of ForCES packet retransmission.  The
        problem is that packets like Heartbeats were retransmitted.
        This was an implementation issue regarding SCTP usage
        implementers should keep in mind.  SCTP-PR option was needed to
        be used.  Nothing was needed to be done.

   The interoperability test went so well that an additional extended
   test was added to test for batching messages.  This test was also
   done successfully.

























Haleplidis, et al.      Expires December 29, 2010              [Page 33]


Internet-Draft      Implementation Report for ForCES           June 2010


7.  Acknowledgements

   The authors like to give thanks to Professors Odysseas Koufopavlou
   and Spyros Denazis, and the Department of Electrical and Computer
   Engineering in the University of Patras who hosted the ForCES
   interoperability test.

   Also the authors would like to give thanks to Chuanhuang Li, Ming
   Gao, and other participants from Zhejiang Gongshang University which
   connected remotely.  This allowed the discovery of a series of issues
   that would have been uncaught otherwise.

   The authors would like to thank also Hideaki Iwata and Yoshinobu
   Morimoto for participating locally at the interoperability test and
   also Hiroki Date and Hidefumi Otsuka all part of NTT Japan for
   contributing to the interoperability test.

   Additionally thanks are given to Xinping Wang for her help in writing
   the interoperability draft and Fenggen Jia for extending the Ethereal
   protocol analyzer.































Haleplidis, et al.      Expires December 29, 2010              [Page 34]


Internet-Draft      Implementation Report for ForCES           June 2010


8.  IANA Considerations

   This memo includes no request to IANA.
















































Haleplidis, et al.      Expires December 29, 2010              [Page 35]


Internet-Draft      Implementation Report for ForCES           June 2010


9.  Security Considerations

   No security elements of the protocol or the SCTP TML [RFC5811]
   specification were tested.

   The survey indicated that no security elements were implemented but
   all participants indicated their intention to implement

   For security considerations regarding the ForCES Protocol and the
   SCTP-TML please see [RFC5810] and [RFC5811]









































Haleplidis, et al.      Expires December 29, 2010              [Page 36]


Internet-Draft      Implementation Report for ForCES           June 2010


10.  References

10.1.  Normative References

   [RFC5810]  Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang,
              W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and
              Control Element Separation (ForCES) Protocol
              Specification", RFC 5810, March 2010.

   [RFC5811]  Hadi Salim, J. and K. Ogawa, "SCTP-Based Transport Mapping
              Layer (TML) for the Forwarding and Control Element
              Separation (ForCES) Protocol", RFC 5811, March 2010.

   [RFC5812]  Halpern, J. and J. Hadi Salim, "Forwarding and Control
              Element Separation (ForCES) Forwarding Element Model",
              RFC 5812, March 2010.

10.2.  Informative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC3654]  Khosravi, H. and T. Anderson, "Requirements for Separation
              of IP Control and Forwarding", RFC 3654, November 2003.

   [RFC3746]  Yang, L., Dantu, R., Anderson, T., and R. Gopal,
              "Forwarding and Control Element Separation (ForCES)
              Framework", RFC 3746, April 2004.

   [RFC5657]  Dusseault, L. and R. Sparks, "Guidance on Interoperation
              and Implementation Reports for Advancement to Draft
              Standard", BCP 9, RFC 5657, September 2009.

   [ethereal]
              "Ethereal is a protocol analyzer. The specific ethereal
              that was used is an updated Ethereal, by Fenggen Jia, that
              can analyze and decode the ForCES protocol messages.", <ht
              tp://peach.ease.lsoft.com/scripts/
              wa.exe?A2=ind0906&L=FORCES&T=0&F=&S=&P=1048>.

   [tcpdump]  "Tcpdump is a linux protocol analyzer. The specific
              tcpdump that was used is a modified tcpdump, by Jamal Hadi
              Salim, that can analyze and decode the ForCES protocol
              messages.", <http://peach.ease.lsoft.com/scripts/
              wa.exe?A2=ind0906&L=FORCES&T=0&F=&S=&P=2262>.






Haleplidis, et al.      Expires December 29, 2010              [Page 37]


Internet-Draft      Implementation Report for ForCES           June 2010


Authors' Addresses

   Evangelos Haleplidis
   University of Patras
   Patras,
   Greece

   Email: ehalep@ece.upatras.gr


   Kentaro Ogawa
   NTT Corporation
   Tokyo,
   Japan

   Email: ogawa.kentaro@lab.ntt.co.jp


   Weiming Wang
   Zhejiang Gongshang University
   18, Xuezheng Str., Xiasha University Town
   Hangzhou,   310018
   P.R.China

   Phone: +86-571-28877721
   Email: wmwang@mail.zjgsu.edu.cn


   Jamal Hadi Salim
   Mojatatu Networks
   Ottawa, Ontario,
   Canada

   Phone:
   Email: hadi@mojatatu.com
















Haleplidis, et al.      Expires December 29, 2010              [Page 38]