Skip to main content

Benchmarking Terminology for Protection Performance
RFC 6414

Revision differences

Document history

Date Rev. By Action
2017-05-16
09 (System) Changed document authors from "Scott Poretsky" to "Scott Poretsky, Jay Karthik, Samir Vapiwala, Rajiv Papneja"
2015-10-14
09 (System) Notify list changed from bmwg-chairs@ietf.org, draft-ietf-bmwg-protection-term@ietf.org to (None)
2012-08-22
09 (System) post-migration administrative database adjustment to the No Objection position for Peter Saint-Andre
2012-08-22
09 (System) post-migration administrative database adjustment to the No Objection position for Lars Eggert
2011-11-22
09 Amy Vezza State changed to RFC Published from RFC Ed Queue.
2011-11-21
09 (System) RFC published
2010-08-10
09 (System) IANA Action state changed to No IC from In Progress
2010-08-10
09 (System) IANA Action state changed to In Progress
2010-08-09
09 Cindy Morgan State Changes to RFC Ed Queue from Approved-announcement sent by Cindy Morgan
2010-08-06
09 Cindy Morgan IESG state changed to Approved-announcement sent
2010-08-06
09 Cindy Morgan IESG has approved the document
2010-08-06
09 Cindy Morgan Closed "Approve" ballot
2010-07-12
09 Ron Bonica State Changes to Approved-announcement to be sent from IESG Evaluation::AD Followup by Ron Bonica
2010-07-11
09 Lars Eggert [Ballot Position Update] Position for Lars Eggert has been changed to No Objection from Discuss by Lars Eggert
2010-07-08
09 Peter Saint-Andre [Ballot discuss]
[cleared]
2010-07-08
09 Peter Saint-Andre [Ballot Position Update] Position for Peter Saint-Andre has been changed to No Objection from Discuss by Peter Saint-Andre
2010-07-08
09 (System) Sub state has been changed to AD Follow up from New Id Needed
2010-07-08
09 (System) New version available: draft-ietf-bmwg-protection-term-09.txt
2010-06-20
09 Samuel Weiler Request for Last Call review by SECDIR Completed. Reviewer: Rob Austein.
2010-06-18
09 (System) Removed from agenda for telechat - 2010-06-17
2010-06-17
09 Cindy Morgan State Changes to IESG Evaluation::Revised ID Needed from IESG Evaluation by Cindy Morgan
2010-06-17
09 Dan Romascanu [Ballot comment]
It looks to me that 'Benchmarking Terminology for Performance of sub-IP layer Protection Mechanisms' would be a more apropriate name for this document.
2010-06-17
09 Dan Romascanu [Ballot Position Update] New position, Yes, has been recorded by Dan Romascanu
2010-06-17
09 Gonzalo Camarillo [Ballot Position Update] New position, No Objection, has been recorded by Gonzalo Camarillo
2010-06-17
09 Jari Arkko [Ballot Position Update] New position, No Objection, has been recorded by Jari Arkko
2010-06-16
09 Adrian Farrel [Ballot Position Update] New position, No Objection, has been recorded by Adrian Farrel
2010-06-16
09 Adrian Farrel
[Ballot comment]
I have a huge raft of issues and concerns about this document. In the
end I raised a COMMENT not a DISCUSS because …
[Ballot comment]
I have a huge raft of issues and concerns about this document. In the
end I raised a COMMENT not a DISCUSS because I don't believe that the
publication of this document as an Informational RFC without making
the changes will be harmful. However, attending to these comments would,
I believe significantly improve the value of your work.

The responsible AD may decide that the comments need to be addressed.

---

idnits (http://tools.ietf.org/tools/idnits/) throws up a number of
issues. Although many of these are minor or editorial, it would have
made the document easier to read had you fixed them.

I think that some of the boilerplate issues are more significant and
that the I-D cannot be accepted unless a new revision with the
correct boilerplate is submitted.

The document writeup says:

  (1.g) Has the Document Shepherd personally verified that the
          document satisfies all ID nits? (See
          http://www.ietf.org/ID-Checklist.html and
          http://tools.ietf.org/tools/idnits/). Boilerplate checks are
          not enough; this check needs to be thorough. Has the document
          met all formal review criteria it needs to, such as the MIB
          Doctor, media type and URI type reviews?

There are a few errors in the nits:
http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-bmwg-protection-term-08.txt
These can be corrected after AD-review, but there will be some work
necessary to adopt the new boilerplate, update references, fix
page lengths, etc.

Clearly this did not happen, and I really think we need to push back on
document shepherds to understand that the only acceptable answer to
the question is "Yes, idnits passes cleanly".

---

There is a significant different between the document title
("Benchmarking Terminology for Protection Performance") and the content
of the document as explained in the Abstract and Introduction.
Viz.
  This document provides common terminology and metrics for
  benchmarking the performance of sub-IP layer protection
  mechanisms.

1. The title should mention metrics
2. The title should indicate the scope is sub-IP

---

The Introduction starts off with...

  Technologies
  that function at sub-IP layers can be enabled to provide further
  protection of IP traffic by providing the failure recovery at the
  sub-IP layers so that the outage is not observed at the IP-layer.

This seems a reasonable statement, but isn't it in contradiction with
the whole premise of this document as stated in the Abstract...

  The performance
  benchmarks are measured at the IP-Layer, avoiding dependence on
  specific sub-IP protection mechanisms.

In other words, if the outage is not observed at the IP-layer, how
will you measure the benchmarks at the IP-Layer?

---

I would really have liked this work to be aligned with RFC 4427.
That document sets out terminology for protection and restoration
in GMPLS networks (i.e. sub-IP) and attempts to achieve alignment
with ITU-T Recommendations G.808.1 and G.841.

It is important to use identical rather than similar terms for
protection and restoration techniques across the IETF when we
are referring to sub-IP layers.

Actually, I find the terminology rather woolly. For example, in
Section 1

  The Working Path is the Primary Path prior to the Failover Event and
  the Backup Path after the Failover Event.

Yet in Section 3.1.3
                                                                               
      The Primary Path is the Path that traffic traverses
      prior to a Failover Event.
                                                                               
And it is difficult to read a document where the terms are used well
in advance of their definitions.

It might help if Section 3 was split with sections 3.1 through 3.4
presented as terminology, and sections 3.5 onwards as Test
Considerations.

The definition of "Restoration" in section 3.3.5 is unlike anything
I have seen before and is very much at odds with the way the term is
used in sub-IP networks.

---

Figure 4
There is a spurious arrow head on the left-hand end of the arrow
marked "IP-Layer Forwarding"

---

Section 3.1.1

      a. R1 is the ingress node and forwards IP packets, which input
      into DUT/SUT, to R2 as sub-IP frames over link L12.

      b. Ri is a node which forwards data frames to R[i+1] over Link
      Li[i+1] for all i, 1
2010-06-16
09 Robert Sparks [Ballot Position Update] New position, No Objection, has been recorded by Robert Sparks
2010-06-16
09 Russ Housley [Ballot Position Update] New position, No Objection, has been recorded by Russ Housley
2010-06-16
09 Stewart Bryant
[Ballot comment]
I am not sure that RFC2119 language is appropriate here

  Figures 1 through 5 show models that MAY be used when benchmarking …
[Ballot comment]
I am not sure that RFC2119 language is appropriate here

  Figures 1 through 5 show models that MAY be used when benchmarking
  Sub-IP Protection mechanisms, which MUST use a Protection Switching
  System that consists of a minimum of two Protection-Switching Nodes,
  an Ingress Node known as the Headend Node and an Egress Node known
  as the Merge Node. 

Ideally the RFC2119 definition should appear before first use in the document

  The Protection Switching System MUST include
  either a Primary Path and Backup Path, as shown in Figures 1 through
  4, or a Primary Node and Standby Node, as shown in Figure 5.



I am not a mathematician, can an equation have a pluarity of sub-equations, or is a different mathematical construct needed?

      TBLM as shown in Equation 2:

      (Equation 2)
          (Equation 2a)
          TBLM Failover Time = Time(Failover) - Time(Failover Event)

          (Equation 2b)
          TBLM Reversion Time = Time(Reversion) - Time(Restoration)

Same with Eq3
2010-06-16
09 Stewart Bryant [Ballot Position Update] New position, No Objection, has been recorded by Stewart Bryant
2010-06-16
09 Sean Turner [Ballot Position Update] New position, No Objection, has been recorded by Sean Turner
2010-06-15
09 Tim Polk
[Ballot comment]
I suspect this is clear enough for its intended audience, but a more detailed description of the
calculations in 3.6.1 and 3.6.3 would …
[Ballot comment]
I suspect this is clear enough for its intended audience, but a more detailed description of the
calculations in 3.6.1 and 3.6.3 would help novice readers.  I couldn't quite sort out the
equations.  For example in 3.6.3 (TBM), are the following interpretations correct?

(1) Time(Failover) = Timestamp on first unimpaired packet received at egress node after the
backup path became the working

(2) Time(Failover Event) = Timestamp on the last unimpaired packet received at egress node
on the primary path before failure

I was unable to construct similar statements for the TBLM in 3.6.1 at all.

Nits:

in section 3.1.1, there is a confusing mixture of notation, using parentheses and brackets
interchangeably.  The definition of path uses parentheses, as in "L(n-1)n", but the description
of node in subitem b. uses brackets, as in "Li[i+1]".

In 3.6.3 under discussion, the comma following "observation of unimpaired packets" should be
a period.
2010-06-15
09 Tim Polk
[Ballot comment]
I suspect this is clear enough for its intended audience, but a more detailed description of the
calculations in 3.6.1 and 3.6.3 would …
[Ballot comment]
I suspect this is clear enough for its intended audience, but a more detailed description of the
calculations in 3.6.1 and 3.6.3 would help novice readers.  I couldn't quite sort out the
equations.  For example in 3.6.3 (TBM), are the following interpretations correct?

(1) Time(Failover) = Timestamp on first unimpaired packet received at egress node after the
backup path became the working

(2) Time(Failover Event) = Timestamp on the last unimpaired packet received at egress node
on the primary path before failure

I was unable to construct similar statements for the TBLM in 3.6.1 at all.

Nits:

in section 3.1.1, there is a confusing mixture of notation, using parentheses and brackets
interchangeably.  The definition of path uses parentheses, as in "L(n-1)n", but the description
of node in subitem b. uses brackets, as in "Li[i+1]".

In 3.6.3 under discussion, the comma following "observation of unimpaired packets" should be
a period.
2010-06-15
09 Tim Polk
[Ballot comment]
I suspect this is clear enough for its intended audience, but a more detailed description of the
calculations in 3.6.1 and 3.6.3 would …
[Ballot comment]
I suspect this is clear enough for its intended audience, but a more detailed description of the
calculations in 3.6.1 and 3.6.3 would help novice readers.  I couldn't quite sort out the
equations.  For example in 3.6.3 (TBM), are the following interpretations correct?

(1) Time(Failover) = Timestamp on first unimpaired packet received at egress node after the
failover event

(2) Time(Failover Event) = Timestamp on the last unimpaired packet received at egress node before the failover event

I was unable to construct similar statements for the TBLM in 3.6.1 at all.

Nits:

in section 3.1.1, there is a confusing mixture of notation, using parentheses and brackets
interchangeably.  The definition of path uses parentheses, as in "L(n-1)n", but the description
of node in subitem b. uses brackets, as in "Li[i+1]".

In 3.6.3 under discussion, the comma following "observation of unimpaired packets" should be
a period.
2010-06-15
09 Tim Polk [Ballot Position Update] New position, No Objection, has been recorded by Tim Polk
2010-06-15
09 Peter Saint-Andre
[Ballot discuss]
1. The document does not pass ID-nits.

  http://tools.ietf.org/tools/idnits/

2. This is a very interesting normative reference:

    [6] Not used.

Please …
[Ballot discuss]
1. The document does not pass ID-nits.

  http://tools.ietf.org/tools/idnits/

2. This is a very interesting normative reference:

    [6] Not used.

Please delete that non-reference and renumber accordingly.

3. It appears that references [8] and [9] (RFC 4090 and RFC 2474) are not in fact normative.
2010-06-15
09 Peter Saint-Andre [Ballot Position Update] Position for Peter Saint-Andre has been changed to Discuss from No Objection by Peter Saint-Andre
2010-06-15
09 Peter Saint-Andre [Ballot Position Update] New position, No Objection, has been recorded by Peter Saint-Andre
2010-06-14
09 Lars Eggert
[Ballot comment]
>                    Benchmarking Terminology
>                    for …
[Ballot comment]
>                    Benchmarking Terminology
>                    for Protection Performance

  As stated in the abstract, Section 3.5 actually defines some tests.
  The title should reflect that this document is not only defining
  terminology.


Section 3.1., paragraph 4:
>        b. Ri is a node which forwards data frames to R[i+1] over Link
>        Li[i+1] for all i, 1<i<n, based on information in the sub-IP
>        layer.

  Since R1/L12 are defined in (a) and (c) defines Rn, you probably want
  to change 1
2010-06-14
09 Lars Eggert
[Ballot discuss]
Section 3., paragraph 0:
> 3. Test Considerations

  DISCUSS: Some of the "discussion" sections of the definitions in
  Sections 3.1-3.4 use …
[Ballot discuss]
Section 3., paragraph 0:
> 3. Test Considerations

  DISCUSS: Some of the "discussion" sections of the definitions in
  Sections 3.1-3.4 use RFC2119 terms, which is not appropriate, because
  those paragraphs are clearly informative.
2010-06-14
09 Lars Eggert [Ballot Position Update] New position, Discuss, has been recorded by Lars Eggert
2010-06-01
09 Ron Bonica [Ballot Position Update] New position, Yes, has been recorded for Ronald Bonica
2010-06-01
09 Ron Bonica Ballot has been issued by Ron Bonica
2010-06-01
09 Ron Bonica Created "Approve" ballot
2010-06-01
09 Ron Bonica State Changes to IESG Evaluation from Waiting for AD Go-Ahead by Ron Bonica
2010-06-01
09 Ron Bonica Placed on agenda for telechat - 2010-06-17 by Ron Bonica
2010-05-14
09 (System) State has been changed to Waiting for AD Go-Ahead from In Last Call by system
2010-05-10
09 Amanda Baber IANA comments:

As described in the IANA Considerations section, we understand this
document to have NO IANA Actions.
2010-05-03
09 Samuel Weiler Request for Last Call review by SECDIR is assigned to Rob Austein
2010-05-03
09 Samuel Weiler Request for Last Call review by SECDIR is assigned to Rob Austein
2010-04-30
09 Amy Vezza Last call sent
2010-04-30
09 Amy Vezza State Changes to In Last Call from Last Call Requested by Amy Vezza
2010-04-30
09 Ron Bonica State Changes to Last Call Requested from AD Evaluation by Ron Bonica
2010-04-30
09 Ron Bonica Last Call was requested by Ron Bonica
2010-04-30
09 (System) Ballot writeup text was added
2010-04-30
09 (System) Last call text was added
2010-04-30
09 (System) Ballot approval text was added
2010-04-23
09 Ron Bonica State Changes to AD Evaluation from Publication Requested by Ron Bonica
2010-04-12
09 Amy Vezza
This is a publication request for
http://tools.ietf.org/html/draft-ietf-bmwg-protection-term-08
as an INFORMATIONAL RFC.


    (1.a) Who is the Document Shepherd for this document? Has the
  …
This is a publication request for
http://tools.ietf.org/html/draft-ietf-bmwg-protection-term-08
as an INFORMATIONAL RFC.


    (1.a) Who is the Document Shepherd for this document? Has the
          Document Shepherd personally reviewed this version of the
          document and, in particular, does he or she believe this
          version is ready for forwarding to the IESG for publication?
Al Morton is the shepherd, has read this version and says its ready.

    (1.b) Has the document had adequate review both from key WG members
          and from key non-WG members? Does the Document Shepherd have
          any concerns about the depth or breadth of the reviews that
          have been performed?
Yes, I believe so. This work was initially proposed in 2002, and was
discussed extensively with many members of the community before
working group adoption in 2006. Extensive review was sought in the last
two rounds of WGLC.

    (1.c) Does the Document Shepherd have concerns that the document
          needs more review from a particular or broader perspective,
          e.g., security, operational complexity, someone familiar with
          AAA, internationalization or XML?
No. There will likely be a few more issues raised by the IESG, but
no laws-of-physics class violations that might stall the work.

    (1.d) Does the Document Shepherd have any specific concerns or
          issues with this document that the Responsible Area Director
          and/or the IESG should be aware of? For example, perhaps he
          or she is uncomfortable with certain parts of the document, or
          has concerns whether there really is a need for it. In any
          event, if the WG has discussed those issues and has indicated
          that it still wishes to advance the document, detail those
          concerns here. Has an IPR disclosure related to this document
          been filed? If so, please include a reference to the
          disclosure and summarize the WG discussion and conclusion on
          this issue.
There are no concerns and no IPR disclosures.

    (1.e) How solid is the WG consensus behind this document? Does it
          represent the strong concurrence of a few individuals, with
          others being silent, or does the WG as a whole understand and
          agree with it?
The consensus is solid. Many WG members read and agreed
with this document at various times. Many readers completed the
BMWG Active Review template for this draft.

    (1.f) Has anyone threatened an appeal or otherwise indicated extreme
          discontent? If so, please summarize the areas of conflict in
          separate email messages to the Responsible Area Director. (It
          should be in a separate email because this questionnaire is
          entered into the ID Tracker.)
No.

    (1.g) Has the Document Shepherd personally verified that the
          document satisfies all ID nits? (See
          http://www.ietf.org/ID-Checklist.html and
          http://tools.ietf.org/tools/idnits/). Boilerplate checks are
          not enough; this check needs to be thorough. Has the document
          met all formal review criteria it needs to, such as the MIB
          Doctor, media type and URI type reviews?

There are a few errors in the nits:
http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-bmwg-protection-term-08.txt
These can be corrected after AD-review, but there will be some work
necessary to adopt the new boilerplate, update references, fix
page lengths, etc.

    (1.h) Has the document split its references into normative and
          informative? Are there normative references to documents that
          are not ready for advancement or are otherwise in an unclear
          state? If such normative references exist, what is the
          strategy for their completion? Are there normative references
          that are downward references, as described in [RFC3967]? If
          so, list these downward references to support the Area
          Director in the Last Call procedure for them [RFC3967].
Yes, the references are split.

There is a normative dependency on:
"Benchmarking Terminology for IGP Convergence",
draft-ietf-bmwg-igp-dataplane-conv-term-21, (to be published) which is
also effectively at the Publication Requested state.
(Authors: this reference needs to be updated in the next version)

    (1.i) Has the Document Shepherd verified that the document IANA
          consideration section exists and is consistent with the body
          of the document? If the document specifies protocol
          extensions, are reservations requested in appropriate IANA
          registries? Are the IANA registries clearly identified? If
          the document creates a new registry, does it define the
          proposed initial contents of the registry and an allocation
          procedure for future registrations? Does it suggest a
          reasonable name for the new registry? See [RFC5226]. If the
          document describes an Expert Review process has Shepherd
          conferred with the Responsible Area Director so that the IESG
          can appoint the needed Expert during the IESG Evaluation?
N/A

    (1.j) Has the Document Shepherd verified that sections of the
          document that are written in a formal language, such as XML
          code, BNF rules, MIB definitions, etc., validate correctly in
          an automated checker?
N/A

    (1.k) The IESG approval announcement includes a Document
          Announcement Write-Up. Please provide such a Document
          Announcement Write-Up? Recent examples can be found in the
          "Action" announcements for approved documents. The approval
          announcement contains the following sections:

          Technical Summary
              Relevant content can frequently be found in the abstract
              and/or introduction of the document. If not, this may be
              an indication that there are deficiencies in the abstract
              or introduction.

    The purpose of this document is to provide a single terminology for
    benchmarking sub-IP protection mechanisms.

    Technologies that transport IP packets can be designed to provide
    protection for IP traffic by providing the failure recovery at lower
    layers. Possibly, the outage is not even observed at the IP-layer.
    Sub-IP protection technologies include, but are not limited to,
    High Availability (HA) stateful failover, Virtual Router Redundancy
    Protocol (VRRP), Automatic Link Protection (APS) for SONET/SDH,
    Resilient Packet Ring (RPR) for Ethernet, and Fast Reroute for
    Multi-Protocol Label Switching (MPLS-FRR).

    Benchmarking terminology was defined for IP-layer convergence in
    draft-ietf-bmwg-igp-dataplane-conv-term-21 .
    Different terminology and methodologies specific to
    benchmarking sub-IP layer protection mechanisms are required.  The
    metrics for benchmarking the performance of sub-IP protection
    mechanisms are measured at the IP layer, so that the results are
    independent of the specific protection mechanism being used.

          Working Group Summary
              Was there anything in WG process that is worth noting? For
              example, was there controversy about particular points or
              were there decisions where the consensus was particularly
              rough?
    Working group consensus was smoothly achieved over a period of
    several years, with many controversies ironed-out before WG adoption.
    Several authors came and went, but others were willing to pick-up the
    work and complete it, so here we are.

          Document Quality

Q - Are there existing implementations of the protocol?

A  these documents neither propose a new protocol nor extend any existing
one, hence current implementations are not required to support this
document. The document only attempts to standardize the benchmarking
strategies so that service providers could evaluate multiple protection
implementations using standard figures of merit with an aim of
producing consistent test results across multiple platforms.

Q - Have a significant number of vendors indicated their plan to implement
the specification?

A - Several test vendors, including the leading ones, have
shown their support and interest in the execution of these methodologies.
The authors have regularly acknowledged them in WG status presentations.

Reference/acknowledgements were shared in the following:
http://www.ietf.org/proceedings/06jul/slides/bmwg-3/bmwg-3.ppt

Q - Are there any reviewers that merit special mention as having done a
thorough review, e.g., one that resulted in important changes or a
conclusion that the document had no substantive issues?

A - The following lays down the history of this work item, which
demonstrates substantial revisions of the documents based on comments form
the BMWGers

1.  July 2005: Common protection terminology created due to merger of
draft-kimura-protection-term-02.txt and
draft-poretsky-mpls-protection-meth-02 as result of series of reviews and
comments. New draft-poretsky-protection-term-00.txt created.

2.  December 2005: another effort with additional test scenarios
(draft-vapiwala-bmwg-frr-failover-meth-00.txt) was merged into
draft-poretsky-meth-05. It was felt that additional scenarios proposed were
needed to offer comprehensive MPLS (including the services) based protection
techniques evaluation. Terms checked against this new methodology.

3.  June 2006: Due to additional test items proposal, a new draft was
created  draft-papneja-mpls-protection-meth-merge-00.txt

4.  Also in the combined effort Mr. Vapiwala were included as one of the
co-authors due to his contribution to the effort in the form of additional
test scenarios for the methodology document, and accordingly updating the
terminology document.

5.  The acknowledgement section is included for terminology document to
thank those who significantly helped in shaping the final version. The
document did receive comments from one of co-authors of original fast
reroute RFC 4090 in the early versions of the work item document in addition
to other pioneers in the MPLS field.
2010-04-12
09 Amy Vezza Draft Added by Amy Vezza in state Publication Requested
2010-04-12
09 Amy Vezza [Note]: 'Al Morton (acmorton@att.com) is the shepherd.' added by Amy Vezza
2009-12-21
08 (System) New version available: draft-ietf-bmwg-protection-term-08.txt
2009-10-16
07 (System) New version available: draft-ietf-bmwg-protection-term-07.txt
2009-03-08
06 (System) New version available: draft-ietf-bmwg-protection-term-06.txt
2008-11-03
05 (System) New version available: draft-ietf-bmwg-protection-term-05.txt
2008-02-25
04 (System) New version available: draft-ietf-bmwg-protection-term-04.txt
2007-11-13
03 (System) New version available: draft-ietf-bmwg-protection-term-03.txt
2007-07-09
02 (System) New version available: draft-ietf-bmwg-protection-term-02.txt
2007-03-08
01 (System) New version available: draft-ietf-bmwg-protection-term-01.txt
2006-10-16
00 (System) New version available: draft-ietf-bmwg-protection-term-00.txt