Methodology for Benchmarking MPLS Traffic Engineered (MPLS-TE) Fast Reroute Protection
draft-ietf-bmwg-protection-meth-14

[Ballot comment]
I am updating my comment to clarify it. It is a point that was somewhat obliquely a Discuss last time round, but I did not properly explain the point. I would raise this as a discuss, but I do not think the review rules permit this.

The text that says : "MPLS FRR protection  mechanisms are generally deployed in a network infrastructure where MPLS is used for provisioning of point-to-point traffic engineered tunnels (tunnel)." and the absence of discussion on the topic elsewhere suggests that you have not considered an important (arguably the most important) deployment case.

In an otherwise LDP network a one hop RSVP-TE tunnel is deployed across a link. This tunnel is normally PHPed, but is protected, so normally no label is pushed on the main tunnel. This case will impact the LSR configuration compared to the pure RSVP-TE case and will impact the restoration, because you have three routing process that interact - the IGP, LDP and RSVP-TE. You do the same think for node protection, but cannot of course PHP.

You might want to rule the case out of scope, although this will reduce the utility of the testing, or you could consider it, but you really ought to provide evidence that you have thought about the implications and validity of your tests in this network environment.

[Ballot comment]
  Thanks for addressing the issues raised in the Gen-ART Review by
  Joel Halpern on 9-Mar-2012.  Joel did a second review on 12-Nov-2012.
  Please consider these non-blocking comments from the second review.

  In section 5.1 "some" failure events are listed. It is unclear
  whether this list is the ones to be tested for, or just "some"
  events.  I think it is intended to be comprehensive, so why "some".

  I presume that the use of the term Layer2 VC in section 6 is
  intended to refer to a pt-to-pt layer2 VPN component.  Is there an
  existing document where that term is used that way?  Could it be
  referenced?  For that matter, since the number of labels is the same
  in the Layer3 cases and the Layer2 cases, could the Layer2 cases be
  omitted without loss of generality?

[Ballot discuss]
As far as I can see from reading the document the authors are considering FRR solely in the explicit case of MPLS-TE FRR being used to protect an MPLS-TE LSP. FRR and MPLS protection have developed considerably since that initial concept. Additionally one of the most common uses of MPLS-TE FRR is to protect a one hop tunnel with PHP (i.e. a null tunnel) which in turn is used as a method of protecting a hop in an LDP MPLS network.

Please can the authors make it much clearer in the introduction and scoping of the document that testing issues related to MPLS/LDP, IPFRR applied to MPLS networks (RFC5714 applied to MPLS), and MPLS-TP have not been taken into  consideration, and are thus out of scope for this document.

I think that there is a zero label case that needs to be tested - one hop php tunnel at network egress protected by a TE tunnel.

[Ballot comment]

MPLS FRR protection
mechanisms are generally deployed in a network infrastructure where
MPLS is used for provisioning of point-to-point traffic engineered
tunnels (tunnel). 

True, but one of the most common deployments of those P2P TE tunnels is to protect a ling or node in an MPLS LDP network.

=====

Planned failures are predictable.

I am not sure whether that is true by definition or an oxymoron. Presumable you mean planned exercise of the
protection mechanism, or planned reconfiguration or maintenance
of the network.

=====

          - Interface Shutdown on PLR side with POS Alarm
          - Interface Shutdown on remote side with POS Alarm

I am not sure why the author single out  POS here, surely there are many p2p link types including Ethernet supported by Y.1731?

[Ballot comment]
Minor points.

  Within the context of MPLS
  protection mechanisms, failures that arise due to Shared Risk Link
  Groups (SRLG) [RFC 4090] can be considered as correlated failures.

Isn't the reference rfc4202? RFC4090 doesn't even spell the acronym

Spell out the acronym or give references in (surely the RFC editor would note that as well)

  "Within the context of MPLS
  protection mechanisms, failures that arise due to Shared Risk Link
  Groups (SRLG) [RFC 4090] can be considered as correlated failures."

[Ballot comment]
Aside from Barry's comment about the 2119 template (which I agree is poorly worded and should be changed), I don't really understand why any of the 2119 language in this document is useful. For example, when you say, "The tester MUST record the number of lost, duplicate, and out-of-order packets" in section 4, I am left to ask, "Why MUST the tester do that?" It's not explained, and I suspect all it means is, "The tester has to record those things if it wants to know those things." I'm pretty sure that all of the 2119 language could be removed from the document without changing its utility.

[Ballot discuss]

  Thanks for addressing the issues raised in the Gen-ART Review by
  Joel Halpern on 9-Mar-2012.  Joel did a second review on 12-Nov-2012,
  which raised an additional concern.

  Section 5.7 is still internally inconsistent.  It says that "It is
  suggested that there be three or more traffic streams..." in the
  first paragraph.  The second paragraph then begins "At least 16 flows
  should be used, ..."  I have compared the definitions in RFC 4689,
  and I can not see how 3 streams (A group of packets treated as a
  single entity by the traffic receiver) can be 16 flows (one or more
  packets sharing a common intended pair of ingress and egress
  interfaces.)  I suspect that the issue is what traffic receiver is
  intended.  Is the intent that there be at least 3 LSPs with 16 or
  more flows running through them?

[Ballot comment]

  Thanks for addressing the issues raised in the Gen-ART Review by
  Joel Halpern on 9-Mar-2012.  Joel did a second review on 12-Nov-2012.
  Please consider these non-blocking comments from the second review.

  In section 5.1 "some" failure events are listed. It is unclear
  whether this list is the ones to be tested for, or just "some"
  events.  I think it is intended to be comprehensive, so why "some".

  I presume that the use of the term Layer2 VC in section 6 is
  intended to refer to a pt-to-pt layer2 VPN component.  Is there an
  existing document where that term is used that way?  Could it be
  referenced?  For that matter, since the number of labels is the same
  in the Layer3 cases and the Layer2 cases, could the Layer2 cases be
  omitted without loss of generality?

[Ballot comment]

- s2, saying you cover "all possible failure scenarios" seems
ambitious - is it really true? How about buggy code?  I'd
suggest s/all possible/a set of interesting/

- s4, it'd be good to say what the lines in Figure 1 represent,
e.g. physical connections or routes or tunnels or whatever,
that may be well known for MPLS developers but perhaps less so
for testers?

- s4, probably a dumb question (sorry;-) but is the definition
of the number of re-ordered packets clear?  e.g. does the
sequence 1,4,3,2,5,6 have two or three re-ordered packets?
What about 1,4,4,3,2,5,6? Later sections also talk about
"Out-of-order" packets, is that meant to be the same thing?
Better to use one term only if so.

- s5.7, last para: fwiw, this seems a bit open-ended for a
benchmark to my uneducated eye. If it makes no difference to
the outcome, be good to say that maybe.

[Ballot comment]
Thanks to the authors for addressing most (but not all) of the issues
I raised in my review last time this document came to the IESG.

Although I have a number of comments and concerns about this document
I will not stand in the way.

---

Reading the earlier part of the document, I got the impression that the
mechanism of failure detection and reporting was out of scope.

However, looking at later sections (such as Section 5.9) I see that you
propose to measure based on packet loss or timestamped packets over the
failure. Both of these measures are dependent on the methods of
detection and reporting.

So I don't understand how the test cases work for anything other than
local errors. And even in these cases, it seems you are more likely
testing and recording fault report times in the DUT than measuring the
actual failover times.

---

Section 7.1.1 etc.

      3.  Verify primary and backup LSPs are up and that primary is
          protected.

      4.  Verify Fast Reroute protection is enabled and ready.

I don't find any description of how these items are verified.

---

Abstract

  This draft describes the methodology for benchmarking MPLS Protection
  mechanisms for link and node protection as defined in [RFC 4090].

It would be nice if this was alligned with the document title by
specifically stating "MPLS Fast Reroute Protection"

===

The following are nits that you may want to sort out before passing the
document to the RFC Editor...

---

I think Jean Louis needs to be correclty abreviated on the front page
viz. JL. Le Roux.

---

It would be good to replace "draft" with "document" so that the text is
consistent when published as an RFC

---

Citations should not be pesent in the Abstract.

---

The Abstract suggests that the document considers "all other factors".
This seems a little unlikely :-)

---

Section 3

This section could usefully include a pointer to Appendix B.

---

Section 5

Slightly odd that the 7 points listed don't match the sub-sections of
this section.

[Ballot comment]
Non-blocking comments only.  No need to respond, though I'll be happy to chat with you about them if you like.

The datatracker and shepherd writeup say this is targeted as Informational, but the document says it's Standards Track.  Which is it?

The Introduction says, "For any conflicting content, this document supersedes [RFC 6414]."  Doesn't that mean that this should Update 6414?

The 2119 boilerplate (Section 3) adds this:

  RFC 2119 defines the use of these key words to help make the
  intent of standards track documents as clear as possible.  While this
  document uses these keywords, this document is not a standards track
  document.

That seems *really* odd: you're saying that these words aren't meant for the purpose that 2119 uses them for, but you don't say what purpose they *are* used for.  If you feel you need to explain this, you should focus on explaining, not disclaiming.  Something more like this (word it as you see appropriate):

  While RFC 2119 defines the use of these key words primarily for
  Standards Track documents, this Informational document uses them
  to [...insert explanation here...].

-- Figure 1 --
The text should perhaps say what all the "Rn" boxes are, and make it clear that they are not part of the Tester -- only the TG and TA are.  It might also expand "DUT" on first use (it's not expanded until Section 5.3).

State changed to In Last Call from Last Call Requested

The following Last Call Announcement was sent out:

From: The IESG

To: IETF-Announce

CC:

Reply-To: ietf@ietf.org

Subject: Last Call:  (Methodology for benchmarking MPLS protection mechanisms) to Informational RFC





The IESG has received a request from the Benchmarking Methodology WG

(bmwg) to consider the following document:

- 'Methodology for benchmarking MPLS protection mechanisms'

  as an Informational RFC



The IESG plans to make a decision in the next few weeks, and solicits

final comments on this action. Please send substantive comments to the

ietf@ietf.org mailing lists by 2012-03-20. Exceptionally, comments may be

sent to iesg@ietf.org instead. In either case, please retain the

beginning of the Subject line to allow automated sorting.



Abstract





  This draft describes the methodology for benchmarking MPLS Protection

  mechanisms for link and node protection as defined in [MPLS-FRR-EXT].

  This document provides test methodologies and testbed setup for

  measuring failover times while considering all dependencies that

  might impact faster recovery of real-time applications bound to MPLS

  based traffic engineered tunnels.  The benchmarking terms used in

  this document are defined in [TERM-ID].









The file can be obtained via

http://datatracker.ietf.org/doc/draft-ietf-bmwg-protection-meth/



IESG discussion can be tracked via

http://datatracker.ietf.org/doc/draft-ietf-bmwg-protection-meth/ballot/





No IPR declarations have been submitted directly on this I-D.

This is a Publication Request from bmwg:

Document Title:  Methodology for benchmarking MPLS protection
mechanisms
Filename:        draft-ietf-bmwg-protection-meth-09.txt
Intended Status: Informational


  (1.a) Who is the Document Shepherd for this document? Has the
        Document Shepherd personally reviewed this version of the
        document and, in particular, does he or she believe this
        version is ready for forwarding to the IESG for publication?
Al Morton, who has reviewed the memo. It is ready (+/- a few minor edits,
see the end of this write-up).

  (1.b) Has the document had adequate review both from key WG members
        and from key non-WG members? Does the Document Shepherd have
        any concerns about the depth or breadth of the reviews that
        have been performed?
This memo has been developed over a period of 8 years, beginning with the
merger of two independent efforts, then narrowing of the scope to become a
working group draft, followed by many reviews and WGLCs. No concerns
about the current degree of review and feedback.

  (1.c) Does the Document Shepherd have concerns that the document
        needs more review from a particular or broader perspective,
        e.g., security, operational complexity, someone familiar with
        AAA, internationalization or XML?
No.

  (1.d) Does the Document Shepherd have any specific concerns or
        issues with this document that the Responsible Area Director
        and/or the IESG should be aware of? For example, perhaps he
        or she is uncomfortable with certain parts of the document, or
        has concerns whether there really is a need for it. In any
        event, if the WG has discussed those issues and has indicated
        that it still wishes to advance the document, detail those
        concerns here. Has an IPR disclosure related to this document
        been filed? If so, please include a reference to the
        disclosure and summarize the WG discussion and conclusion on
        this issue.
This memo follows the referenced Terminology RFC as expected,
although some of the terms agreed by the WG still feel awkward to me
(e.g., use of "Failover Event" where cases where "Failure" would do).

No IPR disclosures have been filed.

  (1.e) How solid is the WG consensus behind this document? Does it
        represent the strong concurrence of a few individuals, with
        others being silent, or does the WG as a whole understand and
        agree with it?
The WG reached consensus on this memo years ago, but it was agreed to
hold this draft until the IGP-Dataplane Benchmarking was completed.
Another WGLC was held in December 2011, and there were no additional
comments.

  (1.f) Has anyone threatened an appeal or otherwise indicated extreme
        discontent? If so, please summarise the areas of conflict in
        separate email messages to the Responsible Area Director. (It
        should be in a separate email because this questionnaire is
        entered into the ID Tracker.)
No.

  (1.g) Has the Document Shepherd personally verified that the
        document satisfies all ID nits? (See the Internet-Drafts Checklist
        and http://tools.ietf.org/tools/idnits/). Boilerplate checks are
        not enough; this check needs to be thorough. Has the document
        met all formal review criteria it needs to, such as the MIB
        Doctor, media type and URI type reviews?
The nits check calls out some of the same reference issues mentioned in
the Editorial comments, attached below.

  (1.h) Has the document split its references into normative and
        informative? Are there normative references to documents that
        are not ready for advancement or are otherwise in an unclear
        state? If such normative references exist, what is the
        strategy for their completion? Are there normative references
        that are downward references, as described in [RFC3967]? If
        so, list these downward references to support the Area
        Director in the Last Call procedure for them [RFC3967].
The references are split, and no DownRefs.
Many of the Informative References provide essential terminology definitions,
and are therefore Normative (see below).

  (1.i) Has the Document Shepherd verified that the document IANA
        consideration section exists and is consistent with the body
        of the document? If the document specifies protocol
        extensions, are reservations requested in appropriate IANA
        registries? Are the IANA registries clearly identified? If
        the document creates a new registry, does it define the
        proposed initial contents of the registry and an allocation
        procedure for future registrations? Does it suggest a
        reasonable name for the new registry? See [RFC5226]. If the
        document describes an Expert Review process has Shepherd
        conferred with the Responsible Area Director so that the IESG
        can appoint the needed Expert during the IESG Evaluation?
IANA section exists, making no IANA requests, as is usual in bmwg.

  (1.j) Has the Document Shepherd verified that sections of the
        document that are written in a formal language, such as XML
        code, BNF rules, MIB definitions, etc., validate correctly in
        an automated checker?
NA

  (1.k) The IESG approval announcement includes a Document
        Announcement Write-Up.

      Technical Summary
Service providers and testing organizations need to benchmark the
performance of network protection mechanisms. The BMWG took-up this work
and defined a general terminology and set of benchmarks in RFC 6414.
This memo describes the methodology for benchmarking MPLS Protection
mechanisms for link and node protection. It provides test methodologies
and test setups for measuring failover times while considering all
dependencies that might impact the recovery of real-time applications
carried in MPLS-based traffic engineered tunnels.
The test procedures in this document cover local failure or remote failure
scenarios for comprehensive benchmarking and to evaluate failover
performance independent of the failure detection techniques.


      Working Group Summary
This memo has been developed over a period of 8 years, beginning with the
merger of two independent efforts, then narrowing of the scope to become a
working group draft, followed by many reviews and WGLCs.
The WG reached consensus on this memo years ago, but it was agreed to
hold this draft until the IGP-Dataplane Benchmarking was completed.
Another WGLC was held in December 2011, and there were no additional
comments.

      Document Quality
The reviews and suggestions of Jean Philip Vasseur, Curtis Villamizar, and
Bhavani Parise are acknowledged.


-=-=-=-=-=-=-=-=-=-=-=- MINOR EDITORIAL COMMENTS -=-=-=-=-=-=-=-=-=-=-=-

Abstract and everywhere else:
s/draft/document/
s/[TERM-ID]/[RFC6414]

Sec 1, 5th para
s/assurance/continuity/

Sec 1, 7th para
s/General Model/General Model (step 3b below)/

Sec 2, 3rd para (BFD mentioned in many places, but out of scope)
s/this document/the test procedures, but mentioned in discussion sections/

Sec 3,
These may not be Primary References, should be replaced with Primary RFC:
                  Out-of-order Packet      [Ref.[Po06], section 3.3.2]
                  Duplicate Packet          [Ref.[Po06], section 3.3.3]

Sec 4
s/stack is dependent of/stack is dependent on/

Spell-out PLR at first use, which is here:
      (2)  # of remaining hops of the primary tunnel from the PLR

Sec 5.7, 2nd para (very redundantly worded)
    ... At least 16 flows should be used, and more if possible.  Prefix-
    dependency behaviors are key in IP and tests with route-specific
    flows spread across the routing table will reveal this dependency.
    Generating traffic to all of the prefixes reachable by the protected
    tunnel (probably in a Round-Robin fashion, where the traffic is
    destined to all the prefixes but one prefix at a time in a cyclic
    manner) is not recommended.  The reason why traffic generation is not
    recommended in a Round-Robin fashion to all the prefixes, one at a
    time is that if there are many prefixes reachable through the LSP the
    time interval between 2 packets destined to one prefix may be
    significantly high and may be comparable with the failover time being
    measured which does not aid in getting an accurate failover
    measurement.

Sec 7.1
spell out "pps" at first use.

Sec 7.1.1 (and 7.1.2 and 7.1.3)
modify steps as follows:

      6.  Send the required MPLS traffic load over the primary LSP
            to achieve the Throughput supported by the DUT (determined using
            section X of RFC 2544).

...
      9.  Verify that the offered load gets mapped to the backup tunnel
            and measure the Additive Backup Delay (as defined in section YY
            of RFC 6414).

...

      12.  Record the final Throughput, which corresponds to the
offered load that
            will be used for the Headend PLR failover test cases.



Sec 7.1.2 only
s/of the 9 from section 6/of the 8 from section 6/

Sec 7.4 and 7.5, Procedure
      4.  Verify Fast Reroute protection. << is enabled?

Sec 8
s/recommended/RECOMMENDED/

s/Offered Load/Offered Load (Throughput)/

References:
Because these refs provide essential terms and definitions,
they should be normative:

    [IGP-METH]
THIS is now RFC 6412
              Poretsky, S., Imhoff, B., and K. Michielsen, "Terminology
              for Benchmarking Link-State IGP Data Plane Route
              Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-23
              (work in progress), February 2011.

    [Br91]    Bradner, S., "Benchmarking terminology for network
              interconnection devices", RFC 1242, July 1991.

    [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

    [MPLS-FRR-EXT]
              Pan, P., Swallow, G., and A. Atlas, "Fast Reroute
              Extensions to RSVP-TE for LSP Tunnels", RFC 4090,
              May 2005.

    [MPLS-FWD] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding
              Benchmarking Methodology for IP Flows", RFC 5695,
              November 2009.

    [RFC6414]  Papneja, R., Poretsky, S., Vapiwala, S., and J. Karthik,
              "Benchmarking Terminology for Protection Performance",
              RFC 6414, October 2011.