Benchmarking Methodology for Link-State IGP Data-Plane Route Convergence
draft-ietf-bmwg-igp-dataplane-conv-meth-23

[Ballot comment]
This document seems to be misusing RFC 2119 language. They don't seem to follow the admonition in section 6 of 2119:

  Imperatives of the type defined in this memo must be used with care
  and sparingly.  In particular, they MUST only be used where it is
  actually required for interoperation or to limit behavior which has
  potential for causing harm (e.g., limiting retransmisssions)  For
  example, they must not be used to try to impose a particular method
  on implementors where the method is not required for
  interoperability.

[Ballot discuss]
I reviewed this draft before it was put on the agenda, and the
authors agreed to address the following concerns in a new version,
but as far as I can see there is neither new version or editor's note.

========

  The distributed
  forwarding tables are typically maintained in hardware.

The forwarding is typically in h/w, but it's more normal
for them to be loaded and changed in s/w.

=======

In section 7

Seconds seems long for convergence time, certainly for IGP
convergence time. I would have expected answers in the
range 50ms to 10s of seconds.

You say fractions in the terms document. Perhaps a note
somewhere to say that by seconds you mean seconds and
fractions is needed in this document.


=====

[Ballot discuss]
I reviewed this draft before it was put on the agenda, and the
authors agreed to address the following concerns in a new version,
but as far as I can see there is neither new version or editor's note.

========

  The distributed
  forwarding tables are typically maintained in hardware.

The forwarding is typically in h/w, but it's more normal
for them to be loaded and changed in s/w.

=======

In section 7

Seconds seems long for convergence time, certainly for IGP
convergence time. I would have expected answers in the
range 50ms to 10s of seconds.

You say fractions in the terms document. Perhaps a note
somewhere to say that by seconds you mean seconds and
fractions is needed in this document.


=====

[Ballot discuss]
This is much improved since the previous version.

I reviewed this draft before it was put on the agenda, and the
authors agreed to address the following concerns in a new version,
but as far as I can see there is neither new version or editor's note.

========

  The distributed
  forwarding tables are typically maintained in hardware.

The forwarding is typically in h/w, but it's more normal
for them to be loaded and changed in s/w.

=======

In section 7

Seconds seems long for convergence time, certainly for IGP
convergence time. I would have expected answers in the
range 50ms to 10s of seconds.

You say fractions in the terms document. Perhaps a note
somewhere to say that by seconds you mean seconds and
fractions is needed in this document.


=====

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Benchmarking Methodology for Link-State IGP Data Plane Route Convergence) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Benchmarking Methodology for Link-State IGP Data Plane Route
  Convergence'
  as an Informational
RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-08-19. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


This document describes the methodology for benchmarking Link-State
Interior Gateway Protocol (IGP) Route Convergence.  The methodology
is to be used for benchmarking IGP convergence time through
externally observable (black box) data plane measurements.  The
methodology can be applied to any link-state IGP, such as IS-IS and
OSPF.



The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-igp-dataplane-conv-meth/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-igp-dataplane-conv-meth/


No IPR declarations have been submitted directly on this I-D.

[Ballot comment]
Section 1

  The test cases in this document are black-box
  tests that emulate the network events that cause convergence, as
  described in [Po09a].

Do the event cause convergence or necessitate convergence?

---

Please also address the large number of minor issues raised in the Routing Area Directorate review from Julien Meuric as follows...

Hello,

I have been selected as the Routing Directorate reviewer for this draft.
The Routing Directorate seeks to review all routing or routing-related
drafts as they pass through IETF last call and IESG review. The purpose
of the review is to provide assistance to the Routing ADs. For more
information about the Routing Directorate, please see
http://www.ietf.org/iesg/directorate/routing.html

Although these comments are primarily for the use of the Routing ADs, it
would be helpful if you could consider them along with any other IETF
Last Call comments that you receive, and strive to resolve them through
discussion or by updating the draft.

Document: draft-ietf-bmwg-igp-dataplane-conv-meth-21.txt
Reviewer: Julien Meuric (with the help of an anonymous colleague eating
IGPs at breakfast)
Review Date: 06/30/2010
Intended Status: Informational

*Summary:*
I have some minor concerns about this document that I think should be
resolved before publication.

*Comments:*
The document is rather heavy: it covers multiple scenarios, gives
several sequences of testing actions, analyses details about
uncertainty... As a result, for someone not used to the BMWG (please
keep in mind that this is my 1st review on a document from BMWG) it is
not so easy to follow in every detail and it requires some back-up
reading (draft-ietf-bmwg-igp-dataplane-conv-term for instance).

*Major Issues:*
No major issues found.

*Minor Issues:*
---
1/ I imagine it has already been discussed on the WG (sorry if I bring
back a troll), but it seems unusual to use RFC 2119 language for an
Informational document, and that is why it is explicitly stated in
section 2. Considering the status remains the same, instead of
advertising that fact, would not it be simpler to avoid the capital
letters in the corresponding words?
---
2/ My GMPLS background brings me to think that an IGP adjacency may be
independent from the corresponding data link. The document seems to
focus on the classical IGP use, but it would be better to make that
context clearer through a simple sentence than considering it is the
default.
---
3/ There is unfortunately no reference to traffic-engineering
extensions, while it might impact IGP convergence. Adding a few words on
this so as to state it is out of scope (if so) would be welcome.
---
4/ By reading section 3, we understand that the causes considered for
testing in this methodology concern failures and administrative changes
(status, costs). Therefore, the link insertion/recovery is apparently
not part of the testing. However, we can find it in section 8 if we take
a close look to the procedure steps. As a consequence, in order to stay
clearly consistent to draft-ietf-bmwg-igp-dataplane-conv-app-17
referenced here, it would be useful to clarify somewhere in section 3
that interface or link insertion/recovery is treated along with the
failure events and is therefore taken into account.
---
5/ The document will also gain in stating from the introduction the
scope of this methodology regarding router stress in front of
convergence performance (i.e. what is addressed in section 5). For
example, add something like:
"Convergence performance is tightly linked to the number of tasks a
router has to deal with. As the most impacting tasks are mainly related
to the control plane and the data plane, the more the DUT is stressed as
in a live environment, the more accurate performance results (i.e. the
ones that would be observed in a live environment) will be. Section 5
gives detailson the recommended environment for IGP convergence
performance benchmarking."
---

*Nits:*
---
Even though it may be usual in the WG, the way document references are
built ("AuthID#") is much less readable than "Summarized-Title" as used
in some places else. Let us hope most of them will be update with RFC
numbers (not more convenient in fact, but stable reference).
---
The phrase "next-hop router" may be confusing (at least until going into
the details), especially because in some contexts like BGP, a next-hop
router may not be adjacent but remote. How about "adjacent routers" to
reuse IS-IS terminology or "neighbor routers" to reuse OSPF terminology?
---
The "ECMP" acronym is expanded in section 3.4 (where it is actually
tested) while it has been used since section 3.1: expansion should be
moved (or duplicated) there.
---
A mix of "Loss of connectivity" and "LoC" acronym are used
alternatively: strict consistency along the document may not be a goal,
but association between them should at least be explicit at 1st use
(section 4).
---
"IS-IS" is always referred to as "ISIS", I would add the dash.
---
Some titles on figures (e.g. 9) and equations (e.g. 3) are closer to the
following paragraph than the corresponding item, swapping or reducing
the amount of blank lines would be easier to read.
---
Section 2:
s/in other BMWG work/in other documents issued by the Benchmarking
Methodology Working Group/
---
Section 3.1:
At 1st occurence, it might be more accurate to specify that "N >= 1" or
"N > 0".
---
Section 3.4:
"the tester emulates N next-hop routers"
Whitout the figure, it is difficult to quickly picture the
configuration. I may ease the understanding by adding something like
"(N-1 adjacent to R1; 1 adjacent to R2)".
---
Section 5.4: "LSA", "LSP" and "SPF" are not expanded: they may be usual,
but IGP is expanded in the abstract and introduction (and "LSP" has 2
usual meanings in the Routing area)... The same question may raise for
"IS-IS" and "OSPF" expansion, but they are considered as "well-known" on
http://www.rfc-editor.org/rfc-style-guide/abbrev.expansion.txt (while
the formers are not).
---
Section 5.6:
s/topologies 3, 4, and 6/topologies 3, 4 and 6/
s/packets are transmitted/packets be transmitted/
---
Section 5.9:
s/test case has/test case have/
---
Section 7:
s/loss or not./loss or not?/
s/Complete the table below/The table below should be completed/
---
Section 8:
"DUT's" and "Tester's" read weird to me with respect to what I was
taught at school, but someone put "the car's wheels" on Wikipedia. I
thus leave this issue to native English speakers. :-)
---
Section 8.1.4:
s/may influenced/may be influenced/
---

[Ballot discuss]
This document certainly seems to have had a ride! Revision 21 is
definitely an experienced draft. Thanks to the authors for sticking
with it and addressing the previous issues and concerns.

In addition to the minor comments provided in my Comment, I have a
couple of larger issues I would like to discuss before recommending
publication of this document.

---

Thanks for calling out the normative reference to draft-ietf-bmwg-igp-
dataplane-conv-app in the proto write-up. I understand that author of
that document intends to revive it, but I worry that it will mean that
this document will sit in the RFC Editor Queue blocked on a reference
for an unknown length of time.

But my main concern is the difficulty I have in reviewing this document
in the absence of an up-to-date version of the normative reference that
defines "motivation and applicability for this benchmarking."

Wouldn't it have been better to present the documents for publication
in the right order?

---

I found Figure 1 confusing in its simplicity! It appears to offer
parallel outgoing interfaces, and the text states that the Tester
emulates two routers. But I think the Tester emulates three routers
as in the figure below.

                                              ----------
                                              | Tester |
    ---------      Ingress Interface        |  ----  |
    |      |<--------------------------------+-| R1 | |
    |      |                                |  ----  |
    |      |    Preferred Egress Interface  |  ----  |
    |  DUT  |---------------------------------+>| R2 | |
    |      |                                |  ----  |
    |      |    Next-Best Egress Interface  |  ----  |
    |      |---------------------------------+>| R3 | |
    |      |                                |  ----  |
    ---------                                ----------

Alternatively, it might be best to show the emulated topology
such as
                                ----- Preferred Egress Interface  ----
      ----  Ingress Interface  |    |--------------------------->| R2 |
    | R1 |------------------->| DUT |                            ----
      ----                    |    |                            ----
                              |    |--------------------------->| R3 |
                                ----- Next-Best Egress Interface  ----

This could usefully be applied to all other figures as well.

---

I understand the value of most of the tests, but I don't think they
really test IGP convergence.

In the case of the local tests it seems that you are testing something
very useful (for example, the time taken to introduce an interface
failure into the local RIB, compute a new next hop, and apply it to the
FIB. An operator would very probably be interested in this measurement,
but it isn't actually anything to do with the IGP itself.

There is nothing wrong with these test cases per se, but the document
is scoped to link state IGP convergence.  The best approach is probably
to rescope the document in terms of what it actually tests, and to make
IGP convergence just a part of those tets.

---

Can you clarify what is being tested in 8.2.2. Is this equivalent to an
IGP software failure or the deconfiguration of an interface from the
IGP? Maybe you could add a word or two.

---

Section 7

I know this was raised in a previous Discuss, but I lack the
correspondence to see the resolution. It looks to me that the
results are reported in seconds, and that seems to be too coarse
grained for some of the local convergence tests.

I note that there is nothing in Section 6 that implies that the
results should be reported as coarsely as in units of seconds.

---         

Section 9

Isn't there a need to also run these tests in the presence of inter-
router IGP security?

[Ballot comment]
Please address the large number of minor issues raised in the Routing Area Directorate review from Julien Meuric as follows...

Hello,

I have been selected as the Routing Directorate reviewer for this draft.
The Routing Directorate seeks to review all routing or routing-related
drafts as they pass through IETF last call and IESG review. The purpose
of the review is to provide assistance to the Routing ADs. For more
information about the Routing Directorate, please see
http://www.ietf.org/iesg/directorate/routing.html

Although these comments are primarily for the use of the Routing ADs, it
would be helpful if you could consider them along with any other IETF
Last Call comments that you receive, and strive to resolve them through
discussion or by updating the draft.

Document: draft-ietf-bmwg-igp-dataplane-conv-meth-21.txt
Reviewer: Julien Meuric (with the help of an anonymous colleague eating
IGPs at breakfast)
Review Date: 06/30/2010
Intended Status: Informational

*Summary:*
I have some minor concerns about this document that I think should be
resolved before publication.

*Comments:*
The document is rather heavy: it covers multiple scenarios, gives
several sequences of testing actions, analyses details about
uncertainty... As a result, for someone not used to the BMWG (please
keep in mind that this is my 1st review on a document from BMWG) it is
not so easy to follow in every detail and it requires some back-up
reading (draft-ietf-bmwg-igp-dataplane-conv-term for instance).

*Major Issues:*
No major issues found.

*Minor Issues:*
---
1/ I imagine it has already been discussed on the WG (sorry if I bring
back a troll), but it seems unusual to use RFC 2119 language for an
Informational document, and that is why it is explicitly stated in
section 2. Considering the status remains the same, instead of
advertising that fact, would not it be simpler to avoid the capital
letters in the corresponding words?
---
2/ My GMPLS background brings me to think that an IGP adjacency may be
independent from the corresponding data link. The document seems to
focus on the classical IGP use, but it would be better to make that
context clearer through a simple sentence than considering it is the
default.
---
3/ There is unfortunately no reference to traffic-engineering
extensions, while it might impact IGP convergence. Adding a few words on
this so as to state it is out of scope (if so) would be welcome.
---
4/ By reading section 3, we understand that the causes considered for
testing in this methodology concern failures and administrative changes
(status, costs). Therefore, the link insertion/recovery is apparently
not part of the testing. However, we can find it in section 8 if we take
a close look to the procedure steps. As a consequence, in order to stay
clearly consistent to draft-ietf-bmwg-igp-dataplane-conv-app-17
referenced here, it would be useful to clarify somewhere in section 3
that interface or link insertion/recovery is treated along with the
failure events and is therefore taken into account.
---
5/ The document will also gain in stating from the introduction the
scope of this methodology regarding router stress in front of
convergence performance (i.e. what is addressed in section 5). For
example, add something like:
"Convergence performance is tightly linked to the number of tasks a
router has to deal with. As the most impacting tasks are mainly related
to the control plane and the data plane, the more the DUT is stressed as
in a live environment, the more accurate performance results (i.e. the
ones that would be observed in a live environment) will be. Section 5
gives detailson the recommended environment for IGP convergence
performance benchmarking."
---

*Nits:*
---
Even though it may be usual in the WG, the way document references are
built ("AuthID#") is much less readable than "Summarized-Title" as used
in some places else. Let us hope most of them will be update with RFC
numbers (not more convenient in fact, but stable reference).
---
The phrase "next-hop router" may be confusing (at least until going into
the details), especially because in some contexts like BGP, a next-hop
router may not be adjacent but remote. How about "adjacent routers" to
reuse IS-IS terminology or "neighbor routers" to reuse OSPF terminology?
---
The "ECMP" acronym is expanded in section 3.4 (where it is actually
tested) while it has been used since section 3.1: expansion should be
moved (or duplicated) there.
---
A mix of "Loss of connectivity" and "LoC" acronym are used
alternatively: strict consistency along the document may not be a goal,
but association between them should at least be explicit at 1st use
(section 4).
---
"IS-IS" is always referred to as "ISIS", I would add the dash.
---
Some titles on figures (e.g. 9) and equations (e.g. 3) are closer to the
following paragraph than the corresponding item, swapping or reducing
the amount of blank lines would be easier to read.
---
Section 2:
s/in other BMWG work/in other documents issued by the Benchmarking
Methodology Working Group/
---
Section 3.1:
At 1st occurence, it might be more accurate to specify that "N >= 1" or
"N > 0".
---
Section 3.4:
"the tester emulates N next-hop routers"
Whitout the figure, it is difficult to quickly picture the
configuration. I may ease the understanding by adding something like
"(N-1 adjacent to R1; 1 adjacent to R2)".
---
Section 5.4: "LSA", "LSP" and "SPF" are not expanded: they may be usual,
but IGP is expanded in the abstract and introduction (and "LSP" has 2
usual meanings in the Routing area)... The same question may raise for
"IS-IS" and "OSPF" expansion, but they are considered as "well-known" on
http://www.rfc-editor.org/rfc-style-guide/abbrev.expansion.txt (while
the formers are not).
---
Section 5.6:
s/topologies 3, 4, and 6/topologies 3, 4 and 6/
s/packets are transmitted/packets be transmitted/
---
Section 5.9:
s/test case has/test case have/
---
Section 7:
s/loss or not./loss or not?/
s/Complete the table below/The table below should be completed/
---
Section 8:
"DUT's" and "Tester's" read weird to me with respect to what I was
taught at school, but someone put "the car's wheels" on Wikipedia. I
thus leave this issue to native English speakers. :-)
---
Section 8.1.4:
s/may influenced/may be influenced/
---

[Ballot discuss]
As with the accompanying terms document this could use input from RTGWG. if it has not been last called in the LS IGP groups it should be sent there.

Although this is entitled LS IGP convergence, there is nothing in the documented methodology that is particularly LS specific, so I do not understand why it is not a general IGP test methodology.

It should also take into account the recent work on IPFRR and loop prevention (or explicitly note that new test are needed)

Please clarify that the arrows in the figs show flows and that interfaces are bidirectional.

The most of the document is written in terms of bad news events only. You can have to consider good news events as well since these can have different convergence times and different impacts on the traffic.

The figures should really be written in terms of pre and post convergence interfaces and both good news and bad news events considered.

WRT Fig 4
In this topology you will get different (worse) results if the tester tells R2 of the failure first, but that does not seem possible in this configuration.

The same consideration applies to Fig 6

The topologies should consider the LAN case as well as the pt-pt case.

ECMP tests seem to be DA based, but the ECMP selection criteria is more complex than simply DA.


The test parameters in Section 7 indicate that the methodology is not considering the impact of BFD on improving path change detection time. Because BFD is faster than hellos this may flush out other convergence issues. Fast hellos can be sub second.

Convergence times will be sub second, and I agree with David Wards earlier advice to the authors to just work in microseconds.

There are many valid DISCUSS comments that were raised in 2007 that do nit seem to have been addressed by this version. Please consider such items not addressed as part of this discuss.

[Ballot discuss]
As with the accompanying terms document this could use input from RTGWG. if it has not been last called in the LS IGP groups it should be sent there.

Although this is entitled LS IGP convergence, there is nothing in the documented methodology that is particularly LS specific, so I do not understand why it is not a general IGP test methodology.

It should also take into account the recent work on IPFRR and loop prevention (or explicitly note that new test are needed)

Please clarify that the arrows in the figs show flows and that interfaces are bidirectional.

The most of the document is written in terms of bad news events only. You can have to consider good news events as well since these can have different convergence times and different impacts on the traffic.

The figures should really be written in terms of pre and post convergence interfaces and both good news and bad news events considered.

WRT Fig 4
In this topology you will get different (worse) results if the tester tells R2 of the failure first, but that does not seem possible in this configuration.

The same consideration applies to Fig 6

The topologies should consider the LAN case as well as the pt-pt case.

ECMP tests seem to be DA based, but the ECMP selection criteria is more complex than simply DA.


The test parameters in Section 7 indicate that the methodology is not considering the impact of BFD on improving path change detection time. Because BFD is faster than hellos this may flush out other convergence issues. Fast hellos can be sub second.

Convergence times will be sub second, and I agree with David Wards earlier advice to the authors to just work in microseconds.

    (1.a) Who is the Document Shepherd for this document? Has the
          Document Shepherd personally reviewed this version of the
          document and, in particular, does he or she believe this
          version is ready for forwarding to the IESG for publication?
Al Morton is the shepherd, has read virtually every version since these
drafts were adopted on the charter, and believes they are now ready for
publication.

    (1.b) Has the document had adequate review both from key WG members
          and from key non-WG members? Does the Document Shepherd have
          any concerns about the depth or breadth of the reviews that
          have been performed?
These drafts have undergone extensive cross-area review, and a previous
IESG review.

    (1.c) Does the Document Shepherd have concerns that the document
          needs more review from a particular or broader perspective,
          e.g., security, operational complexity, someone familiar with
          AAA, internationalization or XML?
No.

    (1.d) Does the Document Shepherd have any specific concerns or
          issues with this document that the Responsible Area Director
          and/or the IESG should be aware of? For example, perhaps he
          or she is uncomfortable with certain parts of the document, or
          has concerns whether there really is a need for it. In any
          event, if the WG has discussed those issues and has indicated
          that it still wishes to advance the document, detail those
          concerns here. Has an IPR disclosure related to this document
          been filed? If so, please include a reference to the
          disclosure and summarize the WG discussion and conclusion on
          this issue.
There are no concerns and no IPR disclosures.

    (1.e) How solid is the WG consensus behind this document? Does it
          represent the strong concurrence of a few individuals, with
          others being silent, or does the WG as a whole understand and
          agree with it?
Over the last seven years, I'm quite sure that many WG members read and agreed
with the documents at various times.  This is a measurement topic where the
"state of the art" has steadily advanced, as evidenced by a useful
set of comments
in response to the 2nd-to-last WGLC from a new member of the working group
(December 2009).

    (1.f) Has anyone threatened an appeal or otherwise indicated extreme
          discontent? If so, please summarize the areas of conflict in
          separate email messages to the Responsible Area Director. (It
          should be in a separate email because this questionnaire is
          entered into the ID Tracker.)
Most discontent can be traced to the length of time this development took
and the head-of-line blocking it caused. The draft development did not
keep pace with some parts of the measurement community, and this caused a
major setback in the first IESG review (among other issues). All of the DISCUSS
points were addressed a year ago (March 2009), but there was still some
participant comments to address.  All seem to have been resolved now.

    (1.g) Has the Document Shepherd personally verified that the
          document satisfies all ID nits? (See
          http://www.ietf.org/ID-Checklist.html and
          http://tools.ietf.org/tools/idnits/). Boilerplate checks are
          not enough; this check needs to be thorough. Has the document
          met all formal review criteria it needs to, such as the MIB
          Doctor, media type and URI type reviews?
Yes.


    (1.h) Has the document split its references into normative and
          informative? Are there normative references to documents that
          are not ready for advancement or are otherwise in an unclear
          state? If such normative references exist, what is the
          strategy for their completion? Are there normative references
          that are downward references, as described in [RFC3967]? If
          so, list these downward references to support the Area
          Director in the Last Call procedure for them [RFC3967].
The references are split.
There is one normative reference to a draft that expired a year ago.
The author would like to resuscitate it...
No down-refs, all are drafts in this series are informational.

    (1.i) Has the Document Shepherd verified that the document IANA
          consideration section exists and is consistent with the body
          of the document? If the document specifies protocol
          extensions, are reservations requested in appropriate IANA
          registries? Are the IANA registries clearly identified? If
          the document creates a new registry, does it define the
          proposed initial contents of the registry and an allocation
          procedure for future registrations? Does it suggest a
          reasonable name for the new registry? See [RFC5226]. If the
          document describes an Expert Review process has Shepherd
          conferred with the Responsible Area Director so that the IESG
          can appoint the needed Expert during the IESG Evaluation?
N/A

    (1.j) Has the Document Shepherd verified that sections of the
          document that are written in a formal language, such as XML
          code, BNF rules, MIB definitions, etc., validate correctly in
          an automated checker?
N/A

    (1.k) The IESG approval announcement includes a Document
          Announcement Write-Up. Please provide such a Document
          Announcement Write-Up? Recent examples can be found in the
          "Action" announcements for approved documents. The approval
          announcement contains the following sections:

          Technical Summary
              Relevant content can frequently be found in the abstract
              and/or introduction of the document. If not, this may be
              an indication that there are deficiencies in the abstract
              or introduction.
    Convergence Time is a critical performance parameter.  Customers
    of Service Providers use convergence packet loss due to
    Interior Gateway Protocol (IGP) convergence as a key metric of
    their network service quality.  Service Providers use IGP
    Convergence time as a key metric of router design and architecture
    for any IGP such as Intermediate System - Intermediate System
    (ISIS) and Open-Shorted Path first (OSPF).
    Fast network convergence can be optimally achieved through deployment
    of fast converging routers.  These documents
    describe the terminology and methodology for benchmarking Link-State
    IGP Convergence time, measured on the data plane by observing packet loss
    through the Device under test.  The methodology and terminology can be
    used for benchmarking IGP Convergence can be applied to both
    IPv4 and IPv6 traffic.


          Working Group Summary
              Was there anything in WG process that is worth noting? For
              example, was there controversy about particular points or
              were there decisions where the consensus was particularly
              rough?
This set of drafts was reviewed by the IESG in 2007, and a design
team was formed
to help address the DISCUSSes. In effect, the working group took the documents
back for more work. The Design Team was disbanded in 2008, and remaining
work was conducted in the working group (a total of 8 more revisions).
The final last call ended quietly after many, many, calls with comments,
so the chair declared "Working Group Consensus" (resulting in off-list
celebrations).

There was some controversy about authorship. The list was expanded to include
a new/leading author in 2009. Early in 2010, a new participant wished to become
an author in return for his comments, but seemed satisfied with the explanation
provided by the WG chair.

          Document Quality
              Are there existing implementations of the protocol? Have a
              significant number of vendors indicated their plan to
              implement the specification? Are there any reviewers that
              merit special mention as having done a thorough review,
              e.g., one that resulted in important changes or a
              conclusion that the document had no substantive issues? If
              there was a MIB Doctor, Media Type or other expert review,
              what was its course (briefly)? In the case of a Media Type
              review, on what date was the request posted?
IGP-Dataplane Convergence time testing has been conducted in various labs
for many years. It was this experience that prompted standards work.
New test equipment capabilities brought improvements in the state-of-the-art.
Recently, test equipment vendors have embraced these methods more completely,
and this yielded the last round of major comments in December 2009.

[Ballot comment]
1. The Abstract says 'The methodology can be applied to any link-state IGP, such as ISIS and OSPF.' Is it true that the methodology applies only to link-state IGPs? If true, I would suggest that the title is change to add 'link-state'. Else strike out 'link-state' from the Abstract.

2. Section 3.2.2 - 'To obtain results similar to those that would be
  observed in an operational network, it is recommended that the
  number of installed routes closely approximates that the network.'

Probably '... that of the network' 

An indication of the deegree of magnitude of this number also seems to be in place here.

3. Section 4.2 - what does 'remove layer 2 session' mean? I read layer 2 failure a failure that is detected at layer 2, but can reflect a fault that happens in the lower layer and can be as trivial as a cable failure. Am I wrong?

[Ballot discuss]
A few comments on this draft:

0) it is unclear why RIP isn't covered

1) it is unclear why the recommended timer values in 3.2.4 do not correspond to typical values configured in the network but, the route scaling numbers do

2) the packet sampling time in 3.2.5 seems out of date. IGPs converge faster than the packet sample time today.

3) it is recommended that the results are in usecs only

4) it is unclear if there are packet order test requirements for ECMP paths. Since the many ECMP tests are called out is there any 'correct' test or outcome that is desired for selection of ECMP path?

5) On bcast interfaces it is unclear if both p2p/nbma and bcast needs to be configured

6) why don't the tests include generation of LSA/LSP as well as change in data plane. IOW, what is SUT/DUT in Fig2 as well as 4.1.3.

7) The results from the remote failure case 4.1.3 aren't quite correct:

"The additional
  convergence time contributed by LSP Propagation can be
  obtained by subtracting the Rate-Derived Convergence Time
  measured in 4.1.2 (Convergence Due to Neighbor Interface
  Failure) from the Rate-Derived Convergence Time measured in
  this test case."

Though the point is mostly academic, it isn't technically correct.

8) why don't we include fiber pull test and/or enable disable interface

9) Do we want to specify tests for "link up" vs just "link down?" Link up is a critical event in the network and frequently causes loops/microloops.

10) In the "metric change" test in 4.5 since traffic is moving from one intf to another there should be an observable convergence event unlike what is stated in expected results:

"There should be no externally observable IGP Route Convergence ..."


11) In general the results section of each tests should state what should be observed during the test, packet loss, packets tx/rx between any SUT and required systems, etc. Right now, there is a very brief description of the influencing variables of the test. It would not be possible to verify a true positive on a test w/ current text.

12) There needs to be a notion and testing of specific prefixes:

first, last and then a median and mean

13) There needs to be a notion of important prefixes or those that are biased for prioritized convergence. E.g. BGP Nexthops.

14) There should be a measure of any microloops formed and duration of any loops|microloops

15) Is graceful restart and time to restore FIB considered a convergence event? If not, why?

[Ballot discuss]
A few comments on this draft:

0) it is unclear why RIP isn't covered

1) it is unclear why the recommended timer values in 3.2.4 do not correspond to typical values configured in the network but, the route scaling numbers do

2) the packet sampling time in 3.2.5 seems out of date. IGPs converge faster than the packet sample time today.

3) it is recommended that the results are in usecs only

4) it is unclear if there are packet order test requirements for ECMP paths. Since the many ECMP tests are called out is there any 'correct' test or outcome that is desired for selection of ECMP path?

5) On bcast interfaces it is unclear if both p2p/nbma and bcast needs to be configured

6) why don't the tests include generation of LSA/LSP as well as change in data plane. IOW, what is SUT/DUT in Fig2 as well as 4.1.3.

7) The results from the remote failure case 4.1.3 aren't quite correct:

"The additional
  convergence time contributed by LSP Propagation can be
  obtained by subtracting the Rate-Derived Convergence Time
  measured in 4.1.2 (Convergence Due to Neighbor Interface
  Failure) from the Rate-Derived Convergence Time measured in
  this test case."

Though the point is mostly academic, it isn't technically correct.

8) why don't we include fiber pull test and/or enable disable interface

9) Do we want to specify tests for "link up" vs just "link down?" Link up is a critical event in the network and frequently causes loops/microloops.

10) In the "metric change" test in 4.5 since traffic is moving from one intf to another there should be an observable convergence event unlike what is stated in expected results:

"There should be no externally observable IGP Route Convergence ..."


11) In general the results section of each tests should state what should be observed during the test, packet loss, packets tx/rx between any SUT and required systems, etc. Right now, there is a very brief description of the influencing variables of the test. It would not be possible to verify a true positive on a test w/ current text.

12) There needs to be a notion and testing of specific prefixes:

first, last and then a median and mean

13) There needs to be a notion of important prefixes or those that are biased for prioritized convergence. E.g. BGP Nexthops.

14) There should be a measure of any microloops formed and duration of any loops|microloops

[Ballot discuss]
A few comments on this draft:

0) it is unclear why RIP isn't covered

1) it is unclear why the recommended timer values in 3.2.4 do not correspond to typical values configured in the network but, the route scaling numbers do

2) the packet sampling time in 3.2.5 seems out of date. IGPs converge faster than the packet sample time today.

3) it is recommended that the results are in usecs only

4) it is unclear if there are packet order test requirements for ECMP paths. Since the many ECMP tests are called out is there any 'correct' test or outcome that is desired for selection of ECMP path?

5) On bcast interfaces it is unclear if both p2p/nbma and bcast needs to be configured

6) why don't the tests include generation of LSA/LSP as well as change in data plane. IOW, what is SUT/DUT in Fig2 as well as 4.1.3.

7) The results from the remote failure case 4.1.3 aren't quite correct:

"The additional
  convergence time contributed by LSP Propagation can be
  obtained by subtracting the Rate-Derived Convergence Time
  measured in 4.1.2 (Convergence Due to Neighbor Interface
  Failure) from the Rate-Derived Convergence Time measured in
  this test case."

Though the point is mostly academic, it isn't technically correct.

8) why don't we include fiber pull test and/or enable disable interface

9) Do we want to specify tests for "link up" vs just "link down?" Link up is a critical event in the network and frequently causes loops/microloops.

10) In the "metric change" test in 4.5 since traffic is moving from one intf to another there should be an observable convergence event unlike what is stated in expected results:

"There should be no externally observable IGP Route Convergence ..."


11) In general the results section of each tests should state what should be observed during the test, packet loss, packets tx/rx between any SUT and required systems, etc. Right now, there is a very brief description of the influencing variables of the test. It would not be possible to verify a true positive on a test w/ current text.

12) There needs to be a notion and testing of specific prefixes:

first, last and then a median and mean

13) There needs to be a notion of important prefixes or those that are biased for prioritized convergence. E.g. BGP Nexthops.

[Ballot discuss]
A few comments on this draft:

0) it is unclear why RIP isn't covered

1) it is unclear why the recommended timer values in 3.2.4 do not correspond to typical values configured in the network but, the route scaling numbers do

2) the packet sampling time in 3.2.5 seems out of date. IGPs converge faster than the packet sample time today.

3) it is recommended that the results are in usecs only

4) it is unclear if there are packet order test requirements for ECMP paths. Since the many ECMP tests are called out is there any 'correct' test or outcome that is desired for selection of ECMP path?

5) On bcast interfaces it is unclear if both p2p/nbma and bcast needs to be configured

6) why don't the tests include generation of LSA/LSP as well as change in data plane. IOW, what is SUT/DUT in Fig2 as well as 4.1.3.

7) The results from the remote failure case 4.1.3 aren't quite correct:

"The additional
  convergence time contributed by LSP Propagation can be
  obtained by subtracting the Rate-Derived Convergence Time
  measured in 4.1.2 (Convergence Due to Neighbor Interface
  Failure) from the Rate-Derived Convergence Time measured in
  this test case."

Though the point is mostly academic, it isn't technically correct.

8) why don't we include fiber pull test and/or enable disable interface

9) Do we want to specify tests for "link up" vs just "link down?" Link up is a critical event in the network and frequently causes loops/microloops.

10) In the "metric change" test in 4.5 since traffic is moving from one intf to another there should be an observable convergence event unlike what is stated in expected results:

"There should be no externally observable IGP Route Convergence ..."


11) In general the results section of each tests should state what should be observed during the test, packet loss, packets tx/rx between any SUT and required systems, etc. Right now, there is a very brief description of the influencing variables of the test. It would not be possible to verify a true positive on a test w/ current text.

PROTO Write-up

> (1.a) Who is the Document Shepherd for this document? Has the
> Document Shepherd personally reviewed this version of the
> document and, in particular, does he or she believe this
> version is ready for forwarding to the IESG for publication?

Al Morton is the Document Shepherd.
Yes, for both review and publication.


> (1.b) Has the document had adequate review both from key WG members
> and from key non-WG members? Does the Document Shepherd have
> any concerns about the depth or breadth of the reviews that
> have been performed?

WG review has been extensive, and there are no concerns.
Cross-area review with routing area directorate has been completed
by Sue Hares.


> (1.c) Does the Document Shepherd have concerns that the document
> needs more review from a particular or broader perspective,
> e.g., security, operational complexity, someone familiar with
> AAA, internationalization or XML?

No



> (1.d) Does the Document Shepherd have any specific concerns or
> issues with this document that the Responsible Area Director
> and/or the IESG should be aware of? For example, perhaps he
> or she is uncomfortable with certain parts of the document, or
> has concerns whether there really is a need for it. In any
> event, if those issues have been discussed in the WG and the
> WG has indicated that it still wishes to advance the document,
> detail those concerns here.

No specific or general concerns.



> 1.e) How solid is the WG consensus behind this document? Does it
> represent the strong concurrence of a few individuals, with
> others being silent, or does the WG as a whole understand and
> agree with it?

Many active WG members affirmed that this set of drafts
were ready for publication (WGLC in October, 2005).



> (1.f) Has anyone threatened an appeal or otherwise indicated extreme
> discontent? If so, please summarise the areas of conflict in
> separate email messages to the Responsible Area Director. (It
> should be in a separate email because this questionnaire will
> be entered into the ID Tracker.)

No.



> (1.g) Has the Document Shepherd verified that the document satisfies
> all ID nits? (See http://www.ietf.org/ID-Checklist.html and
> http://tools.ietf.org/tools/idnits/). Boilerplate checks are
> not enough; this check needs to be thorough.

Yes, except that draft-ietf-bmwg-igp-dataplane-conv-term-11.txt has
a single formatting nit that should not affect the review, and can be
corrected in the next revision:

* There are 3 instances of too long lines in the document, the longest one
being 4 characters in excess of 72.




> (1.h) Has the document split its references into normative and
> informative? Are there normative references to documents that
> are not ready for advancement or are otherwise in an unclear
> state? If such normative references exist, what is the
> strategy for their completion? Are there normative references
> that are downward references, as described in [RFC3967]? If
> so, list these downward references to support the Area
> Director in the Last Call procedure for them [RFC3967].
>

There are normative references to the other IDs in this 3 draft set,
therefore they should be advanced together.
There are no downward references.



> (1.i) The IESG approval announcement includes a Document
> Announcement Write-Up. Please provide such a Document
> Announcement Write-Up. Recent examples can be found in the
> "Action" announcements for approved documents. The approval
> announcement contains the following sections:
>
> Technical Summary
> Relevant content can frequently be found in the abstract
> and/or introduction of the document. If not, this may be
> an indication that there are deficiencies in the abstract
> or introduction.
> Working Group Summary
> Was there anything in WG process that is worth noting? For
> example, was there controversy about particular points or
> were there decisions where the consensus was particularly
> rough?
> Document Quality
> Are there existing implementations of the protocol? Have a
> significant number of vendors indicated their plan to
> implement the specification? Are there any reviewers that
> merit special mention as having done a thorough review,
> e.g., one that resulted in important changes or a
> conclusion that the document had no substantive issues?

All BMWG RFCs are Informational, but they are implemented by
test equipment vendors and cited in trade publications and
advertisements.

Technical Summary


This set of memos describes the process for benchmarking IGP
Route Convergence as described in the Applicability memo.
This approach measures convergence time in the dataplane,
and treats the Device Under Test as a Black Box.
The methodology and terminology memos define the metrics and
process for benchmarking route convergence that can be applied
to any link-state IGP such as ISIS and OSPF.


WG Summary

The drafts received extensive comment and review since their
initial acceptance on the WG charter in 2003.
Many active WG members affirmed that this set of drafts
were ready for publication (WGLC in October, 2005).
There was a subsequent cross-area review that resulted in
additional minor revisions, discussed and agreed by the WG.


Protocol Summary

These methods have been performed in at least one lab,
and review comments were posted based on that experience.




> + Have a significant number of vendors indicated they
> plan to implement the specification?

Several test equipment vendors commented actively during the WG development.




> + Are there any reviewers that merit special mention as
> having done a thorough review (i.e., that resulted in
> important changes or a conclusion that the document
> had no substantive issues)?

Robert Holley (Cisco) and Tanju Cataltepe (formerly AT&T)
completed the BMWG Active Review Templates for these drafts.
Cross-Area Review was provided by Sue Hares.




> The questionnaire and write-up is sent to the ADs and
> iesg-secretary@ietf.org with a request to publish the document. The
> questionnaire and write-up, minus any discussion of possible appeals,
> is also sent to the working group mailing list. The publication
> request SHOULD also indicate which chair will be shepherding the
> document (this will be entered into the ID Tracker [IDTRACKER]). In
> addition to making life easier for the ADs, this lets the IETF chair
> know where to send Gen-ART [GEN-ART] reviews.