Terminology for Benchmarking Link-State IGP Data-Plane Route Convergence
draft-ietf-bmwg-igp-dataplane-conv-term-23

[Ballot comment]
Like Ralph, I am very confused by the use of 2119 language in this document. I don't understand what its necessity is or who it is aimed at.

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Terminology for Benchmarking Link-State IGP Data Plane Route Convergence) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Terminology for Benchmarking Link-State IGP Data Plane Route
  Convergence'
  as an Informational
RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-08-19. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


This document describes the terminology for benchmarking link-state
Interior Gateway Protocol (IGP) route convergence.  The terminology
is to be used for benchmarking IGP convergence time through
externally observable (black box) data plane measurements.  The



The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-igp-dataplane-conv-term/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-igp-dataplane-conv-term/


No IPR declarations have been submitted directly on this I-D.

[Ballot comment]
In several definitions:

  Measurement Units:

  hh:mm:ss:nnn:uuu, where 'nnn' is milliseconds and 'uuu' is
  microseconds.

The "Measurement Units" are microseconds, while "hh:mm:ss:nnn:uuu" is a representation.  Elsewhere, "Measurement Units" are defined as, e.g., "seconds"

I don't understand the requirements language (this example from section 3.1.2):

  Discussion:

  Full Convergence MUST occur after a Convergence Event.

"MUST occur" for compliance or interoperability with what, exactly?

[Ballot discuss]
It is disappointing that a number of David Wards DISCUSS comments have not been taken on board with this version of the document.

I am concerned that a lot is known about IPRFF and the states of partial
convergence and convergence control that is neither reflected in this
document nor explicitly declared out of scope.

I think that the document would benefit from review an input from the RTGWG who have spend a lot of time working on convergence.

The term convergence event is used many times before it is defined

The term  microloop is used but not defined.


"IGP convergence time is measured on the data plane at the Tester by observing packet loss through the DUT."

Packet loss is only one aspect of convergence. You can have misorder
without packet loss during a convergence event.

You can also have convergence without packet loss, but with impact
on delay sensitive protocols.

Convergence Type definitions and methodologies given in this document are all bad news definitions, but you also need to consider convergence under good news conditions. You also need to provide a more complete consideration of various types of ECMP.


"Completion of Network Convergence can be observed by recovery of the
network Forwarding Rate to equal the Offered Load, with no Stale
Forwarding, and no Blenders [Ca01][Ci03]."

You can have convergence without the offered load being satisfied.
Equally you can satisfy the offered load without being fully converged
(consider ECMP) and the effect on packet order and delay.

The term Blenders should be defined


"3.2.2.  Convergence Event Instant

"  If the Convergence Event causes instantaneous traffic loss"

-- you need to consider loss and misorder and latency change events



"3.4.2.  Remote Interface

"  Definition:

"  An interface on a neighboring router that is not directly connected
"  to any interface on the DUT."

Surely a remote interface may NOT be on a neighoring router.



"3.5.  Benchmarking Methods

"  If packets are going over multiple ECMP members and one or more of
"  the members has failed then the number of received packets during
"  each Packet Sampling Interval may vary, even excluding presence of
"  IPDV.  To prevent fluctuation of the number of received packets
"  during each Packet Sampling Interval for this reason, the Packet
"  Sampling Interval duration SHOULD be a whole multiple of the time
"  between two consecutive packets sent to the same destination.

ECMP may consider factors other than DA.


"3.6.  Benchmarks

"3.6.1.  Full Convergence Time

"  Definition:

"  The time duration of the period between the Convergence Event Instant
"  and the Convergence Recovery Instant as observed using the Rate-
"  Derived Method.

This is one measure of convergence time, but you can have convergence
events that do not effect macro rate


"3.6.3.  Route-Specific Convergence Time

"  Definition:

"  The amount of time it takes for Route Convergence to be completed for
"  a specific route, as calculated from the amount of packet loss during
"  convergence for a single route entry.

You may have route specific convergence even that has no packet loss. But does have other negative impact on traffic (for example delay or misorder)


There are lots of convergence events measure in Measurement Units: seconds. This is orders of magnitude too high for an IGP with fast convergence or IPFRR running.

    (1.a) Who is the Document Shepherd for this document? Has the
          Document Shepherd personally reviewed this version of the
          document and, in particular, does he or she believe this
          version is ready for forwarding to the IESG for publication?
Al Morton is the shepherd, has read virtually every version since these
drafts were adopted on the charter, and believes they are now ready for
publication.

    (1.b) Has the document had adequate review both from key WG members
          and from key non-WG members? Does the Document Shepherd have
          any concerns about the depth or breadth of the reviews that
          have been performed?
These drafts have undergone extensive cross-area review, and a previous
IESG review.

    (1.c) Does the Document Shepherd have concerns that the document
          needs more review from a particular or broader perspective,
          e.g., security, operational complexity, someone familiar with
          AAA, internationalization or XML?
No.

    (1.d) Does the Document Shepherd have any specific concerns or
          issues with this document that the Responsible Area Director
          and/or the IESG should be aware of? For example, perhaps he
          or she is uncomfortable with certain parts of the document, or
          has concerns whether there really is a need for it. In any
          event, if the WG has discussed those issues and has indicated
          that it still wishes to advance the document, detail those
          concerns here. Has an IPR disclosure related to this document
          been filed? If so, please include a reference to the
          disclosure and summarize the WG discussion and conclusion on
          this issue.
There are no concerns and no IPR disclosures.

    (1.e) How solid is the WG consensus behind this document? Does it
          represent the strong concurrence of a few individuals, with
          others being silent, or does the WG as a whole understand and
          agree with it?
Over the last seven years, I'm quite sure that many WG members read and agreed
with the documents at various times.  This is a measurement topic where the
"state of the art" has steadily advanced, as evidenced by a useful
set of comments
in response to the 2nd-to-last WGLC from a new member of the working group
(December 2009).

    (1.f) Has anyone threatened an appeal or otherwise indicated extreme
          discontent? If so, please summarize the areas of conflict in
          separate email messages to the Responsible Area Director. (It
          should be in a separate email because this questionnaire is
          entered into the ID Tracker.)
Most discontent can be traced to the length of time this development took
and the head-of-line blocking it caused. The draft development did not
keep pace with some parts of the measurement community, and this caused a
major setback in the first IESG review (among other issues). All of the DISCUSS
points were addressed a year ago (March 2009), but there was still some
participant comments to address.  All seem to have been resolved now.

    (1.g) Has the Document Shepherd personally verified that the
          document satisfies all ID nits? (See
          http://www.ietf.org/ID-Checklist.html and
          http://tools.ietf.org/tools/idnits/). Boilerplate checks are
          not enough; this check needs to be thorough. Has the document
          met all formal review criteria it needs to, such as the MIB
          Doctor, media type and URI type reviews?
Yes.


    (1.h) Has the document split its references into normative and
          informative? Are there normative references to documents that
          are not ready for advancement or are otherwise in an unclear
          state? If such normative references exist, what is the
          strategy for their completion? Are there normative references
          that are downward references, as described in [RFC3967]? If
          so, list these downward references to support the Area
          Director in the Last Call procedure for them [RFC3967].
The references are split.
There is one normative reference to a draft that expired a year ago.
The author would like to resuscitate it...
No down-refs, all are drafts in this series are informational.

    (1.i) Has the Document Shepherd verified that the document IANA
          consideration section exists and is consistent with the body
          of the document? If the document specifies protocol
          extensions, are reservations requested in appropriate IANA
          registries? Are the IANA registries clearly identified? If
          the document creates a new registry, does it define the
          proposed initial contents of the registry and an allocation
          procedure for future registrations? Does it suggest a
          reasonable name for the new registry? See [RFC5226]. If the
          document describes an Expert Review process has Shepherd
          conferred with the Responsible Area Director so that the IESG
          can appoint the needed Expert during the IESG Evaluation?
N/A

    (1.j) Has the Document Shepherd verified that sections of the
          document that are written in a formal language, such as XML
          code, BNF rules, MIB definitions, etc., validate correctly in
          an automated checker?
N/A

    (1.k) The IESG approval announcement includes a Document
          Announcement Write-Up. Please provide such a Document
          Announcement Write-Up? Recent examples can be found in the
          "Action" announcements for approved documents. The approval
          announcement contains the following sections:

          Technical Summary
              Relevant content can frequently be found in the abstract
              and/or introduction of the document. If not, this may be
              an indication that there are deficiencies in the abstract
              or introduction.
    Convergence Time is a critical performance parameter.  Customers
    of Service Providers use convergence packet loss due to
    Interior Gateway Protocol (IGP) convergence as a key metric of
    their network service quality.  Service Providers use IGP
    Convergence time as a key metric of router design and architecture
    for any IGP such as Intermediate System - Intermediate System
    (ISIS) and Open-Shorted Path first (OSPF).
    Fast network convergence can be optimally achieved through deployment
    of fast converging routers.  These documents
    describe the terminology and methodology for benchmarking Link-State
    IGP Convergence time, measured on the data plane by observing packet loss
    through the Device under test.  The methodology and terminology can be
    used for benchmarking IGP Convergence can be applied to both
    IPv4 and IPv6 traffic.


          Working Group Summary
              Was there anything in WG process that is worth noting? For
              example, was there controversy about particular points or
              were there decisions where the consensus was particularly
              rough?
This set of drafts was reviewed by the IESG in 2007, and a design
team was formed
to help address the DISCUSSes. In effect, the working group took the documents
back for more work. The Design Team was disbanded in 2008, and remaining
work was conducted in the working group (a total of 8 more revisions).
The final last call ended quietly after many, many, calls with comments,
so the chair declared "Working Group Consensus" (resulting in off-list
celebrations).

There was some controversy about authorship. The list was expanded to include
a new/leading author in 2009. Early in 2010, a new participant wished to become
an author in return for his comments, but seemed satisfied with the explanation
provided by the WG chair.

          Document Quality
              Are there existing implementations of the protocol? Have a
              significant number of vendors indicated their plan to
              implement the specification? Are there any reviewers that
              merit special mention as having done a thorough review,
              e.g., one that resulted in important changes or a
              conclusion that the document had no substantive issues? If
              there was a MIB Doctor, Media Type or other expert review,
              what was its course (briefly)? In the case of a Media Type
              review, on what date was the request posted?
IGP-Dataplane Convergence time testing has been conducted in various labs
for many years. It was this experience that prompted standards work.
New test equipment capabilities brought improvements in the state-of-the-art.
Recently, test equipment vendors have embraced these methods more completely,
and this yielded the last round of major comments in December 2009.

[Ballot comment]
In section 3.5 I do not understand why the measurment unit reads:

'Number of N-octet offered packets that are not forwarded'

Why not just?

'Number of packets that are not forwarded'

If the definition is packet loss for a packet of length N, then it is the definition field that needs to be changed.

[Ballot discuss]
A few items:

0) we need to have a definition of remote vs local failure

1) Why isn't a convergence event defined as any local or remote trigger that causes a route recalculation vs one in which fwding is effected. If convergence is only to be defined by a change in forwarding what is the term that the authors recommend for an event in which a route calculation has to be made but, in fact forwarding is not changed? To the control plane of the router, the work is the same and given a catastrophic network event; a "queue" of calculations that cause no forwarding change in front of a calculation that would cause a forwarding plane change is critical to define, understand and place as a variable in the convergence equation.

2) There should be a definition of prioritized convergence in which "important prefixes" (e.g. loopbacks that are BGP NHs) are measured vs "unimportant prefixes." In addition, the important prefixes should

3) There have been alternative definitions and terminology for convergence that the authors should cite and rectify. Many of these docs have been discussed in rtgwg.

4) loops and microloops should be defined

5) units of measurement are wrong order of magnitude

6) Restoration Convergence time is unclear. The IGP sees only individual convergence events.

[Ballot comment]
Based on Gen-ART Review by Vijay K. Gurbani.

  This draft is ready for publication as an Informational RFC.
  There are a few editorial changes that should be made:

  - Section 1: s/of the DUT and the/of the DUT, and the/

  - Section 3.1: DUT is expanded here; if it should be expanded
    anywhere, it should be in Section 1.

PROTO Write-up

> (1.a) Who is the Document Shepherd for this document? Has the
> Document Shepherd personally reviewed this version of the
> document and, in particular, does he or she believe this
> version is ready for forwarding to the IESG for publication?

Al Morton is the Document Shepherd.
Yes, for both review and publication.



> (1.b) Has the document had adequate review both from key WG members
> and from key non-WG members? Does the Document Shepherd have
> any concerns about the depth or breadth of the reviews that
> have been performed?

WG review has been extensive, and there are no concerns.
Cross-area review with routing area directorate has been completed
by Sue Hares.


> (1.c) Does the Document Shepherd have concerns that the document
> needs more review from a particular or broader perspective,
> e.g., security, operational complexity, someone familiar with
> AAA, internationalization or XML?

No



> (1.d) Does the Document Shepherd have any specific concerns or
> issues with this document that the Responsible Area Director
> and/or the IESG should be aware of? For example, perhaps he
> or she is uncomfortable with certain parts of the document, or
> has concerns whether there really is a need for it. In any
> event, if those issues have been discussed in the WG and the
> WG has indicated that it still wishes to advance the document,
> detail those concerns here.

No specific or general concerns.



> 1.e) How solid is the WG consensus behind this document? Does it
> represent the strong concurrence of a few individuals, with
> others being silent, or does the WG as a whole understand and
> agree with it?

Many active WG members affirmed that this set of drafts
were ready for publication (WGLC in October, 2005).



> (1.f) Has anyone threatened an appeal or otherwise indicated extreme
> discontent? If so, please summarise the areas of conflict in
> separate email messages to the Responsible Area Director. (It
> should be in a separate email because this questionnaire will
> be entered into the ID Tracker.)

No.



> (1.g) Has the Document Shepherd verified that the document satisfies
> all ID nits? (See http://www.ietf.org/ID-Checklist.html and
> http://tools.ietf.org/tools/idnits/). Boilerplate checks are
> not enough; this check needs to be thorough.

Yes, except that draft-ietf-bmwg-igp-dataplane-conv-term-11.txt has
a single formatting nit that should not affect the review, and can be
corrected in the next revision:

* There are 3 instances of too long lines in the document, the longest one
being 4 characters in excess of 72.




> (1.h) Has the document split its references into normative and
> informative? Are there normative references to documents that
> are not ready for advancement or are otherwise in an unclear
> state? If such normative references exist, what is the
> strategy for their completion? Are there normative references
> that are downward references, as described in [RFC3967]? If
> so, list these downward references to support the Area
> Director in the Last Call procedure for them [RFC3967].
>

There are normative references to the other IDs in this 3 draft set,
therefore they should be advanced together.
There are no downward references.



> (1.i) The IESG approval announcement includes a Document
> Announcement Write-Up. Please provide such a Document
> Announcement Write-Up. Recent examples can be found in the
> "Action" announcements for approved documents. The approval
> announcement contains the following sections:
>
> Technical Summary
> Relevant content can frequently be found in the abstract
> and/or introduction of the document. If not, this may be
> an indication that there are deficiencies in the abstract
> or introduction.
> Working Group Summary
> Was there anything in WG process that is worth noting? For
> example, was there controversy about particular points or
> were there decisions where the consensus was particularly
> rough?
> Document Quality
> Are there existing implementations of the protocol? Have a
> significant number of vendors indicated their plan to
> implement the specification? Are there any reviewers that
> merit special mention as having done a thorough review,
> e.g., one that resulted in important changes or a
> conclusion that the document had no substantive issues?

All BMWG RFCs are Informational, but they are implemented by
test equipment vendors and cited in trade publications and
advertisements.

Technical Summary


This set of memos describes the process for benchmarking IGP
Route Convergence as described in the Applicability memo.
This approach measures convergence time in the dataplane,
and treats the Device Under Test as a Black Box.
The methodology and terminology memos define the metrics and
process for benchmarking route convergence that can be applied
to any link-state IGP such as ISIS and OSPF.


WG Summary

The drafts received extensive comment and review since their
initial acceptance on the WG charter in 2003.
Many active WG members affirmed that this set of drafts
were ready for publication (WGLC in October, 2005).
There was a subsequent cross-area review that resulted in
additional minor revisions, discussed and agreed by the WG.


Protocol Summary

These methods have been performed in at least one lab,
and review comments were posted based on that experience.




> + Have a significant number of vendors indicated they
> plan to implement the specification?

Several test equipment vendors commented actively during the WG development.




> + Are there any reviewers that merit special mention as
> having done a thorough review (i.e., that resulted in
> important changes or a conclusion that the document
> had no substantive issues)?

Robert Holley (Cisco) and Tanju Cataltepe (formerly AT&T)
completed the BMWG Active Review Templates for these drafts.
Cross-Area Review was provided by Sue Hares.




> The questionnaire and write-up is sent to the ADs and
> iesg-secretary@ietf.org with a request to publish the document. The
> questionnaire and write-up, minus any discussion of possible appeals,
> is also sent to the working group mailing list. The publication
> request SHOULD also indicate which chair will be shepherding the
> document (this will be entered into the ID Tracker [IDTRACKER]). In
> addition to making life easier for the ADs, this lets the IETF chair
> know where to send Gen-ART [GEN-ART] reviews.