Device Reset Characterization
draft-ietf-bmwg-reset-06

Received changes through RFC Editor sync (changed abstract to 'An operational forwarding device may need to be restarted (automatically or manually) for a variety of reasons, an event called a "reset" in this document. Since there may be an interruption in the forwarding operation during a reset, it is useful to know how long a device takes to resume the forwarding operation.

This document specifies a methodology for characterizing reset (and reset time) during benchmarking of forwarding devices and provides clarity and consistency in reset test procedures beyond what is specified in RFC 2544. Therefore, it updates RFC 2544. This document also defines the benchmarking term "reset time" and, only in this, updates RFC 1242. This document is not an Internet Standards Track specification; it is published for informational purposes.')

[Ballot comment]
Thanks for addressing my specific Discuss points.

My personal belief remains that it is the recovery time you should measure not the down time plus recovery time, but if this is what the WG thinks is useful, I'll let it go.

The WG chair and responsible AD need to consider whether the WG need to sign off on the substantive definition of "Reset" added to this document?

[Ballot discuss]
The Gen-ART Review by Joel Halpern on 27-Nov-2010 included two
  questions.  I think that the should be answered before the
  document is approved:

  Is there a reason that section 4.1.1.1 on reset of a single-RP device
  refers to simple removal and re-insertion, without explaining the
  caveats that section 4.1.2 (line card removal and re-insertion)
  provides about using a method to avoid having the human time for
  the removal and reinsertion have a significant effect on the
  operation?

  Also, should the same caveat not be applied to section 4.3.1 on
  Power Interruption?  There seems to be an implicit assumption that
  the human operation will be sufficiently fast, and the recovery
  sufficiently slow, that the human element does not matter.  At the
  very least, it would seem that this should be articulated.

[Ballot comment]
I have no issues with this document, but Ari Keränen found some small
problems in his review:

2.2. Recovery Time Measurement Methods

    There are two accepted methods to measure the 'recovery time':

Is the meaning of "accepted" as in "no other method SHOULD be used" or
as in "currently widely used" (or something else)?


4.2. Software Reset Test

    A software reset is initiated for example from the DUT's Command
    Line Interface (CLI).

The CLI abbreviation is used already in Section 4.1. Would make sense to
move the expanded form there.

The second part of the test "Procedure" (6 lines of text) seems to be
identical for all the test cases. Perhaps this text could be in the
draft just once and have that single instance referenced in the test
case descriptions.

[Ballot discuss]
"The tester / operator MAY use either method for recovery time
  measurement depending on the test tool capability. However, the
  Frame-loss method SHOULD be used if the test tool is capable of (a)
  counting the number of lost frames per stream, and (b) transmitting
  test frame despite the physical link status, whereas Time-stamp
  method SHOULD be used if the test tool is capable of (a) time-
  stamping each frame, (b) monitoring received frame's timestamp, and
  (c) transmitting frames only if the physical link status is UP."

The document should explain

1) Why is the phy link status is only applicable in the second case?
2) Any guidance on choice of method if an equipment can do both?

====

The scaling considerations consider the RIB not the FIB. To some extent this is reasonable in that the number of routes is a topology factor and the number of FIB entries is an implementation issue. However given that in many routers it is FIB update time that dominates the topic is worthy of greater discussion. One particular case where FIB entries come into play is when you get ECMPs (which increases FIB entries) another is BGP free core (which reduces them).

====

When power cycle tests (and maybe some of the other tests) are conducted, I would expect that  the outage would be dominated by the specifics of the routing protocols, which are not particularly well characterized by this test? In particular it's difficult to see how to fairly test this without some characteristic of the routing behavior the routing peers being reported in the test results.

[Ballot comment]
"BMWG" is used without expansion.

---                                                                       

Section 3

  In order to provide consistent and fairness                               

s/constistent/consistence/

[Ballot discuss]
It looks to me as though a number of these tests might require that
forwarding state is re-established from an external source (e.g.,
re-learned from neighbors). Are these situations explicitly excluded
from the set of benchmarking methodologies described in this document,
in which case it would be really nice to call this out in the Scope
section.

If control plane interactions with neighbors were needed, you would
have a much wider field to describe in order to ensure "fair" reuslts
for the DUT.

---

Further to Russ's Discuss, I find the following paragraph in Section
4.1 confusing:

  Reset procedures that do not require the physical removal and
  insertion of a hardware component are RECOMMENDED. These include
  using the CLI or a physical switch or button. If such procedures
  cannot be performed (e.g., for lack of platform support, or because
  the corresponding Test Case calls for them), human operation time
  SHOULD be minimized across different platforms and Test Cases as
  much as possible, and variation in human operator time SHOULD also
  be minimized across different vendors products as much as practical,
  by having the same person perform the operation, and by practicing
  the operation.

Surely the variations in operator time will not matter. For example,
the time that a card is out for will not change the test, which is
measuring the time to recover from when the card is reinsterted. Thus,
there is no fear of variation between vendors' products becuase the
ease or insertion of a card is not relevant to the atomic event of
the card being (finally) inserted.

More of an issue would be how you synchronize the timing event with
the reinsertion of the card.

That means hat your Recommendation is sound, but the motivation is
suspect.

[Ballot discuss]
The Gen-ART Review by Joel Halpern on 27-Nov-2010 included two
  questions.  I think that the should be answered before the
  document is approved:

  Is there a reason that section 4.1.1.1 on reset of a single-RP device
  refers to simple removal and re-insertion, without explaining the
  caveats that section 4.1.2 (line card removal and re-insertion)
  provides about using a method to avoid having the human time for
  the removal and reinsertion have a significant effect on the
  operation?

  Also, should the same caveat not be applied to section 4.3.1 on
  Power Interruption?  There seems to be an implicit assumption that
  the human operation will be sufficiently fast, and the recovery
  sufficiently slow, that the human element does not matter.  At the
  very least, it would seem that this should be articulated.

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

To: IETF-Announce 
From: The IESG
Reply-to: ietf@ietf.org
CC:
Subject: Last Call: draft-ietf-bmwg-reset (Device Reset Characterization) to Informational RFC

The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:

- 'Device Reset Characterization '
    as an Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action.  Please send substantive comments to the
ietf@ietf.org mailing lists by 2010-12-07. Exceptionally,
comments may be sent to iesg@ietf.org instead. In either case, please
retain the beginning of the Subject line to allow automated sorting.

The file can be obtained via
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-reset-03.txt


IESG discussion can be tracked via
https://datatracker.ietf.org/public/pidtracker.cgi?command=view_id&dTag=19981&rfc_flag=0

(1.a) Who is the Document Shepherd for this document? Has the
Document Shepherd personally reviewed this version of the
document and, in particular, does he or she believe this
version is ready for forwarding to the IESG for publication?
Al Morton is the shepherd, has read every version since the
draft was created and subsequently adopted on the charter,
and believes it is now ready for publication.

(1.b) Has the document had adequate review both from key WG members
and from key non-WG members? Does the Document Shepherd have
any concerns about the depth or breadth of the reviews that
have been performed?
The draft has been reviewed by many active working group members and
benefits from their comments. It has also been circulated to other
performance-oriented working groups.

(1.c) Does the Document Shepherd have concerns that the document
needs more review from a particular or broader perspective,
e.g., security, operational complexity, someone familiar with
AAA, internationalization or XML?
No.

(1.d) Does the Document Shepherd have any specific concerns or
issues with this document that the Responsible Area Director
and/or the IESG should be aware of? For example, perhaps he
or she is uncomfortable with certain parts of the document, or
has concerns whether there really is a need for it. In any
event, if the WG has discussed those issues and has indicated
that it still wishes to advance the document, detail those
concerns here. Has an IPR disclosure related to this document
been filed? If so, please include a reference to the
disclosure and summarize the WG discussion and conclusion on
this issue.
There are no concerns and no IPR disclosures.

(1.e) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with
others being silent, or does the WG as a whole understand and
agree with it?
After several WG Last Calls, the latest going quietly except for some
editorial comments, both the active and the passive seem now to be satisfied.

(1.f) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarize the areas of conflict in
separate email messages to the Responsible Area Director. (It
should be in a separate email because this questionnaire is
entered into the ID Tracker.)
No.
(1.g) Has the Document Shepherd personally verified that the
document satisfies all ID nits? (See
http://www.ietf.org/ID-Checklist.html and
http://tools.ietf.org/tools/idnits/). Boilerplate checks are
not enough; this check needs to be thorough. Has the document
met all formal review criteria it needs to, such as the MIB
Doctor, media type and URI type reviews?
Yes.


(1.h) Has the document split its references into normative and
informative? Are there normative references to documents that
are not ready for advancement or are otherwise in an unclear
state? If such normative references exist, what is the
strategy for their completion? Are there normative references
that are downward references, as described in [RFC3967]? If
so, list these downward references to support the Area
Director in the Last Call procedure for them [RFC3967].
The references are split.
There is one informative reference to a draft that was since updated.
No down-refs, all are drafts in this series are informational.

(1.i) Has the Document Shepherd verified that the document IANA
consideration section exists and is consistent with the body
of the document? If the document specifies protocol
extensions, are reservations requested in appropriate IANA
registries? Are the IANA registries clearly identified? If
the document creates a new registry, does it define the
proposed initial contents of the registry and an allocation
procedure for future registrations? Does it suggest a
reasonable name for the new registry? See [RFC5226]. If the
document describes an Expert Review process has Shepherd
conferred with the Responsible Area Director so that the IESG
can appoint the needed Expert during the IESG Evaluation?
N/A

(1.j) Has the Document Shepherd verified that sections of the
document that are written in a formal language, such as XML
code, BNF rules, MIB definitions, etc., validate correctly in
an automated checker?
N/A

(1.k) The IESG approval announcement includes a Document
Announcement Write-Up. Please provide such a Document
Announcement Write-Up? Recent examples can be found in the
"Action" announcements for approved documents. The approval
announcement contains the following sections:

Technical Summary
Relevant content can frequently be found in the abstract
and/or introduction of the document. If not, this may be
an indication that there are deficiencies in the abstract
or introduction.
An operational forwarding device may need to be re-started
(automatically or manually) for a variety of reasons, an event that
we call a "reset" in this document. Since there may be an
interruption in the forwarding operation during a reset, it is
useful to know how long a device takes to resume the forwarding
operation.

This document specifies a methodology for characterizing reset (and
recovery time) during benchmarking of forwarding devices, and
provides clarity and consistency in reset test procedures beyond
what is specified in RFC2544. It therefore updates RFC2544.

Working Group Summary
Was there anything in WG process that is worth noting? For
example, was there controversy about particular points or
were there decisions where the consensus was particularly
rough?
During the development of MPLS Forwarding Benchmarks as an add-on to
our fundamental forwarding plane memo, RFC 2544, the additional complexities
of RESET testing with current network devices became an issue.
The WG moved quickly to resolve the concerns with a new set of methods,
and this memo is the result. Consensus was achieved after issues were
resolved in list discussions.

Document Quality
Are there existing implementations of the protocol? Have a
significant number of vendors indicated their plan to
implement the specification? Are there any reviewers that
merit special mention as having done a thorough review,
e.g., one that resulted in important changes or a
conclusion that the document had no substantive issues? If
there was a MIB Doctor, Media Type or other expert review,
what was its course (briefly)? In the case of a Media Type
review, on what date was the request posted?
Since this is an easily understood topic, many folks read this memo, simply
nodded and moved-on. Resets in hardware, in control software, and interruption
of power supply are addressed in this new methodology.