Benchmarking Terminology for Protection Performance
RFC 6414
Revision differences
Document history
| Date | Rev. | By | Action |
|---|---|---|---|
|
2017-05-16
|
09 | (System) | Changed document authors from "Scott Poretsky" to "Scott Poretsky, Jay Karthik, Samir Vapiwala, Rajiv Papneja" |
|
2015-10-14
|
09 | (System) | Notify list changed from bmwg-chairs@ietf.org, draft-ietf-bmwg-protection-term@ietf.org to (None) |
|
2012-08-22
|
09 | (System) | post-migration administrative database adjustment to the No Objection position for Peter Saint-Andre |
|
2012-08-22
|
09 | (System) | post-migration administrative database adjustment to the No Objection position for Lars Eggert |
|
2011-11-22
|
09 | Amy Vezza | State changed to RFC Published from RFC Ed Queue. |
|
2011-11-21
|
09 | (System) | RFC published |
|
2010-08-10
|
09 | (System) | IANA Action state changed to No IC from In Progress |
|
2010-08-10
|
09 | (System) | IANA Action state changed to In Progress |
|
2010-08-09
|
09 | Cindy Morgan | State Changes to RFC Ed Queue from Approved-announcement sent by Cindy Morgan |
|
2010-08-06
|
09 | Cindy Morgan | IESG state changed to Approved-announcement sent |
|
2010-08-06
|
09 | Cindy Morgan | IESG has approved the document |
|
2010-08-06
|
09 | Cindy Morgan | Closed "Approve" ballot |
|
2010-07-12
|
09 | Ron Bonica | State Changes to Approved-announcement to be sent from IESG Evaluation::AD Followup by Ron Bonica |
|
2010-07-11
|
09 | Lars Eggert | [Ballot Position Update] Position for Lars Eggert has been changed to No Objection from Discuss by Lars Eggert |
|
2010-07-08
|
09 | Peter Saint-Andre | [Ballot discuss] [cleared] |
|
2010-07-08
|
09 | Peter Saint-Andre | [Ballot Position Update] Position for Peter Saint-Andre has been changed to No Objection from Discuss by Peter Saint-Andre |
|
2010-07-08
|
09 | (System) | Sub state has been changed to AD Follow up from New Id Needed |
|
2010-07-08
|
09 | (System) | New version available: draft-ietf-bmwg-protection-term-09.txt |
|
2010-06-20
|
09 | Samuel Weiler | Request for Last Call review by SECDIR Completed. Reviewer: Rob Austein. |
|
2010-06-18
|
09 | (System) | Removed from agenda for telechat - 2010-06-17 |
|
2010-06-17
|
09 | Cindy Morgan | State Changes to IESG Evaluation::Revised ID Needed from IESG Evaluation by Cindy Morgan |
|
2010-06-17
|
09 | Dan Romascanu | [Ballot comment] It looks to me that 'Benchmarking Terminology for Performance of sub-IP layer Protection Mechanisms' would be a more apropriate name for this document. |
|
2010-06-17
|
09 | Dan Romascanu | [Ballot Position Update] New position, Yes, has been recorded by Dan Romascanu |
|
2010-06-17
|
09 | Gonzalo Camarillo | [Ballot Position Update] New position, No Objection, has been recorded by Gonzalo Camarillo |
|
2010-06-17
|
09 | Jari Arkko | [Ballot Position Update] New position, No Objection, has been recorded by Jari Arkko |
|
2010-06-16
|
09 | Adrian Farrel | [Ballot Position Update] New position, No Objection, has been recorded by Adrian Farrel |
|
2010-06-16
|
09 | Adrian Farrel | [Ballot comment] I have a huge raft of issues and concerns about this document. In the end I raised a COMMENT not a DISCUSS because … [Ballot comment] I have a huge raft of issues and concerns about this document. In the end I raised a COMMENT not a DISCUSS because I don't believe that the publication of this document as an Informational RFC without making the changes will be harmful. However, attending to these comments would, I believe significantly improve the value of your work. The responsible AD may decide that the comments need to be addressed. --- idnits (http://tools.ietf.org/tools/idnits/) throws up a number of issues. Although many of these are minor or editorial, it would have made the document easier to read had you fixed them. I think that some of the boilerplate issues are more significant and that the I-D cannot be accepted unless a new revision with the correct boilerplate is submitted. The document writeup says: (1.g) Has the Document Shepherd personally verified that the document satisfies all ID nits? (See http://www.ietf.org/ID-Checklist.html and http://tools.ietf.org/tools/idnits/). Boilerplate checks are not enough; this check needs to be thorough. Has the document met all formal review criteria it needs to, such as the MIB Doctor, media type and URI type reviews? There are a few errors in the nits: http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-bmwg-protection-term-08.txt These can be corrected after AD-review, but there will be some work necessary to adopt the new boilerplate, update references, fix page lengths, etc. Clearly this did not happen, and I really think we need to push back on document shepherds to understand that the only acceptable answer to the question is "Yes, idnits passes cleanly". --- There is a significant different between the document title ("Benchmarking Terminology for Protection Performance") and the content of the document as explained in the Abstract and Introduction. Viz. This document provides common terminology and metrics for benchmarking the performance of sub-IP layer protection mechanisms. 1. The title should mention metrics 2. The title should indicate the scope is sub-IP --- The Introduction starts off with... Technologies that function at sub-IP layers can be enabled to provide further protection of IP traffic by providing the failure recovery at the sub-IP layers so that the outage is not observed at the IP-layer. This seems a reasonable statement, but isn't it in contradiction with the whole premise of this document as stated in the Abstract... The performance benchmarks are measured at the IP-Layer, avoiding dependence on specific sub-IP protection mechanisms. In other words, if the outage is not observed at the IP-layer, how will you measure the benchmarks at the IP-Layer? --- I would really have liked this work to be aligned with RFC 4427. That document sets out terminology for protection and restoration in GMPLS networks (i.e. sub-IP) and attempts to achieve alignment with ITU-T Recommendations G.808.1 and G.841. It is important to use identical rather than similar terms for protection and restoration techniques across the IETF when we are referring to sub-IP layers. Actually, I find the terminology rather woolly. For example, in Section 1 The Working Path is the Primary Path prior to the Failover Event and the Backup Path after the Failover Event. Yet in Section 3.1.3 The Primary Path is the Path that traffic traverses prior to a Failover Event. And it is difficult to read a document where the terms are used well in advance of their definitions. It might help if Section 3 was split with sections 3.1 through 3.4 presented as terminology, and sections 3.5 onwards as Test Considerations. The definition of "Restoration" in section 3.3.5 is unlike anything I have seen before and is very much at odds with the way the term is used in sub-IP networks. --- Figure 4 There is a spurious arrow head on the left-hand end of the arrow marked "IP-Layer Forwarding" --- Section 3.1.1 a. R1 is the ingress node and forwards IP packets, which input into DUT/SUT, to R2 as sub-IP frames over link L12. b. Ri is a node which forwards data frames to R[i+1] over Link Li[i+1] for all i, 1 |
|
2010-06-16
|
09 | Robert Sparks | [Ballot Position Update] New position, No Objection, has been recorded by Robert Sparks |
|
2010-06-16
|
09 | Russ Housley | [Ballot Position Update] New position, No Objection, has been recorded by Russ Housley |
|
2010-06-16
|
09 | Stewart Bryant | [Ballot comment] I am not sure that RFC2119 language is appropriate here Figures 1 through 5 show models that MAY be used when benchmarking … [Ballot comment] I am not sure that RFC2119 language is appropriate here Figures 1 through 5 show models that MAY be used when benchmarking Sub-IP Protection mechanisms, which MUST use a Protection Switching System that consists of a minimum of two Protection-Switching Nodes, an Ingress Node known as the Headend Node and an Egress Node known as the Merge Node. Ideally the RFC2119 definition should appear before first use in the document The Protection Switching System MUST include either a Primary Path and Backup Path, as shown in Figures 1 through 4, or a Primary Node and Standby Node, as shown in Figure 5. I am not a mathematician, can an equation have a pluarity of sub-equations, or is a different mathematical construct needed? TBLM as shown in Equation 2: (Equation 2) (Equation 2a) TBLM Failover Time = Time(Failover) - Time(Failover Event) (Equation 2b) TBLM Reversion Time = Time(Reversion) - Time(Restoration) Same with Eq3 |
|
2010-06-16
|
09 | Stewart Bryant | [Ballot Position Update] New position, No Objection, has been recorded by Stewart Bryant |
|
2010-06-16
|
09 | Sean Turner | [Ballot Position Update] New position, No Objection, has been recorded by Sean Turner |
|
2010-06-15
|
09 | Tim Polk | [Ballot comment] I suspect this is clear enough for its intended audience, but a more detailed description of the calculations in 3.6.1 and 3.6.3 would … [Ballot comment] I suspect this is clear enough for its intended audience, but a more detailed description of the calculations in 3.6.1 and 3.6.3 would help novice readers. I couldn't quite sort out the equations. For example in 3.6.3 (TBM), are the following interpretations correct? (1) Time(Failover) = Timestamp on first unimpaired packet received at egress node after the backup path became the working (2) Time(Failover Event) = Timestamp on the last unimpaired packet received at egress node on the primary path before failure I was unable to construct similar statements for the TBLM in 3.6.1 at all. Nits: in section 3.1.1, there is a confusing mixture of notation, using parentheses and brackets interchangeably. The definition of path uses parentheses, as in "L(n-1)n", but the description of node in subitem b. uses brackets, as in "Li[i+1]". In 3.6.3 under discussion, the comma following "observation of unimpaired packets" should be a period. |
|
2010-06-15
|
09 | Tim Polk | [Ballot comment] I suspect this is clear enough for its intended audience, but a more detailed description of the calculations in 3.6.1 and 3.6.3 would … [Ballot comment] I suspect this is clear enough for its intended audience, but a more detailed description of the calculations in 3.6.1 and 3.6.3 would help novice readers. I couldn't quite sort out the equations. For example in 3.6.3 (TBM), are the following interpretations correct? (1) Time(Failover) = Timestamp on first unimpaired packet received at egress node after the backup path became the working (2) Time(Failover Event) = Timestamp on the last unimpaired packet received at egress node on the primary path before failure I was unable to construct similar statements for the TBLM in 3.6.1 at all. Nits: in section 3.1.1, there is a confusing mixture of notation, using parentheses and brackets interchangeably. The definition of path uses parentheses, as in "L(n-1)n", but the description of node in subitem b. uses brackets, as in "Li[i+1]". In 3.6.3 under discussion, the comma following "observation of unimpaired packets" should be a period. |
|
2010-06-15
|
09 | Tim Polk | [Ballot comment] I suspect this is clear enough for its intended audience, but a more detailed description of the calculations in 3.6.1 and 3.6.3 would … [Ballot comment] I suspect this is clear enough for its intended audience, but a more detailed description of the calculations in 3.6.1 and 3.6.3 would help novice readers. I couldn't quite sort out the equations. For example in 3.6.3 (TBM), are the following interpretations correct? (1) Time(Failover) = Timestamp on first unimpaired packet received at egress node after the failover event (2) Time(Failover Event) = Timestamp on the last unimpaired packet received at egress node before the failover event I was unable to construct similar statements for the TBLM in 3.6.1 at all. Nits: in section 3.1.1, there is a confusing mixture of notation, using parentheses and brackets interchangeably. The definition of path uses parentheses, as in "L(n-1)n", but the description of node in subitem b. uses brackets, as in "Li[i+1]". In 3.6.3 under discussion, the comma following "observation of unimpaired packets" should be a period. |
|
2010-06-15
|
09 | Tim Polk | [Ballot Position Update] New position, No Objection, has been recorded by Tim Polk |
|
2010-06-15
|
09 | Peter Saint-Andre | [Ballot discuss] 1. The document does not pass ID-nits. http://tools.ietf.org/tools/idnits/ 2. This is a very interesting normative reference: [6] Not used. Please … [Ballot discuss] 1. The document does not pass ID-nits. http://tools.ietf.org/tools/idnits/ 2. This is a very interesting normative reference: [6] Not used. Please delete that non-reference and renumber accordingly. 3. It appears that references [8] and [9] (RFC 4090 and RFC 2474) are not in fact normative. |
|
2010-06-15
|
09 | Peter Saint-Andre | [Ballot Position Update] Position for Peter Saint-Andre has been changed to Discuss from No Objection by Peter Saint-Andre |
|
2010-06-15
|
09 | Peter Saint-Andre | [Ballot Position Update] New position, No Objection, has been recorded by Peter Saint-Andre |
|
2010-06-14
|
09 | Lars Eggert | [Ballot comment] > Benchmarking Terminology > for … [Ballot comment] > Benchmarking Terminology > for Protection Performance As stated in the abstract, Section 3.5 actually defines some tests. The title should reflect that this document is not only defining terminology. Section 3.1., paragraph 4: > b. Ri is a node which forwards data frames to R[i+1] over Link > Li[i+1] for all i, 1<i<n, based on information in the sub-IP > layer. Since R1/L12 are defined in (a) and (c) defines Rn, you probably want to change 1 |
|
2010-06-14
|
09 | Lars Eggert | [Ballot discuss] Section 3., paragraph 0: > 3. Test Considerations DISCUSS: Some of the "discussion" sections of the definitions in Sections 3.1-3.4 use … [Ballot discuss] Section 3., paragraph 0: > 3. Test Considerations DISCUSS: Some of the "discussion" sections of the definitions in Sections 3.1-3.4 use RFC2119 terms, which is not appropriate, because those paragraphs are clearly informative. |
|
2010-06-14
|
09 | Lars Eggert | [Ballot Position Update] New position, Discuss, has been recorded by Lars Eggert |
|
2010-06-01
|
09 | Ron Bonica | [Ballot Position Update] New position, Yes, has been recorded for Ronald Bonica |
|
2010-06-01
|
09 | Ron Bonica | Ballot has been issued by Ron Bonica |
|
2010-06-01
|
09 | Ron Bonica | Created "Approve" ballot |
|
2010-06-01
|
09 | Ron Bonica | State Changes to IESG Evaluation from Waiting for AD Go-Ahead by Ron Bonica |
|
2010-06-01
|
09 | Ron Bonica | Placed on agenda for telechat - 2010-06-17 by Ron Bonica |
|
2010-05-14
|
09 | (System) | State has been changed to Waiting for AD Go-Ahead from In Last Call by system |
|
2010-05-10
|
09 | Amanda Baber | IANA comments: As described in the IANA Considerations section, we understand this document to have NO IANA Actions. |
|
2010-05-03
|
09 | Samuel Weiler | Request for Last Call review by SECDIR is assigned to Rob Austein |
|
2010-05-03
|
09 | Samuel Weiler | Request for Last Call review by SECDIR is assigned to Rob Austein |
|
2010-04-30
|
09 | Amy Vezza | Last call sent |
|
2010-04-30
|
09 | Amy Vezza | State Changes to In Last Call from Last Call Requested by Amy Vezza |
|
2010-04-30
|
09 | Ron Bonica | State Changes to Last Call Requested from AD Evaluation by Ron Bonica |
|
2010-04-30
|
09 | Ron Bonica | Last Call was requested by Ron Bonica |
|
2010-04-30
|
09 | (System) | Ballot writeup text was added |
|
2010-04-30
|
09 | (System) | Last call text was added |
|
2010-04-30
|
09 | (System) | Ballot approval text was added |
|
2010-04-23
|
09 | Ron Bonica | State Changes to AD Evaluation from Publication Requested by Ron Bonica |
|
2010-04-12
|
09 | Amy Vezza | This is a publication request for http://tools.ietf.org/html/draft-ietf-bmwg-protection-term-08 as an INFORMATIONAL RFC. (1.a) Who is the Document Shepherd for this document? Has the … This is a publication request for http://tools.ietf.org/html/draft-ietf-bmwg-protection-term-08 as an INFORMATIONAL RFC. (1.a) Who is the Document Shepherd for this document? Has the Document Shepherd personally reviewed this version of the document and, in particular, does he or she believe this version is ready for forwarding to the IESG for publication? Al Morton is the shepherd, has read this version and says its ready. (1.b) Has the document had adequate review both from key WG members and from key non-WG members? Does the Document Shepherd have any concerns about the depth or breadth of the reviews that have been performed? Yes, I believe so. This work was initially proposed in 2002, and was discussed extensively with many members of the community before working group adoption in 2006. Extensive review was sought in the last two rounds of WGLC. (1.c) Does the Document Shepherd have concerns that the document needs more review from a particular or broader perspective, e.g., security, operational complexity, someone familiar with AAA, internationalization or XML? No. There will likely be a few more issues raised by the IESG, but no laws-of-physics class violations that might stall the work. (1.d) Does the Document Shepherd have any specific concerns or issues with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here. Has an IPR disclosure related to this document been filed? If so, please include a reference to the disclosure and summarize the WG discussion and conclusion on this issue. There are no concerns and no IPR disclosures. (1.e) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it? The consensus is solid. Many WG members read and agreed with this document at various times. Many readers completed the BMWG Active Review template for this draft. (1.f) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarize the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is entered into the ID Tracker.) No. (1.g) Has the Document Shepherd personally verified that the document satisfies all ID nits? (See http://www.ietf.org/ID-Checklist.html and http://tools.ietf.org/tools/idnits/). Boilerplate checks are not enough; this check needs to be thorough. Has the document met all formal review criteria it needs to, such as the MIB Doctor, media type and URI type reviews? There are a few errors in the nits: http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-bmwg-protection-term-08.txt These can be corrected after AD-review, but there will be some work necessary to adopt the new boilerplate, update references, fix page lengths, etc. (1.h) Has the document split its references into normative and informative? Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the strategy for their completion? Are there normative references that are downward references, as described in [RFC3967]? If so, list these downward references to support the Area Director in the Last Call procedure for them [RFC3967]. Yes, the references are split. There is a normative dependency on: "Benchmarking Terminology for IGP Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-21, (to be published) which is also effectively at the Publication Requested state. (Authors: this reference needs to be updated in the next version) (1.i) Has the Document Shepherd verified that the document IANA consideration section exists and is consistent with the body of the document? If the document specifies protocol extensions, are reservations requested in appropriate IANA registries? Are the IANA registries clearly identified? If the document creates a new registry, does it define the proposed initial contents of the registry and an allocation procedure for future registrations? Does it suggest a reasonable name for the new registry? See [RFC5226]. If the document describes an Expert Review process has Shepherd conferred with the Responsible Area Director so that the IESG can appoint the needed Expert during the IESG Evaluation? N/A (1.j) Has the Document Shepherd verified that sections of the document that are written in a formal language, such as XML code, BNF rules, MIB definitions, etc., validate correctly in an automated checker? N/A (1.k) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up? Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections: Technical Summary Relevant content can frequently be found in the abstract and/or introduction of the document. If not, this may be an indication that there are deficiencies in the abstract or introduction. The purpose of this document is to provide a single terminology for benchmarking sub-IP protection mechanisms. Technologies that transport IP packets can be designed to provide protection for IP traffic by providing the failure recovery at lower layers. Possibly, the outage is not even observed at the IP-layer. Sub-IP protection technologies include, but are not limited to, High Availability (HA) stateful failover, Virtual Router Redundancy Protocol (VRRP), Automatic Link Protection (APS) for SONET/SDH, Resilient Packet Ring (RPR) for Ethernet, and Fast Reroute for Multi-Protocol Label Switching (MPLS-FRR). Benchmarking terminology was defined for IP-layer convergence in draft-ietf-bmwg-igp-dataplane-conv-term-21 . Different terminology and methodologies specific to benchmarking sub-IP layer protection mechanisms are required. The metrics for benchmarking the performance of sub-IP protection mechanisms are measured at the IP layer, so that the results are independent of the specific protection mechanism being used. Working Group Summary Was there anything in WG process that is worth noting? For example, was there controversy about particular points or were there decisions where the consensus was particularly rough? Working group consensus was smoothly achieved over a period of several years, with many controversies ironed-out before WG adoption. Several authors came and went, but others were willing to pick-up the work and complete it, so here we are. Document Quality Q - Are there existing implementations of the protocol? A these documents neither propose a new protocol nor extend any existing one, hence current implementations are not required to support this document. The document only attempts to standardize the benchmarking strategies so that service providers could evaluate multiple protection implementations using standard figures of merit with an aim of producing consistent test results across multiple platforms. Q - Have a significant number of vendors indicated their plan to implement the specification? A - Several test vendors, including the leading ones, have shown their support and interest in the execution of these methodologies. The authors have regularly acknowledged them in WG status presentations. Reference/acknowledgements were shared in the following: http://www.ietf.org/proceedings/06jul/slides/bmwg-3/bmwg-3.ppt Q - Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? A - The following lays down the history of this work item, which demonstrates substantial revisions of the documents based on comments form the BMWGers 1. July 2005: Common protection terminology created due to merger of draft-kimura-protection-term-02.txt and draft-poretsky-mpls-protection-meth-02 as result of series of reviews and comments. New draft-poretsky-protection-term-00.txt created. 2. December 2005: another effort with additional test scenarios (draft-vapiwala-bmwg-frr-failover-meth-00.txt) was merged into draft-poretsky-meth-05. It was felt that additional scenarios proposed were needed to offer comprehensive MPLS (including the services) based protection techniques evaluation. Terms checked against this new methodology. 3. June 2006: Due to additional test items proposal, a new draft was created draft-papneja-mpls-protection-meth-merge-00.txt 4. Also in the combined effort Mr. Vapiwala were included as one of the co-authors due to his contribution to the effort in the form of additional test scenarios for the methodology document, and accordingly updating the terminology document. 5. The acknowledgement section is included for terminology document to thank those who significantly helped in shaping the final version. The document did receive comments from one of co-authors of original fast reroute RFC 4090 in the early versions of the work item document in addition to other pioneers in the MPLS field. |
|
2010-04-12
|
09 | Amy Vezza | Draft Added by Amy Vezza in state Publication Requested |
|
2010-04-12
|
09 | Amy Vezza | [Note]: 'Al Morton (acmorton@att.com) is the shepherd.' added by Amy Vezza |
|
2009-12-21
|
08 | (System) | New version available: draft-ietf-bmwg-protection-term-08.txt |
|
2009-10-16
|
07 | (System) | New version available: draft-ietf-bmwg-protection-term-07.txt |
|
2009-03-08
|
06 | (System) | New version available: draft-ietf-bmwg-protection-term-06.txt |
|
2008-11-03
|
05 | (System) | New version available: draft-ietf-bmwg-protection-term-05.txt |
|
2008-02-25
|
04 | (System) | New version available: draft-ietf-bmwg-protection-term-04.txt |
|
2007-11-13
|
03 | (System) | New version available: draft-ietf-bmwg-protection-term-03.txt |
|
2007-07-09
|
02 | (System) | New version available: draft-ietf-bmwg-protection-term-02.txt |
|
2007-03-08
|
01 | (System) | New version available: draft-ietf-bmwg-protection-term-01.txt |
|
2006-10-16
|
00 | (System) | New version available: draft-ietf-bmwg-protection-term-00.txt |