Internet Engineering Task Force Mark Allman INTERNET DRAFT ICIR File: draft-allman-icar-wg-revcomm-00.txt James Kempf DoCoMo Labs USA April, 2004 Expires: October, 2004 Using Working Group Review Committees Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of [RFC2026]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document sketches a potential quality control mechanism for the IETF in the form of working group review committees. The idea is to form a small committee per working group that will provide document review and advice throughout the working group's lifetime. 1 Introduction In this document we sketch a possible quality control mechanism for the IETF that is based on forming a small review committee for each working group. While the members of this committee would have no special privledges within the IETF process we believe that a small group may aid the WG in several ways: + Provide architectural advice from a broader vantage point than the WG's focused viewpoint to help the WG understand the interactions of their work with the entire system. + Provide a view point from a different area of the IETF (e.g., a security viewpoint in an applications area working group). + Provide a place for vetting the WGs output to a broader group Expires: October 2004 [Page 1]
draft-allman-icar-wg-revcomm-00.txt April 2004 than the WG itself before the output is sent to the IESG. The review committee notion is in some ways a refinement of the "stuckee" notion [Har03]. While the stuckee notion is focused on (slightly) formalizing roles within a WG, a review committee is mainly focused on obtaining "outside" input (important input that generally comes from people who would otherwise not be deeply involved with the WG). The ideas outlined in this document attempt to provide a similar kind of quality control provided by the SIRS proposal [CC03]. Individual SIR members are logical candidates for the review committee, but the committee concept goes further in that the review committee may draw in people who are not traditionally active in IETF, like implementors, academics, or people active in other fora who may have an interest in the work and the experience to judge it. An additional difference between review committees and SIRS is in the collection of the reviews undertaken by given individuals. With a review committee a group of related WG documents will be considered by the same group of reviewers. There is no such grouping of documents for a given reviewer in the SIRS concept. The notion of review committees also is an extension of the "technical advisor" that has traditionally been appointed to various IETF WGs. In some WGs the notion of a "review committee" is already informally put to use. For instance, when documents are at a critical stage a WG chair will ask several people from around the IETF to read and comment on the draft. This document formalizes this notion so that the committee is setup apriori for the WG chair(s) to utilize. 2 Forming a Review Committee Each WG forms a review committee whose size is dictated by the WG's workload and breadth. There are no formal rules about who can sit on a review committee. Likewise, there is no explicit rule on the size of the review committee, but the size should be somewhat dependent on the number of work items a particular WG has (an initial touchstone would be 2 members per document). The committee is chosen by the WG chair(s) and the area directors. While there are no hard and fast rules there are some guidelines in choosing committee members: + A security guru is a likely must for groups outside the security area. + If the WG is involved in writing MIBs, a MIB doctor would be useful. + A transport guru is probably necessary for those groups where the choice of transport protocol is not obvious. Beyond obvious choices the WG chairs and ADs should pick several people from different areas of the IETF to ensure the documents within the group received a thorough review from a number of angles. Expires: October 2004 [Page 2]
draft-allman-icar-wg-revcomm-00.txt April 2004 Particular attention should be paid to choosing people who are otherwise not engaged with the working group -- to ensure that a truely different perspective is gained from employing the review committees. In other words, review committees should not be constituted of the "usual suspects" who will review a given WGs documents anyway. Finally, viewpoints from outside the IETF may also be useful to include (e.g., from operators that do not normally participate but have a particularly interesting vantage point on some work). A committee is formed by consultation between the WG chairs and shepherding AD. The WG chairs generate a list of potential members, and discuss with the AD about who might be appropriate. After an agreed upon list of members has been generated, the WG chairs contact the committee members to assess their willingness to serve. 3 Role of the Review Committee The review committee looks at documents as requested by the WG chair(s). All reviews are sent to the WG mailing list for public comment and archiving. The comments from the review committee are to be treated no differently than reviews from WG members or any other member of the community. The review committee members are not expected to closely track the WG if they do not have the time or inclination. Some members of the committee may want to engage the WG on the mailing list with regards to the work and their reviews, while other members may not. Either of these is fine (although, the WG chair(s) and AD(s) may consider a potential member's stated participation plan when forming the committee). Should a committee member not directly participate on the mailing list or in meetings they should be available for questions and clarifications about their reviews to the WG chair(s). Review committee members contributions are acknowledged in two ways: + By placing their names on a Working Group's Web page. + Within the Contributions section of the Working Group documents they have reviewed. In order for a review committee member to qualify for acknowledgment, they must generate at least one detailed review of a working group draft. People who agree to particate but don't contribute will not be listed as contributors. WG chairs are encouraged to utilize review committees as early as possible to review WG documents (or, even potential WG work items). This early vetting is designed to illuminate large issues in the documents before much time and effort is spent polishing some facet of a protocol that will ultimately be fundamentally changed. 4 Experience with Review Committees Expires: October 2004 [Page 3]
draft-allman-icar-wg-revcomm-00.txt April 2004 Review committees have been used in the Seamoby WG and SEND WG. In both cases, high quality, detailed reviews of protocol documents were generated by most of the review committee members. Only a few of the reviews were perfunctory. In the Seamboy case, two reviewers were selected for their extensive expertise in implementing Mobile IP, though they did not typically participate in the IETF. Several of the reviewers on the Seamoby committee followed up with discussion on the mailing list, even though they did not typically participate in the WG. One issue that arose with the Seamoby review committees is that because the WG is not obliged to accept the review committee's opinion, even if the review committee members are senior and respected people, there is no guarantee that problems identified by the review committee will be addressed by the WG. This is in contrast to review committees for academic conferences and journals, where authors are typically required to address points raised during peer review, or their paper will not be published. An attempt was made to limit the demands on the review committee members, so reviews were only solicited at the semi-final and final stages of document preparation. Since review committee members may not be full time WG members, gauging exactly how much to ask of them is critical to maintaining their commitment and interest. All in all, the experience with review committees in these two cases has been extremely positive. The amount and quality of the technical input obtained from the review committees far and away exceeded the amount of input typically generated when reviews are not directly solicited, such as by issuance of a WG last call. To be effective, reviews need to be accompanied by effective WG issue management. Each review should be used to generate a list of issues recorded, along with other issues raised by the WG, on an issues Web page. Document editors, or, in the absence of strong editorship, the WG chairs, then become responsible for posting issues on the mailing list and monitoring discussion until consensus is achieved on the issues. The results of this discussion then need to be edited back into the WG documents. Editorial issues, which are also raised by reviewers, typically don't need as strict discussion, though editors and WG chairs need to assess carefully whether an editorial issue will affect interpretation of the document, and discuss it with the WG if so. 5 A Review Committee Experiment We believe that the review committee concept can be accomplished within the given IETF procedures. We, however, encourage WG chairs and area directors to form WG review committees, utilize them and report the results back to the community (via the ICAR working group). Non-Normative References [CC03] Brian Carpenter, Dave Crocker. Careful Additional Review of Expires: October 2004 [Page 4]
draft-allman-icar-wg-revcomm-00.txt April 2004 Documents (CARD) by Senior IETF Reviewers (SIRS). Internet-Draft draft-carpenter-icar-sirs-00.txt, March 2004. Work in progress. [Har03] Ted Hardie. Working Groups and Their Stuckees. Internet-Draft draft-hardie-wg-stuckees-00.txt, February 2003. Work in progress. Authors' Addresses: Mark Allman ICSI Center for Internet Research 1947 Center Street, Suite 600 Berkeley, CA 94704-1198 Phone: 216-243-7361 mallman@icir.org http://www.icir.org/mallman/ James Kempf DoCoMo Labs USA 180 Metro Drive, Suite 300 San Jose, CA 95430 Phone: +1 408 451 4711 Email: kempf@docomolabs-usa.com Expires: October 2004 [Page 5]