Network Working Group B. Carpenter Internet Draft IBM draft-carpenter-solution-sirs-01.txt D. Crocker Expires: <12-03> Brandenburg June 2003 Careful Additional Review of Documents (CARD) by Senior IETF Reviewers (SIRS) (draft-carpenter-solution-sirs-01.txt) COPYRIGHT NOTICE If published as an RFC this document will become Copyright (C) The Internet Society (2003). All Rights Reserved. ABSTRACT IETF specifications may not receive significant review, especially beyond a single working group, until they are submitted to the IESG. Hence, significant problems with a specification often are not detected until a problematic approach has been adopted and considerable effort has been spent developing and documenting this approach. Major changes to address problems identified late in the process take considerable effort and significantly delay a document that its authors believed to be ready for publication. The procedure described in this document is intended to solve, or palliate, a number of related problems that have been observed in the IETF process. The basic model is to create a team of Senior IETF Reviewers (SIRS), and have all documents receive a certain number of reviews by SIRs, prior to being submitted for publication. Review by SIRs at a very early stage is strongly encouraged. STATUS OF THIS MEMO This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. TABLE OF CONTENTS 1. Introduction 2. SIRs 2.1. The Body of Senior Internet Reviewers 2.2. Obtaining SIR Participation 3. Carding 3.1. Reviewing in Public 3.2. Spreading the load among SIRs 3.3. Form of a Review 3.4. Iterative Carding 4. Security considerations 5. Acknowledgements 6. Informative References 7. Authors' Addresses 8. Intellectual Property 9. Full Copyright Statement 1. INTRODUCTION IETF specifications may not receive significant review, especially beyond a single working group, until they are submitted to the IESG or the RFC Editor. Hence, significant problems with a specification often are not detected until a problematic approach has been adopted and considerable effort has been spent developing and documenting this approach. Major changes to address problems identified late in the process take considerable effort and significantly delay a document that its authors believed to be ready for publication. We can speculate that this problem explains a large part of the long tail in the distribution of IESG document approval times, as well as a perception that the IESG has too much authority. The procedure described in this document is intended to solve, or palliate, a number of related problems that have been observed in the IETF process [PROBLEM]: * perception that authority and influence in the IETF are concentrated in too few hands * lack of explicit quality auditing throughout the standards development process. * although there may be large attendances at many WG meetings, in many cases 5% or less of the participants have read the drafts which are under discussion or have a bearing on the decisions to be made. * commitments to review a document are not carried out in a timely fashion * little or no response is seen when a request for 'last- call' review is issued either at WG or IETF level. * lack of recognition of review work as a valuable contribution * submission of documents to the IESG that still have significant problems (leading to overload and delay) * failure to detect fundamental problems and Internet- wide issues at an early stage; the IETF is not consistently effective at resolving issues that cross WG or area boundaries. Particularly because of the last point, it is impossible to resolve all these problems simply by giving additional responsibility to working groups themselves and relying on them for additional efforts. An additional procedure is needed. The procedure specified here calls for a team of 'SIRs' to 'CARD'. The verb 'card' applies both to textiles and public establishments. The former usage describes the removal of detritus from textiles and their preparation for weaving. The latter describes the checking of participants' credentials at the entrance. The term is also an acronym for 'Careful Additional Review of Documents.' The carding procedure initially makes no change to the formal process of IETF document development, review, approval, and publication. It is an additional procedure intended to tackle the problems listed above. If successful, it can be transformed into a formal process in due course. For the moment, it is not compulsory. The basic model is that the IETF will create a team of Senior IETF Reviewers (SIRs, who need be neither male nor knighted) chosen in a way designed to create trust, and that all documents should receive a certain number of reviews by SIRs, prior to being submitted to the IESG or the RFC Editor for publication. Furthermore, review at a very early stage is strongly encouraged. The model is not intended to replace existing mechanisms such as the various Directorates within the IETF and the MIB Doctor system. It will in fact build on them. The remainder of this document described how the team of SIRs is created and refreshed, how the review process works, and how it is used by document authors and working groups to achieve their objectives. 2. SIRS 2.1. The Body of Senior Internet Reviewers 2.1.1. Initial Set of SIRs With approximately 200 RFCs published per year, and with most of them having existed in multiple Internet Drafts, a large team of SIRs is needed. An initial target of 100 people is suggested. It is recognized that this may be insufficient, so the number should be revised in the light of experience. Q. Is this anything like the right number? A. Assume we need 9 reviews per year (3 review cycles, 3 reviewers each time) for each of 200 RFCs. That gives each SIR one review every 20 days, approximately. This seems high, and suggests that the final target should be 200 people. The goal is to identify a team of people with adequate experience contributing to IETF technical work, who are likely to be trusted as a group to be both technically expert and unbiased. The initial set of candidates to be SIRS is defined by objective criteria, to avoid any bias in their selection. It will consist of: * all current IAB members * all former IAB and IESG members, and former WG Chairs * all current MIB Doctors * all members of existing IETF Directorates * all authors of at least three RFCs * (other suggestions???) Candidates must, of course, agree to participate and agree to the terms of serving as a SIR. Current IESG Area Directors are excluded from the pool. Current WG Chairs are not explicitly included, but they may qualify under other criteria. Q: Why are all IAB members automatically included? A: Because the IAB is in any case supposed to carry out cross-area review. There should be no significant extra workload for an IAB member to act as a SIR. Q: Why are MIB Doctors automatically included? A: They already do the SIR job today for MIBs. There is essentially no change for them. Q: Why are current Directorates automatically included? A: Because they have been identified as experts by Area Directors and their role normally includes review duties already. Q: Should all types of RFC count, or should it be restricted to standards-track and BCP? A: Some very valuable RFCs are Informational or Experimental. It would seem unreasonable to exclude their authors. On the other hand, not all such RFCs have lasting value. Suggestions on how to adjust this criterion are welcome. Q: Why are ADs excluded? A: Partly because they have to review all final drafts anyway, partly to avoid potential conflict of interest between reviewing and approving, and partly because they are already very busy with IESG work. Q: Why are current WG Chairs not explicitly included? A: Many of them will qualify under other criteria. But the authors are open to suggestions on this point - comments welcome. 2.1.2. Addition and Removal of SIRs The team of SIRs is increased at least once each year by a public nomination process and a voting procedure among the existing SIRs. Q. Why, since we have pre-qualified a team already? A. Undoubtedly people will drop out, so new ones will be needed. Renewal by open nomination seems the most appropriate mechanism to ensure trust. The nomination period will last one month, immediately after the completion of the IESG and IAB nomination cycle, i.e. it will start during the first IETF of each calendar year. Anybody can nominate a SIR candidate, and self-nominations are allowed. The list of nominees will be public (on the IETF web site). After the nomination period, a secret vote will be conducted among the existing SIRs. Each SIR can vote for none, some or all of the nominees. Nominees who receive at least ten votes are elected. Q: Why ten? A: The basic criterion for being a SIR is community agreement that one is, in fact, technically "senior"; so someone who isn't viewed positively by at least ten of his or her peers probably isn't suitable. If the total number of SIRs becomes too great, this rule would need to be revised. SIRs who have been inactive for a full year will be automatically retired from the team. Q. Isn't this too tolerant? People may become SIRs just to get it on their resume. A. It allows for people who need to take a sabbatical for whatever personal or professional reason. But certainly the criterion for automatic retirement should be reviewed when we have some experience. In extremis, SIR status may be removed by a simple majority vote of the team of SIRs. Such a vote will be triggered by a recall petition from at least ten SIRs. The nomination, voting, retirement and recall procedures will be managed by the IETF Executive Director. Any appeals under these procedures will be handled by the IAB. 2.2. Obtaining SIR Participation 2.2.1. Working Group documents For documents being processed by a Working Group (WG), the WG solicits the assistance of SIRs. That is, the general IETF community controls who is authorized as a SIR, but WGs control which specific SIRs provides the formal review that is needed for a given document. However, SIRs may spontaneously and voluntarily review a draft. A primary goal of this proposal is to ensure that Working Groups benefit from broad experience in the design of Internet technology. Hence it is entirely reasonable that some SIRs reviewing a given document should be subject matter experts. However the full set of input from SIRs is substantially more useful when it includes SIRs from other areas. In particular, cross- area review makes it more likely that architectural and operational impacts outside of the subject matter will be detected. It is therefore strongly recommended that WGs seek a diverse set of SIRs to participate in evaluations, able to cover most if not all IETF Areas between them. Each WG will make its own decision about how its SIRs are selected (e.g. chosen by the WG Chairs, chosen by the document authors concerned, etc.). Q. If a SIR is active in the same WG, isn't there a possible conflict of interest? A. SIRs provide two benefits. One is expertise and the other is independence. Expertise is always helpful. However, yes, it will be important for some of the SIR feedback to be independent of the working group. In addition, reviewing will be a public activity, so that any conflict will be visible to the whole WG and can be dealt with openly. But... SIRs should not formally review documents in which they have a direct interest as a contributor, or as a contributor to a competing document. There is no fixed number of SIR reviews required prior to submission to the IESG. However, it is likely that drafts with at least three positive reviews from SIRs in different areas will experience much shorter IESG review cycles than drafts with fewer positive reviews. Other common sense rules will apply; for example a MIB that has not been reviewed by a MIB Doctor is unlikely to be published. In all likelihood, Drafts without SIR reviews will get worse IESG response time than today, whereas Drafts with strongly positive SIR reviews will be processed much more rapidly, especially as the IESG's confidence in the SIR procedure increases. However, since the SIR process does not change the formal document approval process, the IESG will still be obliged to weigh Last Call comments carefully, especially if they conflict with the SIR reviews. 2.2.2. Individual submissions For individual submissions, the document author(s) should solicit SIR reviews, according to the same principles applied to Working Group documents, and SIRs may spontaneously and voluntarily review a draft.. When a draft is submitted to the RFC Editor as an individual submission, the SIR reviews are available to the RFC Editor, and later to the IESG, to assist the evaluation process. Again, in all likelihood, Drafts without reviews will get worse RFC Editor response time than today, whereas Drafts with reviews will be processed much more rapidly, especially as the RFC Editor's confidence in the SIR procedure increases. It should be clear that negative reviews do not automatically lead to rejection and positive reviews do not automatically lead to acceptance. The RFC Editor's independence is intended to ensure that startling and innovative ideas may be published if appropriate. 3. CARDING 3.1. Reviewing in Public The current list of SIRs will be available on the IETF web site, along with information about their areas of expertise. All reviews will be published on the web site, with adequate tooling (linked to the ID Tracker) so that SIRs can publish reviews without adminstrative overhead, every review can be readily found, and all reviews will be automatically archived. In fact, a WG or document author in need of reviews should be able to request them through the web site. SIR reviews are not a formal part of the standards process and do not have formal authority, so they are not subject to formal appeal. However, since the reviews are public, authors are able to include rebuttals in subsequent drafts, or to issue rebuttals on the appropriate mailing list. SIR reviews are nevertheless contributions to the IETF in the sense of [TECHNO] and therefore fall under the relevant intellectual property rules of the IETF. 3.2. Spreading the load among SIRs No doubt some SIRs will be asked for too many reviews, and others will be asked for too few. SIRs are expected to manage their own workload by declining to commit to too many reviews. The mechanism for requesting reviews should, if possible, display each SIR's current review backlog. 3.3. Form of a Review Each review must start with a summary statement chosen from or adapted from the following list: * This draft is ready for publication as a [type] RFC, where [type] is Informational, Experimental, etc. (In some cases, the review might recommend publication as a different [type] than requested by the author.) * This draft is on the right track but has open issues, described in the review. * This draft has serious issues, described in the review, and needs to be rethought. * This draft has very fundamental issues, described in the review, and further work is not recommended. * Unfortunately, I don't have the expertise to review this draft. The length of a review will vary greatly according to circumstances, and it is acceptable for purely editorial comments to be sent privately. All substantive comments must be included in the public review. Wherever possible, they should be written as suggestions for improvement rather than as simple criticism. Explicit references to prior work and prior IETF discussion should be given. SIRs should review for all kinds of problems, from basic architectural or security issues, Internet-wide impact, technical nits, problems of form and format (such as IANA Considerations or incorrect references), and editorial issues. As a draft progresses from its initial, "-00" version towards one that is ready for submission, successive SIR reviews should progress from the general architectural level to the editorial level. The intention is that before a draft is submitted by a WG to the IESG, or by an individual to the RFC Editor, it has already benefited from a level of review equivalent to that traditionally applied by the IESG. In fact, SIRs should apply generally agreed IETF criteria, such as [RFC1958] The Architectural Principles of the Internet [RFC3426] General Architectural and Policy Considerations [RFC3439] Some Internet Architectural Guidelines and Philosophy [NITS] The "ID Nits" document maintained by the IESG [RFC2223] Instructions to RFC Authors [BCP26] Guidelines for Writing an IANA Considerations Section in RFCs [SECCONS] Guidelines for Writing RFC Text on Security Considerations as well as any other applicable architectural or procedural documents. It is important that reviews give precise references to such criteria when relevant. Q: Should we have a complete set of guidelines for reviewers? A: Yes. Please start writing :-) Q: Should a SIR engage actively in improving the document? A: It's not forbidden, but then s/he probably morphs into a contributor and a new SIR will be needed. 3.4. Iterative Carding The carding of textiles is an iterative process, and so is the carding of documents by SIRs. It is not expected that every version of an Internet Draft should be submitted for SIR review. However, it is advisable to request reviews at the very beginning (to check for fundamental issues), as major technical issues are resolved, and again just before the document is submitted for IESG approval (or to the RFC Editor). Thus three SIR review cycles per document may be considered the minimum. The document author should ensure that the SIRs reviewing a document understand what stage in its life cycle it has reached, so that they can review it appropriately. Both Working Groups and individual submitters should realise that carding should start early (to detect and hopefully fix fundamental problems) and be repeated as often as needed (to avoid submitting inadequate documents to the IESG). By these means, it should be possible to avoid most cases where a document spends a long time in IESG review or, worse, is fundamentally unacceptable to the IESG. The feedback from carding is intended to be helpful and beneficial to authors. Authors are encouraged to stay with the same SIRs if possible. SIRs are encouraged to stay with a given document as it evolves and to mentor its author(s), especially if the latter are new to the IETF process. Generally, SIRs are encouraged to participate in the education of less experienced IETF participants. 4. SECURITY CONSIDERATIONS This document does not directly impact the operational security of the Internet. 5. ACKNOWLEDGEMENTS Valuable comments and ideas have come from many sources, especially an earlier draft by Ted Hardie and many members of the IETF 'problem' working group. Specific thanks go to Mark Allman, Senthilkumar Ayyasamy, Aaron Falk, Mike Heard, John Loughney, Jonne Soininen. Spencer Dawkins gets extra credit for having thoroughly carded the first draft of this document and is hereby appointed as the honorary First Sir. 6. INFORMATIVE REFERENCES [PROBLEM] IETF Problem Statement, E. Davies (ed.), draft-ietf-problem-issue-statement- 01.txt, work in progress. [SECCONS] Guidelines for Writing RFC Text on Security Considerations, E.Rescorla, B.Korver, work in progress. [TECHNO] Intellectual Property Rights in IETF Technology, S.Bradner, work in progress. [BCP26] Guidelines for Writing an IANA Considerations Section in RFCs, T. Narten, H. Alvestrand, RFC 2434, 1998. [RFC1958] Architectural Principles of the Internet, IAB, RFC 1958, 1996. [RFC2223] Instructions to RFC Authors. J. Postel, J. Reynolds, RFC 2223, 1997. [RFC3426] General Architectural and Policy Considerations, IAB, RFC 3426, 2002 [RFC3439] Some Internet Architectural Guidelines and Philosophy, R.Bush, D.Meyer, RFC 3439, 2002 [NITS] AD Review of I-Ds, IESG, <http://www.ietf.org/ID-nits.html> 7. AUTHORS' ADDRESSES Brian E. Carpenter IBM Zurich Research Laboratory Saeumerstrasse 4 8803 Rueschlikon Switzerland Email: brian@hursley.ibm.com Dave Crocker Brandenburg InternetWorking 675 Spruce Drive Sunnyvale, CA 94086 USA Tel: +1.408.246.8253 dcrocker@brandenburg.com 8. INTELLECTUAL PROPERTY PLACEHOLDER for full IETF IPR Statement if needed. 9. FULL COPYRIGHT STATEMENT Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.