ICAR                                                          D. Partain
Internet-Draft                                                  Ericsson
Expires: November 30, 2004                                     June 2004


         An Experiment in Early Cross-Area Review for the IETF
             draft-ietf-icar-experiment-early-review-00.txt

Status of this Memo

   By submitting this Internet-Draft, I certify that any applicable
   patent or other IPR claims of which I am aware have been disclosed,
   and any of which I become aware will be disclosed, in accordance with
   RFC 3668.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at http://
   www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on November 30, 2004.

Copyright Notice

   Copyright (C) The Internet Society (2004). All Rights Reserved.

Abstract

   This memo captures current working group rough consensus on
   procedures for improved cross-area early review in the IETF.  It is a
   product of the ICAR working group.

   This memo describes an experiment to improve early cross-area review
   in the IETF.  Early cross-area review aims to improve quality of IETF
   work by identifying problems early in document development. Improved
   quality may, in turn, mean that documents can be processed faster in
   the IESG.





Partain                Expires November 30, 2004                [Page 1]


Internet-Draft              ICAR Experiment                    June 2004


Table of Contents

   1.  Introduction to ICAR . . . . . . . . . . . . . . . . . . . . .  3
   2.  Overview of this memo  . . . . . . . . . . . . . . . . . . . .  3
   3.  Prerequisites to the ICAR early review process . . . . . . . .  4
   4.  ICAR review pool . . . . . . . . . . . . . . . . . . . . . . .  4
     4.1   Makeup of the review pool  . . . . . . . . . . . . . . . .  4
       4.1.1   Seeding the initial reviewer pool  . . . . . . . . . .  5
       4.1.2   Objective criteria for entering the review pool  . . .  5
       4.1.3   Subjective criteria for entering the review pool . . .  5
   5.  Requesting a review  . . . . . . . . . . . . . . . . . . . . .  5
     5.1   Who requests a review  . . . . . . . . . . . . . . . . . .  5
     5.2   Available information about reviewers  . . . . . . . . . .  6
     5.3   How the review is requested  . . . . . . . . . . . . . . .  6
     5.4   Review announcements . . . . . . . . . . . . . . . . . . .  7
   6.  Results of a review  . . . . . . . . . . . . . . . . . . . . .  7
     6.1   Openly published and archived  . . . . . . . . . . . . . .  7
     6.2   Review is not binding on the WG  . . . . . . . . . . . . .  7
     6.3   WG's response to the review  . . . . . . . . . . . . . . .  7
   7.  An initial experiment  . . . . . . . . . . . . . . . . . . . .  8
   8.  Open issues  . . . . . . . . . . . . . . . . . . . . . . . . .  8
   9.  Security Considerations  . . . . . . . . . . . . . . . . . . .  9
       Author's Address . . . . . . . . . . . . . . . . . . . . . . .  9
       Intellectual Property and Copyright Statements . . . . . . . . 10



























Partain                Expires November 30, 2004                [Page 2]


Internet-Draft              ICAR Experiment                    June 2004


1.  Introduction to ICAR

   Improved cross-area review (ICAR) aims to improve the overall quality
   of the work produced by the IETF's working groups and to increase the
   speed of that work's progression through the standardization process.

   This memo is an attempt to capture in a coherent form the rough
   consensus from the working group as well as the state of the on-going
   debate.

   Cross-area review is not a new idea in the IETF. Most particularly,
   the IESG has carried out this kind of review when documents have been
   put on their table.  The IETF as a whole could function as a large
   review body and exactly this kind of review is theoretically
   solicited during a document's IETF Last Call.

   However, current review practices appear to have significant
   problems.  Little or no feedback is the norm during an IETF Last
   Call.  Waiting until the IESG review is deemed to be too late to
   correct significant problems with a document.

   ICAR proposes mechanisms by which early cross-area review is
   solicited, the review is carried out within an IETF-wide pool of
   reviewers, and the review itself as well as the working group's
   response to the review are publicly documented.

   The goal of the ICAR early review process is to ensure that reviews
   actually happen and the results are adequately considered by the
   working group.

2.  Overview of this memo

   This document begins by outlining the necessary prerequisites for a
   successful ICAR early review process in Section 3.

   Thereafter, Section 4 is a discussion of the makeup of the ICAR
   review pool, how it is constituted, how reviewers are added, and how
   reviewers are removed from the pool.

   Section 5 then discusses who requests a review and how they do so.

   Section 6 discusses how the results are handled and how the working
   group requesting the review is expected to respond to the review.

   In order to prove the value of ICAR reviews, the ICAR working group
   aims to start a small-scale experiment, which is outlined in Section
   7.




Partain                Expires November 30, 2004                [Page 3]


Internet-Draft              ICAR Experiment                    June 2004


   Finally, Section 8 points out known open issues (to the editor...)
   that have yet to be resolved.

3.  Prerequisites to the ICAR early review process

   In order for the ICAR early review process to be effective, the
   following need to be in place.

   1.  An open mailing list will be created on which requests for ICAR
       reviews will be posted. This will not be the mechanism by which
       the reviews are requested but is instead the way that interested
       parties and members of the IETF community who are not official
       reviewers can find out what reviews are needed.  Quite apart from
       the ICAR process, any member of the IETF community may, of
       course, review a WG document at any time s/he wishes and provide
       that input to the WG.
   2.  A pool of volunteers needs to be available for reviews.  This is
       discussed in Section 4.
   3.  The processes for handling review requests and results need to be
       ironed out (the goal of this memo).
   4.  There need to be tools available for managing the review flow.
       This includes information about reviewers (rough availability,
       review history, areas of expertise, etc.).  The current tools at
       publication time are available at http://www.machshav.com/~icar/
       reviews/ but are rough and are expected to evolve with the
       experiment.
   5.  The open issues listed in Section 8 should be resolved.

4.  ICAR review pool

   A pool of volunteers who can review documents is needed.     The WG
   considered the alternatives of one pool covering all IETF work and
   multiple pools, each covering an area of the IETF.  The WG consensus
   is to have an IETF-wide review pool.

4.1  Makeup of the review pool

   The initial pool of reviewers when starting the ICAR early review
   process will be constructed as detailed in the following two
   subsections.

   The consensus of the ICAR working group is that there needs to be
   some form of admission control to ensure that those in the review
   pool have sufficient technical expertise to provide high-quality
   reviews as well as an adequate understanding of the way that the IETF
   works. The criteria listed below are tools to ensure that both of
   these are true.




Partain                Expires November 30, 2004                [Page 4]


Internet-Draft              ICAR Experiment                    June 2004


4.1.1  Seeding the initial reviewer pool

   In order to seed the review pool, ADs will suggest 5-6 people from
   their area whom the ADs will "sponsor" and who will commit to
   engaging in this activity.  This would form the minimum pool that is
   needed to get ICAR reviews underway.

   In addition, there will be an open call for reviewers to join the
   pool.  Those who volunteer will need to meet one of the objective or
   subjective criteria listed in the following subsections.

4.1.2  Objective criteria for entering the review pool

   The following criteria will be used as one way to determine
   eligibility for the review pool (and is based on the SIRS experiment
   - editor: add reference).  The criteria listed in the bullets below
   will apply both when the initial review pool is constructed and on a
   continuing basis for those who wish to be added later.

   o  people who have authored 2 IETF RFCs approved for publication by
      the IESG, which includes non-WG documents submitted directly to
      the IESG, but not documents submitted directly to the RFC editor
   o  current/former IAB members
   o  former IESG members
   o  former WG chairs
   o  current WG chairs whose WG has produced 2 RFCs
   o  MIB doctors
   o  directorate members

4.1.3  Subjective criteria for entering the review pool

   There is some push within the ICAR working group to include a more
   subjective method of getting into the review pool.  However, there is
   not yet consensus as to how this would happen in practice. This
   section is simply a placeholder for the practice that the WG agrees
   to.

5.  Requesting a review

5.1  Who requests a review

   In general, it is up to the working group in question to request an
   ICAR review.  However, an AD may also request a review.

   ICAR reviews for working group documents are requested by a WG chair
   / secretary.  If a document falls into two or more working group's
   charters, the working group chairs are expected to coordinate their
   calls for an ICAR review.



Partain                Expires November 30, 2004                [Page 5]


Internet-Draft              ICAR Experiment                    June 2004


   A working group may request a review for any document it deems
   relevant to its work.  This may be official working group drafts, but
   it may also include documents named as personal drafts but which are
   functioning as working group drafts or are under consideration to be
   WG items.

   An AD may request an ICAR review, which would typically be when a
   document does not clearly fall into a specific WG's domain.

   Those requesting a review are expected to request reviews from people
   with different areas of expertise.

5.2  Available information about reviewers

   The following information is available about reviewers in the pool:

   o  Contact information
   o  Area(s) of expertise
   o  Some approximation of availability
   o  Archive of previous reviews
   o  Total number of pending requests to reviewer

5.3  How the review is requested

   It is up to the WG chair(s) / secretary to choose the reviewers who
   review the document. In order to do this, reviewers in the pool must
   be clearly labeled with their area(s) of expertise they feel
   qualified to provide .

   The ICAR WG does not believe that a broadcast request for reviews
   works.  As such, when the requester decides to request an ICAR
   review, s/he will look through the list of reviewers and the
   information about them (see Section 5.2) and determine which
   reviewers are available and have the appropriate technical expertise.

   It is then the responsibility of the requester to make contact with
   these individuals to request a review.  If the reviewer cannot do the
   review, the requester will have to find an alternative reviewer.

   When a requester and a reviewer agree on doing a review then the
   "ICAR infrastructure" should be notified (see Section 3 on tools for
   the ICAR experiment) so that information about the review and the
   review itself can be updated.

   Editor:  Some in the WG have suggested that a management function is
   needed to shepherd the early review process.  There is not yet any
   consensus in that regard.




Partain                Expires November 30, 2004                [Page 6]


Internet-Draft              ICAR Experiment                    June 2004


5.4  Review announcements

   In parallel to the closed ICAR reviewer pool, requests for reviews
   will be sent to an announce list, allowing the community to track
   which documents are actively being reviewed and encouraging people
   outside the ICAR reviewer pool to submit their own reviews of those
   documents.

   In addition, the web infrastructure set up to aid the ICAR early
   review process will include information about which documents are
   currently under review.

6.  Results of a review

6.1  Openly published and archived

   All ICAR reviews should be openly published (on the WG list at a
   minimum).

   All ICAR reviews should be archived.

   Reviews by individuals in response to ICAR review request
   announcements are not archived in the ICAR tools infrastructure.
   They are assumed to be archived in the normal WG mailing list
   archives.

6.2  Review is not binding on the WG

   The ICAR reviewers do not have any kind of veto power over the
   working groups that request reviews.  Reviewers can in no way force
   working groups to follow their advice.  That is, the reviews are not
   binding on WGs.

6.3  WG's response to the review

   Although reviews are not binding, WGs are nonetheless obligated to
   consider and discuss each suggestion from the review.  Not only must
   it be discussed, but there must also be adequate discussion
   documented to demonstrate why a working group chose not to implement
   a particular change requested by the reviewer.

   Like the review, the WG's response to the review should be archived.

   This is important, because the reasons for a WG's decision to
   disagree with a recommendation from a reviewer can be important input
   to the IESG when the document is submitted to the IESG for
   consideration.




Partain                Expires November 30, 2004                [Page 7]


Internet-Draft              ICAR Experiment                    June 2004


   At least during the initial ICAR experiment, it would be useful for
   the WG to note explicitly in documents that they have undergone an
   ICAR review. For example, this could be noted in the "Status of this
   Memo" section or in the abstract.

   The ICAR process has no way of influencing how non-ICAR reviews are
   handled by WGs and cannot force WGs to respond in a similar fashion
   to these reviews.  However, the IETF spirit is that all comments
   should be addressed in an appropriate manner.  Documentation of the
   WG's response to such reviews is of course desirable.

7.  An initial experiment

   The ICAR WG intends to recommend that an "experiment" be carried out
   with a group of willing participating reviewers and working groups.
   This section outlines the steps in carrying out that initial
   experiment.

   o  Ask the ADs to choose one or two WGs per area whose chairs are
      amenable to using an ICAR review pool and to helping the ICAR
      working group gauge how the experiment is working.
   o  Have the WGs in the WG pool solicit reviews on their current
      documents, any new documents, etc. Of course, there will be a
      mechanism by which we ensure that the initial burst will not be
      overwhelming.  Editor:  two per week max?
   o  As we gain experience we can try to add more groups.
   o  WGs from outside the initial WG pool may request reviews if they
      are so inclined.  However, these WGs will not be expected to make
      such requests, whereas the WGs participating in the experiment
      will be expected to use the ICAR review-pool.

8.  Open issues

   Editor:  This list is composed of issues that seem to still be open
   in the working group.  This list needs to be confirmed within the
   working group, and, if necessary, addressed in this document.

   1.  What guidelines should we set, if any, for the form of the
       review?  Can we say that reviewers should provide something like
       a numbered list of concise comments and not a rambling stream?
       This facilitates solid responses from the WG and nicely separates
       issues that will inevitably be dealt with differently.
   2.  Do we need addition information available about potential
       reviewers as discussed in Section 5.2?  For example, do we want
       history of number of requests to this reviewer?
   3.  What are the subjective criteria to be used for admittance to the
       pool (to be put into Section 4.1.3)?




Partain                Expires November 30, 2004                [Page 8]


Internet-Draft              ICAR Experiment                    June 2004


   4.  How are people removed from the pool or is such a process needed
       at all?
   5.  How does one measure the success/failure of ICAR experiments?
   6.  What level of involvement in the ICAR early review process can we
       expect / hope for from the IESG?  In what way should the
       reviewers, prospective reviewers, and working groups interact
       with the IESG?
   7.  Do we want to stipulate the form in which a review and the WG's
       response to it are documented?

9.  Security Considerations

   None.


Author's Address

   David Partain
   Ericsson AB
   P.O. Box 1248
   SE-581 12 Linkoping,
   Sweden

   EMail: david.partain@ericsson.com



























Partain                Expires November 30, 2004                [Page 9]


Internet-Draft              ICAR Experiment                    June 2004


Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights. Information
   on the IETF's procedures with respect to rights in IETF Documents can
   be found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard. Please address the information to the IETF at
   ietf-ipr@ietf.org.


Disclaimer of Validity

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.


Copyright Statement

   Copyright (C) The Internet Society (2004). This document is subject
   to the rights, licenses and restrictions contained in BCP 78, and
   except as set forth therein, the authors retain all their rights.


Acknowledgment

   Funding for the RFC Editor function is currently provided by the
   Internet Society.




Partain                Expires November 30, 2004               [Page 10]