Network Working Group S. Krishnan
Internet-Draft Ericsson
Intended status: Informational P. Eronen
Expires: November 26, 2008 Nokia
E. Gray
Ericsson
May 25, 2008
Guidelines to authors and reviewers regarding the IETF review process
draft-krishnan-review-process-00
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on November 26, 2008.
Abstract
This document describes the IETF review process and provides
guidelines to draft authors and reviewers on how to effectively
participate in it.
Krishnan, et al. Expires November 26, 2008 [Page 1]
Internet-Draft IETF Review Guidelines May 2008
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Sources of reviews . . . . . . . . . . . . . . . . . . . . . . 3
3. Guidelines for authors . . . . . . . . . . . . . . . . . . . . 3
3.1. Who responds to reviews . . . . . . . . . . . . . . . . . 3
3.2. Before reading the review . . . . . . . . . . . . . . . . 4
3.3. Responding to reviews . . . . . . . . . . . . . . . . . . 4
3.3.1. Timeframe for response . . . . . . . . . . . . . . . . 4
3.3.2. Contents of a response . . . . . . . . . . . . . . . . 5
3.4. Keeping track of the issues . . . . . . . . . . . . . . . 6
3.5. New version of the draft . . . . . . . . . . . . . . . . . 6
3.6. Acknowledging the reviewer . . . . . . . . . . . . . . . . 7
4. Guidelines for reviewers . . . . . . . . . . . . . . . . . . . 7
4.1. Level of review . . . . . . . . . . . . . . . . . . . . . 7
4.2. Prioritization of the issues . . . . . . . . . . . . . . . 7
4.3. Reviewing changes . . . . . . . . . . . . . . . . . . . . 8
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
7. Security Considerations . . . . . . . . . . . . . . . . . . . 8
8. Normative References . . . . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 9
Intellectual Property and Copyright Statements . . . . . . . . . . 10
Krishnan, et al. Expires November 26, 2008 [Page 2]
Internet-Draft IETF Review Guidelines May 2008
1. Introduction
An Internet Draft's life cycle begins when the author(s) submit the
document as an individual submission, and usually ends when the draft
is published as an RFC, or is abandoned and eventually expires.
During its lifetime, a draft is likely to receive review comments
from a number people.
In our experience, inefficient handling of review comments is one big
source of delay and frustration in the IETF standards process. The
goal of this document is to help document authors in dealing with
reviews in effective manner.
2. Sources of reviews
An internet draft gets reviewed at different stages of its lifecycle
by different sets of people. When the draft is in the initial stages
it gets reviewed mostly by the members of the working group that the
draft targets. But as the draft progresses, it gets reviewed by a
much more varied set of people with differing fields of expertise.
These reviews are performed with a very different point of view than
the working group reviews and are likely to bring out cross-area
issues. The different sources from which the author can receive
reviews are
o Working Group participants:
o Other IETF participants -- especially during IETF last call
o Review from the area specific directorates. Some directorates try
to review all documents -- these include Gen-ART (General Area
Review Team) assisting the General Area Director, and SecDir
(Security Directorate) assisting the Security Area Directors.
Other directorates are more specialized: for example, MIB Doctors
review only MIB documents.
o Reviews from IESG members: after IETF last call,
o Reviews from the IAB
3. Guidelines for authors
3.1. Who responds to reviews
There are several persons who can respond to a received review. If
the document is an individual document, it is customary for the
author to respond to the review. If the document has several
authors, they can co-ordinate among themselves when they respond.
Otherwise the reviewer may receive conflicting responses from
different authors. If the document is a working group document,
Krishnan, et al. Expires November 26, 2008 [Page 3]
Internet-Draft IETF Review Guidelines May 2008
ideally the working group chairs or the document shepherd should
coordinate things, since the author and reviewer cannot unilaterally
decide to make substantial changes.
3.2. Before reading the review
Receiving a long review, or a critical review for your favorite
document can be disheartening. Before reading the review -- or at
the very least, before starting to reply -- it's good to remind
yourself about a crucial difference between Internet Draft reviews,
and say, reviews about books or movies.
A good movie review attempts to give a balanced view: if the reviewer
liked the movie, the review will say good things about it. However,
a typical review of an Internet Draft will comment only the parts
that the reviewer did not like and wants to see changed.
The reviewer may well think that your work is excellent quality, and
a valuable contribution to Internet engineering -- but it is not
common to say this in a review. Thus, when you start reading a
review, imagine that the review started with a positive comment --
after all, the reviewer considered this important enough to review
it!
Hint for reviewers: it helps if you start your review with a positive
overall opinion. Although it may not seem so, there's a big
difference between saying "The following things have to be fixed:"
and "This is a well-written and valuable document; however, I'd like
to propose some minor improvements for the following things:"
3.3. Responding to reviews
Reviewers spend a non-trivial amount of time in performing a review,
and reviews also play an important part in how IETF determines
consensus. Thus, the single most important rule for handling reviews
is the following: every review merits a response.
Even if the review said "everything looks ok", saying "thank you for
reviewing this" is part of common courtesy. If the reviewer raised
issues, a response is deserved, irrespective whether the authors
agree with the reviewer or not.
3.3.1. Timeframe for response
The authors are expected to respond to the reviews within a
reasonable amount of time. While there is no hard and fast rules
concerning this period, two or three business days might be a
reasonable amount of time to take for a response. It is possible
Krishnan, et al. Expires November 26, 2008 [Page 4]
Internet-Draft IETF Review Guidelines May 2008
that the authors may be unavailable to respond to a review within
this timeline since they may be sick/on vacation/busy with dayjob
etc. In this case, the document shepherd (if any) can respond to the
review and follow up with the authors at a later time.
3.3.2. Contents of a response
3.3.2.1. The initial response
The initial response that the author sends might be as simple as a
single mail that acknowledges the receipt of the review and the
intent of the author to look into the contents of the review. It
would be nice for the author to come up with a timeline on when he/
she would be able to respond to the comments/issues contained in the
review.
3.3.2.2. The substantial response
In the substantial response the author tries to address the concerns
raised in the review. There are several ways to do this.
o Accept the concern as valid and propose a solution
o Accept the concern as valid and propose a timeline for a solution
o Accept the concern as valid and solicit text from the reviewer for
a solution
o Ask for more clarifying information to understand the concern
o Dismiss the concern as invalid (out of scope, already covered in
the document, misunderstanding etc.)
o Any combination of the above
The substantial response needs to be sent at least to the reviewer,
the relevant working groups and the mailing list of the reviewing
body (if any). It can also be sent to the other recipients of the
original review.
3.3.2.3. Common issues with the substantial response
This section tries to document some common mistakes that authors make
while responding to a review.
o If the reviewer has not understood some part of the draft, a very
common response is to explain the topic in email. However, often
that explanation should really be in the draft itself, not just in
emails to the authors, so that future readers will also understand
the text.
o Soliciting text from the reviewer should be used sparingly. It
can be helpful, if the authors, even after asking for more
information, do not understand the reviewer's concern. However,
Krishnan, et al. Expires November 26, 2008 [Page 5]
Internet-Draft IETF Review Guidelines May 2008
if the author understands the concern, it is primarily his/her job
to propose text.
o Simply accepting the concern as valid is a response that is often
seen, but is really good only when the solution is totally obvious
(fixing a typo, etc.). For complex topics, saying "I'll change
this in the next draft version" is not very good, since it makes
life difficult for the reviewer to verify the fix.
3.4. Keeping track of the issues
When the author makes an attempt in good faith to resolve all the
issues raised by the reviewer, it is possible that some of the issues
are left unresolved in the revised version of the draft. In order to
prevent this from happening, it is useful to keep a systematic record
of the issues and the associated resolutions. One common way of
doing this is buy using an issue tracker. The IETF tools team
provides an issue tracker on request to any of the working groups and
the chairs can add authorized users for the tracker. When the issues
are raised by the reviewer, the author can open the issue on the
issue tracker and close it when it is resolved. The author can also
add the suggested resolution text into the tracker. This way both
the author and the reviewer can keep tabs on the change process
without missing out any issues.
For a document in Working Group Last Call or IETF wide Last Call it
is considered good practice for the working group chair or the
document shepherd to post a summary of all the comments received
during the last call, and current proposal for handling them. Even
if the working group chair or shepherd does not do this, it's highly
recommended that the author does this.This summary could include
links to mailing list archives, or if an issue tracker is used, issue
numbers or links to "tickets".
3.5. New version of the draft
One the author feels that all the open issues have been addressed she
can submit a new revision of the draft that incorporates the changes.
It is considered common courtesy to inform the reviewers that a new
version of the draft is available. It is desirable for the author to
summarize the list of changes in the new version of the draft either
in the mail informing the reviewers or in a "Changes since version X"
section of the draft. This makes it easier for the reviewers to go
through just the changes instead of rereading the complete draft.
Alternately or additionally, the rfcdiff tool can be used to identify
the changes made in the latest version of the draft and the output of
this tool can be provided to the reviewer.
After the author and reviewer agree that the issues have been fixed,
Krishnan, et al. Expires November 26, 2008 [Page 6]
Internet-Draft IETF Review Guidelines May 2008
for working group documents, substantial issues need to be confirmed
on the mailing list. Once the document has entered IESG processing,
new versions should not be posted unless the document shepherd or AD
explicitly says so.
3.6. Acknowledging the reviewer
If the reviewer has contributed substantially to the improvement of
the draft it is considered polite to add the reviewer's name to the
Acknowledgements section of the draft, if there is one. There is no
hard and fast rule about when to acknowledge a reviewer and hence the
author should use her own judgement to decide whether or not to do
this.
4. Guidelines for reviewers
4.1. Level of review
The level of review performed on the draft varies greatly based on
the source of the review and on the stage of the document lifecycle.
In the earlier stages of the lifecycle, the document is usually
reviewed by the members of the relevant working group(s) only. Let's
call these reviews "early reviews". At this point the comments on
the draft are deeply technical and mainly remove obvious errors from
the draft. The reviewer and the author are usually from similar
backgrounds and are able to understand the underlying technologies
and jargon in a consistent fashion. The reviews that occur after the
draft has been progressed from the working group have a very
different perspective on the contents of the draft. They explore
very different aspects of the draft such as security aspects,
operational aspects, readability etc. that are not usually on the top
of the list of priorities of the authors and the early reviewers.
They also explore how the mechanisms proposed in the document fit
into the Internet architecture as a whole and what detrimental
effects they will have, if any. Let's call these "late reviews".
The person performing the review should consider this when the review
is being performed and concentrate her efforts on the right parts of
the document.
4.2. Prioritization of the issues
A review results in a list of issues or requested clarifications.
The reviewer can choose to list them in the order they appear in the
document. Alternately, and preferably, the reviewer can use another
classification scheme where the issues are grouped together by their
severity. This makes it easier for the eventual recipients of the
review to attach the appropriate importance for resolving these
Krishnan, et al. Expires November 26, 2008 [Page 7]
Internet-Draft IETF Review Guidelines May 2008
issues. If the review is sent on behalf of a reviewing body this
also helps the beneficiary of the review in taking a position on the
document. e.g. The reviewer can classify the issues into the
following categories
o Major
o Minor
o Editorial
o Nits
4.3. Reviewing changes
Once the authors submit a new revision of the draft, the reviewer can
look over the new revision to see if the agreed changes have been
made to the draft. If the reviewer finds out that some changes have
not been made, he/she can alert the authors of this fact. There are
tools available that can make the reviewer's life easier in this
regard. The rfcdiff tool can be used to identify the changes made in
the latest version of the draft. The reviewer can just look over
these changes instead of rereading the entire draft. The reviewers
can also keep track of issues using the issue tracker used by the
authors (if one was used). If the reviewer is satisfied with the
changes, he/she can send out a mail to the original recipients of the
review mentioning this. If not, a new generation of the review cycle
is started and the steps described earlier are redone.
5. Acknowledgements
The authors would like to thank Bernard Aboba and Thomas Narten for
their contributions to this document.
6. IANA Considerations
This document does not require any action from the IANA.
7. Security Considerations
This document does not create any new security issues.
8. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
Krishnan, et al. Expires November 26, 2008 [Page 8]
Internet-Draft IETF Review Guidelines May 2008
Authors' Addresses
Suresh Krishnan
Ericsson
8400 Decarie Blvd.
Town of Mount Royal, QC
Canada
Phone: +1 514 345 7900 x42871
Email: suresh.krishnan@ericsson.com
Pasi Eronen
Nokia Research Center
P.O. Box 407
FI-00045 Nokia Group
Finland
Email: pasi.eronen@nokia.com
Eric Gray
Ericsson
900 Chelmsford Street
Lowell, MA
USA
Email: eric.gray@ericsson.com
Krishnan, et al. Expires November 26, 2008 [Page 9]
Internet-Draft IETF Review Guidelines May 2008
Full Copyright Statement
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Krishnan, et al. Expires November 26, 2008 [Page 10]