Network Working Group S. Krishnan
Internet-Draft Ericsson
Intended status: Informational P. Eronen
Expires: January 14, 2009 Nokia
E. Gray
Ericsson
July 13, 2008
Guidelines to authors and reviewers regarding the IETF review process
draft-krishnan-review-process-01
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on January 14, 2009.
Abstract
This document describes the authors' experiences with the IETF review
process and provides guidelines to draft authors and reviewers on how
to effectively participate in it. This document does not define the
IETF review process itself.
Krishnan, et al. Expires January 14, 2009 [Page 1]
Internet-Draft IETF Review Guidelines July 2008
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Sources of reviews . . . . . . . . . . . . . . . . . . . . . . 3
3. Guidelines for authors . . . . . . . . . . . . . . . . . . . . 4
3.1. Who responds to reviews . . . . . . . . . . . . . . . . . 4
3.2. Before reading the review . . . . . . . . . . . . . . . . 4
3.3. Responding to reviews . . . . . . . . . . . . . . . . . . 4
3.3.1. Timeframe for response . . . . . . . . . . . . . . . . 5
3.3.2. Contents of a response . . . . . . . . . . . . . . . . 5
3.4. Keeping track of the issues . . . . . . . . . . . . . . . 7
3.5. New version of the draft . . . . . . . . . . . . . . . . . 7
4. Guidelines for reviewers . . . . . . . . . . . . . . . . . . . 8
4.1. Level of review . . . . . . . . . . . . . . . . . . . . . 8
4.2. Recipients of the review . . . . . . . . . . . . . . . . . 9
4.3. Classification of the issues . . . . . . . . . . . . . . . 10
4.4. Prioritization of the issues . . . . . . . . . . . . . . . 10
4.5. Reviewing changes . . . . . . . . . . . . . . . . . . . . 11
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 12
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
7. Security Considerations . . . . . . . . . . . . . . . . . . . 12
8. Normative References . . . . . . . . . . . . . . . . . . . . . 12
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12
Intellectual Property and Copyright Statements . . . . . . . . . . 14
Krishnan, et al. Expires January 14, 2009 [Page 2]
Internet-Draft IETF Review Guidelines July 2008
1. Introduction
The Tao of the IETF [RFC4677] provides an informal overview of the
inner workings of the IETF. This document is intended to augment the
information available in [RFC4677] and is mainly concerned about
reviews.
An Internet Draft's life cycle begins when the author(s) submit the
document as a personal draft; it may become a Working Group draft,
and usually ends when the draft is published as an RFC, or is
abandoned and eventually expires. During its lifetime, a draft is
likely to receive review comments from a number people.
In our experience, inefficient handling of review comments is one big
source of delay and frustration in the IETF standards process. The
goal of this document is to help document authors in dealing with
reviews in an effective manner.
2. Sources of reviews
An internet draft gets reviewed at different stages of its lifecycle
by different sets of people. When the draft is in the initial stages
it gets reviewed mostly by the members of the working group that the
draft targets. But as the draft progresses, it gets reviewed by a
much more varied set of people with differing fields of expertise.
These reviews are performed with a very different point of view than
the wg reviews and are likely to bring out cross-area issues. The
different sources from which the author can receive reviews are
o Working Group participants: Reviews from working group
participants are not just reviews, but also an important method of
participating in the working group. This document deals primarily
with late reviews though some of the concepts may be applicable to
early reviews (such as reviews from WG participants) as well.
o Other IETF participants -- especially during IETF last call
o Review from the area specific review teams. Some review teams try
to review all documents -- these include Gen-ART (General Area
Review Team) assisting the General Area Director, and SecDir
(Security Directorate) assisting the Security Area Directors.
Other review teams are more specialized: for example, MIB Doctors
review only MIB documents.
o Reviews from IESG members: after IETF last call,
o Reviews from the IAB
Krishnan, et al. Expires January 14, 2009 [Page 3]
Internet-Draft IETF Review Guidelines July 2008
3. Guidelines for authors
3.1. Who responds to reviews
There are several persons who can respond to a received review. If
the document is an individual document, it is customary for the
author to respond to the review. If the document has several
authors, they should co-ordinate among themselves before responding.
Otherwise the reviewer may receive conflicting responses from
different authors. If the document is a working group document,
ideally the WG chairs or the document shepherd should coordinate
things, since the author and reviewer cannot unilaterally decide to
make substantial changes. The coordination effort (by WG chairs
and/or document shepard) may be more or less active, depending on the
working group.
3.2. Before reading the review
Receiving a long review, or a critical review for your favorite
document can be disheartening. Before reading the review -- or at
the very least, before starting to reply -- it's good to remind
yourself about a crucial difference between Internet Draft reviews,
and say, reviews about books or movies.
A good movie review attempts to give a balanced view: if the reviewer
liked the movie, the review will say good things about it. However,
a typical review of an Internet Draft will comment only the parts
that the reviewer did not like and wants to see changed.
The reviewer may well think that your work is excellent quality, and
a valuable contribution to Internet engineering -- but it is not
common to say this in a review. Thus, when you start reading a
review, imagine that the review started with a positive comment --
after all, the reviewer considered this important enough to review
it!
Hint for reviewers: it helps if you start your review with a positive
overall opinion. Although it may not seem so, there's a big
difference between saying "The following things have to be fixed:"
and "This is a well-written and valuable document; however, I'd like
to propose some minor improvements for the following things:"
3.3. Responding to reviews
Reviewers spend a non-trivial amount of time in performing a review,
and reviews also play an important part in how IETF determines
consensus. Thus, the single most important rule for handling reviews
is the following: every review merits a response.
Krishnan, et al. Expires January 14, 2009 [Page 4]
Internet-Draft IETF Review Guidelines July 2008
Even if the review said "everything looks ok", saying "thank you for
reviewing this" is part of common courtesy. If the reviewer raised
issues, a response is deserved, irrespective whether the authors
agree with the reviewer or not.
3.3.1. Timeframe for response
The authors are expected to respond to the reviews within a
reasonable amount of time. What constitutes a reasonable amount of
time is left to the judgement of the persons involved. e.g. If a
document is up for IESG review on a certain date, it may be useful to
respond before that date. It is possible that the authors may be
unavailable to respond to a review within a given timeline since they
may be sick/on vacation/busy with dayjob etc. In this case, the
document shepherd (if any) can respond to the review and follow up
with the authors at a later time. If the review is not responded to
in a timely fashion, then other folks are likely to assume that there
is a real problem that needs to be addressed, rather than finding out
that the reviewer simply misread the document (reviewers make
mistakes too).
3.3.2. Contents of a response
3.3.2.1. The initial response
The initial response that the author sends might be as simple as a
single mail that acknowledges the receipt of the review and the
intent of the author to look into the contents of the review. It is
often helpful if the responding author(s) can also provide some
indication of when a subsequent response will be provided to the
comments/issues contained in the review -- assuming that a response
cannot be provided at once. This sort of response can be helpful in
telling the difference between a review that has been forgotten, and
one that is being actively worked on. e.g. If the author is busy
with his/her day job or on vacation, the document shepherd can ask
some other working group member to evaluate the review.
3.3.2.2. The substantial response
In the substantial response the author tries to address the concerns
raised in the review. There are several ways to do this:
o Accept the concern as valid and propose a solution
o Accept the concern as valid and propose a timeline for a solution
o Accept the concern as valid and solicit text from the reviewer for
a solution.
Krishnan, et al. Expires January 14, 2009 [Page 5]
Internet-Draft IETF Review Guidelines July 2008
o Ask for clarifying information to better understand the concern
o Explain why the concern is invalid (out of scope, already covered
in the document, misunderstanding, etc.)
o Other responses that may apply in special cases
o Any combination of the above
The substantial response needs to be sent at least to the reviewer,
the relevant working groups and the mailing list of the reviewing
body(if any). It can also be sent to the other recipients of the
original review.
3.3.2.2.1. Common issues with the substantial response
This section tries to document some common mistakes that authors make
while responding to a review.
o If the reviewer has not understood some part of the draft, a very
common response is to explain the topic in email. However, often
that explanation should really be in the draft itself, not just in
emails to the authors, so that future readers will also understand
the text.
o Soliciting text from the reviewer should not be used as a default
response. It can be helpful, if the authors, even after asking
for more information, do not understand the reviewer's concern.
It can also be used as a technique for reducing the number of
iterations required to arrive at text that is acceptable to the
reviewer, or to help in producing review comments that are direct,
with a specific focus toward some resolution. Note that, if the
author understands the concern, it may be best if an author
proposes text -- since text proposed by an author may be less
likely to introduce stylistic discontinuities, or result in other
issues that an author (who is likely to be more familar with the
history of the draft) might avoid.
o The reviewer should not be pushy while trying to suggest new text.
Instead of writing "Here is the fix for my problem" that imposes
the reviewer's will on the author, he or she might write "Here is
my proposal for one way to fix this issue; there may of course be
other ways to do so. Please consult with the WG and decide on the
text to adopt".
o Simply accepting the concern as valid is a response that is often
seen, but is really good enough only when the solution is obvious
(fixing a typo, etc.). For complex comments, saying "I'll change
this in the next draft version" may not be enough, since it it may
make it difficult for the reviewer to verify the fix, or may
result in a fix that is not acceptable to the reviewer.
Krishnan, et al. Expires January 14, 2009 [Page 6]
Internet-Draft IETF Review Guidelines July 2008
3.4. Keeping track of the issues
When the author makes an attempt in good faith to resolve all the
issues raised by the reviewer, it is possible that some of the issues
are left unresolved in the revised version of the draft. In order to
prevent this from happening, it is useful to keep a systematic record
of the issues and the associated resolutions. One common way of
doing this is by using an issue tracker. The IETF tools team
provides an issue tracker on request to any of the working groups and
the chairs can add authorized users for the tracker. When the issues
are raised by the reviewer, the author can open the issue on the
issue tracker and close it when it is resolved. The author can also
add the suggested resolution text into the tracker. This way both
the author and the reviewer can keep tabs on the change process
without missing out any issues.
For a document in Working Group Last Call or IETF wide Last Call it
is considered good practice for the WG chair or the document shepherd
to ensure that someone posts a summary of all the comments received
during the last call, and current proposal for handling them. If the
WG chair or shepherd does not do this, the task of producing a
summary falls to the author, or authors. Note that - if no such
summary is produced - it will be very difficult for working group
members (or the IETF at large) to determine that the changes made
were in line with the changes discussed and agreed to. This summary
could include links to mailing list archives, or if an issue tracker
is used, issue numbers or links to "tickets".
3.5. New version of the draft
The details of dealing with response resolutions may vary slightly
from working group to working group, but -- generally -- the
following approaches are used at the indicated stages:
o Prior to acceptance as a WG draft, Author(s) are free to publish a
new version with or without including specific resolutions -
bearing in mind that acceptance by a WG is only likely if a) the
text is essentially headed in the right direction and b) the WG is
inclined to believe the current Author(s) are the right ones for
the job.
o Once accepted as a WG draft, and through completion of a final WG
Last Call, the Author(s) need confirmation (typically from
discussions on the working group mailing list, or at meetings and
subsequently confirmed on the mailing list) of all substantive
changes prior to publishing a modified version. Prior to last
call, the necessary confirmation may simply be a general direction
proposed and accepted at an IETF meeting and not refuted in
minutes published. Often prior to last call, the Author(s) may
Krishnan, et al. Expires January 14, 2009 [Page 7]
Internet-Draft IETF Review Guidelines July 2008
explicitly solicit help from WG members who've expressed concerns
about specific portions of draft in an effort to reach convergence
in a subsequent IETF WG meeting. Once Last Call has started, each
substantive change will typically require explicit confirmation on
the WG mailing list.
o Once the document has entered IESG processing (post WG Last Call),
new versions should not be posted unless the document shepherd or
(responsible) AD explicitly says so. Typically, this will occur
only after the responsible individual is certain that the Authors
have addressed all of the outstanding comments - both substantive
and editorial.
o At any point after WG Last Call, the IESG may decide that all
remaining comments may be addressed via a note to the RFC Editor.
Typically, this would occur if all remaining comments are
relatively minor, strictly editorial or of a type that would best
be dealt with by the RFC Editor in any case.
4. Guidelines for reviewers
4.1. Level of review
The level of review performed on the draft varies greatly based on
the source of the review and on the stage of the document lifecycle.
In the earlier stages of the lifecycle, the document is usually
reviewed by the members of the relevant working group(s) only,
although the working group chairs and (usually) ADs should take at
least a brief look at the draft. Let's call these reviews "early
reviews".
At this point the comments on the draft are deeply technical and
mainly remove obvious errors from the draft. It is also a good idea
for working group members, working group chairs and/or ADs, to review
early drafts for "meta issues" (such as Internet Architecture impact,
special security exposures, manageablity, scale, consistency with
chartered goals, etc.) that may be a problem later and result in
potentially going back to the drawing board. Reviewer(s) and
author(s) are usually from similar backgrounds and are able to
understand the underlying technologies and jargon in a consistent
fashion.
The reviews that occur after the draft has been progressed from the
working group have a very different focus with respect to the
contents of the draft. They explore broader aspects of the draft
(such as security, operations, readability, etc.) that are not
usually on the top of the list of priorities of the authors and the
early reviewers - at least during most reviews. They also explore
how the mechanisms proposed in the document fit into the Internet
Krishnan, et al. Expires January 14, 2009 [Page 8]
Internet-Draft IETF Review Guidelines July 2008
architecture as a whole and what detrimental effects they will have,
if any. Note that these reviews are where it will come out if the
very earliest reviews did not consider where the drafts were going
because it is at this point that a poorly directed draft is very
likely to stall. Let's call these "late reviews".
The person performing the review should consider this when the review
is being performed and concentrate their efforts on the right parts
of the document.
4.2. Recipients of the review
The list of recipients of the review is tricky to get right. The
main idea is to make sure all the relevant people receive the review.
The recipient list is determined mainly by the following factors.
o The timeframe of the review (early vs. late)
o The contents of the review (editorial vs. technical)
Early reviews are usually performed by active participants of a
working group. The preferred destination for these reviews is the
working group mailing list since it can be reasonably assumed that
the persons interested in the document are subscribed to the mailing
list. This applies for both technical and editorial issues.
Alternately editorial issues can be resolved using a private mail to
the author(s).
Late reviews are usually performed by persons who are not active
participants of the working group and who usually review the draft
from a different point of view than the working group. If the
contents of the review are mainly editorial in nature, the reviews
can be sent just to the authors, the working group chair(s), the
document shepherd(s). If the review is of a more technical nature it
is considered polite to include the working group mailing list and/or
the IETF discussion list. As it is not reasonable to assume that the
reviewer will subscribe to the working group mailing list just for
discussing this issue, the working group chair(s) need to make sure
that this review will get past any moderation controls imposed on
non-subscribers by the working group mailing list.
Krishnan, et al. Expires January 14, 2009 [Page 9]
Internet-Draft IETF Review Guidelines July 2008
4.3. Classification of the issues
While writing a review, the reviewer needs to distinguish between
different types of comments. On a really high level review comments
may be classified into two types
o Actionable comments: These comments contain statements that can be
either proved or disproved. e.g. "The protocol described in this
document will not work in networks with lossy link layers". This
creates the opportunity for the reviewer and working group to
discuss the issue in terms that can be tested. In order to ask
the working group to do so, a reviewer should be able to back his/
her statement with a reasonably clear, worked set of reasons for
the assertion. A working group can go about disproving the
statement, and if it fails to do so, it can either make the
necessary technical changes or write up an applicability statement
that documents the cases where the protocol works well enough to
be useful.
o Observations: These comments are not testable statements but
merely opinion/commentary of the reviewer. e.g. " I think this
protocol should run on top of TCP rather than on top of UDP
because it needs reliable transport". This kind of comments may
be very valuable as well, but they should not blindly overrule the
consensus of the working group. These should be treated as issues
that the wg and the author need to evaluate carefully.
Reviewers should be sensitive to the difference between their
personal opinions (and preferences) and issues that will affect the
correct operation or interoperation of the documents under review.
4.4. Prioritization of the issues
A review often results in a list of issues or requested
clarifications. The reviewer may choose to list them in the order
they appear in the document. For long drafts, where structure and
content maturity is fairly well established, this approach is easier
to follow. Alternatively (and possibly preferably, depending on the
maturity of the draft and other factors), the reviewer can use
another classification scheme where the issues are grouped together
by importance and/or potential impact on the draft. This makes it
easier for eventual recipients of the review to attach the
appropriate priority for resolving the issues. At early stages of
draft maturity, this approach can be critical, because resolving some
issues may impact on appropriate resoluton alternatives for other
issues. If a review is sent on behalf of a reviewing body (such as
might be the case with a review reported by a liaison statement),
this also helps the beneficiary of the review in taking a position on
the document. For example, A reviewer might classify issues into the
Krishnan, et al. Expires January 14, 2009 [Page 10]
Internet-Draft IETF Review Guidelines July 2008
following categories
o Major
o Minor
o Editorial
o Nits
The purpose of such a prioritization scheme might be to indicate the
level of discomfort the reviewing entity will feel with the results
of any resolution that does not address the concerns at each level of
priority.
It is almost always a good idea to separate at least the editorial
comments (and NITs) from those impacting on the substance of the
draft at any stage of draft maturity. A key reason for doing this is
that this allows most people who will read the review to concentrate
on the portions of the review that impact on the substance of the
document, primarily to ensure that they do not have fundamental
issues with the review comments or proposed resolutions. Note that
this is not a license to put forward omission of key words (such as
"not") as minor editorial comments (or NITs).
If comments are grouped in any way, however, the grouping may very
well be "of the essence" of the comments. In other words, if the
responder does not agree with the distinctions made by the reviewer,
they should be clear about this. This is, in part, to ensure that
something that a reviewer thought to be minor -- but which an author
felt to be fundamental -- receives the right amount of attention on
reading by others.
4.5. Reviewing changes
Once the authors submit a new revision of the draft, the reviewer can
look over the new revision to see if the agreed changes have been
made to the draft. If the reviewer finds out that some changes have
not been made, he/she can alert the authors of this fact. There are
tools available that can make the reviewer's life easier in this
regard. The rfcdiff tool can be used to identify the changes made in
the latest version of the draft. The reviewer can just look over
these changes instead of rereading the entire draft. The reviewers
can also keep track of issues using the issue tracker used by the
authors(if one was used). If the reviewer is satisfied with the
changes, he/she can send out a mail to the original recipients of the
review mentioning this. If not, a new generation of the review cycle
is started and the steps described earlier are redone.
Krishnan, et al. Expires January 14, 2009 [Page 11]
Internet-Draft IETF Review Guidelines July 2008
5. Acknowledgements
The authors would like to thank Bernard Aboba, Thomas Narten, Frank
Ellermann, Ted Hardie, Paul Hoffman, Thomas Goldbeck-Lowe, Scott
Brim, Joel Halpern, Brian Carpenter, Iain Calder, Dan Romascanu and
SM for their contributions to this document.
6. IANA Considerations
This document does not require any action from the IANA.
7. Security Considerations
This document does not create any new security issues.
8. Normative References
[RFC4677] Hoffman, P. and S. Harris, "The Tao of IETF - A Novice's
Guide to the Internet Engineering Task Force", RFC 4677,
September 2006.
Authors' Addresses
Suresh Krishnan
Ericsson
8400 Decarie Blvd.
Town of Mount Royal, QC
Canada
Phone: +1 514 345 7900 x42871
Email: suresh.krishnan@ericsson.com
Pasi Eronen
Nokia Research Center
P.O. Box 407
FI-00045 Nokia Group
Finland
Email: pasi.eronen@nokia.com
Krishnan, et al. Expires January 14, 2009 [Page 12]
Internet-Draft IETF Review Guidelines July 2008
Eric Gray
Ericsson
900 Chelmsford Street
Lowell, MA
USA
Email: Eric.Gray@Ericsson.com
Krishnan, et al. Expires January 14, 2009 [Page 13]
Internet-Draft IETF Review Guidelines July 2008
Full Copyright Statement
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Krishnan, et al. Expires January 14, 2009 [Page 14]