Network Working Group A. Doria (editor)
Internet-Draft ETRI
Expires: April 15, 2006 H. Alvestrand
Cisco Systems
B. Carpenter
IBM Zurich Research Laboratory
October 12, 2005
Experience with the General Area Review Team
draft-doria-genart-experience-01.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on April 15, 2006.
Copyright Notice
Copyright (C) The Internet Society (2005).
Abstract
The General Area Review team has been doing Late Reviews of Internet
Drafts since 2004. This draft discusses the experience and the
lessons learned in the first 18 months of this process.
Doria (editor), et al. Expires April 15, 2006 [Page 1]
Internet-Draft GenART October 2005
Table of Contents
1. Discussion Venue . . . . . . . . . . . . . . . . . . . . . . 3
2. What is GenART . . . . . . . . . . . . . . . . . . . . . . . 3
3. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4. GenART Reviews . . . . . . . . . . . . . . . . . . . . . . . 3
4.1 IESG Telechat Review Process . . . . . . . . . . . . . . . 3
4.2 IETF LC Assignments . . . . . . . . . . . . . . . . . . . 5
4.3 Form of review . . . . . . . . . . . . . . . . . . . . . . 6
4.4 Archiving of reviews . . . . . . . . . . . . . . . . . . . 7
5. Results . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6. Impressions . . . . . . . . . . . . . . . . . . . . . . . . 7
6.1 Reviewers' Impressions . . . . . . . . . . . . . . . . . . 8
6.2 One General Area Director's Impressions . . . . . . . . . 9
6.3 One GenART Secretary's Impressions . . . . . . . . . . . . 10
7. Needed Improvements . . . . . . . . . . . . . . . . . . . . 11
8. Applicability . . . . . . . . . . . . . . . . . . . . . . . 11
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 11
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
10.1 Informative References . . . . . . . . . . . . . . . . . 12
10.2 Normative References . . . . . . . . . . . . . . . . . . 12
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 12
A. IANA considerations . . . . . . . . . . . . . . . . . . . . 12
Intellectual Property and Copyright Statements . . . . . . . 14
Doria (editor), et al. Expires April 15, 2006 [Page 2]
Internet-Draft GenART October 2005
1. Discussion Venue
Discussion of this proposal is intended to take place on the IETF
mailing list <mailto: ietf@ietf.org> in the absence of a better home.
2. What is GenART
The General Area Review team was created personally by the former
General Area Director and retained by the current one. It has no
official role in the IETF standards process, except as a set of
individuals entitled, like everyone, to comment on Internet-Drafts.
Its Secretary, and the team of volunteer reviewers, serve at the
invitation of the General AD. The original secretary was Avri Doria,
recently replaced by Mary Barnes, who previously served as a
reviewer. The current General AD was also previously a reviewer.
3. Goals
The original and continuing goal of the GenART team was, and is, to
offload some of the burden from the General Area AD of IESG reviews.
The load for the bi-weekly IESG reviews is often quite large;
occasionally there are 20 drafts or more scheduled for discussion in
a single telechat.
GenART was based on a model that had proved productive in the Ops
Directorate: Quick review close to telechat time, to advise the AD on
issues that remain serious. By having a trusted group of reviewers
read and evaluate the drafts, the General Area AD would be able to
concentrate on those drafts where there was an concern expressed by
the reviewer.
4. GenART Reviews
4.1 IESG Telechat Review Process
The process for reviewing documents when they appear on the IESG
agenda:
o The nearly final IESG meeting agenda generally appears on Thursday
night, one week before the IESG telechat. The GenART secretary
uses this as the input for the assignment process.
o For documents reviewed at IETF Last Call, a new review is only
asked for if the document is revised. In this case the reviewer,
if possible the person who did the last call review, only needs to
check that any open issues were resolved. Often the draft will
not have changed between IETF LC and the IESG telechat review.
o The GenART secretary names a reviewer per document, more or less
randomly. This assignment process is currently very labor
intensive and is completely manual (although much of it could, in
Doria (editor), et al. Expires April 15, 2006 [Page 3]
Internet-Draft GenART October 2005
theory, be automated).
1. Remove everyone who has announced they are on vacation, or is
otherwise unavailable for reviewing, from the resource list.
2. All previously reviewed drafts are assigned to the reviewer
who reviewed them previously, assuming that reviewer is
available. Otherwise, these documents are assigned to a new
person in the process described below.
3. Make note of the groups of drafts that are being grouped as a
single ballot. If the load is not too great, an attempt is
made to assign all the documents in a single ballot to a
single reviewer. Sometimes, given the length or number of
documents, this is unreasonable and the group is split between
several reviewers.
4. Make a note of whether any of the reviewers is involved with
the draft as either an author or the chair of the WG. This is
sometimes difficult to get right and occasionally, given the
fact that most GenART reviewers are active IETF participants,
sometimes a reassignment is required once the original
assignments are announced.
5. At this point, a very modified round robin process is used to
make the remaining review assignments. The process is
approximately as follows:
+ Begin the round robin from where the previous week's
assignments stopped.
+ If someone has indicated a restricted load, e.g. they have
only agreed to review one draft per cycle, skip them once
that load is met.
+ For every two repeat reviews someone has, skip them in one
cycle of the round robin. The reason for equating two
repeat reviews to a single new review is that there is an
assumption that the task of re-reviewing is lighter then
the process of reviewing. Note: if someone is being
assigned a returning document as if for the first time, it
is treated the same as any new assignment.
+ After the round robin is completed, then the secretary
looks at the list and if it looks very imbalanced with
someone having far too much work to do then some load
shifting may be done. There is no attempt made to actually
equalize the load as the length and complexity of the
drafts is not taken into account in this process. (Thus, a
reviewer could end up with a couple of hundred page
documents, but this is statistically rare.)
o It should be mentioned that assignment is never made according to
a reviewers technical specialty. Even though it happens, when,
for example, routing drafts fall on routing experts or MIBs fall
on MIB doctors, it is coincidental. To the reviewer, the choice
looks random.
Doria (editor), et al. Expires April 15, 2006 [Page 4]
Internet-Draft GenART October 2005
o Once the assignments are made, the web pages that list the reviews
and the assignments are posted. If the reviewers notice any
problems or conflict of interest, a bargaining process, shifting
documents from one reviewer to another, takes place.
o Once the review has been completed the reviewer sends the review
to the GenART list.
o The secretary takes the reviews, sometimes edits them for format,
records the review on the web pages and gives each review a short
synopsis.
o The reviews are then posted on the public web page (this may occur
after the review has been read by the AD, or even after the IESG
telechat).
o If the AD concludes that the concerns raised by the reviewer
warrant placing a DISCUSS comment on the document, the AD will do
so, and the DISCUSS must be resolved before the document advances.
Usually, the reviewer will be involved in the resolution process.
o Occasionally, and more often of late, reviewers have been mailing
reviews to authors, ADs or WGs. This part of the process has been
voluntary, and sometime resulted in confusion among the authors
and working groups. Reviewers have begun pre-pending a
description of GenART and the purpose of the review on these
occasion to lessen the confusion.
o All reviews need to be completed and posted before the Thursday
telechats. For the reviews to be useful to the General area AD
the earlier the posting the better. It should be noted, that this
allows the reviewers only 4 working days and the weekend, for
those who work on the weekend, to complete their reviews. The
secretary generally needs to work late on the Wednesday night
before the telechat to record all the information. In fact the
secretary's job usually requires night work (depending on time
zone effects). It also requires a responsive Internet connection,
even when on travel.
4.2 IETF LC Assignments
While the original process was meant only for late reviews before the
IESG telechat, it was decided to include IETF Last call reviews. In
some ways this proved to be an overloading of the process and has
presented difficulties.
Obviously the greatest utility for IETF LC would be achieved by
having the reviews completed before the end of the last call.
However due to the secretary's schedule and the difficulty of
assigning an IETF LC review as soon as it was announced, these
assignment were often made midway through the LC timing, and
sometimes even after the IETF LC ended. Within GenART practice to
date, the IETF LC assignments took on a different aspect as they were
treated more as early warning of pending IESG reviews then as IETF LC
Doria (editor), et al. Expires April 15, 2006 [Page 5]
Internet-Draft GenART October 2005
requirements. While there were many occasions on which an IETF LC
review was done before the end of the last call, these occasions were
treated as happy coincidences and not as a requirement. The nature
of the IETF LC assignments has remained an issue for the GenART as
the General Area AD would prefer for them to completed by the end of
the IETF LC. There have been frequent conversations on the GenART
mailing list about the nature and timeliness of IETF LC reviews.
IETF LC reviews were assigned on a strictly round robin basis with
the exception that an IETF LC document would not be assigned to
someone who was scheduled to be on vacation for the entire period of
the LC. Otherwise a separate list of the reviewers was maintained
and each assignment was done in order.
4.3 Form of review
Rather than invent new guidelines, the GenART requirements for the
form of a review stole liberally from
draft-carpenter-solutions-sirs-01, making adaptations for the
special "late, quick review" case and the nature of the General
area's concerns.
Each review must start with a summary statement chosen from or
adapted from the following list:
o This draft is ready for publication as a [type] RFC, where [type]
is Informational, Experimental, etc. (In some cases, the review
might recommend publication as a different [type] than requested
by the author.)
o This draft is basically ready for publication, but has nits that
should be fixed before publication.
o This draft is on the right track but has open issues, described
in the review.
o This draft has serious issues, described in the review, and needs
to be rethought.
o This draft has very fundamental issues, described in the review,
and further work is not recommended.
o Unfortunately, I don't have the expertise to review this draft.
The length of a review can vary greatly according to circumstances,
and it is considered acceptable for purely editorial comments to be
sent privately if it's obvious that the document needs substantial
revision. All substantive comments, however, must be included in the
public review. Wherever possible, comments should be written as
suggestions for improvement rather than as simple criticism.
Explicit references to prior work and prior IETF discussion should be
given whenever possible.
Reviewers are asked to review for all kinds of problems, from basic
Doria (editor), et al. Expires April 15, 2006 [Page 6]
Internet-Draft GenART October 2005
architectural or security issues, Internet-wide impact, technical
nits, problems of form and format (such as IANA Considerations or
incorrect references),and editorial issues. Since these reviews are
on documents that are supposed to be finished, the review should
consider "no issue too small" - but should cover the whole range from
the general architectural level to the editorial level.
All reviews should apply generally agreed IETF criteria, such as:
o [RFC1958] The Architectural Principles of the Internet
o [RFC3426] General Architectural and Policy Considerations
o [RFC3439] Some Internet Architectural Guidelines and Philosophy
o [ID-Checklist] The "ID checklist" document maintained by the IESG
o [RFC2223] Instructions to RFC Authors
o [BCP26] Guidelines for Writing an IANA Considerations Section in
RFCs
o [RFC3552] Guidelines for Writing RFC Text on Security
Considerations
o As well as any other applicable architectural or procedural
documents. It is considered important that reviews give precise
references to such criteria when relevant to a comment.
Of special interest to the GEN area (because it's no area's special
interest) is:
o Clear description of why the document or protocol is useful to
the Internet
o Adherence to IETF formalities such as capitalized MUST/SHOULD
(ID-Checklist)
o Useful and reasonable IANA considerations
o Correct dependencies for normative references
o That it's written in reasonably clear English
4.4 Archiving of reviews
All reviews are archived and publicly visible. The archive can be
found at: http://www.alvestrand.no/ietf/gen/art/gen-art.html
5. Results
Over the last 18 months, the GenART has provided reviewing services
to 2 ADs and has done over 450 publicly available reviews. Each of
these reviews was been done in a short interval of about a week by a
dozen or fewer reviewers.
6. Impressions
This section is divided into 3 subsections, the impressions as
gathered from the GenART review team, the impressions of the ADs for
who they worked, and the impressions of one the secretaries of the
Doria (editor), et al. Expires April 15, 2006 [Page 7]
Internet-Draft GenART October 2005
GenART.
6.1 Reviewers' Impressions
The following list of comments are excerpted and edited from comments
sent in by the reviewers of GenART in response to the request:
"We'd like to ask you each to write a few lines about your personal
experience and lessons learned as a Gen-ART reviewer."
o We really do find problems, but we don't find problems with most
documents.
o Comments seem to be in three areas: editorial/grammar, editorial/
what-the-heck-does-this-mean, and actual problems. I'm seeing
fewer reviews in the first category, which is a good thing.
o Since we're not a documented step in the process, a fair number of
authors/editors don't know what to do with us. Arguing is rare,
but negotiating is more common, and it's pretty frequent that
someone asks "do I rev the document now or wait until the
telechat?" Adding the boilerplate explaining who we are, why we
are annoying this particular document, etc. seems to help, but it
would be nice to agree on a particular boilerplate.
o It is becoming rarer that we hear back "these guys have suffered
enough, I'm voting no objection" (I'm remembering an LDAP document
that had been around so long it had 2119 referenced AS A DRAFT -
some people suffered a lot).
o The direct assignment of reviews is necessary and effective. It
does not matter much as far as I can tell what scheme is used to
actually do the assignment.
o Folks are very open to the reviews that come out of GenART. This
somewhat surprised me because I have seen resistance to outside
reviews in other cases.
o The improvements that have come about (for example one of my
latest, the sipping conference draft - whatever the outcome) have
made a big difference to the comprehensibility and usability of
the documents - and provide a useful incentive to keeping going.
o Some form of review like this is desperately needed. While most
of the stuff we see is good, every once in a while really bad
errors have made their way all the way to IESG vote.
o Reading this stuff is interesting. I like having a reason to read
a wide range of materials.
o I am more than convinced that this can be and is a valuable
process. It is IMO a pity that SIRS and so on did not take off,
because this late stage reviewing is a poor substitute for doing
the same thing at a much earlier stage. Very few of the drafts
that have come past my screen are truly fully ready for IESG
review. It is actually a joy to find the occasional nugget that
is both well written and is a proper technical job, such that the
review really can say 'This is ready'.
Doria (editor), et al. Expires April 15, 2006 [Page 8]
Internet-Draft GenART October 2005
o I have certainly found the process intellectually stimulating! It
encourages me to take a wider interest in what is going on in the
IETF, but consumes a fair bit of time to do a proper job, and
requires a very wide knowledge to be able to properly catch the
cross-area implications: I hope (believe!) that this is something
that one gets better at with experience and doing a few of these
reviews.
o There are probably a very limited pool of people who have both the
time and the inclination to keep on doing these reviews. It does
require a fair bit of dedication.
o It is difficult to avoid correcting the English, even if that is
not really the point: Often really bad English (whether as a
result of non-mother tongue authors with limited grasp or mother
tongue authors using informal language) obscures/corrupts what is
being said or just makes it impossible to read.
o Mostly authors welcome the comments: I think most of them
understand the concept of 'ego-free reviewing' and we have
generally been constructive rather than destructive.
o Part of the job of gen-art is to think the unthinkable from
another point of view, to challenge (apparently undocumented)
assumptions and apply experience from other fields.
6.2 One General Area Director's Impressions
It's essential. The reviewing load for the IESG <shout>DOES NOT
SCALE</shout>.
On a single fortnight example, the IESG had 21 drafts on the agenda.
It is just impossible, and no wonder we sometimes miss serious
issues.
So I think a distributed review team with o(30) trusted reviewers
needs to be institutionalized. I suspect that will need to be
formalized in a BCP sooner or later - with their reviews having a
formal position in the standards process, and the expectation that
the whole IESG truly reviews all documents being relaxed.
I think we've learned that polite, well reasoned, constructive
reviews are very positively received by authors and WGs. Dismissive
reviews are counter-productive. And reviews sent in private
eventually show up in public, so it's better to go public at the
start.
I believe that there is a bit of a problem in the way we handle LC
reviews right now. Sometimes the timing works out well (the LC
review comes in good time, and the draft gets revised prior to
hitting the IESG agenda). But quite often the LC review never
happens, or happens so late that the document still has serious
Doria (editor), et al. Expires April 15, 2006 [Page 9]
Internet-Draft GenART October 2005
issues when it hits the agenda.
The other problem I see is a big detail - between late Thursday or
early Friday when the secretary sends out the assignments, and
Wednesday when I like to start filling in ballots, there are only
three work days (plus possible volunteer time over the weekend). Now
even with only one document to review, that may be a real challenge.
Sometimes, a lucky reviewer will get 130 pages (e.g.
draft-ietf-nntpext-base-27). That doesn't compute.
There are some mechanical issues. The process followed is far too
manual. Everything needs to be robotic except for the judgment calls
about which reviewer gets which draft. Similarly, the reviewer
should be able to just paste the review into a web form, click, and
it's sent off to everyone appropriate and posted to the review site.
That, by the way, would be the same tool that we also need to support
early review, WG Last Call review, etc.
6.3 One GenART Secretary's Impressions
Serving as the secretary of GenART was a worthwhile experience. From
a personal point of view, it gave me an easy way to track all of the
work going through the IESG review process and see how the work
flowed through that process. Also, by reviewing and doing light
editing on all of the reviews in order to create some degree of
uniformity of presentation and to create the one line abstracts that
go on the review web page, I had the opportunity to really get a
survey of the work being approved by the IETF.
One element of the GenART that makes it work is the immediacy of the
review. I think that the fact that these are late reviews that have
a particular advisory role allows a relatively small group of people
to read and review. The nature of these reviews is informal, and
originally the reviews were only intended for the General Area AD,
though they were made public. During 2004 there was little if any
interaction between authors and reviewers.
There was some discussion during 2004 about trying to expand the role
of GenART to a more formal, early review model, i.e to evolve it into
a form of SIRS. I was against such a transformation because I felt
it would risk something that worked. I believed that the risk was
inherent in formalizing the reviews and in adding mechanisms for
standardizing the review mechanisms that would resort from
formalization. Another concern involves the interaction between
reviewers and authors. As discussed above, recently, it has become
the practice to send reviews to the authors with an explanation about
the nature of GenART reviews. While it is clear that this has
Doria (editor), et al. Expires April 15, 2006 [Page 10]
Internet-Draft GenART October 2005
resulted in improved RFCs, it has also resulted in increased work
load for the reviewers. The addition of IETF LC reviews was also a
potential increase in work load for the reviewers. If these review
assignments were only to be used as an early warning for a reviewer
of a draft that would eventually be in IESG review, then I saw it as
a helpful to the reviewers. If, on the other hand, they became an
extra work load that need to be completed by the end of LC then I
thought it was harmful.
I think that GenART is an experiment that works but I believe it is
fragile. As secretary I was often concerned about overburdening
reviewers, and felt it my responsibility to keep them from burn out;
though I was never sure how to achieve that given the work load.
7. Needed Improvements
Automation of the manual processes of assignment, and of mailing and
web-posting the resulting reviews seems, to be the main improvement
needed. For example, the IETF Secretariat system generates an
"Evaluation" message to the IESG whenever a ballot is initiated for a
draft. These messages, and "IETF Last Call" messages, could be used
to trigger assignment of reviews to reviewers, with only exceptional
cases needing manual handling. A web tool could allow reviewers to
upload their reviews and send mail to interested parties.
8. Applicability
As implemented today, the process has no formal role in the IETF
standards process. Whether the reviews appear during IETF Last Call
or during IESG Evaluation, they have no distinguished status. But as
trust in the review team has built, and as the team itself has
learned to deliver reviews that are generally well received, they
have had a significant impact on document quality and on timeliness.
Rather than becoming a roadblock, they have (in general) allowed the
General AD to feel more confident in reaching decisions and be more
precise in resolving issues.
A question that naturally arises is whether a formal or official role
for such a team would bring even more benefits.
9. Acknowledgments
Initial comments were received from the members of the GenART team
and the experiences discussed in this document were derived from
their hard work over the last 18 months as reviewers. We thank the
reviewers of GenART: Mark Allman, Mary Barnes, David Black, Scott
Brim, Elwyn Davies, Spencer Dawkins, Lakshminath Dondeti, Joel
Halpern, John Loughney, Lucy Lynch, Michael Patton, and Suzannne
Doria (editor), et al. Expires April 15, 2006 [Page 11]
Internet-Draft GenART October 2005
Woolf
10. References
10.1 Informative References
10.2 Normative References
Authors' Addresses
Avri Doria
ETRI
Lulea University of Technology
Lulea
SE
Phone: +1 401 663 5024
Email: avri@acm.org
Harald Alvestrand
Cisco Systems
Weidemanns vei 27
Trondheim 7043
NO
Phone:
Email: harald@alvestrand.no
Brian E Carpenter
IBM Zurich Research Laboratory
Saeumerstrasse 4
8803 Rueschlikon
Switzerland
Phone:
Email: brc@zurich.ibm.com
Appendix A. IANA considerations
As this is an informational document about an IETF process, there are
no IANA considerations. However, one of the requirements for an IETF
document is the inclusion of IANA considerations section. One of the
areas that the IESG review includes, and thus the GenART review needs
to include is a reading of the IANA section to make sure that it not
only exists, but that it includes the necessary information for IANA
Doria (editor), et al. Expires April 15, 2006 [Page 12]
Internet-Draft GenART October 2005
to makes its assignements.
Doria (editor), et al. Expires April 15, 2006 [Page 13]
Internet-Draft GenART October 2005
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
The IETF has been notified of intellectual property rights claimed in
regard to some or all of the specification contained in this
document. For more information consult the online list of claimed
rights.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2005). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Doria (editor), et al. Expires April 15, 2006 [Page 14]
Internet-Draft GenART October 2005
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Doria (editor), et al. Expires April 15, 2006 [Page 15]
