Network Working Group M. Barnes
Internet-Draft Nortel
Intended status: Informational A. Doria
Expires: February 16, 2009 Lulea University of Technology
H. Alvestrand
Google
B. Carpenter
University of Auckland
August 15, 2008
General Area Review Team (GenART) Experiences
draft-doria-genart-experience-02
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on February 16, 2009.
Abstract
The General Area Review team has been doing Reviews of Internet
Drafts since 2004. This draft discusses the experience and the
lessons learned over the past 4+ years of this process. The review
team initially did reviews before each of the IESG telechats.
Beginning in late 2005, review team members have been assigned to
review documents during IETF Last Call, unless no IETF Last Call is
necessary for the document. The same reviewer then reviews any
Barnes, et al. Expires February 16, 2009 [Page 1]
Internet-Draft GenART August 2008
updates when the document is placed on an IESG telechat agenda.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Who are the GenART review team members? . . . . . . . . . . . 3
3. Goals of GenART . . . . . . . . . . . . . . . . . . . . . . . 3
4. GenART Reviews . . . . . . . . . . . . . . . . . . . . . . . . 4
4.1. IETF LC Review Process . . . . . . . . . . . . . . . . . . 4
4.2. IESG Telechat Review Process . . . . . . . . . . . . . . . 5
4.3. Form of Review . . . . . . . . . . . . . . . . . . . . . . 5
4.4. GenART Process Overview . . . . . . . . . . . . . . . . . 7
5. Secretarial Process . . . . . . . . . . . . . . . . . . . . . 9
5.1. Maintaining review spreadsheet . . . . . . . . . . . . . . 9
5.2. Last Call Assignment procedure . . . . . . . . . . . . . . 11
5.3. Telechat Assignment procedure . . . . . . . . . . . . . . 12
5.4. Capturing reviews . . . . . . . . . . . . . . . . . . . . 13
6. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
7. Impressions . . . . . . . . . . . . . . . . . . . . . . . . . 14
7.1. Reviewers' Impressions . . . . . . . . . . . . . . . . . . 14
7.2. General Area Directors' Impressions . . . . . . . . . . . 15
7.3. GenART Secretaries' Impressions . . . . . . . . . . . . . 16
8. Needed Improvements . . . . . . . . . . . . . . . . . . . . . 17
9. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 18
10. Security considerations . . . . . . . . . . . . . . . . . . . 18
11. IANA considerations . . . . . . . . . . . . . . . . . . . . . 19
12. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19
13. Changes since Last Version . . . . . . . . . . . . . . . . . . 19
14. Informative References . . . . . . . . . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20
Intellectual Property and Copyright Statements . . . . . . . . . . 22
Barnes, et al. Expires February 16, 2009 [Page 2]
Internet-Draft GenART August 2008
1. Introduction
The General Area Review team was created personally by the General
Area Director in 2004. The review team has been retained by
subsequent General Area Directors. It has no official role in the
IETF standards process, except as a set of individuals entitled, like
everyone, to comment on Internet-Drafts. Its Secretary, and the team
of volunteer reviewers, serve at the invitation of the General AD.
Discussion of this document is intended to take place on the IETF
mailing list <mailto: ietf@ietf.org> in the absence of a better home.
In addition, comments may be specifically sent to the gen-art mailing
list: <mailto: gen-art@ietf.org>.
2. Who are the GenART review team members?
The reviewers are typically individuals that have a fair amount of
experience within various IETF Working Groups (WGs), have authored WG
drafts and RFCs, and are often considered to be subject matter
experts (SMEs) in their particular areas of work. The current review
team is comprised of such technical experts including several WG
chairs and past and current IAB members. Several past and current
ADs have served as reviewers. Two past General ADs have also served
as reviewers, with one currently serving.
Members of the review team sometimes excuse themselves from the team
for various reasons, typically due to "day" job demands. However,
they often rejoin (for periods of time) as their schedules allow.
Also, some reviewers remain on the team, while their review workload
is decreased by assigning them just one document (at Last Call time)
to review each month. Section 12 provides a list of currently active
reviewers, along with those who have served on the review team in the
past.
3. Goals of GenART
The original and continuing goal of the GenART team was, and is, to
offload some of the burden from the General Area AD of IESG reviews.
The load for the bi-weekly IESG reviews is often quite large;
occasionally there are more than 20 drafts scheduled for discussion
in a single telechat. Thus, ADs also have less than a week's notice
for many of the documents on the telechat agenda.
GenART was based on a model that had proved productive in the OPS
Directorate: Quick review close to telechat time, to advise the AD on
issues that remain serious. By having a trusted group of reviewers
Barnes, et al. Expires February 16, 2009 [Page 3]
Internet-Draft GenART August 2008
read and evaluate the drafts, the General Area AD would be able to
concentrate on those drafts where there was an concern expressed by
the reviewer. The reviewers are expected to provide feedback based
on a whole set of criteria including the criteria summarized in
Section 4.3. The overall objective is to ensure that the documents
are well structured, can be easily understood, at least at a high
level, and provides a reasonable basis for implementation (for
standard's track documents).
While other area (and WG) directorates/review teams existed prior to
GenART and more have been established since GenART, the roles of each
are fairly distinct. Thus, there is little overlap between the goals
and review criteria for the various review teams. It is also very
valuable for these other review teams to operate independently. For
example, when both GenART reviews and sec-dir reviews raise the same
sorts of concerns, it's a clear red flag that the document needs more
work before progressing. In addition, due to the typical
thoroughness (and objectiveness) of the various review teams'
reviews, ADs/PROTO shepherds are often able to work with the
editors/WG (and vice versa depending upon area and WG structure) to
improve the overall quality of the final document.
Statistics from the GenART reviews over the past 4+ years show a
trend of increased quality and readiness for progression of documents
by the time they are placed on the telechat agenda. Additional
statistics are discussed in Section 6.
4. GenART Reviews
4.1. IETF LC Review Process
While the original process was meant only for reviews just before the
IESG telechat, it was decided to include IETF Last Call reviews in
early 2005. This initially seemed to be an overloading of the
process and presented some initial difficulties. However, over time
it has proven to be quite effective. Assigning the documents at IETF
LC time typically gives a reviewer more time to review a document.
And, in many cases, the IETF LC version is the one to appear on the
telechat. Thus, by the time documents are added to the telechat
agenda, a majority (typically at least 70%) have already been
reviewed. For those documents that have been up-versioned, the
amount of time dedicated to re-review depends upon the review summary
for the IETF LC review.
The IETF LC assignments evolved to minimize the gap between LC
announcements and assignment time, with the secretary doing LC
assignments every Thursday night. This typically allows the reviewer
Barnes, et al. Expires February 16, 2009 [Page 4]
Internet-Draft GenART August 2008
at least one week and sometimes two to three weeks to complete the
review. The reviews are obviously most helpful when done on or
before the end of IETF LC.
The Last Call assignments are done on a fairly strict round robin
basis to ensure a fair workload amongst all the reviewers. Reviewers
that are unavailable (vacations, etc.) during the review period
timeframe obviously are excluded from that round of assignments, but
remain in the same queue position for the next round. The order is
occasionally modified to avoid assigning an editor/author or WG chair
their own documents. A reviewer may also NACK an assignment if they
feel they may have some bias (although corporate affiliations are not
considered to be sources of bias) or they don't feel they can review
the document in a timely manner.
The assignment process is completely manual, although a spreadsheet,
maintained using Open Office, tremendously facilitates the process.
The details are described in Section 5. Ideally, this process could
be automated. However, manual intervention would still be required
to maintain the appropriate available reviewer list (unless reviewers
took on the task of maintaining their data in some sort of database).
Further details on the tools necessary to automate the entire process
are provided in Section 8.
4.2. IESG Telechat Review Process
The process for reviewing documents when they appear on the IESG
agenda:
o The "nearly final" IESG meeting agenda generally appears on
Thursday night, less than one week before the IESG telechat. The
GenART secretary uses this as the input for the assignment
process.
o For documents reviewed at IETF Last Call, a new review is only
asked for if the document is revised. In this case the reviewer,
typically the person who did the Last Call review, only needs to
check that any open issues were resolved. Often the draft will
not have changed between IETF LC and the IESG telechat review.
Section 4.4 provides the step by step telechat review assignment
process, with specific details on the maintenance of the review
assignment data, maintained in spreadsheets detailed in section
Section 5.
4.3. Form of Review
Rather than invent new guidelines, the GenART requirements for the
form of a review stole liberally from
draft-carpenter-solutions-sirs-01, making adaptations for the special
Barnes, et al. Expires February 16, 2009 [Page 5]
Internet-Draft GenART August 2008
"late, quick review" case and the nature of the General Area's
concerns.
Each review must start with a summary statement chosen from or
adapted from the following list:
o This draft is ready for publication as a [type] RFC, where [type]
is Informational, Experimental, etc. (In some cases, the review
might recommend publication as a different [type] than requested
by the author.)
o This draft is basically ready for publication, but has nits that
should be fixed before publication.
o This draft is on the right track but has open issues, described in
the review.
o This draft has serious issues, described in the review, and needs
to be rethought.
o This draft has very fundamental issues, described in the review,
and further work is not recommended.
o Unfortunately, I don't have the expertise to review this draft.
The length of a review can vary greatly according to circumstances,
and it is considered acceptable for purely editorial comments to be
sent privately if it's obvious that the document needs substantial
revision. All substantive comments, however, must be included in the
public review. Wherever possible, comments should be written as
suggestions for improvement rather than as simple criticism.
Explicit references to prior work and prior IETF discussion should be
given whenever possible.
Reviewers are asked to review for all kinds of problems, from basic
architectural or security issues, Internet-wide impact, technical
nits, problems of form and format (such as IANA Considerations or
incorrect references), and editorial issues. Since these reviews are
on documents that are supposed to be finished, the review should
consider "no issue too small" - but should cover the whole range from
the general architectural level to the editorial level.
All reviews should apply generally agreed IETF criteria, such as:
o [RFC1958]: The Architectural Principles of the Internet
o [RFC3426]: General Architectural and Policy Considerations
o [RFC3439]: Some Internet Architectural Guidelines and Philosophy
o ID-Checklist: The "ID checklist" document maintained by the IESG
o [I-D.rfc-editor-rfc2223bis]: Instructions to RFC Authors
o [RFC5226]: BCP 26 - Guidelines for Writing an IANA Considerations
Section in RFCs
Barnes, et al. Expires February 16, 2009 [Page 6]
Internet-Draft GenART August 2008
o [RFC3552]: Guidelines for Writing RFC Text on Security
Considerations
o As well as any other applicable architectural or procedural
documents. It is considered important that reviews give precise
references to such criteria when relevant to a comment.
Of special interest to the GEN area, because it's no other area's
special interest is:
o Clear description of why the document or protocol is useful to the
Internet.
o Adherence to IETF formalities such as capitalized "must",
"should", etc. in normative statements, per the ID-Checklist.
o Useful and reasonable IANA considerations. Ensure that all
necessary registries are defined/referenced and ensure definition
and compliance with IANA assignment criteria.
o Correct dependencies for normative references.
o That it's written in reasonably clear English.
o Checking the updates/obsoletes information.
o Running idnits and checking the output.
o Checking that things imported by reference especially from other
RFCs make sense (notably definitions of terms, security
considerations, lists of criteria)and ensure they are used as
intended by the refeenced document.
o Examples (eg FQDNs, telephone numbers, IP addresses) are taken
from the right spaces.
4.4. GenART Process Overview
The following provides a general overview of the Gen-ART process
along with some basic rules associated with assignments. The very
precise details of the secretary's process are provided in Section 5.
o A list of the reviewers availability is maintained in a
spreadsheet on the GenART server.
o At telechat assignment time, all previously reviewed drafts are
assigned to the reviewer who reviewed them previously, assuming
that reviewer is available. Otherwise, these documents are
assigned to a new person in the process described below.
o In the case of multiple drafts grouped as a single ballot, those
are typically divided among several reviewers unless they're very
small (i.e., less than 20 pages).
o The secretary does maintain the data as to IETF roles such as WG
chairs, other directorates, etc. to obviously avoid assigning a
document for which an individual has too much involvement. In the
cases where the secretary doesn't note the over-involvement, the
reviewer should notify the secretary and gen-art mailing list so
another reviewer may be assigned.
Barnes, et al. Expires February 16, 2009 [Page 7]
Internet-Draft GenART August 2008
o It should be emphasized that assignment is never made according to
a reviewer's technical specialty. Even though it happens, when,
for example, routing drafts fall on routing experts or MIBs fall
on MIB doctors, it is coincidental. To the reviewer, the choice
looks random.
o There is an attempt to evenly distribute documents amongst
reviewers at LC time by using a round robin process, starting from
where the previous week's assignments stopped.
o Typically, there is no attempt made to actually equalize the load
as the length and complexity of the drafts is not taken into
account in this process. (Thus, a reviewer could end up with a
couple of hundred-page documents, but this is statistically rare.)
However, in the case of a reviewer that might receive more than
one new LC document at one time, the secretary does try to ensure
that both are not large documents.
o Once the assignments are made, the web pages that list the reviews
and the assignments are posted.
o If the reviewers notice any problems or conflict of interest, a
bargaining process, shifting documents from one reviewer to
another, takes place. The secretary updates the assignment files
with any new assignments.
o Once the review has been completed the reviewer sends the review
to the GenART list, ideally using the template provided in the
review assignment emails. Typically, reviews are also sent to
authors, ADs and WG chairs/Proto Shepherds. The only case where
this might not be done is when there are no issues found for a re-
review and none had been found on an initial review. Sending the
review to the authors, ADs and/or WG chairs/Proto Shepherds had
been voluntary but is now considered standard practice. Reviewers
may also send the reviews to the IETF discussion list, but that is
entirely at the discretion of the reviewer, in which case the
author must be copied on the review to ensure they see any
follow-up discussion. Reviewers may also send the comments to the
WG, however, this typically causes the review to end up in the
moderation queue, as most reviewers do not want to subscribe to
the WG lists for the documents they review. Thus, it is expected
that the original recipients (authors, WG chairs/PROTO or AD) may
forward the review to the WG mailing list if they believe it is
necessary. In the past, sending these reviews resulted in
confusion among the authors, who may not have been expecting a
GenART review and may not be familiar with GenART. Thus,
reviewers are reminded to pre-pend the description of GenART and
the purpose of the review. This information is part of the
standard template provided in the review assignment emails.
o The secretary takes the reviews, sometimes edits them for format,
records the review on the web pages, including the synopsis. If
the reviewer has not provided a synopsis ("Summary" field in the
template), the secretary makes a best guess based on the review
Barnes, et al. Expires February 16, 2009 [Page 8]
Internet-Draft GenART August 2008
details. Note that in most cases the reviewers do include a
synopsis.
o The reviews are then posted on the public web page. All reviews
received by COB Tuesday (approximately 8 PM EST) before the
telechat are uploaded, along with the updated spreadsheets
containing review summaries/assignments. It should be noted, that
this allows the reviewers only 3 working days and the weekend, for
those who work on the weekend, to complete their reviews.
However, this is necessary to allow the current General area AD
(whose in the Eastern time zone in the U.S.) time to read all the
reviews. The secretary generally needs to work late on the
Tuesday night before the telechat to record all the information.
In fact the secretary's job usually requires night work (depending
on time zone effects). It also requires a responsive Internet
connection, even when on travel.
o If the AD concludes that the concerns raised by the reviewer
warrant placing a DISCUSS comment on the document, the AD will do
so, and the DISCUSS must be resolved before the document advances.
Usually, the reviewer will be involved in the resolution process,
but the responsibility for the DISCUSS rests with the AD.
o If the reviews are received after Tuesday the review may not be
read by the AD nor uploaded before the IESG telechat, since the
secretary typically performs the updates in batches to minimize
the time spent on this task. Thus, updates typically occur on
Tuesdays during the week of telechats and weekly on the Thursday
evenings prior to the sending of new LC and/or telechat
assignments.
5. Secretarial Process
This section summarizes the details of managing the review materials,
including the spreadsheet used to track all reviews and the HTML
files containing the review assignments.
5.1. Maintaining review spreadsheet
An Open Office spreadsheet is used to enter all the documents at the
time of assignment and to capture all the reviews. For IETF LC
assignments, the assignments are completed before adding the
documents to the spreadsheet as described in Section 5.2. For
telechat assignments documents are obviously only added in the cases
where there is no previous LC assignment. For the other documents,
the appropriate fields are updated as described in Section 5.3
All the reviews can be accessed from the spreadsheet via hyperlinks
from specific fields as summarized below. The following information
is maintained in the spreadsheet (in the order listed):
Barnes, et al. Expires February 16, 2009 [Page 9]
Internet-Draft GenART August 2008
1. "Chat/LC Date": indicates either the date on which the LC review
is due or the date of the telechat.
2. "Document": Filename for the text document. This field also
includes a hyperlink to the IETF I-D tracker.
3. "Assigned": Name of the reviewer assigned to that document.
4. "Category": This field contains one of the following self
explanatory values: "PROTO - WG", "PROTO - Ind/AD", "Doc - WG",
"Doc - Ind/AD", or "IETF LC". Note that GenART does not review
documents submitted directly to the RFC editor. The "IETF LC"
field is entered obviously for all documents at LC time. It is
changed to one of the other appropriate fields, based on the
information in the telechat agenda
5. "Previous Review": This includes a link to any previous reviews.
For example, when a doc appears on a telechat agenda, if an IETF
LC review was done, this field is updated to "IETF-LC" and it has
a hyperlink to the LC review. The field is set to "New" when a
document is first assigned/added to the spreadsheet. In the case
of returns, this field has a value of "Return" or "Return/
IETF-LC" for documents for which there is an LC review. It
should be noted that since GenART started doing reviews at LC
time, there seem to be far fewer returns on the agenda.
6. "Current Review Summary": When the field contains text, it
includes a link to the most recent review - typically IETF LC or
telechat. Occasionally, a reviewer will re-review a document
prior to its telechat assignment, in which case it is added to
the spreadsheet but the date does not change to maintain
consistency in the date field, since the reviews themselves
contain the review date.
The following summarizes the steps to add a new document to the
spreadsheet:
1. In order to optimize steps, blank rows are first inserted for the
number of new documents to be added.
2. To minimize data entry, a row with default fields (including the
links for the hyperlinks) is kept at the end of the file. There
is a separate default row for IETF LC versus Telechat
assignments. This row is copied into each of the new blank rows.
The dates are then entered (this allows the double checking that
all documents are accounted for from the review assignments,
especially LC).
3. The document name is then copied to the name field as well as
being appended to the hyperlink for the "Review Summary" field.
The hyperlink is included as part of the default row. This
minimizes the steps in enter the reviews in the spreadsheet.
4. Once all the new documents for that round of assignments have
been added, depending upon the editor, the font size may need to
be normalized in the spreadsheet. The data is also sorted by
Barnes, et al. Expires February 16, 2009 [Page 10]
Internet-Draft GenART August 2008
"Chat/LC Date", "Assigned" and "Document". The file is then
saved and closed.
5. The file is then reopened and saves as HTML.
6. The file is opened a second time and sorted by "Assigned",
"Chat/LC Date" and "Document"
5.2. Last Call Assignment procedure
The secretary can either cache the Last Call assignments as they are
announced or just check the IETF announcement mailing list archives.
The current secretary does both, double-checking the archives to
ensure no reviews were missed. The assignments are done on Thursday
evening, along with any telechat assignments. This optimizes the
process in terms of batch changes to files.
The assignments are listed in an HTML file. The following are the
steps in creating that file:
1. The order of assignment is actually created the week before, with
the details below. Thus, before starting the new assignments the
current file is saved for editing for the following week. The
current filenaming convention is "reviewersyymmdd-lc.html (e.g.,
for July 10th, the filename was reviewers080710-lc.html, created
in the process of July 3rd assignments, and the file for the
following week is named reviewers080717-lc.html).
2. Since the file is already prepared with the appropriate ordering
of reviewers, the assignments are given in the order of due
dates. The LC announcement text (starting with the document
name) is copied into the assignment file for each of the new LC
documents.
3. The paragraph as to the "Due Date" is shortened with the
following text: "IETF LC ends on:", keeping the date.
4. Depending upon editors and whether text was pulled from email
versus mailing list archives, the text font is normalized.
5. Once the assignment file is complete, the new documents are added
to the spreadsheet as described in above.
6. The assignment file for the next week is then updated to reflect
the next reviewer in the round robin process, by simply cutting
and pasting the names in the list in a block and removing any
"one doc per month" reviewers that have already received their
monthly assignment. If the next round of assignments occurs at
the beginning of a new month, the "one doc per month" reviewers
are added back into the list (in the normal "by first name
alphabetical order").
7. The assignment files and updated spreadsheets are then cached on
the GenART server.
Barnes, et al. Expires February 16, 2009 [Page 11]
Internet-Draft GenART August 2008
8. An email providing a link to the assignment file, along with the
updated spreadsheets is sent to the gen-art mailing list. This
email has a standard form, such that the reviewers can simply cut
and paste the template to include the GenART context statement
and link to FAQ.
5.3. Telechat Assignment procedure
Since LC assignments are now the starting point for GenART document
reviews, the telechat assignments are generally straightforward as
the majority of the documents are already in the spreadsheet. The
following details the steps:
1. The telechat agenda is typically available around 6PM PDT. In
order to create the assignment HTML file, the agenda is accessed
from the IESG webpage and then editted and saved locally with a
filename of "reviewersyymmdd" with the date corresponding to the
telechat date.
2. Rows are added to the agenda for the reviewer's name and to add
a space for readability.
3. The reviewers names are then added to the weekly assignment
file.
4. As each reviewer is added to the assignment file, the review
spreadsheet is updated as follows:
* "Chat/LC Date" is changed to the telechat date.
* The link to the LC review, if available, is copied to past as
the link for the "Previous Review" column.
* If the version for the telechat is different, the link in the
"Current Review" column is updated, so that it will point to
the new review when available (this saves a step because the
updating of file version is done in the same step AFTER the
link is copied. The "text" for the "Current Review" is
cleared (i.e., set to the default "-".
* If the version number is different, the change is also made
to the "Document" field. Note, this is the least critical
step because the link in that field points to the tracker, so
the right version should always be pulled.
5. After all the docs that had been reviewed at LC time have been
updated in the review spreadsheet (and reflected in the
assignment file), the number of documents for each reviewer is
added to the reviewer availability spreadsheet. This allows the
secretary to determine who is available to review any unassigned
documents.
6. Once the subset of folks that have no assignments or few
assignments is determined, the secretary double-checks the most
recent spreadsheet sorted by reviewer to exclude the reviewers
that might be behind or overloaded with LC reviews. Also,
whether the reviewer received an assigment for the previous
Barnes, et al. Expires February 16, 2009 [Page 12]
Internet-Draft GenART August 2008
telechat is considered.
7. Once the reviewer(s) have been determined, the assignment file
is updated.
8. Any new documents are then added to the spreadsheet (and the
updates saved) per the steps as described in Section 5.1.
9. The assignment files and updated spreadsheets are then cached on
the GenART server.
10. An email providing a link to the assignment file, along with the
updated spreadsheets is sent to the gen-art mailing list. This
email has a standard form, such that the reviewers can simply
cut and paste the template to include the GenART context
statement and link to FAQ.
5.4. Capturing reviews
As noted in Section 4.4 the spreadsheet is typically updated with the
review summaries on Tuesdays evenings prior to telechats and on
Thursday evenings just prior to entering the data for that week's
assignments. The following summarizes the steps to capture the
reviews:
1. The current secretary caches the email reviews as they arrive.
2. In the cases where the review is included inline in the body of
the email, the review is cut and pasted into a text file and
saved with the reviewers last name appended to the filename -
e.g., draft-ietf-xyz-00-smith.txt .
3. In the case where the review is included as an attachment to the
email, the file can be directly saved locally.
4. The review summary is entered into the text portion for the
"Current Summary" field. Noting that the hyperlink to the review
(added at assignment time) will automatically work when the file
is uploaded.
5. Once all the reviews have been entered and the spreadsheets
formatted, the review spreadsheet is saved and files uploaded per
the last three steps in Section 5.1.
6. Results
Over the past 4+ years, the GenART has provided reviewing services to
3 ADs and has done over 450 publicly available reviews. Each of
these reviews was been done with a team of a dozen or fewer
reviewers. In terms of improving quality, the number of documents
that are now ready at the time of the telechat since the reviews are
now initiated at LC time has increased. Based on the data from 2007,
there were over 250 documents that were assigned at LC time that went
through IESG review. Of those 250 documents, 80% of the LC reviews
were completed (205 documents). Of the completed reviews about 75%
Barnes, et al. Expires February 16, 2009 [Page 13]
Internet-Draft GenART August 2008
(144 documents) were "Ready" at the time of the telechat. Of those
144 documents, roughly 1/4 had been deemed "Ready" (with no nits) at
LC time (based on a sample of 50 reviews). For the documents that
were not reviewed at LC time, only about 1/4 of those were deemed
"Ready" when they were reviewed for the telechat. So, doing the gen-
art reviews at Last Call time does seem to improve the quality of the
documents for the telechat.
7. Impressions
This section is divided into 3 subsections, the impressions as
gathered from the GenART review team, the impressions of the ADs for
who they worked, and the impressions of the secretaries of Gen-ART.
7.1. Reviewers' Impressions
The following list of comments are excerpted and edited from comments
sent in by the reviewers of GenART in response to the request:
"We'd like to ask you each to write a few lines about your personal
experience and lessons learned as a GenART reviewer."
o We really do find problems, but we don't find problems with most
documents.
o Comments seem to be in three areas: editorial/grammar, editorial/
what-the-heck-does-this-mean, and actual problems. I'm seeing
fewer reviews in the first category, which is a good thing.
o It is becoming rarer that we hear back "these guys have suffered
enough, I'm voting no objection" (I'm remembering an LDAP document
that had been around so long it had 2119 referenced AS A DRAFT -
some people suffered a lot).
o The direct assignment of reviews is necessary and effective. It
does not matter much as far as I can tell what scheme is used to
actually do the assignment.
o Folks are very open to the reviews that come out of GenART. This
somewhat surprised me because I have seen resistance to outside
reviews in other cases.
o The improvements that have come about (for example one of my
latest, the sipping conference draft - whatever the outcome) have
made a big difference to the comprehensibility and usability of
the documents - and provide a useful incentive to keep going.
o Some form of review like this is desperately needed. While most
of the stuff we see is good, every once in a while really bad
errors have made their way all the way to IESG vote.
o Reading this stuff is interesting. I like having a reason to read
a wide range of materials.
Barnes, et al. Expires February 16, 2009 [Page 14]
Internet-Draft GenART August 2008
o I am more than convinced that this can be and is a valuable
process. It is IMO a pity that SIRS and so on did not take off,
because this late stage reviewing is a poor substitute for doing
the same thing at a much earlier stage. Very few of the drafts
that have come past my screen are truly fully ready for IESG
review. It is actually a joy to find the occasional nugget that
is both well written and is a proper technical job, such that the
review really can say 'This is ready'.
o I have certainly found the process intellectually stimulating! It
encourages me to take a wider interest in what is going on in the
IETF, but consumes a fair bit of time to do a proper job, and
requires a very wide knowledge to be able to properly catch the
cross-area implications: I hope (believe!) that this is something
that one gets better at with experience and doing a few of these
reviews.
o There are probably a very limited pool of people who have both the
time and the inclination to keep on doing these reviews. It does
require a fair bit of dedication.
o It is difficult to avoid correcting the English, even if that is
not really the point: Often really bad English (whether as a
result of non-mother tongue authors with limited grasp or mother
tongue authors using informal language) obscures/corrupts what is
being said or just makes it impossible to read.
o Mostly authors welcome the comments: I think most of them
understand the concept of 'ego-free reviewing' and we have
generally been constructive rather than destructive.
o Part of the job of GenART is to think the unthinkable from another
point of view, to challenge (apparently undocumented) assumptions
and apply experience from other fields.
7.2. General Area Directors' Impressions
It should be noted that these impressions are from multiple General
Area Directors' thus the "I"s are not necessarily associated with a
specific AD.
It's essential. The reviewing load for the IESG <shout>DOES NOT
SCALE</shout>.
On a single fortnight example, the IESG had 21 drafts on the agenda.
It is just impossible, and no wonder we sometimes miss serious
issues.
So I think a distributed review team with o(30) trusted reviewers
needs to be institutionalized. I suspect that will need to be
formalized in a BCP sooner or later - with their reviews having a
formal position in the standards process, and the expectation that
the whole IESG truly reviews all documents being relaxed.
Barnes, et al. Expires February 16, 2009 [Page 15]
Internet-Draft GenART August 2008
We've learned that polite, well reasoned, constructive reviews are
very positively received by authors and WGs. Dismissive reviews are
counter-productive. And reviews sent in private eventually show up
in public, so it's better to go public at the start.
Normally, LC reviews are available in good time for the draft to be
revised before reaching the IESG agenda. It is important that this
happens, except for an emergency situation where the responsible AD
has good reason to place the draft on the agenda immediately. In
that case it would be preferable for the AD to inform the GenART
team, so that the review can be expedited.
The other problem is a big detail - between late Thursday or early
Friday when the secretary sends out the assignments, and Wednesday
when the General Area AD likes to start filling in ballots, there are
only three work days (plus possible volunteer time over the weekend).
Now even with only one document to review, that may be a real
challenge. Sometimes, a lucky reviewer will get 130 pages (e.g.
draft-ietf-nntpext-base-27). That doesn't compute.
There are some mechanical issues. The process followed is far too
manual. Everything needs to be robotic except for the judgment calls
about which reviewer gets which draft. Similarly, the reviewer
should be able to just paste the review into a web form, click, and
it's sent off to everyone appropriate and posted to the review site.
7.3. GenART Secretaries' Impressions
Serving as the secretary of GenART is a worthwhile experience. From
a personal point of view, it gives the secretary an easy way to track
all of the work going through the IESG review process and see how the
work flowed through that process. Also, by reviewing and doing light
editing on all of the reviews in order to create some degree of
uniformity of presentation and to create the one line abstracts that
go on the review web page, the secretary has an opportunity to really
get a survey of the work being approved by the IETF.
The nature of these reviews is informal, and originally the reviews
were only intended for the General Area AD, though they were made
public. During 2004 there was little if any interaction between
authors and reviewers. There was some discussion during 2004 about
trying to expand the role of GenART to a more formal, early review
model, i.e to evolve it into a form of SIRS. The original GenART
secretary was against such a transformation because she felt it would
risk something that worked. She believed that the risk was inherent
in formalizing the reviews and in adding mechanisms for standardizing
the review mechanisms that would resort from formalization. Another
concern involves the interaction between reviewers and authors. As
Barnes, et al. Expires February 16, 2009 [Page 16]
Internet-Draft GenART August 2008
discussed above, it has become the practice to send reviews to the
authors with an explanation about the nature of GenART reviews.
While it is clear that this has resulted in improved RFCs, it has
also resulted in increased work load for the reviewers.
The secretary thinks that GenART as an experiment that works well,
but the secretary believes it is fragile. The secretary is often
concerned about overburdening reviewers, and feels it is her
responsibility to keep them from burning out. Adding additional
reviewers to the review team would help to alleviate this concern.
In terms of the process, adding additional reviewers has minimal
impact.
8. Needed Improvements
The current size of the review team introduces a fairly heavy
workload for the individual reviewers that are not on the "one doc
per month" assignment cycle. Additional reviewers would be really
helpful to alleviate this workload. It is also important to note
that having additional reviewers adds minimal workload to the
secretary's process, thus the only blocking point is finding the
right folks that are interested in this type of volunteer role. As
noted in Section 7.2, 30 would be a good size for the review team.
This would cut the workload for an individual reviewer by more than
60% (given the current model of 5 reviewers on the "one doc per
month" assignment cycle).
Obviously, automation of the process would be a good thing. However,
the current secretary is not highly motivated to transition to a more
automated approach until a significant part of the process is
automated. In more recent consideration of this situation, it likely
would be best to first automate the process of entering the reviews,
as that benefits the review team as a whole. This automation should
allow the reviewers to enter the reviews via a web interface that
would automatically generate the appropriate emails - quite similar
to how the draft "Upload" tool currently works. Also, given
consistent naming conventions for the review forms, this step would
automate some of the process for the secretary, as the reviews would
automatically appear via the Spreadsheet hyperlinks, although there
would still be a need to manually enter the summary. But, this would
eliminate the need for the secretary to edit/normalize and upload
files. And, hopefully, eliminate the problem encountered with
unflowed text in emails and getting the review properly formatted
using some text editors.
Section 5 was written to facilitate the process of determining tools
requirements, by providing the very detailed steps currently applied
Barnes, et al. Expires February 16, 2009 [Page 17]
Internet-Draft GenART August 2008
to the process. As noted above, automating the upload of the reviews
could be a good first step. This is somewhat starting at the end of
the process. However, it seems that by automating in this direction,
we may have optimal results, since one of the earliest steps in the
process is the task of assiging reviewers it likely needs the most
manual intervention, even with tools available.
The current security directorate (sec-dir) secretary does use some
tools for assignments and generating assignment emails. These tools
could be considered for use by the GenART secretary. Since the sec-
dir reviews are not cached and the information maintained for those
reviews is less detailed, there would be no reusability of that
aspect. However, if the GenART spreadsheet can be automatically
populated (with assignments and completed reviews), the sec-dir may
be able to make use of that same tool.
A third improvement would be to move the review repository to an IETF
hosted server. This would provide us more reliability in terms of
having a back-up server and is required when we automate the process.
Thus, we should make this step a priority and the first step prior to
any automation.
9. Applicability
As implemented today, the process has no formal role in the IETF
standards process. But as trust in the review team has built, and as
the team itself has learned to deliver reviews that are generally
well received, they have had a significant impact on document quality
and on timeliness. Rather than becoming a roadblock, they have (in
general) allowed the General AD to feel more confident in reaching
decisions and be more precise in resolving issues. Since reviews now
typically appear during IETF Last Call, the reviews like the sec-dir
reviews are now generally expected. So, the role of the team has
evolved to be more formal than in the past (i.e., when this document
was first published in 2005). However, the handling of the reviews
remain entirely within the scope of the ADs, PROTO shepherds, WG and
authors as they deem appropriate.
10. Security considerations
Since this is an informational document about an open process, the
security considerations are specific to the process and users
involved in the process. The primary concern would be to limit the
people that have access to the GenART data/files to ensure that the
integrity of the data is maintained. Also, once the data is moved to
the IETF servers, the normal IETF processes should ensure that only
Barnes, et al. Expires February 16, 2009 [Page 18]
Internet-Draft GenART August 2008
authorized individuals can access the data. For example, each GenART
reviewer should have a unique user name/password, just as folks do to
access any other IETF maintained data and/or tools, as appropriate.
11. IANA considerations
As this is an informational document about an IETF process, there are
no IANA considerations.
12. Acknowledgments
Initial comments were received from the members of the GenART team
and the experiences discussed in this document were derived from
their hard work over the last 4+ years as reviewers. We thank the
past reviewers of the GenART team: Mark Allman, Harald Alvestrand,
Mary Barnes, Ron Bonica, Sharon Chisholm, Lakshminath Dondeti, Avri
Doria, Pasi Eronen, Miguel-Angel Garcia, John Loughney, Lucy Lynch,
Michael Patton, Tom Taylor and Suzannne Woolf As well as the current
team of reviewers: David Black, Scott Brim, Gonzalo Camarillo, Ben
Campbell Brian Carpenter, Elwyn Davies, Spencer Dawkins, Francis
Dupont, Eric Gray, Vijay Gurbani, Joel Halpern, Suresh Krishnan,
Robert Sparks, and Christian Vogt
13. Changes since Last Version
1. Updated to reflect current practices such as assignment at Last
Call time being the norm, standard template for reviews/emails,
new members of team, comments from current secretary, General
Area AD and review team members.
2. Removed previous comments in the "Impressions" section that are
no longer germane such as problems with no standard boilerplate,
problems with LC reviews, etc.
3. Added step by step details on the secretary's process to
facilitate tools requirements.
4. Added the documents which provide the baseline for reviews to the
informal reference section.
14. Informative References
[RFC1958] Carpenter, B., "Architectural Principles of the Internet",
RFC 1958, June 1996.
[RFC3426] Floyd, S., "General Architectural and Policy
Considerations", RFC 3426, November 2002.
Barnes, et al. Expires February 16, 2009 [Page 19]
Internet-Draft GenART August 2008
[RFC3439] Bush, R. and D. Meyer, "Some Internet Architectural
Guidelines and Philosophy", RFC 3439, December 2002.
[I-D.rfc-editor-rfc2223bis]
Reynolds, J. and R. Braden, "Instructions to Request for
Comments (RFC) Authors", draft-rfc-editor-rfc2223bis-08
(work in progress), July 2004.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 5226,
May 2008.
[RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC
Text on Security Considerations", BCP 72, RFC 3552,
July 2003.
Authors' Addresses
Mary Barnes
Nortel
2201 Lakeside Blvd
Richardson, TX
Email: mary.barnes@nortel.com
Avri Doria
Lulea University of Technology
Arbetsvetenskap
Lulea
SE-97187
Email: avri@acm.org
Harald Alvestrand
Google
Beddingen 10
Trondheim 7014
NO
Email: harald@alvestrand.no
Barnes, et al. Expires February 16, 2009 [Page 20]
Internet-Draft GenART August 2008
Brian E Carpenter
University of Auckland
PB 92019
Auckland, 1142
New Zealand
Phone:
Email: brian.e.carpenter@gmail.com
Barnes, et al. Expires February 16, 2009 [Page 21]
Internet-Draft GenART August 2008
Full Copyright Statement
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Barnes, et al. Expires February 16, 2009 [Page 22]
