Tracking Reviews of Documents
draft-sparks-genarea-review-tracker-00
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft that was ultimately published as RFC 7735.
|
|
|---|---|---|---|
| Authors | Robert Sparks , Tero Kivinen | ||
| Last updated | 2015-01-27 | ||
| RFC stream | (None) | ||
| Formats | |||
| Reviews |
GENART Telechat review
(of
-03)
by Suresh Krishnan
Ready w/nits
GENART Last Call review
(of
-02)
by Suresh Krishnan
Ready w/nits
|
||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | Became RFC 7735 (Informational) | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-sparks-genarea-review-tracker-00
Network Working Group R. Sparks
Internet-Draft Oracle
Intended status: Informational T. Kivinen
Expires: July 30, 2015 INSIDE Secure
January 26, 2015
Tracking Reviews of Documents
draft-sparks-genarea-review-tracker-00
Abstract
Several review teams ensure specific types of review are performed on
Internet Drafts as they progress towards becoming RFCs. The tools
used by these teams to assign and track reviews would benefit from
tighter integration to the Datatracker. This document discusses
requirements for improving those tools without disrupting current
work flows.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 30, 2015.
Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
Sparks & Kivinen Expires July 30, 2015 [Page 1]
Internet-Draft Review Tracking January 2015
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Overview of current workflows . . . . . . . . . . . . . . . . 3
3. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1. Secretary focused . . . . . . . . . . . . . . . . . . . . 5
3.2. Reviewer focused . . . . . . . . . . . . . . . . . . . . 6
3.3. Review Requester and Consumer focused . . . . . . . . . . 8
3.4. Statistics focused . . . . . . . . . . . . . . . . . . . 8
4. Security Considerations . . . . . . . . . . . . . . . . . . . 9
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 9
7. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 9
7.1. 00 . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8. Informative References . . . . . . . . . . . . . . . . . . . 9
Appendix A. A starting point for Django models supporting the
review tool . . . . . . . . . . . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11
1. Introduction
As Internet Drafts are processed, reviews are requested from several
review teams. For example, the General Area Review Team (Gen-Art)
and the Security Directorate (Secdir) perform reviews of documents
that are in IETF Last Call. Gen-art performs a follow-up review when
the document is scheduled for an IESG telechat. These teams also
perform earlier reviews of documents on demand. There are several
other teams that perform similar services, often focusing on specific
areas of expertise.
The secretaries of these teams manage a pool of volunteer reviewers.
Documents are assigned to reviewers, taking various factors into
account. For instance, a reviewer will not be assigned a document
for which he is an author or shepherd. Reviewers are given a
deadline, usually driven by the end of last call or a telechat date.
The reviewer sends each completed review to the team's mailing list,
and any other lists that are relevant for document being reviewed.
Often, a thread ensues on one or more of those lists to resolve any
issues found in the review.
The secretaries and reviewers from several teams are using a tool
developed and maintained by Tero Kivinen. Much of its design
predates the modern Datatracker. The application currently keeps its
own data store, and learns about documents needing review by
inspecting Datatracker and tools.ietf.org pages. Most of those pages
Sparks & Kivinen Expires July 30, 2015 [Page 2]
Internet-Draft Review Tracking January 2015
are easy to parse, but the last-call pages, in particular, require
some effort. Tighter integration with the Datatracker would simplify
the logic used to identify documents ready for review, make it
simpler for the Datatracker to associate reviews with documents, and
allow users to reuse their Datatracker credentials. It would also
make it easier to detect other potential review-triggering events,
such as a document entering working group last call, or when a RFC's
standard level is being changed without revising the RFC. Tero
currently believes this integration is best achieved by a new
implementation of the tool. This document captures requirements for
that reimplementation, with a focus on the workflows that the new
implementation must take care not to disrupt. It also discusses new
features, including changes suggested for the existing tool at its
issue tracker [art-trac].
For more information about the various review teams, see the
following references
+---------+---------------------+
| Gen-Art | [Gen-Art] [RFC6385] |
| Secdir | [Secdir] |
+---------+---------------------+
2. Overview of current workflows
This section gives a high-level overview of how the review team
secretaries and reviewers use the existing tool. It is not intended
to be comprehensive documentation of how review teams operate.
Please see the references for those details.
A team's secretary periodically (typically once a week) checks the
tool for documents it has identified as ready for review. The tool
has compiled this list from Last Call announcements and telechat
agendas. The secretary creates a set of assignments from this list
into the reviewer pool, choosing the reviewers in roughly a round-
robin order. That order can be perturbed by several factors.
Reviewers have different levels of availability. Some are willing to
review multiple documents a month. Others may only be willing to
review a document every other month. The assignment process takes
exceptional conditions such as reviewer vacations into account.
Furthermore, secretaries are careful not to assign a document to a
reviewer that is an author, shepherd, responsible WG chair, or has
some other already existing association with the document. The
preference is to get a reviewer with a fresh perspective. The
secretary may discover reasons to change assignments while going
through the list of documents. In order to not cause a reviewer to
make a false start on a review, the secretaries complete the full
list of assignments before sending notifications to anyone. This
Sparks & Kivinen Expires July 30, 2015 [Page 3]
Internet-Draft Review Tracking January 2015
assignment process can take several minutes, and it is possible for
new last calls to be issued while the secretary is making
assignments. The secretary typically checks to see if new documents
are ready for review just before issuing the assignments, and updates
the assignments if necessary. The issued assignments are sent to the
review team list and are reflected in the tool. For those teams
handling different types of reviews (Last Call vs Telechat for
example), the secretary typically processes the documents for each
type of review separately, and potentially with different assignment
criteria. In Gen-art, for example, the Last Call reviewer for a
document will almost always get the follow-up Telechat review
assignment. Similarly, Secdir assigns any re-reviews of a document
to the same reviewer. Other teams may choose to assign a different
reviewer.
Reviewers discover their assignments through the announcement to the
list or by looking at their queue in the tool. (Most reviewers only
check the tool when they see they have an assignment via the list).
A reviewer has the opportunity to reject the assignment for any
reason. The secretary will find another volunteer for any rejected
assignments. The reviewer can indicate that the assignment is
accepted in the tool before starting the review.
The reviewer sends a completed review to the team's list, and any
other lists relevant to the review. For instance, many last call
reviews are also sent to the IETF general list. The teams typically
have a template format for the review. Those templates usually start
with a summary, describing the conclusion of the review. Typical
summaries are "Ready for publication" or "On the right track, but has
open issues". The reviewer uses the tool to indicate that the review
is complete, provides the summary, and has an opportunity to provide
a link to the review in the archives. (Note, however, that having to
wait for the document to appear in the archive to know the link to
paste into the tool is a significant enough impedance that this link
is often not provided by the reviewer. The Secdir secretary manually
collects these links from the list and adds them to the tool.)
Occasionally, a document is revised between when a review assignment
is made and when the reviewer starts the review. Different teams can
have different policies about whether the reviewer should review the
assigned version or the current version.
3. Requirements
Sparks & Kivinen Expires July 30, 2015 [Page 4]
Internet-Draft Review Tracking January 2015
3.1. Secretary focused
o A secretary must be able to see what documents are ready for
review of a given type (such as a Last Call review)
o A secretary must be able to assign reviews for documents that may
not have been automatically identified as ready for a review of a
given type. (This allows the secretary to work around errors and
handle special cases, including early review requests.)
o A secretary must be able to work on and issue a set of assignments
as an atomic unit. No assignment should be issued until the
secretariat declares the set of assignments complete.
o It must be easy for the secretary to discover that more documents
have become ready for review while working on an assignment set.
o The tool should make preparing the assignment email to the team's
list easy. For instance, the tool could prepare the message, give
the secretary an opportunity to edit it, and handle sending it to
the list.
o A secretary must be able to easily see who the next available
reviewers are, in order.
o A secretary must be able to edit a Reviewer's availability, both
in terms of frequency, not-available-until-date, and skip-next-
n-assignments. (See the description of these settings in the
Reviewer focused section.)
o The tool should make it easy for the secretary to identify that a
reviewer is already involved with a document. The current tool
allows the secretary to provide a regular expression to match
against the document name. If the expression matches, the
document is not available for assignment to this reviewer. For
example, Tero will not be assigned documents matching '^draft-
(kivinen|ietf-tcpinc)-.*$'. The tool should also take any roles,
such as document shepherd, that the Datatracker knows about into
consideration.
o The tool should make it easy for the secretary to see key features
of a document ready for assignment, such as its length, its
authors, the group and area it is associated with, its title and
abstract, and any other personnel (such as the shepherd and
reviewers already assigned from other teams) involved in the
draft.
Sparks & Kivinen Expires July 30, 2015 [Page 5]
Internet-Draft Review Tracking January 2015
o The tool must make it easy for the secretary to detect and process
re-review requests on the same version of a document (such as when
a document has an additional last call only to deal with new IPR
information).
o Common operations to groups of documents should be easy for the
secretary to process as a group with a minimum amount of
interaction with the tool. For instance, it should be possible to
process all the of documents described by the above bullet with
one action. Similarly, for teams that assign re-reviews to the
same reviewer, issuing all re-review requests should be a simple
action.
o The tool must make it easy for the secretary to see the result of
previous reviews from this team. In Secdir, for example, if the
request is for a revision that has only minor differences, and the
previous review result was "Ready", a new assignment will not be
made.
o The tools must make it easy for the secretary to see the recent
performance of a reviewer while making an assignment (see
Section 3.4). This allows the secretary to detect overburdened or
unresponsive volunteers earlier in the process.
o A secretary must be able to withdraw a review assignment.
3.2. Reviewer focused
o A reviewer must be able to indicate availability, both in
frequency of reviews, and as "not available until this date." The
current tool speaks of frequency in these terms:
Assign at maximum one new review per week
Assign at maximum one new review per fortnight
Assign at maximum one new review per month
Assign at maximum one new review per two months
Assign at maximum one new review per quarter
o Reviewers must be able to indicate that they should be skipped the
next n times they would normally have received an assignment.
Sparks & Kivinen Expires July 30, 2015 [Page 6]
Internet-Draft Review Tracking January 2015
o A reviewer must be able easily discover new review assignments.
(The tool might send email directly to an assigned reviewer in
addition to sending the set of assignments to the team's list.
The tool might also use the Django Message framework to let a
reviewer that's logged into the Datatracker know a new review
assignment has been made.
o Reviewers must be able to see their current set of outstanding
assignments, completed assignments, rejected assignments. The
presentation of those sets should either be separate, or, if
combined, the sets should be visually distinct.
o A reviewer must be able to reject a review assignment, optionally
providing the secretary with an explanation for the rejection.
o A reviewer must be able to indicate that have accepted and are
working on an assignment
o It should be possible for a reviewer to reject or accept a review
either by using the tool's web interface, or by replying to the
review assignment email.
o A reviewer must be able to indicate that a review is complete,
capturing where the review is in the archives, and the high-level
review-result summary
o It must be possible for a reviewer to clearly indicate which
version of a document was reviewed. Documents are sometimes
revised between when a review was assigned and when it is due.
The tool should note the current version of the document, and
highlight when the review is not for the current version.
o It must be easy for a reviewer to submit a completed review.
- The current workflow, where the reviewer sends email to the
team list (possibly copying other lists) and then indicates
where to find that review must continue to be supported. The
tool should make it easier to capture the link to review in the
list archives (perhaps by suggesting links based on a search
into the archives).
- The tool should allow the reviewer to enter the review into the
tool via a web form. The tool will ensure the review is posted
to the appropriate lists, and will construct the links to those
posts in the archives.
Sparks & Kivinen Expires July 30, 2015 [Page 7]
Internet-Draft Review Tracking January 2015
- The tool could also allow the reviewer to submit the review to
the tool by email (perhaps by replying to the assignment). The
tool would then ensure the review is posted to the appropriate
lists.
3.3. Review Requester and Consumer focused
o It should be easy for an AD or group chair to request any type of
review, but particularly an early review, from a review team.
o It should be possible for that person to withdraw a review
request.
o It must be easy to find all reviews of a document when looking at
the document's main page in the Datatracker. The reference to the
review must make it easy to see any responses to the review on the
lists it was sent to.
o It must be easy to find all reviews of a document when looking at
search result pages, and other lists of documents such as the
documents on a telechat agenda.
3.4. Statistics focused
o It must be easy to see the following, across all teams, a given
team, or a given reviewer, and independently across all time, or
across configurable recent periods of time:
- How many reviews have been completed
- How many reviews are in progress
- How many in progress reviews are late
- How many completed reviews were late
- How many reviews were not completed at all
- Average time to complete reviews (from assignment to
completion)
o It must be easy to see, for all teams, for a given team, or for a
given reviewer, across all time, or across configurable recent
periods:
Sparks & Kivinen Expires July 30, 2015 [Page 8]
Internet-Draft Review Tracking January 2015
- Total counts of reviews in each review state (done, rejected,
etc.)
- Total counts of completed reviews by result (ready, ready with
nits, etc.)
o Where possible, the above statistics should be visible as a time-
series graph.
o It should be possible view the correlation between completed
reviews results with number of discuss positions taken on a given
document. Future enhancements may allow ADs to indicate their
position was informed by a given review to help make this view
more meaningful.
4. Security Considerations
This document discusses requirements for tools that assist review
teams. These requirements do not affect the security of the Internet
in any significant fashion. The tools themselves have authentication
and authorization considerations (team secretaries will be able to do
different things than reviewers). None of these have been identified
as non-obvious.
5. IANA Considerations
This document has no actions for IANA.
6. Acknowledgments
Tero Kivinen and Henrik Levkowetz were instrumental in forming this
set of requirements and in developing the initial Django models in
the appendix.
7. Changelog
7.1. 00
o Initial Version
8. Informative References
[Gen-Art] "General Area Review Team Guidelines", Work in Progress ,
January 2015,
<http://trac.tools.ietf.org/area/gen/trac/wiki>.
Sparks & Kivinen Expires July 30, 2015 [Page 9]
Internet-Draft Review Tracking January 2015
[RFC6385] Barnes, M., Doria, A., Alvestrand, H., and B. Carpenter,
"General Area Review Team (Gen-ART) Experiences", RFC
6385, October 2011.
[Secdir] "Security Directorate", Work in Progress , January 2015,
<http://trac.tools.ietf.org/area/sec/trac/wiki/
SecurityDirectorate>.
[art-trac]
"Area Review Team Tool - Active Tickets", Work in Progress
, January 2015,
<https://wiki.tools.ietf.org/tools/art/trac/report/1>.
Appendix A. A starting point for Django models supporting the review
tool
from django.db import models
from ietf.doc.models import Document
from ietf.person.models import Email
from ietf.group.models import Group, Role
from ietf.name.models import NameModel
class ReviewRequestStateName(NameModel):
""" Requested, Accepted, Rejected, Withdrawn, Overcome By Events,
No Response , Completed """
class ReviewTypeName(NameModel):
""" Early Review, Last Call, Telechat """
class ReviewResultName(NameModel):
"""Almost ready, Has issues, Has nits, Not Ready,
On the right track, Ready, Ready with issues,
Ready with nits, Serious Issues"""
class Reviewer(models.Model):
"""
These records associate reviewers with review team, and keeps track
of admin data associated with the reviewer in the particular team.
There will be one record for each combination of reviewer and team.
"""
role = models.ForeignKey(Role)
frequency = models.IntegerField(help_text=
"Can review every N days")
available = models.DateTimeField(blank=True,null=True, help_text=
"When will this reviewer be available again")
filter_re = models.CharField(blank=True)
skip_next = models.IntegerField(help_text=
Sparks & Kivinen Expires July 30, 2015 [Page 10]
Internet-Draft Review Tracking January 2015
"Skip the next N review assignments")
class ReviewResultSet(models.Model):
"""
This table provides a way to point out a set of ReviewResultName
entries which are valid for a given team, in order to be able to
limit the result choices that can be set for a given review, as a
function of which team it is related to.
"""
team = models.ForeignKey(Group)
valid = models.ManyToManyField(ReviewResultName)
class ReviewRequest(models.Model):
"""
There should be one ReviewRequest entered for each combination of
document, rev, and reviewer.
"""
# Fields filled in on the initial record creation:
time = models.DateTimeField(auto_now_add=True)
type = models.ReviewTypeName()
doc = models.ForeignKey(Document,
related_name='review_request_set')
team = models.ForeignKey(Group)
deadline = models.DateTimeField()
requested_rev = models.CharField(verbose_name="requested_revision",
max_length=16, blank=True)
state = models.ForeignKey(ReviewRequestStateName)
# Fields filled in as reviewer is assigned, and as the review
# is uploaded
reviewer = models.ForeignKey(Reviewer, null=True, blank=True)
review = models.OneToOneField(Document, null=True,
blank=True)
reviewed_rev = models.CharField(verbose_name="reviewed_revision",
max_length=16, blank=True)
result = models.ForeignKey(ReviewResultName)
Authors' Addresses
Robert Sparks
Oracle
7460 Warren Parkway
Suite 300
Frisco, Texas 75034
USA
Email: rjsparks@nostrum.com
Sparks & Kivinen Expires July 30, 2015 [Page 11]
Internet-Draft Review Tracking January 2015
Tero Kivinen
INSIDE Secure
Eerikinkatu 28
HELSINKI FI-00180
FI
Email: kivinen@iki.fi
Sparks & Kivinen Expires July 30, 2015 [Page 12]