Network Working Group J. Klensin
Internet-Draft S. Dawkins
Expires: August 8, 2004 February 08, 2004
A model for IETF Process Experiments
draft-klensin-process-july14-00.txt
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 7, 2004.
Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract
In the last two years, the IETF has initiated a number of
interrelated efforts to improve or fine-tune its standards process
and its internal procedures using the procedures intended for
development of protocol specifications. None of these efforts has
had an observable impact on the quality or timeliness of IETF
outputs, and, based on the proposed charter milestones now under
discussion, approval to try to improve things is still between six
and eighteen months away. This document proposes a radically
different approach to the system of making changes to IETF process,
one that relies heavily on a "propose and carry out an experiment,
evaluate the experiment, and then establish permanent procedures
based on operational experience" model rather than the ones that have
been attempted previously.
Klensin & Dawkins Expires August 8, 2004 [Page 1]
Internet-Draft Process Experiments February 2004
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Not an Accident, Not a Coincidence . . . . . . . . . . . . . . 5
3. Evidence We Are Headed in the Wrong Direction . . . . . . . . 6
4. Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5. Implications for the Present Paralysis . . . . . . . . . . . . 10
6. Security Considerations . . . . . . . . . . . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 11
A. References . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Intellectual Property and Copyright Statements . . . . . . . . 13
Klensin & Dawkins Expires August 8, 2004 [Page 2]
Internet-Draft Process Experiments February 2004
1. Introduction
Starting with IETF 54 in Yokohama in July 2002, the IETF has been
spending significant time and energy on analysis of actual and
perceived problems that keep us from producing high quality standards
on a timely and efficient basis. Ironically, these reform activities
have been impeded by some of the same difficulties that impact the
standards process itself, e.g.,
o Seemingly interminable discussion about charter details that
derails a focus on the work to be done and the problems to be
solved.
o Discussion that doesn't converge, because it relies on logic and
persuasion, instead of experiments and experience.
o Proposals for "improvements" based on intuition, not experience. A
significant percentage of the postings about how the IETF
standards process should work are coming from participants who
have never authored an RFC. A large percentage of the MPOWR
postings about what WG chairs should do is coming from people who
have never had the experience of being WG chairs. An overwhelming
percentage of the postings on several mailing lists about
offloading the IESG is coming from people who are not present or
former IESG members and have no other direct information about how
the IESG functions internally.
o There has been uncertainty about the level of IESG enthusiasm for
some of the proposals and doubts about whether they were worth the
investment to develop if the IESG was not enthusiastic.
In addition, these activities have shared two unfortunate
characteristics with previous major process initiatives in the IETF:
o There is an inevitable time conflict -- whether in following and
contributing to mailing lists, developing and evaluating documents
and proposals, or scheduling face to face meetings -- between
process efforts and actual IETF technical/ engineering work.
Especially when process efforts drag out for many months, there is
a tendency for people whose primary commitment is to the
development of technical specifications to go back to doing that,
leaving process efforts to those who are strongly interested in
process issues and, to some measure, less inclined or able to do
the technical work. This creates a situation in which the process
groups and efforts may be significantly unrepresentative of the
range of IETF participants who are materially concerned with their
outcomes. The meeting time conflict portion of the problem is the
reason why the then co-Chairs of POISSON tried to avoid face to
Klensin & Dawkins Expires August 8, 2004 [Page 3]
Internet-Draft Process Experiments February 2004
face meetings entirely: a situation in which IETF participants
needed to choose between attending standards-oriented WGs and a
process-oriented one would ultimately benefit no one.
o With protocol standards, the IETF understands, as a community, the
dangers of standardization in the absence of implementation,
deployment, and operational experience. As a result, we have
traditionally had Experimental RFCs, a standards track with
multiple maturity levels, and even implementations from Internet
Drafts that could then be evaluated as definition and
standardization progressed. Except, possibly, when the IESG has
initiated process changes on its own initiative without going
through a community-wide review first, we have had no parallel for
procedures. We try to develop them (usually in Working Groups),
instantiate them in BCPs, and then put them into effect. This
appears to be unwise. It is certainly inefficient if efficiency
is measured by the length of time between achievement of some
level of community consensus and putting a procedure into effect.
This document is specifically addressed to that set of problems with
proposed modifications to procedures, especially the last one.
Klensin & Dawkins Expires August 8, 2004 [Page 4]
Internet-Draft Process Experiments February 2004
2. Not an Accident, Not a Coincidence
To be very clear - the problems encountered using procedures intended
for protocol specification development to fix procedural problems are
systemic, and have existed since the POISED working group in the
early 1990s. These problems will not go away.
We recognize the efforts that have been made to make this work by a
number of contributors, including current IESG members. At this time,
the longest-serving Area Director the IETF has had is shepherding
NEWTRK, and both ICAR and MPOWR are being shepherded by current Area
Directors. The quality of these contributors argues strongly that
it's just not possible to use the existing working group process to
accomplish significant process improvements in the IETF.
Klensin & Dawkins Expires August 8, 2004 [Page 5]
Internet-Draft Process Experiments February 2004
3. Evidence We Are Headed in the Wrong Direction
[[Note in draft: This section, and some other parts of the document,
are very specific to observations about events in the year or two
preceding its writing. If the document evolves in the direction of
an RFC, the sections will, at least, need to be recast to put the
remarks in historical context.]]
The authors observe that, in the year prior to the creation of this
draft,
o The Senior Internet Reviewer effort fizzled out, largely due to a
"chicken and egg problem". It became clear that it was not
possible to get an adequate number of volunteers and generate
high-quality reviews unless there was a strong commitment to using
the results, but it also became clear that there couldn't be a
strong commitment to integrating the SIR reviews into the
specification review process unless there were adequate volunteers
and it had been demonstrated that reviews of sufficient quality
were being produced.
o Proposals to create new process WGs [NEWTRK] and [ICAR] to try to
devise and define solutions to particular problems became bogged
down for extended periods in discussions of charter details,
discussions that did nothing to either address the problems or to
quickly generate alternatives for solving them. Five proposals to
modify the existing standards track were under active discussion
on [SOLUTIONS] before IETF 58, where they were presented at the
NEWTRK BoF [NEWTRK58], but since IETF 58 the NEWTRK focus for an
entire IETF meeting cycle has been on charter discussion, not on
proposing changes that might improve document quality or
timeliness. None of the existing proposals have been discussed on
the mailing list or updated, no new proposals have been discussed
on the mailing list or posted, and two-thirds of the NEWTRK
postings have discussed the charter and planning for the upcoming
IETF meeting.
o Other proposals for ways to address procedural issues, or to
schedule BOFs to discuss some of them, have deteriorated into
arguments (or whining) about boundaries, definitions, priorities,
etc.
o We have not been successful in modularizing these efforts - taking
SOLUTIONS, NEWTRK, and ICAR mailing lists as one example, only one
active participant has stayed exclusively on one list. Anyone who
cares about overall IETF process improvement is forced to
subscribe to an increasingly large set of semi-connected mailing
lists (SOLUTIONS, NEWTRK, ICAR, MPOWR, PROTO, EDU,...). We believe
Klensin & Dawkins Expires August 8, 2004 [Page 6]
Internet-Draft Process Experiments February 2004
there is already a "backlash" of people who have contributed
previously, but don't think continued participation is worth the
time and effort it takes to keep track of all the semi-independent
proposals.
o Finally, and most damaging - the draft charters for all of these
efforts feature schedules in which the first milestone that
produces a possible improvement is six to eighteen months away.
If the problems involved are important enough to need solving --
keeping in mind that we have already been at the problem
definition and evaluation process for eighteen months -- another
six to eighteen months before we can start to do anything is far
too long - and this assumes no slippages from the milestones as
proposed. If they are not that important, then we should stop
wasting time on them.
In some cases, these lengthy discussions about how to organize work
(rather than about the work itself) may have been completely
appropriate. But they are not an efficient way to improve document
quality or timeliness. It cannot be overstated that no one,
regardless of depth or breadth of experience, really "knows" the
effect that these proposals will have on the IETF's ability to
produce quality specifications in a timely manner. After all of the
discussion is complete, we still will not know with certainty whether
a proposal is a good idea, and we will still need experience to know
whether we have improved anything at all.
Klensin & Dawkins Expires August 8, 2004 [Page 7]
Internet-Draft Process Experiments February 2004
4. Proposal
Since the 1992 changes, the IESG has adopted a number of procedural
changes on its own initiative and documented them informally,
utilizing their wide discretion implicit in RFC 2026. This document
proposes to regularize and formalize that mechanism as a means of
moving forward with procedural changes that might prove valuable. We
note that, if the procedures the IESG has adopted (and procedural
exceptions it has made) over the last decade are legitimate, then the
IESG has the authority to institute the changes proposed here by
bootstrapping the proposed process.
We propose to permit (and encourage) the IESG to adopt and institute
"process experiments" using the following procedure:
1. An I-D is written that describes what the proposed new or altered
procedure is about and how it works. A statement of what problem
it is expected to solve would be desirable, but is not a
requirement (the intent is to keep the firm requirements for such
an experiment as lightweight as possible). The I-D must state an
explicit "sunset" timeout, typically not to exceed one year after
adoption.
2. If the IESG believes the proposal is plausible and plausibly
useful, a four week IETF Last Call is initiated.
3. At the conclusion of the Last Call, the IESG reevaluates the
plausibility and appropriateness of the proposal. If they
conclude that the proposed experiment is appropriate, a second
I-D is generated (either by the IESG or by the original authors
with IESG advice) that cleans up any definitional issues exposed
in the Last Call and that explicitly identifies and responds to
issues raised during that Last Call.
4. The document and experiment are then announced, the experiment is
begun, and the document is forwarded for RFC publication.
The IESG could, of course, reach a "bad idea" conclusion at any stage
in this process and abandon the experiment. It might recommend
publication of the experimental document, with a discussion of why it
was a bad idea, but is not required to do so. The list above is
deliberately agnostic about where the I-Ds come from: a WG, design
team, individual contribution, editing group, or other mechanism,
could be used in the first and/or third steps, but no specific
mechanisms are required and the IESG is explicitly permitted to
generate such proposals internally.
In each case, the IESG's making of the decisions to go forward (or
Klensin & Dawkins Expires August 8, 2004 [Page 8]
Internet-Draft Process Experiments February 2004
not) with a procedural experiment, or the steps leading up to one, is
expected to reflect their judgment of the existence of rough
consensus in the community. That judgment may be appealed using the
usual procedures, but the IESG and the community are reminded that an
experimental attempt to try something for a fixed period is typically
a better engineering approach than extended philosophical discussion
without any experience to back it up.
Nothing above is to be construed as a requirement that any given
process experiment be attempted IETF-wide. A proposal for such an
experiment may specify selected areas, selected working groups,
working groups meeting some specific criteria (such as those created
after a particular time or more than a specified number of years
old), or be specific in other ways.
At or before the end of the "sunset" timeout, the IESG would either
revise (or cause to be revised) the document into a BCP RFC or the
procedure would expire and, presumably, not be tried again unless
something changed radically. A document describing why the
experiment had succeeded or failed would be desirable but could not,
realistically, be a requirement. If the procedure went to BCP, the
BCP would reflect what we would call "operational experience" in the
real world.
Klensin & Dawkins Expires August 8, 2004 [Page 9]
Internet-Draft Process Experiments February 2004
5. Implications for the Present Paralysis
On the basis of this model, if the IESG believes that ICAR is
probably a useful idea and that the community is at least tentatively
in favor of trying it, then the first steps are not community review
and IESG approval of a charter for a WG to study the issue. Instead,
the first step is a proposal to do something and IESG consensus that
the "something" is worth trying. A year later, if it seems to have
helped and not caused harm, it gets cast in concrete. If not, we
discard it and move on. Similar comments apply to other proposed
WGs (or BOFs) concerned with process, such as MPOWR: if the IESG is
convinced that something is needed, and that there is community
backing for that "something", then it should be documented and
deployed experimentally, with formal procedural changes, if any,
occurring only after experience had been accumulated, observed, and
evaluated.
This model can, and in the opinion of the authors, probably should,
be applied to changes in the standards process as well. Specifically,
we tune the standards process in place, experimentally, largely by
changes in what IESG _does_, rather than the language used to
describe things. If those actions produce favorable results or
directions, we _then_ go off and write up the details, rather than
trying to make changes based on speculation about what might or might
not work better (much less spending months on debates about the
charter for considering those changes).
Klensin & Dawkins Expires August 8, 2004 [Page 10]
Internet-Draft Process Experiments February 2004
6. Security Considerations
This document specifies a mechanism for evolving IETF procedures. It
does not raise or consider any protocol-specific security issues. In
considering experimental changes to procedures, the IESG should, of
course, exercise due caution that such changes not reduce the quality
of security review and consideration for protocols or, at least, that
the process experiment proposals contain early detection and
correction mechanisms should quality deterioration occur.
Authors' Addresses
John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA 02140
USA
Phone: +1 617 491 5735
EMail: john-ietf@jck.com
Spencer Dawkins
1547 Rivercrest Blvd.
Allen, TX 75002
USA
Phone: +1 469 330 3616
EMail: spencer@mcsr-labs.org
Klensin & Dawkins Expires August 8, 2004 [Page 11]
Internet-Draft Process Experiments February 2004
Appendix A. References
[EDU]: Edu-Discuss Mailing List, https://www1.ietf.org/mail-archive/
working-groups/edu-discuss/current/threads.html
[ICAR]: Improved Cross-Area Review, https://www1.ietf.org/mailman/
listinfo/icar
[MPOWR]: Management Positions -- Oversight, Work and Results, https:/
/www1.ietf.org/mailman/listinfo/mpowr
[NEWTRK]: New IETF Standards Track mailing list, http://
darkwing.uoregon.edu/~llynch/newtrk/threads.html
[NEWTRK58]: New IETF Standards Track BoF at IETF 58, https://
www1.ietf.org/proceedings/03nov/130.htm
[SOLUTION]: Solutions Mailing List, http://eikenes.alvestrand.no/
mailman/listinfo/solutions
Klensin & Dawkins Expires August 8, 2004 [Page 12]
Internet-Draft Process Experiments February 2004
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on the
IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in BCP-11. Copies of
claims of rights made available for publication and any assurances of
licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such
proprietary rights by implementors or users of this specification can
be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.
Full Copyright Statement
Copyright (C) The Internet Society (2004). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assignees.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
Klensin & Dawkins Expires August 8, 2004 [Page 13]
Internet-Draft Process Experiments February 2004
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Klensin & Dawkins Expires August 8, 2004 [Page 14]
