Network Working Group J. Klensin
Internet-Draft S. Dawkins
Expires: September 16, 2004 March 18, 2004
A model for IETF Process Experiments
draft-klensin-process-july14-01.txt
Status of this Memo
By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed,
and any of which I become aware will be disclosed, in accordance with
RFC 3667.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on September 16, 2004.
Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract
In the last two years, the IETF has initiated a number of
interrelated efforts to improve or fine-tune its standards process
and its internal procedures using the procedures intended for
development of protocol specifications. None of these efforts has
had an observable impact on the quality or timeliness of IETF
outputs, and, based on the proposed charter milestones now under
discussion, approval to try to improve things is still between six
and eighteen months away. This document proposes a radically
different approach to the system of making changes to IETF process,
one that relies heavily on a "propose and carry out an experiment,
evaluate the experiment, and then establish permanent procedures
Klensin & Dawkins Expires September 16, 2004 [Page 1]
Internet-Draft Process Experiments March 2004
based on operational experience" model rather than the ones that have
been attempted previously.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Not an Accident, Not a Coincidence . . . . . . . . . . . . . . 5
3. Evidence We Are Headed in the Wrong Direction . . . . . . . . 6
4. Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5. Implications for the Present Paralysis . . . . . . . . . . . . 11
6. Security Considerations . . . . . . . . . . . . . . . . . . . 12
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 13
A. References . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Intellectual Property and Copyright Statements . . . . . . . . 15
Klensin & Dawkins Expires September 16, 2004 [Page 2]
Internet-Draft Process Experiments March 2004
1. Introduction
Starting with IETF 54 in Yokohama in July 2002, the IETF has been
spending significant time and energy on analysis of actual and
perceived problems that keep us from producing high quality standards
on a timely and efficient basis. Ironically, these reform activities
have been impeded by some of the same difficulties that impact the
standards process itself, e.g.,
o Seemingly interminable discussion about charter details that
derails a focus on the work to be done and the problems to be
solved.
o Discussion that doesn't converge, because it relies on logic and
persuasion, instead of experiments and experience.
o Proposals for "improvements" based on intuition, not experience. A
significant percentage of the postings about how the IETF
standards process should work are coming from participants who
have never authored an RFC. A large percentage of the MPOWR
postings about what WG chairs should do is coming from people who
have never had the experience of being WG chairs. An overwhelming
percentage of the postings on several mailing lists about
offloading the IESG is coming from people who are not present or
former IESG members and have no other direct information about how
the IESG functions internally.
o There has been uncertainty about the level of IESG enthusiasm for
some of the proposals and doubts about whether they were worth the
investment to develop if the IESG was not enthusiastic.
In addition, these activities have shared two unfortunate
characteristics with previous major process initiatives in the IETF:
o There is an inevitable time conflict -- whether in following and
contributing to mailing lists, developing and evaluating documents
and proposals, or scheduling face to face meetings -- between
process efforts and actual IETF technical/ engineering work.
Especially when process efforts drag out for many months, there is
a tendency for people whose primary commitment is to the
development of technical specifications to go back to doing that,
leaving process efforts to those who are strongly interested in
process issues and, to some measure, less inclined or able to do
the technical work. This creates a situation in which the process
groups and efforts may be significantly unrepresentative of the
range of IETF participants who are materially concerned with their
outcomes. The meeting time conflict portion of the problem is the
reason why the then co-Chairs of POISSON tried to avoid face to
Klensin & Dawkins Expires September 16, 2004 [Page 3]
Internet-Draft Process Experiments March 2004
face meetings entirely: a situation in which IETF participants
needed to choose between attending standards-oriented WGs and a
process-oriented one would ultimately benefit no one.
o With protocol standards, the IETF understands, as a community, the
dangers of standardization in the absence of implementation,
deployment, and operational experience. As a result, we have
traditionally had Experimental RFCs, a standards track with
multiple maturity levels, and even implementations from Internet
Drafts that could then be evaluated as definition and
standardization progressed. Except, possibly, when the IESG has
initiated process changes on its own initiative without going
through a community-wide review first, we have had no parallel for
procedures. We try to develop them (usually in Working Groups),
instantiate them in BCPs, and then put them into effect. This
appears to be unwise. It is certainly inefficient if efficiency
is measured by the length of time between achievement of some
level of community consensus and putting a procedure into effect.
This document is specifically addressed to that set of problems with
proposed modifications to procedures, especially the last one.
In reading the balance of this document, it is worth noting that the
difference between a procedural document and a protocol standard is
that the former comes with a body (the IESG), that is responsible
both for its adoption and for its ongoing and dynamic interpretation.
For protocol specifications, the specification is required to be good
enough to enable interoperable independent implementations (without a
lot of shared oral tradition, especially beyond the first maturity
level). That will typically require getting more details nailed down,
and nailed down fairly precisely. That difference implies that we
can be a lot more relaxed about fine details of procedural documents,
especially experimental ones, at least as long as we can trust the
IESG to document their interpretations so that those can be
incorporated into a final document should the "experiment" succeed.
Klensin & Dawkins Expires September 16, 2004 [Page 4]
Internet-Draft Process Experiments March 2004
2. Not an Accident, Not a Coincidence
To be very clear - the problems encountered using procedures intended
for protocol specification development to fix procedural problems are
systemic, and have existed since the POISED working group in the
early 1990s. These problems will not go away.
We recognize the efforts that have been made to make this work by a
number of contributors, including current IESG members. At this time,
the longest-serving Area Director the IETF has had is shepherding
NEWTRK, and both ICAR and MPOWR are being shepherded by current Area
Directors. The quality of these contributors argues strongly that
it's not possible to use the existing working group process to
accomplish significant process improvements in the IETF and that, if
it is possible, it is probably not efficient.
Klensin & Dawkins Expires September 16, 2004 [Page 5]
Internet-Draft Process Experiments March 2004
3. Evidence We Are Headed in the Wrong Direction
[[Note in draft: This section, and some other parts of the document,
are very specific to observations about events in the year or two
preceding its writing. If the document evolves in the direction of
an RFC, the sections will, at least, need to be recast to put the
remarks in historical context.]]
The authors observe that, in the year prior to the creation of the
initial version of this draft,
o The Senior Internet Reviewer effort fizzled out, largely due to a
"chicken and egg problem". It became clear that it was not
possible to get an adequate number of volunteers and generate
high-quality reviews unless there was a strong commitment to using
the results, but it also became clear that there couldn't be a
strong commitment to integrating the SIR reviews into the
specification review process unless there were adequate volunteers
and it had been demonstrated that reviews of sufficient quality
were being produced.
o Proposals to create new process WGs [NEWTRK] and [ICAR] to try to
devise and define solutions to particular problems became bogged
down for extended periods in discussions of charter details,
discussions that did nothing to either address the problems or to
quickly generate alternatives for solving them. Five proposals to
modify the existing standards track were under active discussion
on [SOLUTIONS] before IETF 58, where they were presented at the
NEWTRK BoF [NEWTRK58], but since IETF 58 the NEWTRK focus for an
entire IETF meeting cycle has been on charter discussion, not on
proposing changes that might improve document quality or
timeliness. None of the existing proposals have been discussed on
the mailing list or updated, no new proposals have been discussed
on the mailing list or posted, and two-thirds of the NEWTRK
postings have discussed the charter and planning for the upcoming
IETF meeting.
o Other proposals for ways to address procedural issues, or to
schedule BOFs to discuss some of them, have deteriorated into
arguments (or whining) about boundaries, definitions, priorities,
etc.
o We have not been successful in modularizing these efforts - taking
SOLUTIONS, NEWTRK, and ICAR mailing lists as one example, only one
active participant has stayed exclusively on one list. Anyone who
cares about overall IETF process improvement is forced to
subscribe to an increasingly large set of semi-connected mailing
lists (SOLUTIONS, NEWTRK, ICAR, MPOWR, PROTO, EDU,...). We believe
Klensin & Dawkins Expires September 16, 2004 [Page 6]
Internet-Draft Process Experiments March 2004
there is already a "backlash" of people who have contributed
previously, but don't think continued participation is worth the
time and effort it takes to keep track of all the semi-independent
proposals.
o Finally, and most damaging - the draft charters for all of these
efforts feature schedules in which the first milestone that
produces a possible improvement is six to eighteen months away.
If the problems involved are important enough to need solving --
keeping in mind that we have already been at the problem
definition and evaluation process for eighteen months -- another
six to eighteen months before we can start to do anything is far
too long - and this assumes no slippages from the milestones as
proposed. If they are not that important, then we should stop
wasting time on them.
In some cases, these lengthy discussions about how to organize work
(rather than about the work itself) may have been completely
appropriate. But they are not an efficient way to improve document
quality or timeliness. It cannot be overstated that no one,
regardless of depth or breadth of experience, really "knows" the
effect that these proposals will have on the IETF's ability to
produce quality specifications in a timely manner. After all of the
discussion is complete, we still will not know with certainty whether
a proposal is a good idea, and we will still need experience to know
whether we have improved anything at all.
Klensin & Dawkins Expires September 16, 2004 [Page 7]
Internet-Draft Process Experiments March 2004
4. Proposal
Since the 1992 changes, the IESG has adopted a number of procedural
changes on its own initiative and documented them informally,
utilizing the wide discretion implicitly granted them by RFC 2026.
This document proposes to regularize and formalize that mechanism as
a means of moving forward with procedural changes that might prove
valuable. We note that, if the procedures the IESG has adopted (and
procedural exceptions it has made) over the last decade are
legitimate, then the IESG has the authority to institute the changes
proposed here by bootstrapping the proposed process.
The mechanisms outlined here are not intended to be exclusive: they
add to the IESG's range of tools for dealing with process issues on
an ongoing basis, rather that replacing those tools with a single
"magic bullet". The choice as to whether to use the procedure
outlined in this document, if it is adopted, or other mechanisms
available to the IESG and the community --present or future-- remains
in the IESG's hands. If the IESG does not exercise that discretion
wisely, this document provides no additional remedies.
Some have read the current procedures as giving the IESG all of the
capabilities outlined in this document, i.e., that it changes almost
nothing. If this is true, this document only encourages the IESG to
use this type of mechanism more frequently rather than less
streamlined ones, and to more explicitly document what it is doing
and what decisions it is making.
We propose to permit (and encourage) the IESG to adopt and institute
"process experiments" using the following procedure:
1. An I-D is written that describes what the proposed new or altered
procedure is about and how it works. A statement of what problem
it is expected to solve would be desirable, but is not a
requirement (the intent is to keep the firm requirements for such
an experiment as lightweight as possible). Similarly, specific
experimental or evaluation criteria are very desirable, but not
required -- for some of the process changes we anticipate, having
the IESG reach a conclusion at the end of the sunset period that
the community generally believes things to be better (or worse)
will be both adequate and sufficient. The I-D must state an
explicit "sunset" timeout, typically not to exceed one year after
adoption.
2. If the IESG believes the proposal is plausible and plausibly
useful, a four week IETF Last Call is initiated. The IESG can
institute whatever procedures it wishes to make that
determination and to avoid denial of service attacks from large
Klensin & Dawkins Expires September 16, 2004 [Page 8]
Internet-Draft Process Experiments March 2004
numbers of spurious or unimportant proposals. In particular,
they might institute a procedure requiring some number of
endorsements, or endorsements of a particular type, before the
IESG considers the draft. The IESG is, however, expected to
understand that procedures or review processes that act as a
mechanism for significant delays do not fall within the intent of
this proposal.
3. At the conclusion of the Last Call, the IESG reevaluates the
plausibility and appropriateness of the proposal. If they
conclude that the proposed experiment is appropriate, a second
I-D is generated (either by the IESG or by the original authors
with IESG advice) that cleans up any definitional issues exposed
in the Last Call and that explicitly identifies and responds to
issues raised during that Last Call.
4. The document and experiment are then announced, the experiment is
begun, and the document is forwarded for RFC publication.
The IESG could, of course, reach a "bad idea" conclusion at any stage
in this process and abandon the experiment. It might recommend
publication of the experimental document, with a discussion of why it
was a bad idea, but is not required to do so. The list above is
deliberately agnostic about where the I-Ds come from: a WG, design
team, individual contribution, editing group, or other mechanism,
could be used in the first and/or third steps, but no specific
mechanisms are required and the IESG is explicitly permitted to
generate such proposals internally.
In each case, the IESG's making of the decisions to go forward (or
not) with a procedural experiment, or the steps leading up to one, is
expected to reflect their judgment of the existence of rough
consensus in the community. That judgment may be appealed using the
usual procedures, but the IESG and the community are reminded that an
experimental attempt to try something for a fixed period is typically
a better engineering approach than extended philosophical discussion
without any experience to back it up.
Nothing above is to be construed as a requirement that any given
process experiment be attempted IETF-wide. A proposal for such an
experiment may specify selected areas, selected working groups,
working groups meeting some specific criteria (such as those created
after a particular time or more than a specified number of years
old), or be specific in other ways.
At or before the end of the "sunset" timeout, the IESG would either
revise (or cause to be revised) the document into a BCP RFC or the
procedure would expire and, presumably, not be tried again unless
Klensin & Dawkins Expires September 16, 2004 [Page 9]
Internet-Draft Process Experiments March 2004
something changed radically. A document describing why the
experiment had succeeded or failed would be desirable but could not,
realistically, be a requirement. If the procedure went to BCP, the
BCP would reflect what we would call "operational experience" in the
real world.
Klensin & Dawkins Expires September 16, 2004 [Page 10]
Internet-Draft Process Experiments March 2004
5. Implications for the Present Paralysis
On the basis of this model, if the IESG believes that ICAR is
probably a useful idea and that the community is at least tentatively
in favor of trying it, then the first steps are not community review
and IESG approval of a charter for a WG to study the issue. Instead,
the first step is a proposal to do something and IESG consensus that
the "something" is worth trying. A year later, if it seems to have
helped and not caused harm, it gets cast in concrete. If not, we
discard it and move on. Similar comments apply to other proposed
WGs (or BOFs) concerned with process, such as MPOWR: if the IESG is
convinced that something is needed, and that there is community
backing for that "something", then it should be documented and
deployed experimentally, with formal procedural changes, if any,
occurring only after experience had been accumulated, observed, and
evaluated.
This model can, and in the opinion of the authors, probably should,
be applied to changes in the standards process as well. Specifically,
we tune the standards process in place, experimentally, largely by
changes in what IESG _does_, rather than the language used to
describe things. If those actions produce favorable results or
directions, we _then_ go off and write up the details, rather than
trying to make changes based on speculation about what might or might
not work better (much less spending months on debates about the
charter for considering those changes).
Klensin & Dawkins Expires September 16, 2004 [Page 11]
Internet-Draft Process Experiments March 2004
6. Security Considerations
This document specifies a mechanism for evolving IETF procedures. It
does not raise or consider any protocol-specific security issues. In
considering experimental changes to procedures, the IESG should, of
course, exercise due caution that such changes not reduce the quality
of security review and consideration for protocols or, at least, that
the process experiment proposals contain early detection and
correction mechanisms should quality deterioration occur.
Klensin & Dawkins Expires September 16, 2004 [Page 12]
Internet-Draft Process Experiments March 2004
7. Acknowledgements
The first revision of this document benefited significantly from
suggestions and comments from Avri Doria, Margaret Wasserman, and
Harald Alvestrand, and from discussions with the General Area
Directorate and at its open meeting during IETF 59.
Authors' Addresses
John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA 02140
USA
Phone: +1 617 491 5735
EMail: john-ietf@jck.com
Spencer Dawkins
1547 Rivercrest Blvd.
Allen, TX 75002
USA
Phone: +1 469 330 3616
EMail: spencer@mcsr-labs.org
Klensin & Dawkins Expires September 16, 2004 [Page 13]
Internet-Draft Process Experiments March 2004
Appendix A. References
[EDU]: Edu-Discuss Mailing List, https://www1.ietf.org/mail-archive/
working-groups/edu-discuss/current/threads.html
[ICAR]: Improved Cross-Area Review, https://www1.ietf.org/mailman/
listinfo/icar
[MPOWR]: Management Positions -- Oversight, Work and Results, https:/
/www1.ietf.org/mailman/listinfo/mpowr
[NEWTRK]: New IETF Standards Track mailing list, http://
darkwing.uoregon.edu/~llynch/newtrk/threads.html
[NEWTRK58]: New IETF Standards Track BoF at IETF 58, https://
www1.ietf.org/proceedings/03nov/130.htm
[SOLUTION]: Solutions Mailing List, http://eikenes.alvestrand.no/
mailman/listinfo/solutions
Klensin & Dawkins Expires September 16, 2004 [Page 14]
Internet-Draft Process Experiments March 2004
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the IETF's procedures with respect to rights in IETF Documents can
be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2004). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Klensin & Dawkins Expires September 16, 2004 [Page 15]
