[Search] [txt|pdfized|bibtex] [Tracker] [Email] [Nits]
Versions: 00                                                            
Network Working Group                                      John Loughney
INTERNET-DRAFT                                                     Nokia
Category: Informational                                    Bernard Aboba
<draft-loughney-coach-00.txt>                                  Microsoft
23 June 2003



              A Comprehensive Approach to Quality (COACH)

This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC 2026.

Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups.  Note that other groups
may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time.  It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.

Copyright Notice

Copyright (C) The Internet Society (2003).  All Rights Reserved.

Abstract

This document describes an approach to improving the quality of IETF
documents which is based on quality plans developed by Working Groups,
and approved by the Area Director.  This document describes the
components of a quality plan, as well as a list of materials which need
to be compiled in order to assist Working Groups in developing their
plans.

1.  Introduction

This document describes an approach to improving the quality of IETF
documents that centers around the development and execution of Working
Group quality plans.  Since in the IETF Working Groups are responsible
for completing work,  they are also accountable for the quality of that
work.  If the quality of IETF documents is to improve, the Working



Loughney, et al.              Informational                     [Page 1]


INTERNET-DRAFT                    COACH                     23 June 2003


Groups must plan for, and carry out, that quality improvement.

This document describes how Working Group quality plans can be used to
assist in that effort - and what materials the IETF needs to put
together in order to assist Working Groups in developing their plans.

The Working Group quality plan is a public document no more than 5 pages
long that is developed in concert with the Working Group charter and is
made available on the IETF web site.  The quality plan lays out the
challenges that the Working Group faces, and describes how it plans to
overcome those challenges in order to deliver documents of high quality.
The plan includes pointers to processes, tools, and templates that will
assist in the effort, as well as the plan for assessing the quality of
work at several checkpoints.

Working Group quality plans are made publicly available so as to
encourage sharing of best practices.  Since each Working Group is
different, there is no "one size fits all" quality plan, although it is
expected that over time, as Working Groups share their plans with each
other, that best practices will be more widely adopted.

Typically, the quality plan will be revised when the Working Group
Charter or schedule changes significantly.  For example, if the Working
Group has fallen badly behind in its schedule, then a revised Challenge
Assessment may be required in order to determine where the Working Group
resources have fallen short of those required to complete the work.

2.  The quality plan

The Working Group quality plan includes (but is not limited to) the
following:

[a]  Working Group Charter.  The Working Group Charter is a summary of
     what the Working Group plans to do, who will do it, and by when.
     It includes a description of the documents that the Working Group
     plans to produce, the individuals who have signed up to do the
     work, and the schedule for when the work will be completed.  Since
     the Charter is a summary of the goals and objectives, as well as a
     description of the team, it is the foundation for the Working Group
     quality plan.

[b]  Challenge Assessment. This part of the quality plan describes the
     challenges that the Working Group faces in accomplishing the work
     outlined in the charter.  For example, this part of the plan
     describes the technical issues that need to be solved in order to
     complete the work, the estimated size of the documents that are to
     be produced, the documents on which the work depends, and the areas
     of knowledge, skill or experience in which the working group is



Loughney, et al.              Informational                     [Page 2]


INTERNET-DRAFT                    COACH                     23 June 2003


     weak.  In general, it is expected that the Challenge Assessment
     will incorporate feedback from the Area Director and the IETF
     community, so as to ensure that the Working Group has a realistic
     view of the difficulty of the task.

[c]  The Tools Plan.  This part of the plan describes the tools that the
     Working Group intends to use in order to manage the work.  This
     includes the Working Group mailing list and web site; issue
     tracking and reporting tools; document revision control systems;
     and document production and build environments.

[d]  The Review plan.  This part of the quality plan describes the
     checkpoints at which the work will be reviewed, the review process
     to be used, and how the reviewers will be recruited.

The elements of the Working Group quality plan are discussed in more
detail in the sections that follow.

2.1.  Challenge Assessment

The Working Group Charter is the statement of what the Working Group is
attempting to achieve, as well as the resources at its disposal.  The
Charter is essentially a promise to deliver (the goals) and a down
payment on that promise (the resources).

The question is whether the promise is realistic or not, and whether the
down payment is sufficient collateral against default.  The purpose of
the Challenge Assessment is to provide a realistic assessment of what is
going to be necessary to deliver on the promises that the Working Group
has made - with high quality.

There are many reasons why Working Groups deliver poor quality
documents.  However, in most cases where poor quality documents are
produced, it is unlikely that the plan was to deliver poor quality
documents all along.  Rather, it is more likely that the level of
effort, skill and experience required to produce high quality documents
was beyond the capability of the Working Group participants, and that
poor quality was the result, not of bad intentions, but of insufficient
resources.

At the start of a project, it is easy to underestimate what will be
required to finish the job.  This seems particularly true in the IETF,
where industries go from birth to maturity, and where individuals change
jobs, change their marital status, buy or build a house, go through a
mid-life crisis (or two), have children, or go trekking in Nepal, all in
the time it takes to bring an initial Internet-Draft to the point where
it is approved for publication as an RFC.




Loughney, et al.              Informational                     [Page 3]


INTERNET-DRAFT                    COACH                     23 June 2003


Often the group that is least able to estimate the resources required to
finish the proposed work are the core Working Group participants - if
only because a certain degree of naive optimism is required to start the
process in the first place.

The purpose of the Challenge Assessment is not to destroy that naive
optimism, but rather to augment it with an external assessment of
whether the resources identified in the Charter fall short of those
required to complete the work successfully.

An important element of the Challenge Assessment is a review of the
extent to which the charter goes beyond the current state of the art.
For example, are the security problems likely to be encountered ones
which can be solved by the application of existing security
technologies?  Or are there unique aspects of the security problem for
which existing techniques are inapplicable?  Does the Charter require
advances in research as well as engineering?

To a large degree, the Challenge Assessment focuses on risks outside the
core area of expertise of the Working Group.  In general, IETF Working
Groups tend to do a reasonable job within their area, but weaknesses,
where they arise, are often the result of missing expertise in other
areas.  For example, a Working Group in the Security Area will tend
handle security issues well, but may stumble over transport issues;  an
Internet Area Working Group will tend to handle things correctly at the
IP layer, but may mishandle security;  a Routing Area Working Group will
tend to properly handle routing, but may have difficulty with Operations
and Management.

The goal of the Challenge Assessment is to garner feedback from the IETF
community as a whole, in order to anticipate challenges that will need
to be overcome in order to complete the proposed work successfully.
Ideally, the Challenge Assessment will answer the question: "what does
the Working Group need to do in order to have a high propability of
completing the proposed work on schedule and with high quality?"

As a result of the Challenge Assessment, the Working Group charter and
quality plan may be modified so as to include additional resources in
certain areas.  For example, if security issues are anticipated, then
security reviews may be added to the quality plan, and a security
advisor may be named as part of the Working Group Charter.  If
production of large and complex documents is anticipated, then the Tools
plan will include use of sophisticated issue tracking and document
revision control systems, and the Working Group charter will include
additional editors with experience producing documents of that
complexity.





Loughney, et al.              Informational                     [Page 4]


INTERNET-DRAFT                    COACH                     23 June 2003


In addition to assessing the challenges that need to be overcome, the
Challenge Assessment will also identify key assumptions that if wrong,
would require the quality plan to be revised.  For example,  a key
assumption might be that transmission layer security (IPsec, TLS) will
be sufficient to address the security issues.  If that assumption were
to be invalidated, then the significantly greater security resources
would be required - and the plan will need to be revised to take this
into account.

2.2.  Tools Plan

The purpose of the Tools Plan is to identify the tools with which the
work will be carried out.

This includes:

[a]  The technology and host for the Working Group mailing list.  While
     every Working Group has a mailing list, mailing list maintenance
     has become increasingly complex as issues of spam control and
     subscription spoofing have arisen.  As a result, these and other
     issues need to be taken into account in selecting how the mailing
     list is run.

[b]  The Working Group web site. Many Working Groups now use their own
     web sites in order to track issues and provide access to work in
     progress in formats not available on the IETF archive, such as XML
     or HTML.  In some cases, these web sites become integral parts of
     the document production environment.  The quality plan should
     include a section on the plan for the web site.  Ideally, the web
     site will be hosted in a location that would not be disrupted by a
     change in a participant's home address (home DSL or Cable links
     have proven to be particularly vulnerable in this regard).

[c]  The document production environment.  Over the years, editors often
     develop their own document production environments.  These can vary
     from the quick and dirty, suitable for the production of small
     documents, to comprehensive environments that handle the building
     of complex documents in multiple formats (e.g. text, HTML, XML,
     etc.)  as well as generation of difference files and propagation of
     documents to the Working Group web site.  Since each document is
     different, there is no single right answer to the problem of
     document production - but it does seem likely that Working Groups
     can benefit by sharing sample build environments and templates and
     documenting best practices.

[d]  The revision control system.  Not every Working Group will need to
     maintain a revision control system, but where this is useful it is
     probably a good idea to describe how revision control will be



Loughney, et al.              Informational                     [Page 5]


INTERNET-DRAFT                    COACH                     23 June 2003


     handled.

[e]  The issue tracking and reporting system.  The use of issue tracking
     tools has had a substantial impact on Working Group productivity
     and document quality in a number of Working Groups.  By focusing
     effort on the identification and resolution of issues, tracking
     tools encourage discussions that lead to document improvements.  In
     addition, tracking tools make it possible to avoid rehashing old
     arguments and enable the quantitative measurement of progress
     towards a goal.  Given the increasing popularity of document
     tracking tools, Working Groups are now experimenting with a number
     of tools, and as part of the quality plan, it is helpful for the
     chosen tools to be described along with the reasoning behind the
     selection and pointers to relevant templates.  As Issue Tracking
     tools gain in popularity, use of reporting tools is also likely to
     become more common.  Reporting tools are essential for quality
     improvement, since they supply the metrics by which progress can be
     gauged.  Use of quality metrics is not yet widespread with the
     IETF, but where they are used, they should be discussed within the
     quality plan.

2.3.  Review Plan

The Review plan includes (but is not limited to) a description of the
plans for the following:

[a]  Challenge Assessment review. This is plan for the challenge
     assessment review, which is based on the Working Group Charter.
     Ideally this plan should include reviewers from outside the area of
     the Working Group.

[b]  Work item review.  In some cases, the Working Group may wish to
     review documents before they become work items in order to identify
     issues up front.  If this is done, then the process by which these
     reviews are conducted should be described.

[c]  Midterm assessment.  This is a review conducted when the document
     is mid-way through its life.

[d]  Late review.  This is a review conducted late in the process,
     within a few months of the start of Working Group last call.

As part of the review plan, the sources of reviewers should be
identified.  This could be members of the relevant directorates, noted
experts, members of the IAB or IESG, volunteers in the process described
in [SIRS], etc.





Loughney, et al.              Informational                     [Page 6]


INTERNET-DRAFT                    COACH                     23 June 2003


3.  Post Mortems

In order to increase the rate of quality improvement, it is important
for Working Groups to assess their quality plans after the fact.  Prior
to dissolution, Working Groups are encouraged to produce a post mortem
no longer than three pages including the following sections:

[a]  Quality assessment. This is an evaluation of the quality of working
     group output, written by the Area Director.

[b]  Challenge Assessment. This covers the unforseen challenges that the
     Working Group encountered.

[c]  Tools assessment. This is an evaluation of the tools that were used
     and how well they worked.

[d]  Reviews assessment. This is an assessment of the review process.

[e]  Recommendations. This section includes recommendations to future
     working groups looking to avoid similar problems.

Post mortems will be made publicly available along with the working
group quality plans.

4.  Required materials

In order to assist Working Groups in putting together their quality
plans, the following materials would be helpful:

[a]  An archive of sample document build environments and templates.

[b]  A website covering issue tracking and reporting tools.

[c]  A document on use of tracking tools for document management,
     including metrics and reporting.

[d]  A document on the review process, including Challenge Assessment.

[e]  A website including sample quality plans and post-mortems.

5.  Normative References

[SIRS]         Carpenter, B. and D. Crocker, "Careful Additional Review
               of Documents (CARD) by Senior IETF Reviewers (SIRS)",
               draft-carpenter-solution-sirs-01.txt, Internet draft
               (work in progress), June 2003.





Loughney, et al.              Informational                     [Page 7]


INTERNET-DRAFT                    COACH                     23 June 2003


Acknowledgments

The authors would like to acknowledge Margaret Wasserman and the members
of the IETF Problem Working Group for discussions relating to document
quality.

Authors' Addresses

John Loughney
Nokia
It204merenkatu 11-13
00180 Helsinki
Finland

Email: john.loughney@nokia.com

Bernard Aboba
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052

EMail: bernarda@microsoft.com

Intellectual Property Statement

The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to  pertain
to the implementation or use of the technology described in this
document or the extent to which any license under such rights might or
might not be available; neither does it represent that it has made any
effort to identify any such rights.  Information on the IETF's
procedures with respect to rights in standards-track and standards-
related documentation can be found in BCP-11.  Copies of claims of
rights made available for publication and any assurances of licenses to
be made available, or the result of an attempt made to obtain a general
license or permission for the use of such proprietary rights by
implementors or users of this specification can be obtained from the
IETF Secretariat.

The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights
which may cover technology that may be required to practice this
standard.  Please address the information to the IETF Executive
Director.







Loughney, et al.              Informational                     [Page 8]


INTERNET-DRAFT                    COACH                     23 June 2003


Full Copyright Statement

Copyright (C) The Internet Society (2003).  All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it or
assist in its implementation may be prepared, copied, published and
distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are included
on all such copies and derivative works.  However, this document itself
may not be modified in any way, such as by removing the copyright notice
or references to the Internet Society or other Internet organizations,
except as needed for the purpose of developing Internet standards in
which case the procedures for copyrights defined in the Internet
Standards process must be followed, or as required to translate it into
languages other than English.  The limited permissions granted above are
perpetual and will not be revoked by the Internet Society or its
successors or assigns.  This document and the information contained
herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Expiration Date

This memo is filed as <draft-loughney-coach-00.txt>,  and  expires
December 22, 2003.
























Loughney, et al.              Informational                     [Page 9]