Skip to main content

RFC Evaluation Project - First Step
draft-huitema-rfc-eval-project-00

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft that was ultimately published as RFC 8963.
Author Christian Huitema
Last updated 2019-07-31
RFC stream (None)
Formats
IETF conflict review conflict-review-huitema-rfc-eval-project, conflict-review-huitema-rfc-eval-project, conflict-review-huitema-rfc-eval-project, conflict-review-huitema-rfc-eval-project, conflict-review-huitema-rfc-eval-project
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state Became RFC 8963 (Informational)
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-huitema-rfc-eval-project-00
Network Working Group                                         C. Huitema
Internet-Draft                                      Private Octopus Inc.
Intended status: Informational                             July 31, 2019
Expires: February 1, 2020

                  RFC Evaluation Project - First Step
                   draft-huitema-rfc-eval-project-00

Abstract

   This document presents a first attempt at evaluating the production
   of the IETF.  We analyze a set of randomly chosen RFC approved in
   2018, looking for history and delays, and using Google Scholar as a
   proxy for the RFC popularity.  The results are interesting, and
   inform further evaluation efforts.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on February 1, 2020.

Copyright Notice

   Copyright (c) 2019 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Huitema                 Expires February 1, 2020                [Page 1]
Internet-Draft                  RFC-Eval                       July 2019

Table of Contents

   1.  RFC Evaluation project  . . . . . . . . . . . . . . . . . . .   2
   2.  Methodology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Analysis of 20 selected RFC . . . . . . . . . . . . . . . . .   6
     3.1.  8411  . . . . . . . . . . . . . . . . . . . . . . . . . .   6
     3.2.  8456  . . . . . . . . . . . . . . . . . . . . . . . . . .   7
     3.3.  8446  . . . . . . . . . . . . . . . . . . . . . . . . . .   7
     3.4.  8355  . . . . . . . . . . . . . . . . . . . . . . . . . .   8
     3.5.  8441  . . . . . . . . . . . . . . . . . . . . . . . . . .   9
     3.6.  8324  . . . . . . . . . . . . . . . . . . . . . . . . . .  10
     3.7.  8377  . . . . . . . . . . . . . . . . . . . . . . . . . .  10
     3.8.  8498  . . . . . . . . . . . . . . . . . . . . . . . . . .  11
     3.9.  8479  . . . . . . . . . . . . . . . . . . . . . . . . . .  11
     3.10. 8453  . . . . . . . . . . . . . . . . . . . . . . . . . .  12
     3.11. 8429  . . . . . . . . . . . . . . . . . . . . . . . . . .  12
     3.12. 8312  . . . . . . . . . . . . . . . . . . . . . . . . . .  13
     3.13. 8492  . . . . . . . . . . . . . . . . . . . . . . . . . .  14
     3.14. 8378  . . . . . . . . . . . . . . . . . . . . . . . . . .  15
     3.15. 8361  . . . . . . . . . . . . . . . . . . . . . . . . . .  16
     3.16. 8472  . . . . . . . . . . . . . . . . . . . . . . . . . .  16
     3.17. 8466  . . . . . . . . . . . . . . . . . . . . . . . . . .  17
     3.18. 8362  . . . . . . . . . . . . . . . . . . . . . . . . . .  17
     3.19. 8468  . . . . . . . . . . . . . . . . . . . . . . . . . .  18
   4.  Observations  . . . . . . . . . . . . . . . . . . . . . . . .  19
     4.1.  Publication delays  . . . . . . . . . . . . . . . . . . .  19
     4.2.  Citation Counts . . . . . . . . . . . . . . . . . . . . .  24
     4.3.  Preparation and Publication delays  . . . . . . . . . . .  30
     4.4.  Copy editing  . . . . . . . . . . . . . . . . . . . . . .  34
     4.5.  Independent Series  . . . . . . . . . . . . . . . . . . .  36
   5.  Next steps  . . . . . . . . . . . . . . . . . . . . . . . . .  36
   6.  Security considerations . . . . . . . . . . . . . . . . . . .  36
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  36
   8.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  37
   9.  Informative References  . . . . . . . . . . . . . . . . . . .  37
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  39

1.  RFC Evaluation project

   As stated on the organization's web site, "The IETF is a large open
   international community of network designers, operators, vendors, and
   researchers concerned with the evolution of the Internet architecture
   and the smooth operation of the Internet."  In this memo, we start
   exploring how the IETF could possibly be evaluated, and we do so by
   attempting to evaluate the RFC production process.

   The IETF data tracker provides information about RFC and drafts, from
   which we can infer statistics about the production system.  We can

Huitema                 Expires February 1, 2020                [Page 2]
Internet-Draft                  RFC-Eval                       July 2019

   measure how long it takes to drive a proposition from initial draft
   to final publication, and how these delays can be split between
   Working Group discussions, IETF reviews, IESG assessment, RFC Editor
   delays and final reviews by the authors.

   Just measuring production delays may be misleading.  The IETF
   produces standard proposals and informative memos that get published
   in the RFC series, but that is not its only purpose.  Two other
   purposes would be the organization of fruitful discussions between
   members of the technical community, and the filtering of ill-thought
   proposals so they are not endorsed in IETF publications.

   To try assess this other purposes, we have to find ways to measure
   the impact of the documents that were published.  If the
   communication between participants was efficient, the documents
   should be easily accepted by the community.  If bad ideas were
   filtered, the specifications should lead to deployed product and
   services with good feedback.

   A potential failure mode of the IETF is to do too much filtering.
   After all, classifying a proposal as ill-thought or potentially
   harmful is subjective.  A comprehensive analysis would look at the
   fate of proposals that were not accepted by the IETF but got
   developed in other ways and became eventually successful.  We will
   not try to assess that in this memo, but further studies migh
   consider the issue.

2.  Methodology

   In this exploration, we want to evaluate not just the mechanics of
   the RFC production, but also the quality and impact of the results.
   This evaluation of quality and impact is subjective.  We start with
   two ideas:

   1- Use Google Scholar to assess the citation counts of published
   documents

   2- Ask the RFC authors whether the specifications resulted in the
   deployment of products or services

   When accessing Google Scholar, we search for quoted strings of the
   form "RFC xxxx".  This is an arbitrary choice, we could for example
   have chosen to search for "RFCxxx" or a combination of the two forms.
   We retained the simpler alternative, because we don't believe that
   picking one or the other would introduce a significant bias.

   Basic production mechanisms could be evaluated by processing data
   from the IETF tracker, but subjective data requires manual assessment

Huitema                 Expires February 1, 2020                [Page 3]
Internet-Draft                  RFC-Eval                       July 2019

   of results, which can be time consuming.  Google Scholar also
   requires manual access because the site does not offer an open API.
   Since our resources are limited, we will only perform this analysis
   for a small sample of RFC, selected at random from the list of RFC
   approved in 2018.  Specifically, we will pick 10 RFC at random
   between:

   o  RFC 8307, published in January 2018, and

   o  RFC 8511, published December 2018.

   In order to avoid injecting personal bias in the random selecton, we
   use a random selection process similar to the Nomination Committee
   selection process defined in [RFC3797].  The process is seeded with
   the text string "vanitas vanitatum et omnia vanitas", and the results
   are:

   Picking 20 numbers between 8307 and 8511,
   using MD5(vanitas vanitatum et omnia vanitas)
   Rank 1: 8411 -- md5=daba041224a879199b698748808f917d
   Rank 2: 8456 -- md5=f5570484d91ada6a672edbdca61d808c
   Rank 3: 8446 -- md5=8340e918bb8faf69d197f79c9a58d7b8
   Rank 4: 8355 -- md5=19474df74efd9917cf3fe8acce2ac374
   Rank 5: 8441 -- md5=5acce2b730f3c24a4a91a5fc1921d1cd
   Rank 6: 8324 -- md5=411c11a1cf4c292f83458865599c6921
   Rank 7: 8377 -- md5=ac16a89192c0f0727febd35aacbc1f24
   Rank 8: 8498 -- md5=bba44f2ba1ab240a1265a82ab71f7e02
   Rank 9: 8479 -- md5=1653606b0af95d529a473a8f85ffaea4
   Rank 10: 8453 -- md5=0cbfe105667c5a83b027dcfa85062f98
   Rank 11: 8429 -- md5=fa51d7738562d990926a0d199fb060b8
   Rank 12: 8312 -- md5=96d061523b1a57343356ae7a1e498ca5
   Rank 13: 8492 -- md5=1b72b746eb05f79af40ed2bd3faccbe8
   Rank 14: 8378 -- md5=645833b936d36cdcc797256518d7c483
   Rank 15: 8361 -- md5=2064622c868e410beb0d9c18d0cb522c
   Rank 16: 8472 -- md5=ca8a823072a21df011d0ea8b96a6aa47
   Rank 17: 8471 -- md5=01b293a7dd0793e6f3297f2a973cd7e3
   Rank 18: 8466 -- md5=8e411babe271557fe83bcdececc1643f
   Rank 19: 8362 -- md5=8a1ba3efd82856a12b2b35fc5237e1b7
   Rank 20: 8468 -- md5=57ae50ee0e1e0708d356d96d116dbfe1

   When evaluating delays and impact, we will compare the year 2018 to
   2008 and 1998, 10 and 20 years ago.  To drive this comparison, we
   pick 20 RFC at random among those published in 2008, and another 20
   among those published in 1998.  We use the same nomcom-like
   methodology.

   For 2008, we picking random RFC numbers between RFC 5134 (January
   2008) and RFC 5405 (December 2008), using the sentence "sed fugit

Huitema                 Expires February 1, 2020                [Page 4]
Internet-Draft                  RFC-Eval                       July 2019

   interea fugit irreparabile tempus" as a seed.  We actually list here
   21 numbers, because the random draw place RFC 5315 in 20th position,
   but that RFC was never issued.  We replace it by RFC 5301, which came
   in 21st position.

   Picking 21 numbers between 5134 and 5405,
   using MD5(sed fugit interea fugit irreparabile tempus)
   Rank 1: 5227 -- md5=61e5b3cd97fda4e75450e93df0744b6d
   Rank 2: 5174 -- md5=eed49542394e9d392bcd756ee0f5beed
   Rank 3: 5172 -- md5=72055ca953d6a8ee0d60a9fca0b8d738
   Rank 4: 5354 -- md5=7a00ef15897479d1d15255159f6d8674
   Rank 5: 5195 -- md5=54813be7bb56f48a05af8c894799b51f
   Rank 6: 5236 -- md5=153263bbd8c0349501b75a33f3d66f6c
   Rank 7: 5348 -- md5=b2d19aa9c1250ef2ddf169045f6e99d7
   Rank 8: 5281 -- md5=1c3e61643d46d1da4ba16f0fd7a0aff5
   Rank 9: 5186 -- md5=5e87001b830183b1a9427d479a7b0e42
   Rank 10: 5326 -- md5=024347839f83d8082549c08bdfa1b43e
   Rank 11: 5277 -- md5=049a83016ab08552841c59400480cd9d
   Rank 12: 5373 -- md5=a1ce374aaebdacca2e7d6eeff039d970
   Rank 13: 5404 -- md5=fb0d6b582a27ce34175e39de33598556
   Rank 14: 5329 -- md5=df043ef1f9d42ba12a03a84434d26ead
   Rank 15: 5283 -- md5=c40d3f966bc7800d6508d3d82df2371d
   Rank 16: 5358 -- md5=6fea5bdb26b19e68befd409a09cb335d
   Rank 17: 5142 -- md5=a8844b73287781762e6548fc6f533508
   Rank 18: 5271 -- md5=c19eb02984265ecfe4ca076f2c160cfa
   Rank 19: 5349 -- md5=33d756f81bf6e40ff344cf6ccaf29f13
   Rank 20: 5315 -- md5=d4c30875f88328d72c9f78def2d1dde5
   Rank 21: 5301 -- md5=3356419e5560901f0d31309b39d14a80

   For 1998 we picking random RFC numbers between RFC 2257 (January
   1998) and RFC 2479 (December 1998), using the sentence "pulvis et
   umbra sumus" as a seed.

Huitema                 Expires February 1, 2020                [Page 5]
Internet-Draft                  RFC-Eval                       July 2019

   Picking 20 numbers between 2257 and 2479,
   using MD5(pulvis et umbra sumus)
   Rank 1: 2431 -- md5=6eba444bb3349339fcde7e2be726f0f0
   Rank 2: 2381 -- md5=0ce53f69f1a49a49309054af1e9a1d42
   Rank 3: 2387 -- md5=53726e77ffeed903d244155d28e30f10
   Rank 4: 2348 -- md5=69fcab2d2085555bac9eb0e0f4346523
   Rank 5: 2391 -- md5=93358a26a7fce9fa9d61a31cdfdb87b7
   Rank 6: 2267 -- md5=652dcab91bd9d5c58f98bdab21b23d80
   Rank 7: 2312 -- md5=2505b0bed4af0a00ee663891c6f294d8
   Rank 8: 2448 -- md5=6ede4b0dfa935f6ca656a5104b8dc5d0
   Rank 9: 2374 -- md5=5312bfe5a9ca45563eb0bf35510d8daa
   Rank 10: 2398 -- md5=7748a6deec3860898678f992b0b22792
   Rank 11: 2283 -- md5=d73ff67466eb42d971a0e134b9284f83
   Rank 12: 2382 -- md5=de1f667ac3e4c64aa529872e08823dff
   Rank 13: 2289 -- md5=37773c2569dc25fdd0ab400ea401d5c7
   Rank 14: 2282 -- md5=6b3df671a0a0becf9e42d203b59acd08
   Rank 15: 2404 -- md5=e1b6819e5355924f456eb79f93beb8fd
   Rank 16: 2449 -- md5=4f057df7c226efea773e7013c9c62081
   Rank 17: 2317 -- md5=40eee3b536abe4afdb8834a0650c0a04
   Rank 18: 2394 -- md5=044f09c53fc9fd1c50fe6bd0c39318e1
   Rank 19: 2297 -- md5=78ee7e128436c969c80900fef80c075c
   Rank 20: 2323 -- md5=ea6935bbda5f6d97756d3df5c3e2fdfb

3.  Analysis of 20 selected RFC

   We review each of the RFC listed in (#methodology) for the year 2018,
   trying both to answer the known questions and to gather insight for
   further analyzes.  In many cases, the analysis of the data is
   complemented by direct feedback from the RFC authors.

3.1.  8411

   IANA Registration for the Cryptographic Algorithm Object Identifier
   Range [RFC8411]:

   Informational, 5 pages
   4 drafts (personal),first May 8, 2017. Published August 2018.
   Last call announced 2017/10/09
   IESG evaluation starts 2017/12/28
   Approved 2018/02/26, draft 03
   Auth 48 2018/04/20
   Auth 48 complete 2018/07/17
   Published 2018/08/06
   IANA action: create table

   The draft underwent minor copy edit before publication.

Huitema                 Expires February 1, 2020                [Page 6]
Internet-Draft                  RFC-Eval                       July 2019

   The long delay in Auth48 is probaby due to clustering with [RFC8410],
   which entered AUTH 48 on 06/05.  The MISSREF tracker code was cleared
   then.

   2 references on Google Scholar.

3.2.  8456

   Benchmarking Methodology for Software-Defined Networking (SDN)
   Controller Performance [RFC8456]:

   Informational, 64 pages
   2 personal drafts, 9 WG drafts, first in March 2015
   Last call announced 2018-01-19
   IESG evaluation starts 2018-02-27
   IESG approved 2018-05-25
   Auth 48 2018-08-31
   Auth 48 complete 2018-10-16
   Published 2018-10-30

   The draft underwent very extensive copy editing, covering use of
   articles, turn of phrases, choice of vocabulary.  The changes are
   enough to cause pagination differences.  The "diff" tool marks pretty
   much every page as changed.  Some diagrams see change in protocol
   elements like message names.

   According to the author, the experience of producing this draft
   mirrors a typical one in the Benchmarking Methodologies Working Group
   (BMWG).There were multiple authors in multiple time zones, which
   slowed down the AUTH process somewhat, although the Auth48 delay of
   46 is only a bit longer than the average draft.

   The RFC was part of cluster with [RFC8455].

   Google Scholar shows 3 references.  BMWG publishes informational RFCs
   centered around benchmarking, and the methodologies in RFC 8456 have
   been implemented in benchmarking products.

3.3.  8446

   The Transport Layer Security (TLS) Protocol Version 1.3 [RFC8446], as
   the title indicates, defines the new version of the TLS protocol.
   From the datatracker, we extract the following:

Huitema                 Expires February 1, 2020                [Page 7]
Internet-Draft                  RFC-Eval                       July 2019

   Proposed standard
   160 pages
   29 WG drafts first April 17, 2014.
   Last call announced 2018/02/15
   IESG evaluation starts 2018/03/02
   Approved 2018/03/21, draft 28
   Auth 48 2018/06/14
   Auth 48 complete 2018/08/10
   Published 2018/08/10

   The RFC was a major effort in the IETF.  Working group members
   developed and tested several implementations.  Researchers analyzed
   the specifications and performed to formal verification.  Deployment
   tests outlined issues that caused extra work when the specification
   was almost ready.  These complexity largely explains the time spent
   in the working group.

   Comparing the final draft to the published version, we find
   relatively light copy editing.  It includes explaining acronyms on
   first use, clarifying some definitions standardizing punctiation and
   capitalization, and spelling out some numbers in text.  This
   generally fall in the category of "style", although some of the
   clarifications go into message definitions.  However, that simple
   analysis does not explain why the Auth48 phase took almost two
   months.

   According to the author, the main reason was a requirement to have
   author and AD review each proposed change, down to any single comma.
   The concern there was that any change in a complex specification
   might affect a protocol that was extensively reviewed in the working
   group.  The author asked the RFC editors to back away a number of
   edits that he considered spurious.  This took a long time, partly
   because it was very hard to distinguish the edits that were editorial
   from those that were typesetting.

   The RFC has 123 references in Google Scholar.  There are 21
   implementations listed in the Wiki of the TLS 1.3 project.  It has
   been deployed on major browsers.

3.4.  8355

   Resiliency Use Cases in Source Packet Routing in Networking (SPRING)
   Networks [RFC8355] is an informational RFC.  It originated from a use
   case informational draft that was mostly used for the BOF creating
   the WG, and then to drive initial work/evolutions from the WG.

Huitema                 Expires February 1, 2020                [Page 8]
Internet-Draft                  RFC-Eval                       July 2019

   Informational, 13 pages.
   2 personal drafts (personal), first January 31, 2014. 13 WG drafts.
   Last call announced 2017-04-20
   IESG evaluation starts 2017-05-04, draft 09
   Approved 2017-12-19, draft 12
   Auth 48 2018-03-12
   Auth 48 complete 2018-03-27
   Published 2018-03-28

   According to the author, the delays reflect the relative lack of
   priority of this work in the WG.  One of the authors actually
   mentions "blockage by a minority of politically-savvy persons."  Once
   the WG was created, other work was higher priority than working on
   the use case.

   Minor set of copy edit, mostly for style.

   9 references on Google Scholar.  No implementation of the RFC itself,
   but the technology behind it such as Segment Routing (architecture
   RFC 8402, TI-LFA draft-ietf-rtgwg-segment-routing-ti-lfa) is widely
   implemented and deployment is ongoing.  Even if this specific RFC was
   incurred significant delay, the authors think that the technology was
   implemented and deployed rather quickly.

3.5.  8441

   Bootstrapping WebSockets with HTTP/2 [RFC8441]

   Proposed standard, 8 pages. Updates RFC 6455.
   3 personal drafts (personal), first 10/15/2017. 8 WG drafts.
   Last call announced 2018-05-07, draft 05
   IESG evaluation starts 2018-05-29, draft 06
   Approved 2018-06-07, draft 07
   Auth 48 2018-08-13
   Auth 48 complete 2018-09-15
   Published 2018-09-21
   IANA Action: table entries

   This RFC defines the support of WebSockets in HTTP/2, which is
   different from the mechanism defined for HTTP/1.1 in [RFC6455].  The
   process was relatively straightforward, involving the usual type of
   discussions, some on details and some on important points.

   Comparing final draft and published RFC shows a minor set of copy
   edit, mostly for style.  However, the author recalls a painful
   process.  The RFC includes many charts and graphs that were very
   difficult to format correctly in the author's production process that

Huitema                 Expires February 1, 2020                [Page 9]
Internet-Draft                  RFC-Eval                       July 2019

   involve conversions from markdown to XML, and then from XML to text.
   The author had to get substantial help from the RFC editor.

   No references on Google Scholar.  (RFC 6455 had over 1000 results)

   There are several implementations, including Firefox and Chrome,
   making RFC 8441 a very successful standard.

3.6.  8324

   DNS Privacy, Authorization, Special Uses, Encoding, Characters,
   Matching, and Root Structure: Time for Another Look?  [RFC8324].
   This is an opinion piece on DNS development, published on the
   Independent Stream.

   Informational, 29 pages. Independent stream.
   5 personal drafts (personal), first June 2, 2017.
   ISE review started 2017-07-10, draft 03
   IETF conflict review and IESG review started 2017-10-29
   Approved 2017-12-18, draft 04
   Auth 48 2018-01-29, draft 05
   Auth 48 complete 2018-02-26
   Published 2018-02-27

   This RFC took only 9 months from first draft to publication, which is
   the shortest in the 2018 sample set.  In part, this is because the
   text was privately circulated and reviewed before the first draft was
   published.  The nature of the document is another reason for the
   short delay.  It is an opinion piece, and does not require the same
   type of consensus building and reviews than a protocol specification.

   Comparing the final draft and the published version shows only minor
   copy edit, mostly for style.  According to the author, because this
   is because he knows how to write in RFC Style with the result that
   his documents often need a minimum of editing.  He also makes sure
   that the document on which the Production Center starts working
   already has changes discussed and approved during Last Call and IESG
   review incorporated rather than expecting the Production Center to
   operate off of notes about changed to be made.

   2 references on Google Scholar.

3.7.  8377

   Transparent Interconnection of Lots of Links (TRILL): Multi-Topology
   [RFC8377]

Huitema                 Expires February 1, 2020               [Page 10]
Internet-Draft                  RFC-Eval                       July 2019

   Proposed standard, 20 pages. Updates RFC 6325, 7177.
   3 personal drafts (personal), first September 3, 2013. 7 WG drafts.
   Last call announced 2018-02-19, draft 05
   IESG evaluation starts 2018-03-02, draft 06
   Approved 2018-03-12, draft 05
   Auth 48 2018-04-20, draft 06
   Auth 48 complete 2018-07-31
   Published 2018-07-31
   IANA Table, table entries

   Minor set of copy edit, mostly for style, also clarity.

   1 reference on Google Scholar.

3.8.  8498

   A P-Served-User Header Field Parameter for an Originating Call
   Diversion (CDIV) Session Case in the Session Initiation Protocol
   (SIP) [RFC8498].

   Informational, 15 pages.
   5 personal drafts (personal), first March 21, 2016. 9 WG drafts.
   Last call announced 2018-10-12, draft 05
   IESG evaluation starts 2018-11-28, draft 07
   Approved 2018-12-10, draft 08
   Auth 48 2019-01-28
   Auth 48 complete 2019-02-13
   Published 2019-02-15
   IANA Action, table rows added.

   Copy edit for style, but also clarification of ambiguous sentences.

   No references on Google Scholar.

3.9.  8479

   Storing Validation Parameters in PKCS#8 [RFC8479]

   Informational, 8 pages. Independent stream.
   5 personal drafts (personal), first August 8, 2017.
   ISE review started 2018-03-29, draft 02
   IETF conflict review and IESG review started 2018-03-29
   Approved 2018-08-20, draft 03
   Auth 48 2018-09-20, draft 04
   Auth 48 complete 2018-09-25
   Published 2018-09-26

Huitema                 Expires February 1, 2020               [Page 11]
Internet-Draft                  RFC-Eval                       July 2019

   The goal of the draft was to document what the gnutls implementation
   was using for storing provably generated RSA keys.  This is a short
   RFC that was published relatively quickly, although discussion
   between the author, the Independent Series Editor and the IESG lasted
   several months.  The IESG actually asked the ISE to not publish this
   document, because "it extends an IETF protocol in a way that requires
   IETF review and should therefore not be published without IETF review
   and IESG approval.  The ISE overruled that advice.

   Very minor set of copy edit, moving some references from normative to
   informative.

   No reference on Google Scholar.

   The author is not aware of other implementations than gnutls relying
   on this RFC.

3.10.  8453

   Framework for Abstraction and Control of TE Networks (ACTN) [RFC8453]

   Informational, 42 pages.
   3 personal drafts, first June 15, 2015. 16 WG drafts.
   Out of WG 2018-01-26, draft 11
   Expert review requested, 2018-02-13
   Last call announced 2018-04-16, draft 13
   IESG evaluation starts 2018-05-16, draft 14
   Approved 2018-06-01, draft 15
   Auth 48 2018-08-13
   Auth 48 complete 2018-08-20
   Published 2018-08-20
   IANA Action, table rows added.

   Minor copy editing.

   8 references on Google Scholar.

3.11.  8429

   Deprecate Triple-DES (3DES) and RC4 in Kerberos [RFC8429]

Huitema                 Expires February 1, 2020               [Page 12]
Internet-Draft                  RFC-Eval                       July 2019

   Informational, 42 pages.
   6 WG, first 5/1/2017.
   Last call announced 7/16/2017, draft 03
   IESG evaluation starts 8/18/2017, draft 04
   Approved 5/25/2018, draft 05
   Auth 48 7/24/2018
   Auth 48 complete 10/31/2018
   Published 10/31/2018
   IANA Action, table rows added.

   This RFC recommends to deprecate two encryption algorithms that are
   now considered obsolete and possibly broken.  The document was sent
   back to the WG after the first last call, edited, and then there was
   a second last call.  The delay from first draft to working group last
   call was relatively short, but thenumber may be misleading.  The
   initial draft was a replacement of a similar draft in the KITTEN
   working group, which stagnated for some time before the VURDLE
   working group took up the work.  The deprecation of RC4 was somewhat
   contentious, but the WG had already debated this prior to the
   production of this draft, and the draft was not delayed by this
   debate.

   Most of the 280 days between IETF LC and IESG approval was because
   the IESG had to talk about whether this document should obsolete or
   move to historic RFC 4757, and no one was really actively pushing
   that discussion for a while.

   The 99 days in AUTH48 are mostly because one of the authors was a
   sitting AD, and those duties ended up taking precedence over
   reviewing this document.

   Minor copy editing, for style.

   1 reference on Google Scholar.

   The implementation of the draft would be the actual removal of
   support for 3DES and RC4 in major implementations.  This is
   happening, but very slowly.

3.12.  8312

   CUBIC for Fast Long-Distance Networks [RFC8312]

Huitema                 Expires February 1, 2020               [Page 13]
Internet-Draft                  RFC-Eval                       July 2019

   Informational, 18 pages.
   2 personal drafts, first 9/1/2014. 8 WG drafts
   Last call announced 9/18/2017, draft 06
   IESG evaluation starts 2017-11-14
   Approved 2017-10-04, draft 07
   Auth 48 2018-01-08
   Auth 48 complete 2018-02-07
   Published 2018-02-07
   IANA Action, table rows added.

   Minor copy editing, for style.

   9 references on Google Scholar.

   The TCP congestion control algorithm Cubic was defined first in 2005,
   was implemented in Linux soon after, and was implemented in major
   OSes after that.  After some debates from 2015 to 2015, the TCPM
   working group adopted the draft, with a goal of documenting Cubic in
   the RFc series.  According to the authors, this was not a high
   priority effort, as Cubic was already implemented in multiple OSes
   and documented in research papers.  At some point, only one of the
   authors was actively working on the draft.  Ths may explain why
   another two years was spent progressing the draft after adoption by
   the WG.

   The RFC publication may or may not have triggered further
   implementations.  On the other hand, several OSes picked up bug fixes
   from the draft and the RFC.

3.13.  8492

   Secure Password Ciphersuites for Transport Layer Security (TLS)
   [RFC8492]

   Informational, 40 pages. (Independent Stream)
   10 personal drafts, first 9/2/2012. 8 WG drafts
   ISE review started 2017-05-10, draft 01
   IETF conflict review and IESG review started 2017-09-04
   Approved 2017-10-29, draft 04
   Auth 48 10/19/2018, draft 05
   Auth 48 complete 2/19/2019
   Published 2/21/2019
   IANA Action, table rows added.

   This RFC has a complex history.  The first individual draft was
   submitted to the TLS working group on September 7, 2012.  It
   progressed there, and was adopted by the WG after 3 revisions.  There
   were then 8 revisions in the TLS WG, until the WG decided to not

Huitema                 Expires February 1, 2020               [Page 14]
Internet-Draft                  RFC-Eval                       July 2019

   progress it.  The draft was parked in 2013 by the WG chairs after
   failing to get consensus in WG last call.  The AD finally pulled the
   plug in 2016, and the draft was then resubmitted to the ISE.

   At that point, the author was busy and was treating this RFC with a
   low priority because, in his words, it would not be a "real RFC".
   There were problems with the draft that only came up late.  It was
   written back before TLS1.3 existed and the author had to add TLS1.3
   support onto it late in the process.  He also got a very late comment
   while in AUTH48 that caused some rewrite.  Finally, there was some
   IANA issue with the extension registry where a similar extension was
   added by someone else.  The draft was changed to just use it.

   Changes in AUTH48 include added reference to TLS 1.3, copy-editing
   for style, some added requirements, added paragraphs, and changes in
   algorithms specification.

   2 references on Google Scholar.  Only the author implemented the
   specification.

3.14.  8378

   Signal-Free Locator/ID Separation Protocol (LISP) Multicast [RFC8378]
   is an experimental RFC, defining how to implement Multicast in the
   LISP architecture.

   Experimental, 21 pages.
   5 personal drafts, first 2/28/2014. 10 WG drafts
   Last call announced 2018-02-13, draft 07
   IESG evaluation starts 2018-02-28, draft 08
   Approved 2018-03-12, draft 09
   Auth 48 2018-04-23
   Auth 48 complete 2018-05-02
   Published 2018-05-02

   Preparing the RFC took more than 4 years.  According to the authors,
   they were not aggressive pushing it and just let the working group
   process decide to pace it.  They also did implementations during that
   time.

   Minor copy editing, for style.

   1 reference on Google Scholar.  The RFC was implemented by
   lispers.net and cisco, and was used in doing IPv6 multicast over IPv4
   unicast/multicast at the Olympics for NBC in PyeungChang.  The plan
   is to work on a proposedstandard once the experiment concludes.

Huitema                 Expires February 1, 2020               [Page 15]
Internet-Draft                  RFC-Eval                       July 2019

3.15.  8361

   Transparent Interconnection of Lots of Links (TRILL): Centralized
   Replication for Active-Active Broadcast, Unknown Unicast, and
   Multicast (BUM) Traffic [RFC8361]

   Proposed Standard, 17 pages.
   3 personal drafts, first 11/12/2013. 14 WG drafts
   Last call announced 2017-11-28, draft 10
   IESG evaluation starts 2017-12-18, draft 11
   Approved 2018-01-29, draft 13
   Auth 48 2018-09-17
   Auth 48 complete 4/9/2018
   Published 2018-10-08

   According to the authors, the long delays in producing this RFC was
   due to a slow uptake of the technology in the industry.

   Minor copy editing, for style.

   1 reference on Google Scholar.

   There was at least 1 partial implementation.

3.16.  8472

   Transport Layer Security (TLS) Extension for Token Binding Protocol
   Negotiation [RFC8472]

   Proposed Standard, 8 pages.
   1 personal drafts, first 5/29/2015. 15 WG drafts
   Last call announced 2017-11-13, draft 10
   IESG evaluation starts 2018-03-19
   Approved 2018-07-20, draft 14
   Auth 48 2018-09-17
   Auth 48 complete 2018-09-25
   Published 2018-10-08

   This is a pretty simple document, but it took over 3 years from
   individual draft to RFC.  According to the authors,the biggest
   setbacks occurred at the start: it took a while to find a home for
   this draft.  It was presented in the TLS WG (because it's a TLS
   extension) and UTA WG (because it has to do with applications using
   TLS).  Then the ADs determined that a new WG was needed, so the
   authors had to work through the WG creation process, including
   running a BOF.

Huitema                 Expires February 1, 2020               [Page 16]
Internet-Draft                  RFC-Eval                       July 2019

   Minor copy editing, for style, with the addition of a reference to
   TLS 1.3.

   5 references on Google Scholar.

   Perhaps partially due to the delays, some of the implementers lost
   interest in supporting this RFC.

3.17.  8466

   A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service
   Delivery [RFC8466]

   Proposed Standard, 158 pages.
   5 personal drafts, first 9/1/2016. 11 WG drafts
   Last call announced 2018-02-21, draft 07
   IESG evaluation starts 2018-03-14, draft 08
   Approved 2018-06-25, draft 10
   Auth 48 2018-09-17
   Auth 48 complete 2018-10-09
   Published 2018-10-12

   Copy editing for style and clarity, with also corrections to the yang
   model.

   2 references on Google Scholar.

3.18.  8362

   OSPFv3 Link State Advertisement (LSA) Extensibility [RFC8362] is a
   major extension to the OSPF protocol.  It makes OSPFv3 fully
   extensible.

   Proposed Standard, 33 pages.
   4 personal drafts, first February 17, 2013. 24 WG drafts
   Last call announced 2017-12-19, draft 19
   IESG evaluation starts 2018-01-18, draft 20
   Approved 2018-01-29, draft 23
   Auth 48 2018-03-19
   Auth 48 complete 2018-03-30
   Published 2018-04-03

   The specification was first submitted as a personal draft in the IPv6
   WG, then moved to the OSPF WG.  The long delay of producing this RFC
   is due to the complexity of the problem, and the need to wait for
   implementations.  It is a very important change to OSPF that makes
   OSPFv3 fully extensible.  Since it was a non-backward compatible
   change, the developers started out with some very complex migration

Huitema                 Expires February 1, 2020               [Page 17]
Internet-Draft                  RFC-Eval                       July 2019

   scenarios but ended up with either legacy or extended OSPFv3 LSAs
   within an OSPFv3 routing domain.  The initial attempts to have a
   hybrid mode of operation with both legacy and extended LSAs also
   delayed implementation due to the complexity.

   Copy editing for style and clarity.

   7 references on Google Scholar.  It either was or will be implemented
   by all the router vendors.

3.19.  8468

   IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance
   Metrics (IPPM) Framework [rfc8468].

   Informational, 15 pages.
   3 personal drafts, first August 6, 2015. 7 WG drafts
   Last call announced 2018-04-11, draft 04
   IESG evaluation starts 2018-05-24, draft 05
   Approved 2018-07-10, draft 06
   Auth 48 2018-09-13
   Auth 48 complete 2018-11-05
   Published 2018-11-14

   RFC8468 was somehow special in that there was not a technical reason/
   interest that triggered it, but rather a formal requirement.  While
   writing RFC7312 the IP Performance Metrics working group (IPPM)
   realized that RFC 2330, the IP Performance Metrics Framework
   supported IPv4 only and explicitly excluded support for IPv6.
   Nevertheless, people used the metrics that were defined on top of RFC
   2330 (and, therefore, IPv4 only) for IPv6, too.  Although the IPPM WG
   agreed that the work was needed, the interest of IPPM attendees in
   progressing (and reading/reviewing) the IPv6 draft was limited.
   Resolving the IPv6 technical part was straight-forward, but
   subsequently some people asked for a broader scope (topics like
   header compression, 6lo, etc.) and it took some time to figure out
   and later on convince people that these topics are out of scope.  The
   group also had to resolve contentious topics, for example how to
   measure the processing of IPv6 extension headers, which is sometimes
   non-standard.

   The Auth48 delay for this draft was longer than average.  According
   to the authors, the main reasons include:

   o  Work-load and travel caused by busy-work-periods of all co-authors

   o  Time zone difference between co-authors and editor (at least US,
      Europe, India, not considering travel)

Huitema                 Expires February 1, 2020               [Page 18]
Internet-Draft                  RFC-Eval                       July 2019

   o  Editor proposing and committing some unacceptable modifications
      that needed to be reverted

   o  Lengthy discussions on a new document title (required high effort
      and took a long time, in particular reaching consensus between co-
      authors and editor was time-consuming and involved the AD)

   o  Editor correctly identifying some nits (obsoleted personal
      websites of co-authors) and co-authors attempting to fix them.

   The differences between the final draft and the publish RFC show copy
   editing for style and clarity, but do not account for the back and
   forth between authors and editors mentioned by the authors.

   2 references on Google Scholar.  In contrast, RFC 2330 has more than
   3000 references, including over 70 that are more recent than RFC
   8468.  The authors believe that many of these references include IPv6
   work that should formally reference RFC 8468 in addition to RFC 2330,
   but do not.

4.  Observations

   We examine the 20 RFC in the sample, measuring various
   characteristics such as delay and citation counts, in an attempt to
   identify patterns in the IETF processes.

4.1.  Publication delays

   We look at the distribution of delays between the submission of the
   first draft and the publication of the RFC.  We break out the total
   delay in three components:

   o  The working group delay, from the first draft to the start of the
      IETF last call;

   o  The IETF delay, which lasts from the beginning of the IETF last
      call to the approval by the IESG, including the reviews by various
      directorates;

   o  The RFC production, from approval by the IESG to publication,
      including the Auth48 reviews.

   For submissions to the independent stream, we don't have a working
   group.  We consider instead the progression of the individual draft
   until the adoption by the ISE as the equivalent of the "working
   group" period, and the delay from adoption by the ISE until
   submission to the RFC Editor as the equivalent of the IETF delay.
   The following table shows the delays for the 20 RFC in the sample:

Huitema                 Expires February 1, 2020               [Page 19]
Internet-Draft                  RFC-Eval                       July 2019

         +------+---------+-------+---------+------+------+------+
         |  RFC | Status  | Pages | Overall |   WG | IETF | Edit |
         +------+---------+-------+---------+------+------+------+
         | 8411 | Info    |     5 |     455 |  154 |  140 |  161 |
         |      |         |       |         |      |      |      |
         | 8456 | Info    |    64 |    1107 |  823 |  126 |  158 |
         |      |         |       |         |      |      |      |
         | 8446 | PS      |   160 |    1576 | 1400 |   34 |  142 |
         |      |         |       |         |      |      |      |
         | 8355 | Info    |    13 |    1517 | 1175 |  243 |   99 |
         |      |         |       |         |      |      |      |
         | 8441 | PS      |     8 |     341 |  204 |   31 |  106 |
         |      |         |       |         |      |      |      |
         | 8324 | ISE     |    29 |     270 |   38 |  161 |   71 |
         |      |         |       |         |      |      |      |
         | 8377 | PS      |     8 |    1792 | 1630 |   21 |  141 |
         |      |         |       |         |      |      |      |
         | 8498 | Info    |    15 |    1061 |  935 |   59 |   67 |
         |      |         |       |         |      |      |      |
         | 8479 | ISE     |     8 |     414 |  233 |  144 |   37 |
         |      |         |       |         |      |      |      |
         | 8453 | Info    |    42 |    1162 | 1036 |   46 |   80 |
         |      |         |       |         |      |      |      |
         | 8429 | BCP     |    10 |     548 |   76 |  313 |  159 |
         |      |         |       |         |      |      |      |
         | 8312 | Info    |    18 |    1255 | 1113 |   16 |  126 |
         |      |         |       |         |      |      |      |
         | 8492 | ISE     |    40 |    2358 | 1706 |  172 |  480 |
         |      |         |       |         |      |      |      |
         | 8378 | Exp     |    21 |    1524 | 1446 |   27 |   51 |
         |      |         |       |         |      |      |      |
         | 8361 | PS      |    17 |    1612 | 1477 |   62 |   73 |
         |      |         |       |         |      |      |      |
         | 8472 | PS      |     8 |    1228 |  899 |  249 |   80 |
         |      |         |       |         |      |      |      |
         | 8466 | PS      |   158 |     771 |  538 |  124 |  109 |
         |      |         |       |         |      |      |      |
         | 8362 | PS      |    33 |    1871 | 1766 |   41 |   64 |
         |      |         |       |         |      |      |      |
         | 8468 | Info    |    15 |    1196 |  979 |   90 |  127 |
         |      |         |       |         |      |      |      |
         |      | average |    35 |    1161 |  928 |  110 |  123 |
         +------+---------+-------+---------+------+------+------+

   The average delay from first draft to publication is about 3 years,
   but this varies widely.  Excluding the independent stream
   submissions, the average delay from start to finish is 3 years and 3

Huitema                 Expires February 1, 2020               [Page 20]
Internet-Draft                  RFC-Eval                       July 2019

   months, of which on average 2 years and 8 months are spent getting
   consensus in the working group.

   The longest delay is found for [RFC8492], 6.5 years from start to
   finish.  This is however a very special case, a draft that was
   prepared for the TLS working group and failed to reach consensus.
   After that, it was resubmitted to the ISE, and incurred atypical
   production delays.

   On average, we see that 80% of the delay is incurred in WG
   processing, 10% in IETF review, and 10% for edition and publication.
   We can compare these delays to those observed 10 years ago and 20
   years ago:

Huitema                 Expires February 1, 2020               [Page 21]
Internet-Draft                  RFC-Eval                       July 2019

                  +------------+--------+-------+-------+
                  | RFC (2008) | Status | Pages | Delay |
                  +------------+--------+-------+-------+
                  | 5326       | Exp    |    54 |  1584 |
                  |            |        |       |       |
                  | 5348       | PS     |    58 |   823 |
                  |            |        |       |       |
                  | 5281       | Info   |    51 |  1308 |
                  |            |        |       |       |
                  | 5354       | Exp    |    23 |  2315 |
                  |            |        |       |       |
                  | 5227       | PS     |    21 |  2434 |
                  |            |        |       |       |
                  | 5329       | PS     |    12 |  1980 |
                  |            |        |       |       |
                  | 5277       | PS     |    35 |   912 |
                  |            |        |       |       |
                  | 5236       | ISE    |    26 |  1947 |
                  |            |        |       |       |
                  | 5358       | BCP    |     7 |   884 |
                  |            |        |       |       |
                  | 5271       | Info   |    22 |  1066 |
                  |            |        |       |       |
                  | 5195       | PS     |    10 |   974 |
                  |            |        |       |       |
                  | 5283       | PS     |    12 |  1096 |
                  |            |        |       |       |
                  | 5186       | Info   |     6 |  2253 |
                  |            |        |       |       |
                  | 5142       | PS     |    13 |  1005 |
                  |            |        |       |       |
                  | 5373       | PS     |    24 |  1249 |
                  |            |        |       |       |
                  | 5404       | PS     |    27 |   214 |
                  |            |        |       |       |
                  | 5172       | PS     |     7 |   305 |
                  |            |        |       |       |
                  | 5349       | Info   |    10 |  1096 |
                  |            |        |       |       |
                  | 5301       | PS     |     6 |   396 |
                  |            |        |       |       |
                  | 5174       | Info   |     8 |   427 |
                  +------------+--------+-------+-------+

Huitema                 Expires February 1, 2020               [Page 22]
Internet-Draft                  RFC-Eval                       July 2019

                 +------------+--------+-------+---------+
                 | RFC (1998) | Status | Pages |   Delay |
                 +------------+--------+-------+---------+
                 | 2289       | PS     |    25 |     396 |
                 |            |        |       |         |
                 | 2267       | Info   |    10 | unknown |
                 |            |        |       |         |
                 | 2317       | BCP    |    10 |     485 |
                 |            |        |       |         |
                 | 2404       | PS     |     7 |     488 |
                 |            |        |       |         |
                 | 2374       | PS     |    12 |     289 |
                 |            |        |       |         |
                 | 2449       | PS     |    19 |     273 |
                 |            |        |       |         |
                 | 2283       | PS     |     9 |     153 |
                 |            |        |       |         |
                 | 2394       | Info   |     6 |     365 |
                 |            |        |       |         |
                 | 2348       | DS     |     5 |     699 |
                 |            |        |       |         |
                 | 2382       | Info   |    30 |     396 |
                 |            |        |       |         |
                 | 2297       | ISE    |   109 |      28 |
                 |            |        |       |         |
                 | 2381       | PS     |    43 |     699 |
                 |            |        |       |         |
                 | 2312       | Info   |    20 |     365 |
                 |            |        |       |         |
                 | 2387       | PS     |    10 |     122 |
                 |            |        |       |         |
                 | 2398       | Info   |    15 |     396 |
                 |            |        |       |         |
                 | 2391       | PS     |    10 |     122 |
                 |            |        |       |         |
                 | 2431       | PS     |    10 |     457 |
                 |            |        |       |         |
                 | 2282       | Info   |    14 |     215 |
                 |            |        |       |         |
                 | 2323       | ISE    |     5 | unknown |
                 |            |        |       |         |
                 | 2448       | ISE    |     7 |      92 |
                 +------------+--------+-------+---------+

   We can compare the median delay, and the delays observed by the
   fastest and slowest quartiles in the three years:

Huitema                 Expires February 1, 2020               [Page 23]
Internet-Draft                  RFC-Eval                       July 2019

               +------+-------------+--------+-------------+
               | Year | Fastest 25% | Median | Slowest 25% |
               +------+-------------+--------+-------------+
               | 2018 |         604 |   1179 |        1522 |
               |      |             |        |             |
               | 2008 |         869 |   1081 |        1675 |
               |      |             |        |             |
               | 1998 |         169 |    365 |         442 |
               +------+-------------+--------+-------------+

   The IETF takes three to four times more times to produce an RFC in
   2018 than it did in 1998, but about the sametime as it did in 1998.

   The increased delay does not mean increased work per RFC.  The number
   of RFC published per year remained between 200 and 300 all those
   years, and the number of participants is not greater now than in
   1998.  If we estimated the "level of attention" by dividing the
   number of participants by the number of RFC produced, we would see a
   number that remains stable.  People are probably not working much
   more on each RFC now than they were 20 years ago, but the same amount
   of work is stretched over a much longer period.

4.2.  Citation Counts

   Part of the exercise is to test whether citation counts provide a
   useful measure of the popularity of the IETF production.  These
   citation counts vary widely:

Huitema                 Expires February 1, 2020               [Page 24]
Internet-Draft                  RFC-Eval                       July 2019

                        +------+--------+---------+
                        |  RFC | Status | Scholar |
                        +------+--------+---------+
                        | 8411 | Info   |       2 |
                        |      |        |         |
                        | 8456 | Info   |       3 |
                        |      |        |         |
                        | 8446 | PS     |     123 |
                        |      |        |         |
                        | 8355 | Info   |       9 |
                        |      |        |         |
                        | 8441 | PS     |       0 |
                        |      |        |         |
                        | 8324 | ISE    |       2 |
                        |      |        |         |
                        | 8377 | PS     |       1 |
                        |      |        |         |
                        | 8498 | Info   |       0 |
                        |      |        |         |
                        | 8479 | ISE    |       0 |
                        |      |        |         |
                        | 8453 | Info   |       8 |
                        |      |        |         |
                        | 8429 | BCP    |       1 |
                        |      |        |         |
                        | 8312 | Info   |       9 |
                        |      |        |         |
                        | 8492 | ISE    |       2 |
                        |      |        |         |
                        | 8378 | Exp    |       1 |
                        |      |        |         |
                        | 8361 | PS     |       1 |
                        |      |        |         |
                        | 8472 | PS     |       5 |
                        |      |        |         |
                        | 8466 | PS     |       2 |
                        |      |        |         |
                        | 8362 | PS     |       7 |
                        |      |        |         |
                        | 8468 | Info   |       2 |
                        +------+--------+---------+

   The results indicate that [RFC8446] is by far the most popular of the
   20 RFC in our sample.  This is not surprising, since TLS is a key
   Internet Protocol.  Of the other publications, only 4 have 5 to 9
   citations, and the others have three or less.

Huitema                 Expires February 1, 2020               [Page 25]
Internet-Draft                  RFC-Eval                       July 2019

   In order to get a baseline, we can look at the number of references
   for the RFC published in 2008 and 1998.  However, we need totake time
   into account.  Documents published a long time ago are expected to
   have accrued more references.  We try to address this by looking at
   three counts for each document: the overall number of references over
   the document's lifetime, the number of references obtained in the
   year following publication, and the number of references observed
   since 2018:

Huitema                 Expires February 1, 2020               [Page 26]
Internet-Draft                  RFC-Eval                       July 2019

                +------------+---------+----------+-------+
                | RFC (2008) | Overall | 08 to 09 | >2018 |
                +------------+---------+----------+-------+
                | 5326       |     234 |       27 |    26 |
                |            |         |          |       |
                | 5348       |     364 |       34 |    13 |
                |            |         |          |       |
                | 5281       |     182 |       27 |    12 |
                |            |         |          |       |
                | 5354       |      30 |       15 |     7 |
                |            |         |          |       |
                | 5227       |      81 |        9 |     7 |
                |            |         |          |       |
                | 5329       |      57 |       11 |     5 |
                |            |         |          |       |
                | 5277       |      85 |       10 |     5 |
                |            |         |          |       |
                | 5236       |      48 |        8 |     4 |
                |            |         |          |       |
                | 5358       |      45 |        6 |     3 |
                |            |         |          |       |
                | 5271       |      16 |        3 |     3 |
                |            |         |          |       |
                | 5195       |      21 |       12 |     1 |
                |            |         |          |       |
                | 5283       |      26 |        5 |     1 |
                |            |         |          |       |
                | 5186       |      13 |        4 |     1 |
                |            |         |          |       |
                | 5142       |      42 |       14 |     0 |
                |            |         |          |       |
                | 5373       |      16 |        4 |     0 |
                |            |         |          |       |
                | 5404       |      12 |        3 |     0 |
                |            |         |          |       |
                | 5172       |      11 |        2 |     0 |
                |            |         |          |       |
                | 5349       |      11 |        1 |     0 |
                |            |         |          |       |
                | 5301       |      11 |        3 |     0 |
                |            |         |          |       |
                | 5174       |       1 |        1 |     0 |
                +------------+---------+----------+-------+

Huitema                 Expires February 1, 2020               [Page 27]
Internet-Draft                  RFC-Eval                       July 2019

                +------------+---------+----------+-------+
                | RFC (1998) | Overall | 98 to 99 | >2018 |
                +------------+---------+----------+-------+
                | 2289       |     369 |       13 |    14 |
                |            |         |          |       |
                | 2267       |     607 |       13 |     7 |
                |            |         |          |       |
                | 2317       |      81 |        9 |     6 |
                |            |         |          |       |
                | 2404       |     446 |       20 |     5 |
                |            |         |          |       |
                | 2374       |     280 |       18 |     4 |
                |            |         |          |       |
                | 2449       |      54 |        6 |     3 |
                |            |         |          |       |
                | 2283       |     240 |       25 |     1 |
                |            |         |          |       |
                | 2394       |      69 |        4 |     1 |
                |            |         |          |       |
                | 2348       |      27 |        2 |     1 |
                |            |         |          |       |
                | 2382       |      89 |       30 |     0 |
                |            |         |          |       |
                | 2297       |      68 |       21 |     0 |
                |            |         |          |       |
                | 2381       |      86 |       20 |     0 |
                |            |         |          |       |
                | 2312       |     115 |       18 |     0 |
                |            |         |          |       |
                | 2387       |     292 |        8 |     0 |
                |            |         |          |       |
                | 2398       |      72 |        7 |     0 |
                |            |         |          |       |
                | 2391       |     110 |        5 |     0 |
                |            |         |          |       |
                | 2431       |      25 |        4 |     0 |
                |            |         |          |       |
                | 2282       |      12 |        3 |     0 |
                |            |         |          |       |
                | 2323       |      12 |        1 |     0 |
                |            |         |          |       |
                | 2448       |       5 |        1 |     0 |
                +------------+---------+----------+-------+

   We can compare the median number of references, and the numbers of
   references for the least and most popular quartiles in the three
   years:

Huitema                 Expires February 1, 2020               [Page 28]
Internet-Draft                  RFC-Eval                       July 2019

     +----------------------------+-----------+--------+------------+
     | References                 | Lower 25% | Median | Higher 25% |
     +----------------------------+-----------+--------+------------+
     | RFC (2018)                 |         1 |      2 |          6 |
     |                            |           |        |            |
     | RFC (2008)                 |        12 |     26 |         57 |
     |                            |           |        |            |
     | RFC (2008), until 2009     |         3 |      6 |         12 |
     |                            |           |        |            |
     | RFC (2008), 2018 and after |         0 |      1 |          5 |
     |                            |           |        |            |
     | RFC (1998)                 |        47 |     84 |        250 |
     |                            |           |        |            |
     | RFC (1998), until 1999     |         4 |      9 |         19 |
     |                            |           |        |            |
     | RFC (1998), 2018 and after |         0 |      0 |          3 |
     +----------------------------+-----------+--------+------------+

   The total numbers shows a decreasing number of references over time.
   This can be explain to some degree by the passage of time.  If we
   restrict the analysis to the number of references accrued in the year
   of publishing and the year after that, we still see decreasing
   numbers.  For example, the top quartile of RFC published in 1998 had
   at least 19 references by the end of 1999, while the top quartile of
   RFC published in 2008 only had at least 12 references, which is twice
   the corresponding number for the RFC of 2018.

   We also see that the number of references to RFC fades over time.
   Only the most popular of the RFC produced in 1998 are still
   referenced in 2019.  The overall popularity of the RFC series benefit
   from a history of publishing relevant documents, but over time the
   references to historic documents will decrease and the overall
   popularity will depend on more recent documents.

   We need however to be a bit cautious before asserting that
   publications with a low citation count have limited impact:

   o  some documents may well accumulate more citations over time.  For
      example, [RFC8377] updates [RFC6455].  There are more than 1000
      citations of [RFC6455] on Google Scholar.  We might expect that
      the citation count of [RFC8377] will increase in the coming years.

   o  citation counts largely come from academic publications, and thus
      reflect popularity within researchers more than popularity with
      network operators and vendors.

Huitema                 Expires February 1, 2020               [Page 29]
Internet-Draft                  RFC-Eval                       July 2019

   We should be able to assess the popularity of specifications with
   vendors, operators and designers by asking questions about deployed
   services and products.

4.3.  Preparation and Publication delays

   The preparation and publication delays include three components:

   o  the delay from submission to the RFC Editor to beginning of
      Auth48, during which the document is prepared;

   o  the AUTH48 delay, during which authors review and eventually
      approve the changes proposed by the editors;

   o  the publication delay, from final agreement by authors and editors
      to actual publication.

   The breakdown of the publication delays for each RFC is shown in the
   following table.

Huitema                 Expires February 1, 2020               [Page 30]
Internet-Draft                  RFC-Eval                       July 2019

   +-------+---------+-------+--------+---------+--------+-------------+
   |   RFC | Status  | Pages |    RFC | Auth 48 |    RFC | Edit(total) |
   |       |         |       |   edit |         |    Pub |             |
   +-------+---------+-------+--------+---------+--------+-------------+
   |  8411 | Info    |     5 |     53 |      88 |     20 |         161 |
   |       |         |       |        |         |        |             |
   |  8456 | Info    |    64 |     98 |      46 |     14 |         158 |
   |       |         |       |        |         |        |             |
   |  8446 | PS      |   160 |     85 |      57 |      0 |         142 |
   |       |         |       |        |         |        |             |
   |  8355 | Info    |    13 |     83 |      15 |      1 |          99 |
   |       |         |       |        |         |        |             |
   |  8441 | PS      |     8 |     67 |      33 |      6 |         106 |
   |       |         |       |        |         |        |             |
   |  8324 | ISE     |    29 |     42 |      28 |      1 |          71 |
   |       |         |       |        |         |        |             |
   |  8377 | PS      |     8 |     39 |     102 |      0 |         141 |
   |       |         |       |        |         |        |             |
   |  8498 | Info    |    15 |     49 |      16 |      2 |          67 |
   |       |         |       |        |         |        |             |
   |  8479 | ISE     |     8 |     31 |       5 |      1 |          37 |
   |       |         |       |        |         |        |             |
   |  8453 | Info    |    42 |     73 |       7 |      0 |          80 |
   |       |         |       |        |         |        |             |
   |  8429 | BCP     |    10 |     60 |      99 |      0 |         159 |
   |       |         |       |        |         |        |             |
   |  8312 | Info    |    18 |     96 |      30 |      0 |         126 |
   |       |         |       |        |         |        |             |
   |  8492 | ISE     |    40 |    355 |     123 |      2 |         480 |
   |       |         |       |        |         |        |             |
   |  8378 | Exp     |    21 |     42 |       9 |      0 |          51 |
   |       |         |       |        |         |        |             |
   |  8361 | PS      |    17 |     39 |      31 |      3 |          73 |
   |       |         |       |        |         |        |             |
   |  8472 | PS      |     8 |     59 |       8 |     13 |          80 |
   |       |         |       |        |         |        |             |
   |  8466 | PS      |   158 |     84 |      22 |      3 |         109 |
   |       |         |       |        |         |        |             |
   |  8362 | PS      |    33 |     49 |      11 |      4 |          64 |
   |       |         |       |        |         |        |             |
   |  8468 | Info    |    15 |     65 |      53 |      9 |         127 |
   |       |         |       |        |         |        |             |
   |       | Average |       |   77.3 |    41.2 |    4.2 |       122.7 |
   |       |         |       |        |         |        |             |
   | -8492 | Average |       |     62 |      37 |      4 |         103 |
   +-------+---------+-------+--------+---------+--------+-------------+

Huitema                 Expires February 1, 2020               [Page 31]
Internet-Draft                  RFC-Eval                       July 2019

   On average, the total delay appears to be a bit more than four month,
   but the average is skewed by the extreme values encountered for
   [RFC8492].  If we exclude that RFC from the computations, the average
   delay drops to a just a bit more than 3 months: about 2 months for
   the preparation, a bit more than one month for the Auth48 phase, and
   4 days for the publishing.

   Of course, these delays vary from RFC to RFC.  To try explain the
   causes of the delay, we compute the correlation factor between the
   observed delays and 4 plausible explanation factors:

   o  The number of pages in the document,

   o  The amount of copy edit, as discussed in (#copy-editing),

   o  Whether or not an IANA action was required.

   We find the following values:

            +-------------+----------+---------+-------------+
            | Correlation | RFC edit | Auth 48 | Edit(total) |
            +-------------+----------+---------+-------------+
            | Nb pages    |     0.50 |   -0.04 |        0.21 |
            |             |          |         |             |
            | Copy-Edit   |     0.42 |    0.24 |        0.45 |
            |             |          |         |             |
            | IANA        |    -0.13 |    0.26 |        0.15 |
            +-------------+----------+---------+-------------+

   None of these indicate strong correlations.  The greater number of
   pages will tend to increase the preparation delay, but it does not
   appear to impact the Auth48 delay at all.  The amount of copy editing
   also tend to increase the the preparation delay somewhat and the
   Auth48 delay a little.  The presence or absence of IANA action has
   very ittle correlation with the delays.

   We also observe that the Auth48 delay varies much more than the
   preparation delay, with a standard deviation of 20 days for Auth48
   versus 10 days for the preparation delay.  In theory, Auth48 is just
   a final verification: the authors receive the document prepared by
   the RFC production center, and just have to give their approval, or
   maybe request a last minute correction.  The name indicates that this
   is expected to last just two days, but in average it lasts more than
   a month.

   We tested a variety of hypotheses that might explain the duration of
   AUTH48 by computing the correlation coefficients between various

Huitema                 Expires February 1, 2020               [Page 32]
Internet-Draft                  RFC-Eval                       July 2019

   properties of the RFC and the production delays.  The results are
   listed in the following table:

   +----------------------+----------------+---------+-----------------+
   | Correlation          | RFC production | Auth 48 | RFC Edit(total) |
   +----------------------+----------------+---------+-----------------+
   | Nb pages             |            0.5 |   -0.04 |            0.21 |
   |                      |                |         |                 |
   | Copy-Edit            |           0.42 |    0.24 |            0.45 |
   |                      |                |         |                 |
   | IANA action          |          -0.13 |    0.26 |            0.15 |
   | requested            |                |         |                 |
   |                      |                |         |                 |
   | Nb drafts            |           0.19 |    -0.3 |           -0.17 |
   |                      |                |         |                 |
   | Nb Authors           |            0.4 |   -0.04 |            0.16 |
   |                      |                |         |                 |
   | WG delay             |           0.03 |   -0.16 |           -0.15 |
   +----------------------+----------------+---------+-----------------+

   The results show that there is no simple answer.  The number of
   pages, the required amount of copy editing and to a very small extent
   the number of drafts can help predict the production delay, but there
   is no obvious predictor for the Auth 48 delay.  In particular, there
   is no numerical evidence that the number of authors influences the
   Auth48 delay, or that authors who have spent a long time working on
   the document in the working group somehow spend even longer to answer
   questions during Auth48 - if anything, the numerical results point in
   the opposite direction.

   After asking the authors of the RFC in the sample why the AUTH48
   phase took a long time, and we got three explanations:

   1- Some RFC have multiple authors in multiple time zones.  This slows
   down the coordination required for approving changes.

   2- Some authors found some of the proposed changes unnecessary or
   undesirable, and asked that they be reversed.  This required long
   exchanges between authors and editors.

   3- Some authors were not giving high priority to AUTH48 responses.

   As mentioned above, we were not able to verify these hypotheses by
   looking at the data.

Huitema                 Expires February 1, 2020               [Page 33]
Internet-Draft                  RFC-Eval                       July 2019

4.4.  Copy editing

   We can assess the amount of copy editing applied to each published
   RFC by comparing the text of the draft approved for publication and
   the text of the RFC.  We do expect differences in the "boilerplate"
   and in the IANA section, but we will also see differences due to copy
   editing.  Assessing the amount of copy editing is subjective, and we
   do it using a scale of 1 to 4:

   1- Minor editing

   2- Editing for style, such as capitalization, hyphens, that versus
   which, and expending all acronyms at least one.

   3- Editing for clarity in addition to style, such as rewriting
   ambiguous sentences and clarifying use of internal references.  For
   Yang models, that may include model corrections suggested by the
   verifier.

   4- Extensive editing.

   The following table shows that with about half of the RFC required
   editing for style, and the other half at least some editing for
   clarity.

Huitema                 Expires February 1, 2020               [Page 34]
Internet-Draft                  RFC-Eval                       July 2019

                       +------+--------+-----------+
                       |  RFC | Status | Copy Edit |
                       +------+--------+-----------+
                       | 8411 | Info   |         2 |
                       |      |        |           |
                       | 8456 | Info   |         4 |
                       |      |        |           |
                       | 8446 | PS     |         3 |
                       |      |        |           |
                       | 8355 | Info   |         2 |
                       |      |        |           |
                       | 8441 | PS     |         2 |
                       |      |        |           |
                       | 8324 | ISE    |         2 |
                       |      |        |           |
                       | 8377 | PS     |         3 |
                       |      |        |           |
                       | 8498 | Info   |         3 |
                       |      |        |           |
                       | 8479 | ISE    |         1 |
                       |      |        |           |
                       | 8453 | Info   |         2 |
                       |      |        |           |
                       | 8429 | BCP    |         2 |
                       |      |        |           |
                       | 8312 | Info   |         2 |
                       |      |        |           |
                       | 8492 | ISE    |         3 |
                       |      |        |           |
                       | 8378 | Exp    |         2 |
                       |      |        |           |
                       | 8361 | PS     |         2 |
                       |      |        |           |
                       | 8472 | PS     |         2 |
                       |      |        |           |
                       | 8466 | PS     |         3 |
                       |      |        |           |
                       | 8362 | PS     |         3 |
                       |      |        |           |
                       | 8468 | Info   |         3 |
                       +------+--------+-----------+

   This method of assessment does not take into account the number of
   changes proposed by the editors and eventually rejected by the
   authors, since these changes are not present in either the final
   draft or the published RFC.  It might be possible to get an
   evaluation of these "phantom changes" from the RFC Production Center.

Huitema                 Expires February 1, 2020               [Page 35]
Internet-Draft                  RFC-Eval                       July 2019

4.5.  Independent Series

   Out of 20 randomly selected RFC, 3 were published through the
   "independent series".  One is an independent opinion, another a
   description of a non-IETF protocol format, and the third was
   [RFC8492], which is a special case.  Apart from this special case,
   the publication delays were significantly shorter for the Independent
   Stream than for the IETF stream.  This seems to indicate that the
   Independent Series is functioning as expected.

   The authors of these 3 RFC are regular IETF contributors.  This
   pbservation motivated a secondary analysis of all the RFC published
   in the "independent" stream in 2018.  There are 14 such RFC: 8507,
   8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351,
   8328 and 8324.  (RFC 8367 and 8369 were published on 1 April 2018.)
   We can ask whether the authors of these RFC are these outsiders, part
   of a "wider community" or are people who are also contributing to the
   IETF.  The overwhelming response is, "insiders".  Pretty much all the
   authors are or were involved in the IETF, many of them with a
   prominent track record.  There are just 2 exceptions, a single RFC in
   which only 3 of the 5 authors are well associated with the IETF.

5.  Next steps

   This draft is not really complete.  We have obtained feedback from
   many authors but not all.  We should also get a review from the RFC
   Production Center.  We may want to find a better way of looking at
   citations than simple queries on Google Scholar.

   Even with those limitations, the exercise shows some promise, and
   also shows the interest of doing more studies.  For example, one of
   the plausible questions is whether the IETF impact is increasing or
   decreasing over time.  We could do that by repeating the statistical
   sampling analysis for previous years, for example 2008 and 1998.

6.  Security considerations

   This draft does not specify any protocol.

   We might want to analyze whether security issues were discovered
   after publication of specific standards.

7.  IANA Considerations

   This draft does not require any IANA action.

   Peliminary analysis does not indicate that IANA is causing any
   particular delay in the publication process.

Huitema                 Expires February 1, 2020               [Page 36]
Internet-Draft                  RFC-Eval                       July 2019

8.  Acknowledgements

   Many thanks to the authors of the selected RFC who were willing to
   provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah
   Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini,
   Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak
   Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos
   Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei
   Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao
   Weiguo, and Li Yizhou.

9.  Informative References

   [RFC3797]  Eastlake 3rd, D., "Publicly Verifiable Nominations
              Committee (NomCom) Random Selection", RFC 3797,
              DOI 10.17487/RFC3797, June 2004,
              <https://www.rfc-editor.org/info/rfc3797>.

   [RFC6455]  Fette, I. and A. Melnikov, "The WebSocket Protocol",
              RFC 6455, DOI 10.17487/RFC6455, December 2011,
              <https://www.rfc-editor.org/info/rfc6455>.

   [RFC8312]  Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
              R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
              RFC 8312, DOI 10.17487/RFC8312, February 2018,
              <https://www.rfc-editor.org/info/rfc8312>.

   [RFC8324]  Klensin, J., "DNS Privacy, Authorization, Special Uses,
              Encoding, Characters, Matching, and Root Structure: Time
              for Another Look?", RFC 8324, DOI 10.17487/RFC8324,
              February 2018, <https://www.rfc-editor.org/info/rfc8324>.

   [RFC8355]  Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R.
              Shakir, "Resiliency Use Cases in Source Packet Routing in
              Networking (SPRING) Networks", RFC 8355,
              DOI 10.17487/RFC8355, March 2018,
              <https://www.rfc-editor.org/info/rfc8355>.

   [RFC8361]  Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu,
              "Transparent Interconnection of Lots of Links (TRILL):
              Centralized Replication for Active-Active Broadcast,
              Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361,
              DOI 10.17487/RFC8361, April 2018,
              <https://www.rfc-editor.org/info/rfc8361>.

Huitema                 Expires February 1, 2020               [Page 37]
Internet-Draft                  RFC-Eval                       July 2019

   [RFC8362]  Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and
              F. Baker, "OSPFv3 Link State Advertisement (LSA)
              Extensibility", RFC 8362, DOI 10.17487/RFC8362, April
              2018, <https://www.rfc-editor.org/info/rfc8362>.

   [RFC8377]  Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent
              Interconnection of Lots of Links (TRILL): Multi-Topology",
              RFC 8377, DOI 10.17487/RFC8377, July 2018,
              <https://www.rfc-editor.org/info/rfc8377>.

   [RFC8378]  Moreno, V. and D. Farinacci, "Signal-Free Locator/ID
              Separation Protocol (LISP) Multicast", RFC 8378,
              DOI 10.17487/RFC8378, May 2018,
              <https://www.rfc-editor.org/info/rfc8378>.

   [RFC8410]  Josefsson, S. and J. Schaad, "Algorithm Identifiers for
              Ed25519, Ed448, X25519, and X448 for Use in the Internet
              X.509 Public Key Infrastructure", RFC 8410,
              DOI 10.17487/RFC8410, August 2018,
              <https://www.rfc-editor.org/info/rfc8410>.

   [RFC8411]  Schaad, J. and R. Andrews, "IANA Registration for the
              Cryptographic Algorithm Object Identifier Range",
              RFC 8411, DOI 10.17487/RFC8411, August 2018,
              <https://www.rfc-editor.org/info/rfc8411>.

   [RFC8429]  Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and
              RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429,
              October 2018, <https://www.rfc-editor.org/info/rfc8429>.

   [RFC8441]  McManus, P., "Bootstrapping WebSockets with HTTP/2",
              RFC 8441, DOI 10.17487/RFC8441, September 2018,
              <https://www.rfc-editor.org/info/rfc8441>.

   [RFC8446]  Rescorla, E., "The Transport Layer Security (TLS) Protocol
              Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,
              <https://www.rfc-editor.org/info/rfc8446>.

   [RFC8453]  Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for
              Abstraction and Control of TE Networks (ACTN)", RFC 8453,
              DOI 10.17487/RFC8453, August 2018,
              <https://www.rfc-editor.org/info/rfc8453>.

   [RFC8455]  Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
              and S. Banks, "Terminology for Benchmarking Software-
              Defined Networking (SDN) Controller Performance",
              RFC 8455, DOI 10.17487/RFC8455, October 2018,
              <https://www.rfc-editor.org/info/rfc8455>.

Huitema                 Expires February 1, 2020               [Page 38]
Internet-Draft                  RFC-Eval                       July 2019

   [RFC8456]  Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
              and S. Banks, "Benchmarking Methodology for Software-
              Defined Networking (SDN) Controller Performance",
              RFC 8456, DOI 10.17487/RFC8456, October 2018,
              <https://www.rfc-editor.org/info/rfc8456>.

   [RFC8466]  Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG
              Data Model for Layer 2 Virtual Private Network (L2VPN)
              Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October
              2018, <https://www.rfc-editor.org/info/rfc8466>.

   [rfc8468]  Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V.
              Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for
              the IP Performance Metrics (IPPM) Framework", RFC 8468,
              DOI 10.17487/RFC8468, November 2018,
              <https://www.rfc-editor.org/info/rfc8468>.

   [RFC8472]  Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport
              Layer Security (TLS) Extension for Token Binding Protocol
              Negotiation", RFC 8472, DOI 10.17487/RFC8472, October
              2018, <https://www.rfc-editor.org/info/rfc8472>.

   [RFC8479]  Mavrogiannopoulos, N., "Storing Validation Parameters in
              PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018,
              <https://www.rfc-editor.org/info/rfc8479>.

   [RFC8492]  Harkins, D., Ed., "Secure Password Ciphersuites for
              Transport Layer Security (TLS)", RFC 8492,
              DOI 10.17487/RFC8492, February 2019,
              <https://www.rfc-editor.org/info/rfc8492>.

   [RFC8498]  Mohali, M., "A P-Served-User Header Field Parameter for an
              Originating Call Diversion (CDIV) Session Case in the
              Session Initiation Protocol (SIP)", RFC 8498,
              DOI 10.17487/RFC8498, February 2019,
              <https://www.rfc-editor.org/info/rfc8498>.

Author's Address

   Christian Huitema
   Private Octopus Inc.
   Golfcourse Rd
   Friday Harbor  WA 98250
   U.S.A

   Email: huitema@huitema.net

Huitema                 Expires February 1, 2020               [Page 39]