Low Extra Delay Background Transport (LEDBAT)
draft-ietf-ledbat-congestion-10

[Ballot discuss]
1. Since "LEDBAT is an experimental delay-based congestion control mechanism", I was wondering when the LEDBAT is not appropriate.
And I was very surprised not to see a reference to RFC 6297.  Specifically http://tools.ietf.org/html/rfc6297#section-2.2, "Potential Issues with Delay-Based Congestion Control for LBE Transport".
A sentence such as the following, from RFC 6297, is particularly important to mention: "this makes such protocols highly sensitive to eventual changes in the end-to-end route during the lifetime of the flow [Mo99]."

2. Then I was wondering: what about ECMPs with different delays?
Is LEDBAT appropriate or not? I could not deduced from the document if you keep the list of delay per host, or per transport.
If per transport, LEDBAT is appropriate
If per host IP address, LEDBAT might have some issues in case of significant different delays in the ECMPs
A new section about LEDBAT applicability would be welcome

3. how can I monitor that LEDBAT doesn't do any harm or does actually the job?
I understand that LEDBAT is experimental, but also deployed, but how do we monitor the impact? What if LEDBAT misbehaves: how do we notice?
And I don't see in the charter that you're going to address the manageability and operational aspect in a different document.

[Ballot comment]
OLD:
  The
  International Telecommunication Union's (ITU's) Recommendation G.114
  defines a delay of 150 ms to be acceptable for most user voice
  applications.

NEW:
  The
  International Telecommunication Union's (ITU's) Recommendation G.114
  defines a one way delay of 150 ms to be acceptable for most user voice
  applications.

[Ballot discuss]
I would be somewhat more comfortable with a "warning label", perhaps
as a new section 2.3, that explicitly describes safe deployment
scenarios, identifies and specifies crucial parameters for safe
operation, and explains what experimental results will be produced.

Before requiring any new text, I have a few questions that do not
require any immediate action on the part of the authors.  These
questions may result in an actionable Discuss position.

1. As this document suggests an ongoing experiment that will be run on
the Internet, I'd like to know if the WG chairs and the document
authors see any situation in which deployment of this experimental
protocol could be harmful to the operation of the Internet.

2. Are there specific operational parameters that are crucial to the
safe operation of this protocol?

3. Is there a plan for explicit experimentation to gather results that
will allow this protocol to be published on the Standards Track?

[Ballot comment]
Minor editorial issue: I don't understand this sentence at the end of
section 4.2.1:

  As closer the extra delay gets to the TARGET value, as slower
  LEDBAT will increase the window.

[Ballot discuss]

There are a lot of timing assumptions and many specific references to NTP.
The draft should go to the TICTOC and NTP WGs with a request for their
review of these aspects of the draft.

Detailed comments below:


  Applications.  Thus the extra delay induced by LEDBAT must be below
  150 ms to reduce impact on delay-sentive applications.
SB> I find this a suprising comment given that earlier in the document
SB> you state that LEDBAT is not aimed at this class of application.
SB> Please can the authors provide clarity.

=====

A.2.  Clock skew

  The clock skew manifests in a linearly changing error in the time
  estimate.  For a given pair of clocks, the difference in skews is the
  skew of the one-way delay estimate.  Unlike the offset, this no
  longer cancels in the computation of the queuing delay estimate.  On
  the other hand, while the offset could be huge, with some clocks off
  by minutes or even hours or more, the skew is typically small.  For
  example, NTP is designed to work with most clocks, yet it gives up
  when the skew is more than 500 parts per million (PPM).  Typical
  skews of clocks that have never been trained seem to often be around
  100-200 PPM. 

SB> This needs evidence and qualification

Previously trained clocks could have 10-20 PPM skew due
  to temperature changes.

SB> This also needs evidence and qualification

A 100-PPM skew means accumulating 6
  milliseconds of error per minute.  The base delay updates mostly
  takes care of clock skew unless the skew is unusually high or extreme
  values have been chosen for TARGET and BASE_HISTORY so that the clock
  skew in BASE_DELAY minutes is larger than the TARGET.

SB> This needs to be reviewed by some NTP experts.

[Ballot comment]
Given that the base delay is constant,

SB> It's not completely constant in the face of routing changes

======

  implementation as a reference:
  o  Skew correction with faster virtual clock:
SB> This is the first time you have introduced the reader to a virtual
SB> clock.

[Ballot comment]
A nit:
-- Section 2 --
The second sentence looks like it needs to be split in two after "however", perhaps with a colon.


Mail to Stanislav Shalunov is bouncing.  If you can't get a current address, you should probably remove him from the front-page authors list, and move him to a "Contributors" section to avoid delays during AUTH48.

====================================================
My original DISCUSS, now recorded as a comment for posterity:

I understand that the shepherd says that the algorithm "requires further experimentation and targeted deployments before it is ready for standards track."  But the shepherd also says that there are "several independent implementations" that include a product deployment, that there's been a lot of experimentation and that "parameterization and associated tradeoffs are well understood and documented," that "we got enough (independent) experimental data that guided the design decisions," and that it's been reviewed by experts from TCPM and ICCRG (who I presume are happy with it).  All this over a development period of a year and a half.

It makes me wonder what experimentation is still needed, such that this can't be sent out a Proposed Standard.  In general, have we really gotten to where "Proposed Standard" means "final and perfect"?  For this specific document, is there really not enough data at this point to recommend greatly expanded deployment?  Does this really still have to be carefully rolled out here and there in controlled experiments?  Is it intended that this will eventually become a Proposed Standard?  What data will you collect from limited experimentation at this point, that you haven't already gotten?  How will we know when the experiment is ready to be turned into a Proposed Standard (or declared a failure), how far from that point are we now, and how will be make sure the experiment progresses?

--------
Response from Rolf Winter:

We have seen a large deployment by a single company. While this means that there indeed is massive deployment, it doesn't mean we have (enough) independent data to say it is always safe, efficient, reliable etc. The company in question says they have deployed it with "good results" (no other indicators or data really). I personally feel that this is not enough and (as far as I am concerned) not enough independent experimental results. In particular, the WG made decisions that changed the initial design that was brought to the IETF which is what has been run in the wild I believe.

Generally, I know what you mean when you say that this smells like Proposed Standard rather than Experimental. We should aim to bring good proposals faster to standards track, however this is congestion control and the IETF was always cautious when it came to congestion control (you could argue this as being positive or negative of course). So I think experimental here is the right way to go for now. We have just seen implementations from other people and I would like them to experiment as well (hopefully sharing more real data with the community).

Once we decide to go standards track we would also need to define a framing format. I'd say we should do these two things at the same time. The experiment I believe is over, once people that have implemented the congestion control part are actually asking for a framing format. This hasn't happened as of now.

--------
Further response from Wes Eddy:

One of the open questions that I think would warrant further understanding before going Standards Track is discussed in the Applicability Section 2.2, and summarized at the end:

  Further study is required to fully understand the behaviour of LEDBAT
  with non-drop-tail, non-FIFO queues.

In my opinion, that's one of the important unknowns.  It's related to the question of what happens when queues are very small.

Another question that has received a lot of discussion lately is whether the LEDBAT target delay is really low enough that the people doing competing real-time media flows are totally happy with it, how to know whether we've got this balance right, and how low it can really go without suffering from other issues.

I think Rolf explained well that we tend to be pretty conservative about advancing CC specifications, traditionally.  Look at the progression of 2001 (PS) in 1997 to 5681 (DS) in 2009 as an example! (and of course SS and CA predate RFC 2001 by quite a bit as well).

[Ballot comment]
A nit:
-- Section 2 --
The second sentence looks like it needs to be split in two after "however", perhaps with a colon.


Mail to Stanislav Shalunov is bouncing.  If you can't get a current address, you should probably remove him from the front-page authors list, and move him to a "Contributors" section to avoid delays during AUTH48.

[Ballot discuss]
The answer to this might well be "We're OK as we are," but I want to discuss it:

I understand that the shepherd says that the algorithm "requires further experimentation and targeted deployments before
it is ready for standards track."  But the shepherd also says that there are "several independent implementations" that include a product deployment, that there's been a lot of experimentation and that "parameterization and associated tradeoffs are well understood and documented," that "we got enough (independent) experimental data that guided the design decisions," and that it's been reviewed by experts from TCPM and ICCRG (who I presume are happy with it).  All this over a development period of a year and a half.

It makes me wonder what experimentation is still needed, such that this can't be sent out a Proposed Standard.  In general, have we really gotten to where "Proposed Standard" means "final and perfect"?  For this specific document, is there really not enough data at this point to recommend greatly expanded deployment?  Does this really still have to be carefully rolled out here and there in controlled experiments?  Is it intended that this will eventually become a Proposed Standard?  What data will you collect from limited experimentation at this point, that you haven't already gotten?  How will we know when the experiment is ready to be turned into a Proposed Standard (or declared a failure), how far from that point are we now, and how will be make sure the experiment progresses?

[Ballot comment]
A nit:
-- Section 2 --
The second sentence looks like it needs to be split in two after "however", perhaps with a colon.

[Ballot comment]
One security/2119 question and a bunch of nits on what's a good
document.

section 7 - you say a protocol using LEDBAD "should"
authentiate timestamp and delay fields. Is that a 2119 SHOULD?
If so, then someone who turns up with an I-D that uses LEDBAT
but has no MTI authentication might get DISCUSS comments on
that basis. Is that what you want? If not, then perhaps
s/should/ought/ will reduce the scope for possible future
confusion. You could also mean that if using (D)TLS then
the timestamp and acks SHOULD be protected, but that
its ok for a protocol that has no MTI security (heaven
forbid:-) to use LEDBAT.

(See also the recent endless discussion on case for
2119 keywords on ietf-discuss if you've time to spare:-)

I don't mind how you choose to handle this, since its
really an issue to be handled when LEDBAT is used,
but think that this could usefully be clarified here.

nits:

- list in 2.1 ends with a comma - just checking that a
point 4 wasn't dropped off the end of the list

- 2.2 uses "tail-drop" and "non-drop-tail" - would
"non-tail-drop" be better there?

- p8, there's no mandatory-to-implement FILTER() - is that
ok? You give a bunch of options, but don't say that one of
them MUST be implemented.

- p9, says a maximum value MAY be placed on CTO, then gives a
MUST about that value. Just checking that its intended to be
ok to implement with no maximum CTO at all.

- 4.2.2, a reference to the limited experimented with BT nodes
would be nice (I've a few such comments, the added references
would be good I think if someone's reading this in 10 years
time, which they may do.)

- 4.3 a reference to the G.114 doc would be nice (incl. the
version since that can change and the 150ms recommendation
might also)

- 4.3, references to the DNA and uTorrent implementations
would be nice

- 5.3, typo s/worse case/worst case/

- p16, a reference for NTP would be nice in Appendix A

- p17, typo "condersiderations"

- A.3, earlier you mentioned two BT implementations but
here you talk about *the* BT implementation - which do
you mean? And a reference would be nice.

[Ballot comment]
I am happy to see this document being published.  I only have a minor, non-blocking, comment.

In several places within the document, it says that LEDBAT is "no more aggressive than TCP".  Given the wide range of TCP congestion control algorithms, I think it would be good to indicate which ones are being compared to LEDBAT.  I assume the document is referring to those congestion control algorithms that conform to RFC 5681.

State changed to In Last Call from Last Call Requested

The following Last Call Announcement was sent out:

From: The IESG

To: IETF-Announce

CC:

Reply-To: ietf@ietf.org

Subject: Last Call:  (Low Extra Delay Background Transport (LEDBAT)) to Experimental RFC





The IESG has received a request from the Low Extra Delay Background

Transport WG (ledbat) to consider the following document:

- 'Low Extra Delay Background Transport (LEDBAT)'

  as an Experimental RFC



The IESG plans to make a decision in the next few weeks, and solicits

final comments on this action. Please send substantive comments to the

ietf@ietf.org mailing lists by 2012-05-07. Exceptionally, comments may be

sent to iesg@ietf.org instead. In either case, please retain the

beginning of the Subject line to allow automated sorting.



Abstract





  LEDBAT is an experimental delay-based congestion control algorithm

  that attempts to utilize the available bandwidth on an end-to-end

  path while limiting the consequent increase in queueing delay on the

  path.  LEDBAT uses changes in one-way delay measurements to limit

  congestion that the flow itself induces in the network.  LEDBAT is

  designed for use by background bulk-transfer applications; it is

  designed to be no more aggressive than TCP congestion control and to

  yield in the presence of any competing flows when latency builds,

  thus limiting interference with the network performance of the

  competing flows.









The file can be obtained via

http://datatracker.ietf.org/doc/draft-ietf-ledbat-congestion/



IESG discussion can be tracked via

http://datatracker.ietf.org/doc/draft-ietf-ledbat-congestion/ballot/





No IPR declarations have been submitted directly on this I-D.

1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)? Why
is this the proper type of RFC? Is this type of RFC indicated in the
title page header?

Experimental. The draft proposes a new algorithm for lower than best-effort transport and requires further experimentation and targeted deployments before
it is ready for standards track.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary


This document describes an experimental congestion algorithm for building a lower than best-effort transport. The algorithm attempts to use the residual capacity on an end-to-end path while limiting queuing delay. The algorithm sense increase in one-way delay measurements to yield in the presence of any competing flows thus limiting interference with the network performance of competing flows.


Working Group Summary


We reached consensus on the basic algorithm a long while back while most of the later revisions and discussions were around parameterization and choosing the right defaults. This is expected as this is an experimental algorithm and data needs to guide these decisions rather than theory. In the end we got enough (independent) experimental data that guided the design decisions.


Document Quality

There are several independent implementations including BitTorrent itself which pioneered this algorithm and ship it in their product. A number of universities and more recently Apple prototyped and shared results during the last call. I am confident that parameterization
and associated tradeoffs are well understood and documented.


Personnel

Who is the Document Shepherd? Who is the Responsible Area
Director?

Murari Sridharan (muraris@microsoft.com) is the Document Shepherd.
Wesley Eddy (wes@mti-systems.com) is the AD.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd. If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

I have been a co-chair right from the start and have reviewed this version and all of the previous versions and believe the current version is ready for publication.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?


No. The algorithm from the beginning has gone through thorough reviews including the final version. The chairs also forwarded the document to ICCRG and may folks from ICCRG are also part of the LEDBAT WG.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.


This document should be reviewed by both TCPM and ICCRG as there are broader implications. Several folks who are part of LEDBAT are also part of these other WG and I am also co-chair for ICCRG. The final version of the document has been cross-posted to the relevant WGs. I therefore have no concerns here.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

None. Key concerns have been captured in the doc.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Yes I believe so.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it?


I believe the consensus is quite strong. Several people who are experts in the area commented during the life time of the algorithm development and also specifically during the last call. A document addressing these concerns was posted back to the group and elicited no further comments.


(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

== Unused Reference: 'RFC6298' is defined on line 718, but no explicit
reference was found in the text
'[RFC6298] Paxson, V., Allman, M., Chu, J., and M. Sargent, "Computi...'


(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

N/A

(13) Have all references within this document been identified as
either normative or informative?

Yes. The following aren’t normative and should be moved to informative.

[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, September 2001.

[RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU
Discovery", RFC 4821, March 2007.


(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?
No.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in the
Last Call procedure.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).


IANA section exists but no considerations required for this document.


(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

N/A

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.
N/A