Skip to main content

Last Call Review of draft-ietf-teas-rfc3272bis-24

Request Review of draft-ietf-teas-rfc3272bis
Requested revision No specific revision (document currently at 27)
Type Last Call Review
Team Transport Area Review Team (tsvart)
Deadline 2023-07-11
Requested 2023-06-27
Authors Adrian Farrel
I-D last updated 2023-07-19
Completed reviews Secdir Last Call review of -24 by Shawn M Emery (diff)
Artart Last Call review of -24 by Rich Salz (diff)
Genart Last Call review of -24 by Behcet Sarikaya (diff)
Tsvart Last Call review of -24 by Bob Briscoe (diff)
Intdir Telechat review of -24 by Brian Haberman (diff)
Rtgdir Early review of -21 by John Drake (diff)
Assignment Reviewer Bob Briscoe
State Completed
Request Last Call review on draft-ietf-teas-rfc3272bis by Transport Area Review Team Assigned
Posted at
Reviewed revision 24 (document currently at 27)
Result Ready w/issues
Completed 2023-07-19
This document has been reviewed as part of the transport area review team's
ongoing effort to review key IETF documents. These comments were written
primarily for the transport area directors, but are copied to the document's
authors and WG to allow them to address any issues raised and also to the IETF
discussion list for information.

When done at the time of IETF Last Call, the authors should consider this
review as part of the last-call comments they receive. Please always CC if you reply to or forward this review.

Last call review of

The draft gives a taxonomy of traffic engineering (TE) practices, techniques
and systems, including a review of recent IETF TE projects. It is a bis to
RFC3272, which it obsoletes.

This review categorizes comments into

IMO, the comments and the nits all need to be fixed as well, but it all depends
how much effort you want to put into perfecting this.

I haven't pointed out nits already raised by other reviewers.
And it might be hard to believe, but I have tried to limit the number of
comments I've raised!



1. The "Congestion Problem"
From a transport area perspective, there is one particiularly glaring omission.
A very large majority of Internet traffic is either capacity-seeking or at
least adaptive to available capacity. This traffic intentionally induces
congestion at the path bottleneck, which is 'good congestion', because it
maximizes capacity utilization and minimizes completion time. So when this
draft repeatedly says that the goal of TE is to combat "The Congestion
Problem", it needs to explain why one part of the IETF is trying to induce
congestion while another (this draft) is trying to combat it.

The explanation is that most network operators design their networks with one
node per-customer (or per-customer-site) as the path bottleneck (or two nodes,
if dual-homed). Then this node (typically the multi-service edge or equivalent)
is where operators can focus deployment of traffic management and control
functions including service differentiation, while other nodes can be
overprovisioned so that they either do not need these functions at all, or they
only need much-simplified functions that complement the primary controls at the
edge node.

Once this context has been explained, the goal of TE is indeed to /avoid/
congestion at all these other nodes, while the goal of endpoints is to /induce/
congestion at their bottleneck node (but only when they have something to send
or receive - the rest of the time they are idle).

Each occurrence of 'congestion problem' will then need to be qualified. Eg:

* "Clearly, congestion is highly undesirable."

* "Congestion is one of the most significant problems in an operational IP

* "If traffic from a source to a destination exceeds the capacity of a link
along the shortest path, the link (and hence the shortest path) becomes
congested while a longer path between these two nodes may be under-utilized."
[Given latency is important to many / most applications, if throughput is
sufficient, it would be wrong to 'solve' this 'problem' by using the longer
path. The solution would be to minimize the delay that results from congestion
by using the latest queue management techniques.]

1.1 Definition of Congestion
Quoting 3 instances of similar wording:

* "...traffic incident on the resource exceeds its output capacity over an
interval of time."

* "...for an interval of time, the arrival rate of packets exceeds the output
capacity of the resource."

* "...sustained overload over an interval of time."

These statements define just the extreme of congestion, not all congestion. If
the input exceeded the output for a sustained interval of time, the queue would
very quickly overflow the buffer and there would be sustained high levels of
loss. That is sustained overload, not just 'congestion'.

Congestion controlled traffic ensures that the input and the output are roughly
matched on average, alternating which is highest at a fast timescale. This is
congestion, even though the average queue remains stable. The more congestion
controlled /flows/ there are, the faster the queue cycles, so the loss fraction
is higher. But input and output are still matched on average (contrary to the
definition given).

Even if there is a high proportion of unresponsive traffic alongside the
congestion controlled traffic, input will not exceed output over a sustained
interval unless the sum of the unresponsive traffic /alone/ exceeds capacity.
The normal percentage of unresponsive traffic in today's Internet is probably
under 1% (since QUIC became prevalent, it has not been so easy to measure how
much UDP traffic is unresponsive).




2. Current Practice?

§5.1 on "IETF projects related to TE" needs to say that not all these practices
have been widely adopted in practice, and some are still too immature to know
whether they will be adopted. Otherwise, on reaching §8 "Contemporary TE
practices", an obvious question arises: "Well, what were the previous 56 pages
about?" There are sometimes hints in the wording when a technology is not yet
adopted or deployed, e.g. the use of phrases like 'can be useful', 'may be
used', but I guess it's hard to pass judgement on likely deployment.

Content Distribution (§5.2) is the only TE technique included that is not
within the IETF project section. I suggest the following should also be

* ECMP. This is described rather disparagingly in §6.2 under routing
recommendations as if it is not good enough. However it is widely used with
n-stage Clos topologies (with n=2 or higher), precisely because it is
considered good enough (i.e. cost-effective) by many major operators.

§8 says "Service providers apply many of the TE mechanisms described in this
document..." I think it also needs to be said that many service providers do


§1. Intro
"...a preponderance of Internet traffic tends to originate in one autonomous
system and terminate in another," This assertion (inherited from RFC3272) needs
an up-to-date reference. I thought the opposite had been true for a decade or
more, but I have no hard measurement evidence other than Arbor's study in 2010
(Craig Labovitz et al, ACM CCR), which found that the majority of inter-domain
traffic was flowing to CDNs, and I figured one could assume that most CDN
content would then be served multiple times intra-domain.

§1.1 What is Internet TE?

It might be useful to give examples of practices that are /not/ TE. For
instance, "Other functions that regulate the flow of traffic through the
network" surely includes endpoint congestion control algorithms (CCAs), which I
don't think this definition intended to include. I was surprised to find that
capacity planning is categorized as TE - I thought TE was what you do within
capacity contraints. Similarly, I didn't think queue management and scheduling
would be categorized as TE. Given the document doesn't discuss different types
of scheduling at all, and its discussion of AQM is outdated (see later), it
doesn't seem wise to have cast the scope so wide. Also, although these
definitions are preceded by 'In this document' it might be worth saying what
some other definitions of TE exclude. For instance, until reading this draft, I
thought TE was defined as: "Traffic engineering (TE) is a process whereby a
network operator can engineer the paths used to carry traffic flows that vary
from those chosen automatically by the routing protocol(s) in use in that same
network." [Tom Nadeau, "Offline Traffic Engineering", MPLS Network Management

§1.2 Components of Traffic Engineering

This new section seems at odds with the very similar existing section "Traffic
Engineering Process Models" (§3) and particularly §3.1, which opens with "The
key components of the traffic engineering process model are as follows." Why
was it necessary to add §1.2 with similar scope, but different content? Unlike
§3 it doesn't list 'Measurement' or 'Analysis' as a component of TE, which
surely can't be correct. And it implies that resource reservation is the only
path steering approach.

"Examples of resources are bandwidth, buffers, and queues,..."
A queue is not a resource. The buffer is the resource, and the queue uses it.

"...rate-shaping mechanisms that are typically supported via queuing." In
aggregates, using queuing for rate shaping has become less acceptable than
using dropping nowadays.

On a related note, traffic mapping is first mentioned in §6.3. Should it not be
mentioned earlier as one of the components of TE?

§1.3 Scope

It seemed odd to call the subject of the draft 'Internet TE', then define the
scope as intra-domain, given Internet means Internetwork. It might be worth
explaining that this is intended to mean 'TE of the Internet service' not 'TE
of the Internet'.

The draft focuses nearly exclusively on MPLS as the transport technology
(Ethernet transport is mentioned a couple of times, but only in passing). It
might be worth saying this in the scope section.

§2.4.1 Combating the Congestion Problem

Under short timescale, a lengthy passage about AQM is provided (and no other
examples of short timescale technologies are given). This seems out of place at
this point, where no other technology is described in such depth. It would seem
more consistent to have a subsection on AQM in §5, and refer forward to it from
here. Having said that, the text on AQM is outdated in three respects: RED, TCP
and LQD.

a) RFC7567 (which is a BCP and cited in this section) effectively deprecates
RED, as follows:

   "With an appropriate set of parameters, RED is an effective algorithm.
   However, dynamically predicting this set of parameters was found to
   be difficult.  As a result, RED has not been enabled by default, and
   its present use in the Internet is limited.  Other AQM algorithms
   have been developed since RFC 2309 was published, some of which are
   self-tuning within a range of applicability.  Hence, while this memo
   continues to recommend the deployment of AQM, it no longer recommends
   that RED or any other specific algorithm is used by default.  It
   instead provides recommendations on IETF processes for the selection
   of appropriate algorithms, and especially that a recommended
   algorithm is able to automate any required tuning for common
   deployment scenarios."

However, it is understandable that this draft needs to introduce RED, because
it is the basis of WRED, which this text is working towards introducing as part
of Diffserv AF. So it would be perhaps best to say sthg like:

"RFC7567 recommends self-tuning AQM algorithms like those that the IETF has
since published [RFC8290, RFC8033, RFC8034, RFC9332], but RED is still
appropriate for links with stable bandwidth, if configured carefully."

b) TCP is no longer an appropriate byword for 'responsive traffic', and UDP is
no longer a byword for unresponsive traffic, both given the growing prevalence
of QUIC over UDP (and of SCTP, DCCP). Pls search the draft for multiple

c) Even at the time RFC3272 was written, it says LQD was theoretical. If a
deployed ref would be preferred, I suggest AFD, which was implemented by cisco,
at least, and is still available AFAICT. Pan, R.; Breslau, L.; Prabhaker, B. &
Shenker, S. "Approximate Fairness Through Differential Dropping" ACM SIGCOMM
Computer Communication Review, 2003, 33, 23-40

§5.1. Overview of IETF Projects Related to Traffic Engineering

After §5.1.1 on Intserv (or after the section on scalability within it), it
would surely be worth mentioning Pre-Congestion Notification (PCN) , which solves the scaling
problems of Intserv by using measurement-based admission control (and flow
termination to handle failures) between edge-nodes. Nodes between the edges of
the internetwork have no per-flow operations and the edge nodes can use RSVP
per-flow or per-aggregate. It was implemented by a number of equipment vendors.

Also, in §5.1.5 on DETNET, it would be worth mentioning that DETNET would
suffer from the same scaling problems described in the Intserv section, but
DETNET's domain of applicability is considered small enough for this to be

§ Differentiated Services

"The Diffserv model deals with traffic management issues on a per hop basis.
... Other TE capabilities, such as capacity management (including routing
control), are also required in order to deliver acceptable service quality in
Diffserv networks"

The above-quoted para is problematic:
i) Diffserv is not solely per-hop (which contradicts the complementary mix of
domain edge and per-hop functions explained 2 paras earlier). ii) Routing
control is not /required/ to deliver acceptable service quality - other
techniques, e.g. liberal provisioning, can preserve service after shortest path
reroutes around failures.

§ (Layer-4) Transport-Based TE

When the draft says no support for ATSSS splitting has yet been developed for
QUIC, it would be worth explaining why (e2e cryptographic protection), and
possibly referencing multipath QUIC [draft-ietf-quic-multipath]. It seems
rather odd to say so much about QUIC (which ATSSS does not support) and so
little about MPTCP (which ATSSS does use).

§ Network Slicing

This section seems out of scope or at best aspirational - should it be deleted?
It admits itself that "IETF network slices are not, of themselves, TE
constructs. However, a network operator that offers IETF network slices is
likely to use many TE tools in order to manage their network and provide the
services." Further, it doesn't point to any work on how this might be done,
particularly what information visibility would be necessary to coordinate
multiple slices.

§ Flow Measurement

The RTFM WG concluded in Oct 2000. Should the draft not discuss IPFIX (the open
standards development of Cisco's Netflow), which ran from 2001-2015? See the
comparison with RTFM at Since that
comparison, IPFIX was developed a lot further too; see .

§ Endpoint Congestion Management (and the omission of a section on
multipath L4 transports)

§2.3.1 says endpoint congestion control is not in primary scope. But, surely,
if the draft includes this fairly outdated example of purely endpoint
coordination across flows, there should be a full sub-section on multipath
transport protocols, which are currently used by in-network control as well as
endpoint control (rather than subordinating multipath within the section on
ATSSS, which is just one in-network example of the use of multipath
transports)? Then the ATSSS section could cross-refer to the new multipath
subsection instead of having to include it.

The idea of multipath L4 transport was originally developed as an improvement
over existing TE techniques, whether deployed solely on endpoints, or with
in-network control. At minimum, the original rationale for adding a multipath
capability to L4 transport protocols should be referenced:

    Wischik, D.; Handley, M. & Bagnulo Braun, M. The Resource Pooling Principle
    SIGCOMM Comput. Commun. Rev., ACM, 2008, 38, 47-52

When I was in BT, an ex-colleague, the late Peter Key, calculated that
in-network traffic engineering would become redundant, if at least about 6% of
traffic used multipath at L4. (6% is from memory 'cos I can't find his paper on
it - pls don't quote it.)

§6.1. Generic Non-functional Recommendations

Stability: I suspect this para might be talking about flap, but I don't believe
I've seen the problem spelled out, so perhaps it should be here. That is, when
endpoint CCAs (or TE systems in neighbouring domains) interact with TE such
that, when the TE of one domain moves an aggregate, CCAs rapidly restore the
original imbalance, possibly causing the TE system to flap.

§9. Security Considerations

Shouldn't it be said here that external control interfaces (e.g. ALTO and the
other approaches in §5.1.2) have to trade off providing flexibility to
customers with opening up control of a network's internals to potentially
malicious actors.

General Comment

<RANT> In lists of important traffic characteristic (as in the definition of
QoS) pls consider replacing 'jitter' with '99th percentile delay' or another
high percentile e.g. 99.9th. Jitter was only relevant when many end devices
were analogue. Once the vast majority of devices have memory buffers, the only
relevant delay metrics characterize the tail of the distribution. In contrast,
jitter is overwhelmingly driven by the shape of the body of the delay
distribution, which bears no relation to the tail. Because jitter does not and
cannot characterize the seriousness of the actual delay that a buffered
receiver will play out, it just confuses everyone into seeing problems where
there are none, and missing where the real problems are. </RANT>



§1.1 What is Internet TE?

"...utilizing network resources without waste": too strong; how about "without
excessive waste"?

"while reacting to the real-time statistical behavior of traffic" -> "while
reacting to statistical measures of the real-time behavior of traffic"

(There's a similar problem in § with the phrase "statistical packet

in the later case -> latter

§1.2 Components of Traffic Engineering

Standard TE solutions -> TE solutions

§1.4. Terminology

Please include explanations of the following terms, which have confusable
alternative meanings:

* 'global' is invariably used (17 occurrences) in the sense of domain-wide
(although it clearly means globe-wide in phrases like 'the global Internet
infrastructure', "global network provider" and "globally interconnected
network"). The phrase 'global synchronization' is an exception to both cases.

* 'end-to-end' is used in the sense of edge-to-edge, contrary to the common
IETF (or at least transport and application area) use of the phrase meaning
application endpoint to application endpoint.

* 'transport' is generally used to mean below IP, as one would expect in a TE
document. But it is also used to describe L4 protocols, e.g. the title and
content of § "Transport-Based TE" and in § "Endpoint Congestion
Management". I suggest 'layer-4 transport' is used in these latter cases.

And some comments on existing definitions:

* Effective bandwidth - although the definition given is not incorrect it is
not as precise as the mathematical definition, so perhaps a reference should be
added, e.g.

    F. Kelly. Notes on effective bandwidths. In Stochastic Networks: Theory and
    Applications. Oxford University Press, 1996.

* Hot-spot - better defined as an element or sub-system in a considerably
higher state of congestion /than others/.

* Inter-domain is defined, but intra-domain is not (perhaps both are obvious
and neither is needed?).

* Offline/online TE are defined as "exists outside of/within the network", but
surely they're defined by when they operate, not where (e.g. online TE can be
located outside the network, e.g. SDN).

* Traffic flow: It defines flow as between 2 endpoints, and says a common
classification for a flow is a 5-tuple. However, wherever 'flow' is used in the
body of the document, I'm pretty sure it is more likely to be intended to mean
an aggregate flow. So the way this definition is worded has great potential for

Flow-size distribution.
On a related point, it would be useful to explain that the very large majority
of 5-tuple flows are very brief (mice - indeed single-packet flows massively
predominate). So a number of TE techniques are designed to shift elephant flows
around, or to shift aggregates likely to contain elephants. The practicality of
numerous technologies described in this draft depends heavily on the definition
of 'flow'.

* southbound - perhaps needs a definition?

§2.2 Network Domain Context

"This requirement is clarified in [RFC2475] which also provides an architecture
for Differentiated Services (Diffserv)." Suggest 'also' is removed.

§2.4 Solution Context

"A collection of online and possibly offline tools and mechanisms for
measurement, characterization, modeling, and control of traffic, and control
over the placement and allocation of network resources, as well as control over
the mapping or distribution of traffic onto the infrastructure."

This list needs to be split, 'cos measurement cannot be offline.

§2.4.1 Combating the Congestion Problem

"Many of these adaptive schemes rely on measurement systems." -> "These
adaptive schemes rely on measurement systems." [How could an adaptive scheme
not rely on measurement?]

"RED provides congestion avoidance which is not worse than traditional
Tail-Drop (TD) queue management." not worse -> better [I don't think the
intention was to damn with faint praise].

"RED reduces the possibility of global synchronization where retransmission
bursts become synchronized across the whole network" Global synchronization
only means synchronization between all the flows sharing the same bottleneck,
not the whole network. Also it's primarily about synchronization of the
sawtoothing window variations of each flow; retransmissions will not
synchronize unless the paths all have the same RTT.

"All the policies described above for the long and medium time scales can be
categorized as being reactive." Given the shorter the timescale, the more it is
likely that a solution will be reactive, this odd choice of sentence conveys
the opposite impression, even though it is strictly not incorrect. For
instance, long timescale activities like capacity expansion certainly /can/ be
reactive in theory, but normally they are preventative.

§2.5 Implementation and Operational Context

"The operational context of Internet TE is characterized by constant changes
that occur at multiple levels of abstraction." I think this intended to say
multiple levels of granularity [something can be described or modelled at a
level of abstraction, but surely real changes do not occur at a level of
abstraction]. Similarly, in § 3.1 "Measurement in support of the TE function
can occur at different levels of abstraction." -> granularity

§4.1 Time-Dependent Versus State-Dependent Versus Event-Dependent

"learning models, as in the success- to-the-top (STT) method" [needs a ref &
perhaps a brief description]

"a fully functional TE system is likely to use all aspects of time-dependent,
state-dependent, and event-dependent methodologies as described in Section
4.1." [Shouldn't this point be made in §4.1?]

§4.3.2. Considerations for Software Defined Networking

"...SDN control plane often determines the end-to-end path ..."
[end-to-end implies more than intra-domain - seems too strong]

§4.6. Open-Loop Versus Closed-Loop

"feedback information may be in the form of historical information"
[Surely that would not be described as closed loop?]

§ Application-Layer Traffic Optimization

"...allows a network to publish its network information such as network
locations, costs between them at configurable granularities, and end-host
properties to network applications."

This sentence is hard to parse. Perhaps change the order and/or flag the items
in the list with a), b), c).

§ Network Virtualization and Abstraction

"statistical packet bandwidth" [does this mean some form of effective


"RSVP has been extended to reserve resources for aggregation of flows"
[Cite RFC3175?]


"...the paths of LSPs" -> LSPs [the P already stands for path]

"To determine the paths for P2MP LSPs, selection of the branch points (based on
capabilities, network state, and policies) is key." [This problem is left
dangling. Is there at least a reference giving a solution?]

§ Generalized MPLS (GMPLS)

"TE extensions to MPLS (see Section" ->

"These additions impact basic LSP properties: ..."
[Again, this problem is left dangling. Is there at least a reference giving
solutions to the ensuing list of apparently fundamental problems?]

§ Segment Routing

" context (network wide)" [this potentially gives the impression of an
Internet-wide lookup capability]

"BIER-TE does not of itself offer traffic engineering..." -> BIER-TE does not
offer a complete traffic engineering system... [Rather misleading - perhaps
better would be to move the sentence from the next para here: "...steers the
traffic within the network and forms an element of a traffic engineering

§6. Recommendations for Internet Traffic Engineering

The order of some of the sub-sections seems odd (e.g. 'measurement' and
'traffic mapping' after 'routing') and contrary to more logical ordering of the
TE process model in §3.

§6.1. Generic Non-functional Recommendations

"...a TE system should remain functional as the network expands with regard to
the number of routers and links, and with respect to the traffic volume."
[traffic volume -> number of flows (I don't believe the same number of flows
but carrying more volume would impact TE scaling at all)]

§6.4 Measurement Recommendations

hot-spot -> hot spot [consistent with 2 other occurrences]

§6.5. Policing, Planning, and Access Control

"This is a simple way to check that the actual traffic volumes are consistent
with the planned volumes." [check -> enforce/ensure]

§6.6. Network Survivability

"Network capacity reserved in one layer to provide protection and restoration
is not available to carry traffic in a higher layer: it is not visible as spare
capacity in the higher layer." [Unclear whether these are statements of fact or
recommendations. Perhaps 'is' -> 'should be' (twice)]

§7. Inter-Domain Considerations

a are -> are

"it is generally considered inadvisable for one domain to permit a control
process from another domain to influence the routing and management of traffic
in its network." [Surely this is inescapable, if TE in one domains moves
traffic to a different ingress of the next domain. Or is this intended to mean
'Don't open up an explicit control interface for other domains"?]

Could mention that L4 multipath transport protocols (whether controlled by
endpoints or in-network ) were designed to shift traffic between domains (and
they are doing so).

§8. Overview of Contemporary TE Practices in Operational IP Networks

This section presents rather a large wall of unbroken text. Navigation markers
would be useful, perhaps based on the list in the 2nd para (altho I couldn't
see them all in the text).

§13. Informative References

[Floyd94] -> [RFC3168]

§A.2 This Document

§ & § are missing.

General (multiple sections)

I found a number of cases of repetition - probably just a symptom of age (of
the draft, not the editor ;)

* The definition of congestion was given 3 times, as already listed earlier;

Repeated explanation of the distinction between reactive and

* "...can be both pro-active and reactive. In the pro-active case, the TE
control system takes preventive action..."

* Reactive Versus Preventive Congestion Management Schemes (the main bullet
item on this distinction)

* Network performance optimization can be corrective or perfective. In
corrective optimization,... In perfective...

* Prescriptive TE can be further categorized as either corrective or
perfective. Corrective TE prescribes ... Perfective TE...

(BTW, no hyphen in proactive.)

Extensions to link-state routing protocols are repeatedly listed and explained:

* "Examples of protocol extensions used to advertise network link state
information are defined in [RFC5305], [RFC6119], [RFC7471], [RFC8570], and

* "taking into consideration the prevailing network state as advertised by IGP
extension for IS-IS in [RFC5305], for OSPFV2 in [RFC3630], and for OSPFv3 in

* "[RFC5305] describes the extensions to the Intermediate System to
Intermediate System (IS-IS) protocol to support TE, similarly [RFC3630]
specifies TE extensions for OSPFv2, and [RFC5329] has the same description for

* "A number of enhancements to the link state IGPs allow them to distribute
additional state information required for constraint-based routing. The
extensions to OSPF are described in [RFC3630], and to IS-IS in..."

I found the number of sentences that gratuitously contained 'X or not X'
started to irritate. A taxonomy will naturally divide practice into
alternatives, but there were a number of cases where such phrases were not used
to divide up the taxonomy, but seemed to be just woffle that could be removed
completely without subtracting anything. Some examples (some are only
marginally gratuitous):

Some TE solutions rely on these elements to a lesser or greater extent.
This determination may be made at a very coarse or very fine level.
Metrics that provide quantitative or qualitative measures
may allow the settings of the traffic control mechanisms to be manipulated by
external or internal entities Delivery requirements of a specific set of
packets may be specified explicitly or implicitly. derivation of solutions
which may be implicitly or explicitly formulated This process model may be
enacted explicitly or implicitly An SLA may explicitly or implicitly specify a
Traffic Conditioning Agreement using a set of shared or dedicated network