Internet Engineering Task Force
INTERNET-DRAFT
TE Working Group
                                          Daniel O. Awduche
January 2000                              UUNET (MCI Worldcom)

                                          Angela Chiu
                                          AT&T

                                          Anwar Elwalid
                                          Lucent Technologies

                                          Indra Widjaja
                                          Fujitsu Network Communications

                                          Xipeng Xiao
                                          Global Crossing


              A Framework for Internet Traffic Engineering

                    draft-ietf-tewg-framework-00.txt


Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   To view the list Internet-Draft Shadow Directories, see
   http://www.ietf.org/shadow.html.
















Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 1]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


Abstract

   This memo describes a framework for Traffic Engineering (TE) in the
   Internet.  The framework is intended to promote better understanding
   of the issues surrounding traffic engineering in IP networks, and to
   provide a common basis for the development of traffic engineering
   capabilities for the Internet.  The framework explores the
   principles, architectures, and methodologies for performance
   evaluation and performance optimization of operational IP networks.
   The optimization goals of traffic engineering seek to enhance the
   performance of IP traffic while utilizing network resources
   economically, efficiently, and reliably. The framework includes a set
   of generic requirements, recommendations, and options for Internet
   traffic engineering.  The framework can serve as a guide to
   implementors of online and off-line Internet traffic engineering
   mechanisms, tools, and support systems. The framework can also help
   service providers in devising traffic engineering solutions for their
   networks.


Table of Contents

    1.0 Introduction
       1.1 What is Internet Traffic Engineering?
       1.2 Scope
       1.3 Terminology
    2.0 Background
       2.1 Context of Internet Traffic Engineering
       2.2 Network Context
       2.3 Problem Context
          2.3.1 Congestion and its Ramifications
       2.4 Solution Context
          2.4.1 Combating the Congestion Problem
       2.5 Implementation and Operational Context
    3.0 Traffic Engineering Process Model
       3.1 Components of the Traffic Engineering Process Model
       3.2 Measurement
       3.3 Modeling and Analysis
       3.4 Optimization
    4.0 Historical Review and Recent Developments
       4.1 Traffic Engineering in Classical Telephone Networks
       4.2 Evolution of Traffic Engineering in Packet Networks
          4.2.1 Adaptive Routing in ARPANET
          4.2.2 Dynamic Routing in the Internet
          4.2.3 ToS Routing
          4.2.4 Equal Cost MultiPath
       4.3 Overlay Model
       4.4 Constraint-Based Routing
       4.5 Overview of Recent IETF Projects Related to Traffic
   Engineering
          4.5.1 Integrated Services
          4.5.2 RSVP
          4.5.3 Differentiated Services
          4.5.4 MPLS



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 2]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


          4.5.5 IP Performance Metrics
          4.5.6 Flow Measurement
          4.5.7 Endpoint Congestion Management
       4.6 Overview of ITU Activities Related to Traffic Engineering
    5.0 Taxonomy of Traffic Engineering Systems
       5.1 Time-Dependent Versus State-Dependent
       5.2 Offline Versus Online
       5.3 Centralized Versus Distributed
       5.4 Local Versus Global
       5.5 Prescriptive Versus Descriptive
       5.6 Open-Loop Versus Closed-Loop
    6.0 Requirements for Internet Traffic Engineering
       6.1 Generic Requirements
       6.2 Routing Requirements
       6.3 Traffic Mapping Requirements
       6.4 Measurement Requirements
       6.5 Network Survivability
          6.5.1 Survivability in MPLS Based Networks
       6.6 Content Distribution (Webserver) Requirements
       6.7 Off-line Traffic Engineering Support Systems
       6.8 Traffic Engineering in Diffserv Environments
    7.0 Multicast Considerations
    8.0 Inter-Domain Considerations
    9.0 Conclusion
    10.0 Security Considerations
    11.0 Acknowledgments
    12.0 References
    13.0 Authors' Addresses


1.0 Introduction


   This memo describes a framework for Internet traffic engineering.
   The intent is to articulate the general issues, principles and
   requirements for Internet traffic engineering; and where appropriate
   to provide recommendations, guidelines, and options for the
   development of online and off-line Internet traffic engineering
   capabilities and support systems.

   The framework can assist vendors of networking hardware and software
   in developing mechanisms and support systems for the Internet
   environment that support the traffic engineering function. The
   framework can also help service providers in devising and
   implementing traffic engineering solutions for their networks.

   The framework provides a terminology for describing and understanding
   common Internet traffic engineering concepts.  The framework also
   provides a taxonomy of known traffic engineering styles.  In this
   context, a traffic engineering style abstracts important aspects from
   a traffic engineering methodology. Traffic engineering styles can be
   viewed in different ways depending upon the specific context in which
   they are used and the specific purpose which they serve. The
   combination of styles and views results in a natural taxonomy of



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 3]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   traffic engineering systems.

   Although Internet traffic engineering is most effective when applied
   end-to-end, the initial focus of this framework document is intra-
   domain traffic engineering (that is, traffic engineering within a
   given autonomous system). However, in consideration of the fact that
   a preponderance of Internet traffic tends to be inter-domain (that
   is, they originate in one autonomous system and terminate in
   another), this document provides an overview of some of the aspects
   that pertain to inter-domain traffic engineering.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119.

   This draft is preliminary and will be reviewed and revised over time.


1.1. What is Internet Traffic Engineering?


   Internet traffic engineering is defined as that aspect of Internet
   network engineering that deals with the issue of performance
   evaluation and performance optimization of operational IP networks.
   Traffic Engineering encompasses the application of technology and
   scientific principles to the measurement, characterization, modeling,
   and control of Internet traffic [AWD1, AWD2].

   A major objective of Internet traffic engineering is to enhance the
   performance of an operational network; at both the traffic and
   resource levels. This is accomplished by addressing traffic oriented
   performance requirements, while utilizing network resources
   efficiently, reliably, and economically. Traffic oriented performance
   measures include delay, delay variation, packet loss, and goodput.

   It is worthwhile to emphasize that an important objective of Internet
   traffic engineering is to facilitate reliable network operations
   [AWD1]. Reliable network operations can be facilitated by providing
   mechanisms that enhance network integrity and by embracing policies
   that emphasis network survivability,  so that the vulnerability of
   the network to service outages arising from errors, faults,  and
   failures that occur within the infrastructure can be minimized.

   It is also important to be cognizant of the fact that ultimately,
   what really matters is the performance of the network as seen by end
   users of network services. This crucial aspect should be kept in view
   when developing traffic engineering mechanisms and policies. The
   charateristics that are visible to end users are the emergent
   properties of the network. Emergent properties are the
   characteristics of the network viewed as a whole.

   A significant, but subtle, practical advantage of applying traffic
   engineering concepts to operational networks is that it helps to
   identify and structure goals and priorities in terms of enhancing the



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 4]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   quality of service delivered to end-users of network services, and in
   terms of measuring and analyzing the achievement of these goals.

   The optimization aspects of traffic engineering can be achieved
   through capacity management and traffic management. As used in this
   document, capacity management includes capacity planning, routing
   control, and resource management. Network resources of particular
   interest include link bandwidth, buffer space, and computational
   resources. Likewise, as used in this document, traffic management
   includes traffic conditioning, scheduling, and other functions that
   regulate traffic flow through the network or that arbitrate access to
   network resources between different packets.

   The optimization objectives of Internet traffic engineering should be
   viewed as a continual and iterative process of network performance
   improvement, rather than as a one time goal. Traffic engineering also
   demands continual development of new technologies and new
   methodologies for network performance enhancement.  The optimization
   objectives of Internet traffic engineering may change over time as
   new requirements are imposed, or as new technologies emerge, or as
   new insights are brought to bear on the underlying problems.
   Moreover, different networks may have different optimization
   objectives, depending upon their business models, capabilities, and
   operating constraints. Regardless of the specific optimization goals
   that prevail in any particular environment, for practical purposes,
   the optimization aspects of traffic engineering are ultimately
   concerned with network control.

   Thus, the optimization aspects of traffic engineering can be viewed
   from a control perspective. The control dimension of Internet traffic
   engineering can be pro-active and/or reactive. In the reactive case,
   the control system responds to events that have already transpired in
   the network. In the pro-active case, the control system takes
   preventive action to obviate predicted unfavorable future network
   states, or takes perfective action to induce a more desirable state
   in the future. The control dimension of Internet traffic engineering
   responds at multiple levels of temporal resolution to network events.
   Some aspects of capacity management such as capacity planning
   functions respond at a very coarse temporal level, ranging from days
   to possibly years. The routing control functions operate at
   intermediate levels of temporal resolution, ranging from milliseconds
   to days.  Finally, the packet level processing functions (e.g. rate
   shaping, queue management, and scheduling) operate at very fine
   levels of temporal resolution, responding to the real-time
   statistical characteristics of traffic, ranging from picoseconds to
   milliseconds. The subsystems of Internet traffic engineering control
   include: capacity augmentation, routing control, traffic control, and
   resource control (including control of service policies at network
   elements).  Inputs into the control system include network state
   variables, policy variables, and decision variables.

   For practical purposes, traffic engineering concepts and mechanisms
   must be sufficiently specific and well defined to address known
   requirements, but at the same time must be flexible and extensible to



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 5]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   accommodate unforeseen future demands.

   A major challenge in Internet traffic engineering is the realization
   of automated control capabilities that adapt quickly and at
   reasonable cost to significant changes in network state, while
   maintaining stability.


1.2. Scope


   The scope of this document is intra-domain traffic engineering; that
   is, traffic engineering within a given autonomous system in the
   Internet. The framework will discuss concepts pertaining to intra-
   domain traffic control, including such issues as routing control,
   micro and macro resource allocation, and the control coordination
   problems that arise consequently.

   This document will describe and characterize techniques already in
   use or in advanced development for Internet traffic engineering,
   indicate how they fit together, and identify scenarios in which they
   are useful.

   Although the emphasis is on intra-domain traffic engineering, in
   Section 8.0, however, an overview of the high level considerations
   pertaining to inter-domain traffic engineering will be provided.
   Inter-domain Internet traffic engineering is crucial to the
   performance enhancement of the global Internet infrastructure.

   Whenever possible, relevant requirements from existing IETF documents
   and other sources will be incorporated by reference.


1.3 Terminology


   This subsection provides terminology which is useful for Internet
   traffic engineering. The definitions presented apply to this
   framework document. These terms may have other meanings elsewhere.

     - Baseline analysis:
          A study conducted to serve as a baseline for comparison to the
          actual behavior of the network.

     - Busy hour:
          A one hour period within a specified interval of time
          (typically 24 hours) in which the traffic load in a
          network or subnetwork is greatest.

      - Congestion:
          A state of a network resource in which the traffic incident
          on the resource exceeds its output capacity over an interval
          of time.




Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 6]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


     - Congestion avoidance:
          An approach to congestion management that attempts to obviate
          the occurrence of congestion.

     - Congestion control:
          An approach to congestion management that attempts to remedy
          congestion problems that have already occurred.

     - Constraint-based routing:
          A class of routing protocols that take specified traffic
          attributes, network constraints, and policy constraints into
          account in making routing decisions. Constraint-based routing
          is applicable to traffic aggregates as well as flows. It is a
          generalization of QoS routing.

     - Demand side congestion management:
          A congestion management scheme that addresses congestion
          problems by regulating or conditioning offered load.

     - Effective bandwidth:
          The minimum amount of bandwidth that can be assigned to a flow
          or traffic aggregate in order to deliver 'acceptable service
          quality' to the flow or traffic aggregate.

     - Egress traffic:
          Traffic exiting a network or network element.

     - Ingress traffic:
          Traffic entering a network or network element.

     - Inter-domain traffic:
          Traffic that originates in one Autonomous system and
          terminates in another.

     - Loss network:
          A network that does not provide adequate buffering for
          traffic, so that traffic entering a busy resource within
          the network will be dropped rather than queued.

     - Network Survivability:
          The capability to provide a prescribed level of QoS for
          existing services after a given number of failures occur
          within the network.

     - Off-line traffic engineering:
          A traffic engineering system that exists outside of the
          network.

     - Online traffic engineering:
          A traffic engineering system that exists within the network,
          typically implemented on or as adjuncts to operational network
          elements.

     - Performance measures:



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 7]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


          Metrics that provide quantitative or qualitative measures of
          the performance of systems or subsystems of interest.

     - Performance management:
          A systematic approach to improving effectiveness in the
          accomplishment of specific networking goals related to
          performance improvement.

     - Provisioning:
          The process of assigning or configuring network resources to
          meet certain requests.

     - QoS routing:
          Class of routing systems that selects paths to be used by a
          flow based on the QoS requirements of the flow.

     - Service Level Agreement:
          A contract between a provider and a customer that guarantees
          specific levels of performance and reliability at a certain
          cost.

     - Stability:
          An operational state in which a network does not oscillate
          in a disruptive manner from one mode to another mode.

     - Supply side congestion management:
          A congestion management scheme that provisions additional
          network resources to address existing and/or anticipated
          congestion problems.

     - Transit traffic:
          Traffic whose origin and destination are both outside of
          the network under consideration.

     - Traffic characteristic:
          A description of the temporal behavior or a description of the
          attributes of a given traffic flow or traffic aggregate.

     - Traffic engineering system
          A collection of objects, mechanisms, and protocols that are
          used conjunctively to accomplish traffic engineering
          objectives.

     - Traffic flow:
          A stream of packets between two end-points that can be
          characterized in a certain way. A micro-flow has a more
          specific definition: A micro-flow is a stream of packets with
          a bounded inter-arrival time and with the same source and
          destination addresses, source and destination ports, and
          protocol ID.

     - Traffic intensity:
          A measure of traffic loading with respect to a resource
          capacity over a specified period of time. In classical



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 8]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


          telephony systems, traffic intensity is measured in units of
          Erlang.

     - Traffic matrix:
          A representation of the traffic demand between a set of origin
          and destination abstract nodes. An abstract node can consist
          of one or more network elements.

     - Traffic monitoring:

          The process of observing traffic characteristics at a given
          point in a network and collecting the traffic information for
          analysis and further action.

     - Traffic trunk:
          An aggregation of traffic flows belonging to the same class
          which are forwarded through a common path. A traffic trunk
          may be characterized by an ingress and egress node, and a
          set of attributes which determine its behavioral
          characteristics and requirements from the network.


2.0 Background


   The Internet has quickly evolved into a very critical communications
   infrastructure, supporting significant economic, educational, and
   social activities. At the same time, the delivery of Internet
   communications services has become a very competitive endeavor.
   Consequently, optimizing the performance of large scale IP networks,
   especially public Internet backbones, has become an important
   problem.  Network performance requirements are multidimensional,
   complex, and sometimes contradictory; thereby making the traffic
   engineering problem very challenging.

   The network must convey IP packets from ingress nodes to egress nodes
   efficiently, expeditiously, reliably, and economically. Furthermore,
   in a multiclass service environment (e.g. Diffserv capable networks),
   the resource sharing parameters of the network must be appropriately
   determined and configured according to prevailing policies and
   service models to resolve resource contention issues arising from
   mutual interference between packets traversing through the network.
   Moreover, in multi-class environments, consideration must be given to
   resolving competition for network resources between traffic streams
   belonging to the same service class (intra-class contention
   resolution) and between traffic streams belonging to different
   classes (inter-class contention resolution).


2.1 Context of Internet Traffic Engineering


   The context of Internet traffic engineering pertains to the scenarios
   in which the problems that traffic engineering attempts to solve



Awduche/Chiu/Elwalid/Widjaja/Xiao                               [Page 9]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   manifest. A traffic engineering methodology establishes appropriate
   rules to solve traffic performance problems that occur in a specific
   context. The context of Internet traffic engineering includes:

    (1) A network context which defines the situations in which the
        TE problems occur. The Network context encompasses network
        structure, network policies, network characteristics, network
        constraints, network quality attributes, network optimization
        criteria, etc.

    (2) A problem context which defines the general and concrete
        issues that TE addresses. The problem context encompasses
        identification, abstraction of relevant features,
        representation, formulation, requirements and desirable
        features of solutions, etc.

    (3) A solution context which suggests how to solve the TE
        problems. The solution context encompasses analysis, evaluation
        of alternatives, prescription, and resolution.

    (4) An implementation and operational where the solutions are
        methodologically instantiated. The implementation and
   operational
        context which encompasses planning, organization, and execution.

   In the following subsections, we elaborate on the context of Internet
   traffic engineering.


2.2 Network Context


   IP networks range in size from small clusters of routers situated
   within a given location, to thousands of interconnected routers and
   switches distributed all over the world.

   At the most basic level of abstraction, an IP network can be
   represented as: (1) a constrained system consisting of set of
   interconnected resources which provide transport services for IP
   traffic, (2) a demand system representing the offered load to be
   transported through the network, and (3) a response system consisting
   of network processes, protocols, and related mechanisms which
   facilitate the movement of traffic through the network [see also
   AWD2].

   The network elements and resources may have specific characteristics
   which restrict the way in which they handle the demand. Additionally,
   network resources may be equipped with traffic control mechanisms
   which allow the way in which they handle the demand to be regulated.
   Traffic control mechanisms may also be used to control various packet
   processing activities within the resource, or to arbitrate contention
   for access to the resource by different packets, or to regulate
   traffic behavior through the resource. A configuration management and
   provisioning system may allow the settings of the traffic control



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 10]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   mechanisms to be manipulated by external or internal entities in
   order to constrain or to exercise control over the way in which the
   network element responds to internal and external stimuli.

   The details of how the network provides transport services for
   packets are specified in the policies of the network administrators
   and are installed through network configuration management and
   provisioning systems.  Generally, the types of services provided by
   the network also depends upon the technology and characteristics of
   the network elements, the prevailing policies, as well as the ability
   of the network administrators to translate policies into network
   configurations.

   There are three significant characteristics of contemporary Internet
   networks: (1) they provide real-time services, (2) they have become
   mission critical, and (3) their operating environments are very
   dynamic. The dynamic characteristics of IP networks can be attributed
   in part to fluctuations in demand, to the interaction between various
   network protocols and processes, to the rapid evolution of the
   infrastructure which demands constant insertion of new technologies
   and new network elements, and to transient and persistent impairments
   which occur within the system.

   The most significant function permformed by an IP network is the
   routing of packets from source nodes to destination nodes. Not
   surprisingly, one of the most significant functions performed by
   Internet traffic engineering is the control and optimization of the
   routing function, so as to steer packets in the most effective way
   through the network. As packets are conveyed through the network,
   they contend for the use of network resources. If the arrival rate of
   packets exceed the output capacity of a network resource over an
   interval of time, the resource is said to be congested, and some of
   the arrival packets may be dropped as a result. Congestion also
   increases transit delays, delay variation, and reduces the
   predictability of network service delivery. Thus, congestion is a
   highly undesirable phenomenon. Combating congestion at reasonable
   cost is a major objective of Internet traffic engineering.

   A basic economic premise for packet switched networks in general and
   the Internet in particular is the efficient sharing of network
   resources by multiple traffic streams. One of the fundamental
   challenges in operating a network, especially large scale public IP
   networks, is the need to increase the efficiency of resource
   utilization while minimizing the possibility of congestion.

   Increasingly, the Internet will have to function in the presence of
   different classes of traffic, especially with the advent of
   differentiated services. In practice, a particular set of packets may
   have specific delivery requirements which may be specified explicitly
   or implicitly. Two of the most important traffic delivery
   requirements are (1) capacity constraints which can be expressed
   statistically as peak rates, mean rates, burst sizes, or as some
   deterministic notion of effective bandwidth, and (2) QoS constraints
   which can be expressed in terms of integrity constraints (e.g. packet



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 11]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   loss) and temporal constraints, e.g. timing restrictions for the
   delivery of each packet and timing restrictions for the delivery of
   consecutive packets belonging to the same traffic stream. Packets may
   also be grouped into classes, in such a way that each class may have
   a common set of behavioral characteristics and/or a common set of
   delivery requirements.


2.3 Problem Context


   There are a number of fundamental problems associated with the
   operation of a network described by the simple model of the previous
   subsection. The present subsection reviews the problem context with
   regard to the traffic engineering function.

   One problem concerns how to identify, abstract, represent, and
   measure relevant features of the network which are relevant for
   traffic engineering.

   One particularly important class of problems concerns how to
   explicitely formulate the problems that TE attempts to solve, how to
   identify the requirements on the solution space, how to specify the
   desireable features of good solutions, and how to measure and
   characterize the effectiveness of the solutions.

   Another problem concerns how to measure and estimate relevant network
   state parameters. Effective traffic engineering relies on a good
   estimate of the offered traffic load as well as a view of the
   underlying topology and associated resource constraints.  A network-
   wide view of the topology is also a must for off-line planning.

   Still another problem concerns how to characterize the state of the
   network and how to evaluate its performance under a variety of
   scenarios. There are two aspects to the performance analysis problem.
   One aspect relates to the evaluation of the system level performance
   of the network. The other aspect relates to the evaluation of the
   resource level performance, which restricts attention to the
   performance evaluation of individual network resources. In this memo,
   we shall refer to the system level characteristics of the network as
   the "macro-states" and the resource level characteristics as the
   "micro-states." Likewise, we shall refer to the traffic engineering
   schemes that deal with network performance optimization at the
   systems level as macro-TE and the schemes that optimize at the
   individual resource level as micro-TE. In general, depending upon the
   particular performance measures of interest, the system level
   performance can be derived from the resource level performance
   results using appropriate rules of composition.

   Yet another fundamental problem concerns how to optimize the
   performance of the network. Performance optimization may entail some
   degree of resource management control, routing control, and/or
   capacity augmentation.




Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 12]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   As noted previously, congestion is an undesirable phenomena in
   operational networks. Therefore, we devote the next subsection to the
   issue of congestion and its ramifications within the problem context
   of Internet traffic engineering.


2.3.1 Congestion and its Ramifications


   Congestion is one of the most significant problems in an operational
   IP context. A network element is said to be congested if it
   experiences sustained overload over an interval of time. Almost
   invariably, congestion results in degradation of service quality to
   end users. Congestion control schemes can include demand side
   policies and supply side policies. Demand side policies may restrict
   access to congested resources and/or dynamically regulate the demand
   to alleviate the overload situation. Supply side policies may re-
   allocate network resources by redistributing traffic over the
   infrastructure and/or expand or augment network capacity. In this
   memo, the emphasis is mainly on congestion management schemes that
   fall within the scope of the network, rather than congestion
   management systems that depend on sensitivity and adaptivity from
   end-systems. That is, the focus of this memo with respect to
   congestion management is on those solutions that can be provided by
   control entities operating on the network or by the actions of
   network administrators.


2.4 Solution Context


   The solution context for Internet traffic engineering involves
   analysis, evaluation of alternatives, and choice between alternative
   courses of action.  Generally the solution context is predicated on
   making reasonable inferences about the current or future state of the
   network and possibly making appropriate decisions that may involve a
   preference between alternative sets of action. More specifically, the
   solution context demands good estimates of traffic workload,
   characterization of network state, and possibly a set of control
   actions. Control actions may involve manipulation of parameters
   associated with the routing function, control over tactical capacity
   acquisition, and control over the traffic management functions.

   The following is a subset of the instruments that may be applicable
   to the solution context of Internet TE.

   (1) A set of policies, objectives, and requirements (which may be
       context dependent) for network performance evaluation and
       performance  optimization.

   (2) A collection of online and possibly off-line tools and mechanisms
       for measurement, characterization, modeling, and control
       of Internet traffic and control over the placement and allocation
       of network resources, as well as control over the mapping or



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 13]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


       distribution of traffic onto the infrastructure.

   (3) A set of constraints on the operating environment, the network
       protocols, and the traffic engineering system itself.

   (4) A set of administrative control parameters which may be
       manipulated through a Configuration Management (CM) system. The
   CM
       system itself may include a configuration control subsystem, a
       configuration repository, a configuration accounting subsystem,
       and a configuration auditing subsystem.

   Derivation of traffic characteristics through measurement and/or
   estimation is very useful within the realm of the solution space for
   traffic engineering. Traffic estimates can be derived from customer
   subscription information, from traffic projections, from traffic
   models, or from actual empirical measurements. In order to measure
   and derive traffic matrices at various levels of detail, the
   measurement may be performed at the flow level or at the traffic
   aggregate level.  Measurements at the flow level or on small traffic
   aggregates may be performed at edge nodes, where traffic enters and
   leaves the network [FGLR].

   In order to conduct performance studies and planning on existing or
   future networks, a routing analysis may be performed to determine the
   path(s) which the routing protocols will choose for each traffic
   demand, and the utilization of network resources as traffic is routed
   through the network. The routing analysis needs to capture the
   selection of paths through the network, the assignment of traffic
   across multiple feasible routes , and the multiplexing of IP traffic
   over traffic trunks (if such constructs exists) and over the
   underlying network infrastructure. A topology model for the network
   may be extracted from network architecture documents, or from network
   designs, or from information contained in router configuration files,
   or from routing databases, or from routing tables. Topology
   information may also be derived from servers that monitor network
   state and from servers that perform provisioning functions.

   Routing in operational IP networks can be administratively controlled
   at various level of abstraction, e.g., manipulating BGP attributes;
   manipulating IGP metrics; manipulating  traffic engineering
   parameters, resource parameters, and policy constraints for path
   oriented technologies such as MPLS, etc. Within the context of MPLS,
   the path of an explicit LSP can be computed and established in
   various ways, e.g. (1) manually, (2) automatically online using
   constraint-based routing processes implemented on label switching
   routers, or (3) automatically off-line using a constraint-based
   traffic engineering support systems.


2.4.1 Combating the Congestion Problem


   Minimizing congestion is a significant aspect of traffic engineering.



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 14]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   This subsection gives an overview of the general approaches that have
   been used or proposed to combat congestion problems.

   Congestion management policies can be categorized based upon the
   following criteria (see [YaRe95] for a more detailed taxonomy of
   congestion control schemes): (1) Response time scale, which can be
   characterized as long, medium, or short; (2) reactive versus
   preventive which relates to congestion control and congestion
   avoidance; and (3) supply side versus demand side congestion
   management schemes. These aspects are elaborated upon in the
   following paragraphs.

   (1) Response time scale

   - Long (weeks to months): Capacity planning works over a relatively
   long time scale to expand network capacity based on estimates or
   forecasts of future traffic demand and traffic distribution. Since
   router and link provisioning takes time and are in general expensive,
   these upgrades are typically carried out in the weeks-to-months or
   even years time scale.

   - Medium (minutes to days): Several control policies fall within the
   medium time scale category. Examples include: 1) Adjusting IGP and/or
   BGP parameters to route traffic away or towards certain segment of
   the network; 2) Setting up and/or adjusting some Explicitly-Routed
   Label Switched Paths (ER-LSPs) to route some traffic trunks away from
   possibly congested resources or towards possibly more favorable
   routes; 3) reconfiguring the logical topology of the network to make
   it correlate more closely with the traffic distribution using for
   example some underlying path-oriented technology such as MPLS LSPs,
   ATM PVCs, or optical channel trails (see e.g. [AWD6]).  Many of these
   adaptive medium time scale response schemes rely on a measurement
   system that monitors changes in traffic distribution, traffic shifts,
   and network resource utilization and subsequently provides feedback
   to the online and/or off-line traffic engineering mechanisms and
   tools which employ this feedback information to trigger certain
   control actions to occur within the network. The traffic engineering
   mechanisms and tools can be implemented in a distributed or
   centralized fashion, and may have a hierarchical or flat structure.
   The comparative merits and demerits of distributed and centralized
   control structures for networks are well known. A centralized scheme
   may have global visibility into the network state and may produce
   potentially more optimal solutions. However, centralized schemes are
   prone to single points of failure and may not scale as well as
   distributed schemes. Moreover, the information utilized by a
   centralized scheme may be stale and may not reflect the actual state
   of the network. It is not an objective of this memo to make a
   recommendation between distributed and centralized schemes. This is a
   choice that network administrators must make based on their specific
   needs.

   - Short (picoseconds to minutes): This category includes packet level
   processing functions and events in the order of several round trip
   times. It includes router mechanisms such as passive or active buffer



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 15]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   management which is used to control congestion and/or signal
   congestion to end systems so that they can slow down. One of the most
   popular active management schemes, especially for TCP traffic, is
   Random Early Detection (RED) [FlJa93], which supports congestion
   avoidance by controlling the average queue size. During congestion
   (but before the queue is filled), the RED scheme chooses arriving
   packets to be "marked" according to a probabilistic algorithm which
   takes into account the average queue size. For a router that does not
   utilize explicit congestion notification (ECN) see e.g., [Floy94]),
   the marked packets can simply be dropped to signal the inception of
   congestion to end systems; otherwise, if the router supports ECN,
   then it can set the ECN field in the packet header. Several
   variations of RED have been proposed for use in multiclass
   environments with different drop precedence levels [RFC-2597], e.g.,
   RED with In and Out (RIO) and Weighted RED. It is generally agreed
   that RED provides congestion avoidance performance which is not worse
   than traditional Tail-Drop (TD) (i.e., dropping arriving packets only
   when the queue is full). Importantly, however, RED reduces the
   possibility of global synchronization and improves fairness among
   different TCP sessions. However, RED by itself can not prevent
   congestion and unfairness caused by unresponsive sources, e.g., UDP
   connections, or some misbehaved greedy connections. Other schemes
   have been proposed to improve the performance and fairness in the
   presence of unresponsive connections.  Some of these schemes have
   been proposed as theoretical frameworks and are not typically
   available in existing products. Two such schemes are Longest Queue
   Drop (LQD) and Dynamic Soft Partitioning with Random Drop (RND)
   [SLDC98].

   (2) Reactive versus preventive

   - Reactive (recovery): reactive policies are those that react to
   existing congestion in order to improve it. All the policies
   described in the long and medium time scales above can be categorized
   as being reactive especially if the policies are based on monitoring
   and identifying existing congestion problems and initiating relevant
   actions to ease the situation.

   - Preventive (predictive/avoidance): preventive policies are those
   that take proactive action to prevent congestion based on estimates
   or predictions of potential congestion problems in the future. Some
   of the policies described in the long and medium time scales fall
   under this category. They do not necessarily respond immediately to
   existing congestion problems. Instead they may take into account
   forecasts of future traffic demand and distribution, and may take or
   prescribe actions in order to prevent potential congestion problems
   in the future. The schemes described in short time scale, e.g., RED
   and its variations, ECN, LQD, and RND, are also used for congestion
   avoidance since dropping or marking packets as an early congestion
   notification before queues actually overflow would trigger
   corresponding TCP sources to slow down.

   (3) Supply side versus demand side




Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 16]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   - Supply side: supply side policies are those that seek to increase
   the effective capacity available to traffic in order to control or
   obviate congestion. One way to accomplish this is to minimize
   congestion by having a relatively balanced network. For example,
   capacity planning should aim to provide a physical topology and
   associated link bandwidths that match estimated traffic workload and
   traffic distribution based on forecasting, subject to budgetary and
   other constraints.  However, if actual traffic distribution does not
   match the topology derived from capacity panning (due, for example,
   to forecasting errors or facility constraints), then the traffic can
   be mapped onto the existing topology using routing control mechanisms
   or by modifying the logical topology using path oriented technologies
   (e.g., MPLS, ATM, optical channel trails), or by using some other
   load redistribution mechanisms.

   - Demand side: demand side policies are those that seek to control or
   regulate the offered traffic. For example, some of the short time
   scale mechanisms described earlier (such as RED and its variations,
   ECN, LQD, and RND) as well as policing and rate shaping mechanisms
   attempt to regulate the offered load in various ways. Tariffs may
   also be applied as a demand side instrument. However, to date,
   tariffs have not been used as a means of demand side congestion
   management within the Internet.

   In summary, a variety of mechanisms can be brought to bear to address
   congestion problems in IP networks. These mechanisms may operate at
   multiple time-scales.


2.5 Implementation and Operational Context


   The operational context of Internet traffic engineering is
   characterized by constant change which occur at multiple levels of
   abstraction.  The implementation context demands effective planning,
   organization, and execution. The planning aspects may involve
   determining prior sets of actions in order to achieve desired
   objectives. Organizing involves arranging and assigning
   responsibility to the various components of the traffic engineering
   system and coordinating their activities in order to accomplish the
   desired TE objectives. Execution involves measuring and applying
   corrective or perfective actions to attain and maintain desired TE
   goals.


3.0 Traffic Engineering Process Model(s)


   This section describes a process model that captures the high level
   practical aspects of Internet traffic engineering in an operational
   context. The process model is described in terms of a sequence of
   actions that a traffic engineer, or more generally that a traffic
   engineering system, goes through in order to optimize the performance
   of an operational network (see also [AWD1, AWD2]). Although the



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 17]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   details regarding how traffic engineering is carried out may differ
   from network to network, the process model described here represents
   broad activities which are common to most traffic engineering
   methodologies. This process model may be enacted explicitely or
   implicitely; by an automaton and/or by a human.

   The first phase of the TE process model is to define relevant control
   policies that govern the operation of the network. These policies may
   depend on the prevailing business model, the network cost structure,
   the operating constraints, and one or more optimization criteria, as
   well as other factors.

   The second phase of the process model is a feedback process which
   involves acquiring measurement data from the operational network. If
   empirical data is not readily available from the network, then
   synthetic workloads may be used instead, which reflect either the
   prevailing or the expected workload of the network. Synthetic
   workloads may be derived by estimation or by extrapolation using
   prior empirical data, or by using mathematical models of traffic
   characteristics, or by using some other means.

   The third phase of the process model is to analysis the network state
   and to characterize traffic workload. In general, performance
   analysis may be proactive and/or reactive. Proactive performance
   analysis identifies potential problems that do not exist, but that
   may manifest at some point in the future. Reactive performance
   analysis, on the other hand, identifies existing problems, determines
   their cause through a process of diagnosis, and if necessary
   evaluates alternative approaches to remedy the problem. A number of
   quantitative and qualitative techniques may be used in the analysis
   process, including modeling based analysis and simulation. The
   analysis phase of the process model may involve the following
   actions: (1) investigate the concentration and distribution of
   traffic across the network or relevant subsets of the network, (2)
   identify the characteristics of the offered traffic workload, (3)
   identify existing or potential bottlenecks, and (4) identify network
   pathologies such as ineffective placement of links, single points of
   failures, etc. Network pathologies may result from a number of
   factors such as inferior network architecture, inferior network
   design, configuration problems, and others.  A traffic matrix may be
   constructed as part of the analysis process. Network analysis may
   also be descriptive or prescriptive.

   The fourth phase of the TE process model is concerned with the
   performance optimization of the network. The performance optimization
   phase generally involves a decision process which selects and
   implements a particular set of actions from a choice between
   alternatives.  Optimization actions may include use of appropriate
   techniques to control the offered traffic, or to control the
   distribution of traffic across the network.  Optimization actions may
   also involve increasing link capacity, deploying additional hardware
   such as routers and switches, adjusting parameters associated with
   routing such as IGP metrics and BGP attributes in a systematic way,
   and adjusting traffic management parameters. Network performance



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 18]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   optimization may also involve starting a network planning process to
   improve the network architecture, network design, network capacity,
   network technology, and the configuration of network elements in
   order to accommodate current and future growth.


3.1 Components of the Traffic Engineering Process Model


   As evidenced by the discussion in the previous subsection, the key
   components of the TE process model include a measurement subsystem, a
   modeling and analysis subsystem, and an optimization subsystem. The
   following subsections elaborate on these components as they apply to
   the TE process model.


3.2 Measurement


   Measurement is crucial to the traffic engineering function. The
   operational state of a network can only be conclusively determined
   through measurement. Measurement is also critical to the optimization
   function because it provides feedback data which is used by TE
   control subsystems to adaptively optimize network performance in
   response to events and stimuli that originate within and outside the
   network. Measurement is also needed to ascertain the quality of
   network services and to evaluate the effectiveness of TE policies.
   Experience suggests that measurement is most effective when it is
   applied systematically.

   In developing a measurement system to support the TE function in IP
   networks, the following questions should be considered very
   carefully:  Why is measurement needed in this particular context?
   What parameters are to be measured?  How should the measurement be
   accomplished?  Where should the measurement be performed? When should
   the measurement be performed?  How frequently should the monitored
   variables be measured? What level of measurement accuracy and
   reliability is desirable. What level of measurement accuracy and
   reliability is realistically attainable? To what extent can the
   measurement system permissibly interfere with the monitored network
   components and variables? What is the acceptable cost of measurement?
   To a large degree, the answers to the above questions will determine
   the measurement tools and measurement methodologies that are
   appropriate in any given TE context.

   It is also worthwhile to point out that there is a distinction
   between measurement and evaluation. Measurement provides raw data
   concerning state parameters and variables of monitored elements.
   Evaluation utilizes the raw data to make inferences regarding the
   monitored system.

   Measurement in support of the TE function can occur at different
   levels of abstraction. For example, measurement can be used to derive
   packet level characteristics, flow level characteristics, user or



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 19]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   customer level characteristics, traffic aggregate characteristics,
   component level characteristics, network wide characteristics, etc.


3.3 Modeling and Analysis


   Modeling and analysis are important aspects of Internet traffic
   engineering. Modeling involves constructing an abstract or physical
   representation which depicts relevant traffic and network attributes
   and characteristics.

   Accurate source models for traffic are particularly very useful for
   analysis.  A major research topic in Internet traffic engineering is
   the development of traffic source models that are consistent with
   empirical data obtained from operational networks. Such models should
   also be tractable and amenable to analysis. The topic of source
   models for IP traffic is a research topic and is therefore outside
   the scope of this document; nonetheless its importance cannot be
   over-emphasized.

   A network model is an abstract representation of the network which
   captures relevant network features, attributes, and characteristics,
   such as link and nodal attributes and constraints.  A network model
   may facilitate analysis and/or simulation which can be used to
   predict network performance under various conditions, and also to
   guide network expansion plans.

   Network simulation tools are extremely useful for traffic
   engineering. A good network simulator can be used to mimic and
   visualize network characteristics in various ways under various
   conditions.  For example, a network simulator might be used to depict
   congested resources and hot spots, and to provide hints regarding
   possible solutions to network performance problems. A good simulator
   may also be used to validate the effectiveness of planned solutions
   to network issues without the need to tamper with the operational
   network, or to commence an expensive network upgrade which may not
   achieve the desired objectives. Furthermore, during the process of
   network planning, a network simulator may reveal pathologies such as
   single points of failure which may require additional redundancy, and
   potential bottlenecks and hot spots which may require additional
   capacity. Routing simulators are especially useful. A routing
   simulator may identify planned links which may not actually be used
   to route traffic by the existing routing protocols. Simulators can
   also be used to conduct scenario based and perturbation based
   analysis, as well as sensitivity studies.  Simulation results can be
   used to initiate appropriate actions in various ways. For example, an
   important application of network simulation tools is to investigate
   and identify how best to evolve and grow the network in order to
   accommodate projected future demands.







Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 20]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


3.4 Optimization


   Network performance optimization involves resolving network issues
   into concepts that enable a solution, identifying a solution, and
   implementing the solution. Network performance optimization can be
   corrective or perfective. In corrective optimization, the goal is to
   remedy a problem that has occurred or that is incipient. In
   perfective optimization, the goal is to improve network performance
   even when explicit problems do not exist and are not anticipated.

   Network performance optimization is a continual process, as noted
   previously.  Performance optimization iterations may consist of
   real-time optimization sub-processes and non-real-time network
   planning sub-processes.  The difference between real-time
   optimization and network planning is largely in the relative time-
   scale at they operate and in the granularity of actions.  One of the
   objectives of a real-time optimization sub-process is to control the
   mapping and distribution of traffic over the existing network
   infrastructure to avoid and/or relieve congestion, to assure
   satisfactory service delivery, and to optimize resource utilization.
   Real-time optimization is needed because, no matter how well a
   network is designed, random incidents such as fiber cuts or shifts in
   traffic demand will occur.  When they occur, they can cause
   congestion and other problems to manifest in an operational network.
   Real-time optimization must solve such problems in small to medium
   time-scales ranging from micro-seconds to minutes or hours. Examples
   of real-time optimization include queue management, IGP/BGP metric
   tuning, and using technologies such as MPLS explicit LSPs to change
   the paths of some traffic trunks [XIAO].

   One of the functions of the network planning sub-process is to
   initiate actions to evolve the architecture, technology, topology,
   and capacity of a network in a systematic way. When there is a
   problem in the network, real-time optimization should provide an
   immediate fix. Because of the need to respond promptly, the real-time
   solution may not be the best possible solution.  Network planning may
   subsequently be needed to refine the solution and improve the
   situation.  Network planning is also needed to expand the network to
   support traffic growth and changes in traffic distribution over time.
   As noted previously, the outcome of network planning might be a
   change in the topology and/or capacity of the network.

   It can be seen that network planning and real-time performance
   optimization are mutually complementary activities. A well-planned
   and designed network makes real-time optimization easier, while a
   systematic approach to real-time network performance optimization
   allows network planning to focus on long term issues rather than
   tactical considerations. Systematic real-time network performance
   optimization also provides valuable inputs and insights towards
   network planning.

   Stability is a major consideration in real-time network performance
   optimization. This aspect will be reiterated repeatedly throughout



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 21]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   this memo.


4.0 Historical Review and Recent Developments


   This section presents a brief review of various traffic engineering
   approaches that have been proposed and implemented in
   telecommunications and computer networks. The discussion is not meant
   to be exhaustive, but is primarily intended to illuminate pre-
   existing perspectives and prior art concerning traffic engineering in
   the Internet as well as in legacy telecommunications networks.


4.1 Traffic Engineering in Classical Telephone Networks


   It is useful to begin with a review of traffic engineering in
   telephone networks which often relates to the means by which user
   traffic is steered from the source to the destination.  This
   subsection presents a brief overview of this topic. The book by G.
   Ash [ASH2] contains a detailed description of the various routing
   strategies that have been applied in telephone networks.

   The early telephone network relied on static hierarchical routing,
   whereby routing patterns remained fixed independent of the state of
   the network or time of day. The hierarchy was intended to accommodate
   overflow traffic, improve network reliability via alternate routes,
   and prevent call looping by using strict hierarchical rules.  The
   network was typically over-provisioned since a given fixed route had
   to be dimensioned so that it could carry user traffic during a busy
   hour of any busy day.  Hierarchical routing in the telephony network
   was found to be too rigid with the advent of digital switches and
   stored program control which were able to manage more complicated
   traffic engineering rules.

   Dynamic routing was introduced to alleviate the routing inflexibility
   in the static hierarchical routing so that the network would operate
   more efficiently, thereby resulting in significant economic gains
   [HuSS87].  Dynamic routing typically reduces the overall loss
   probability by 10 to 20 percent as compared to static hierarchical
   routing.  Dynamic routing can also improve network resilience by
   recalculating routes on a per-call basis and periodically updating
   routes.

   There are three main types of dynamic routing in the telephone
   network: time-dependent routing, state-dependent routing (SDR), and
   event dependent routing (EDR).

   In time-dependent routing, regular variations in traffic loads due to
   time of day and seasonality are exploited in pre-planned routing
   tables.  In state-dependent routing, routing tables are updated
   online according to the current state of the network (e.g, traffic
   demand, utilization, etc.).  In event dependent routing, routing



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 22]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   changes are incepted by events, such as call setups encountering
   congested or blocked links, whereupon new paths are searched out
   using learning models.  EDR methods are real-time adaptive, but do
   not require global state information such as is the case with SDR.
   Examples of EDR schemes include the dynamic alternate routing (DAR)
   from BT, the state-and-time dependent routing (STR) from NTT, and the
   success-to-the-top (STT) routing from AT&T.

   Dynamic non-hierarchical routing (DNHR) is an example of dynamic
   routing that was introduced in the AT&T toll network in the 1980's to
   respond to time-dependent information such as regular load variations
   as a function of time.  Time-dependent information in terms of load
   may be divided into three time scales: hourly, weekly, and yearly.
   Correspondingly, three algorithms are defined to pre-plan the routing
   tables.  Network design algorithm operates over a year-long interval
   while demand servicing algorithm operates on a weekly basis to fine
   tune link sizes and routing tables to correct forecast errors on the
   yearly basis. At the smallest time scale, routing algorithm is used
   to make limited adjustments based on daily traffic variations.
   Network design and demand servicing are computed using off-line
   calculations.  Typically, the calculations require extensive search
   on possible routes.  On the other hand, routing may need online
   calculations to handle crankback.  DNHR adopts a "two-link" approach
   whereby a path can consist of two links at most.  The routing
   algorithm presents an ordered list of route choices between an
   originating switch and a terminating switch.  If a call overflows, a
   via switch (a tandem exchange between the originating switch and the
   terminating switch) would send a crankback signal to the originating
   switch which would then select the next route, and so on, until no
   alternative routes are available in which the call is blocked.


4.2 Evolution of Traffic Engineering in Packet Networks


   This subsection reviews related prior work that was intended to
   improve the performance of data networks.  Indeed, optimization of
   the performance of data networks started in the early days of the
   ARPANET. Other early commercial networks such as SNA also recognized
   the importance of performance optimization and service
   differentiation.

   In terms of traffic management, the Internet has been a best effort
   service environment until recently. In particular, very limited
   traffic management capabilities existed in IP networks to provide
   differentiated queue management and scheduling services to packets
   belonging to different classes.

   In the following subsections, we review the evolution of practical
   implementations of traffic engineering mechanisms in IP networks and
   its predecessors.






Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 23]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


4.2.1 Adaptive Routing in ARPANET


   The early ARPANET recognized the importance of adaptive routing where
   routing decisions were based on the current state of the network
   [McQ80].  In the early minimum delay routing approaches, each packet
   was forwarded to its destination along a path for which the total
   estimated transit time is the smallest.  Each node maintained a table
   of network delays, which represented the estimated delay that a
   packet can expect to experience along a given  path toward its
   destination. The minimum delay table was periodically transmitted by
   a node to its neighbors. The shortest path in terms of hop count was
   also propagated to give the connectivity information.

   A drawback of this approach is that dynamic link metrics tend to
   create "traffic magnets" whereby congestion will be shifted from one
   location of a network to another location; essentially creating
   oscillation and instability.


4.2.2 Dynamic Routing in the Internet


   The Internet, which evolved from the APARNET, adopted dynamic routing
   algorithms with distributed control to determine the paths that
   packets should take en-route to their destinations.  The routing
   algorithms themselves are adaptations of shortest path algorithms
   where costs are based on link metrics. In principle, the link metric
   can be based on static or dynamic quantities.  In the static case,
   the link metric may be assigned administratively according to some
   local criteria.  In the dynamic case, the link metric may be a
   function of some congestion measure such as delay or packet loss.

   It was recognized early that static link metric assignment was
   inadequate because it can easily lead to unfavorable scenarios
   whereby some links become congested while some others remain lightly
   loaded.  One of the many reasons for the inadequacy of static link
   metrics is that link metric assignment was often done without
   considering the traffic matrix in the network.  Moreover, the routing
   protocols did not take traffic attributes and capacity constraints
   into account in making routing decisions. The practical implication
   is that traffic concentration is localized in subsets of the network
   infrastructure, potentially causing congestion.  Even if link metrics
   are assigned in accordance with the traffic matrix, unbalanced loads
   in the network can still occur due to a number of reasons, such as:
    - Some resources might not be deployed in the most optimal locations
      from a routing perspective.
    - Forecasting errors in traffic volume and/or traffic distribution.
    - Dynamics in traffic matrix due to the temporal nature of traffic
      patterns, BGP policy change from peers, etc.

   The inadequacy of the legacy Internet interior gateway routing system
   is one of the factors motivating the interest in path oriented
   technologies with explicit routing and constraint-based routing



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 24]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   capability, such as MPLS.


4.2.3 ToS Routing


   In ToS-based routing, different routes to the same destination may be
   selected depending on the Type-of-Service (ToS) field of an IP packet
   [RFC-1349].  The ToS classes may be classified as low delay and high
   throughput.  Each link is associated with multiple link costs, where
   each link cost is used to compute routes for a particular ToS.  A
   separate shortest path tree is computed for each ToS. Since the
   shortest path algorithm has to be run for each ToS, the computation
   may be quite expensive with this approach.  Classical ToS-based
   routing has become outdated as the IP header field has been replaced
   by a Diffserv field. A more serious technical issue with the
   classical TOS based routing concerns the fact that it is difficult to
   perform effective traffic engineering because each class still relies
   exclusively on shortest path routing.


4.2.4 Equal Cost MultiPath


   Equal Cost MultiPath (ECMP) is another technique that attempts to
   address the deficiency in Shortest Path First (SPF) interior gateway
   routing systems [RFC-2178].  In a SPF algorithm, if two or more paths
   to a given destination have the same cost, the algorithm will choose
   one of them.  In ECMP, the algorithm is modified slightly so that if
   two or more equal shortest cost paths exist between two nodes, the
   traffic between the nodes is distributed among the multiple equal-
   cost paths.  Traffic distribution across the equal-cost paths is
   usually done in two ways: 1) packet-based in a round-robin fashion,
   or 2) flow-based using hashing on source and destination IP
   addresses.  Approach 1) can easily cause out-of-order packets while
   approach 2) is dependent on the number and distribution of flows.
   Flow-based load sharing may be unpredictable in an enterprise network
   where the number of flows is relatively small and heterogeneous
   (i.e., hashing may not be uniform), but is generally effective in
   core public networks where the number of flows is very large.

   Because link costs are static and bandwidth constraints are not taken
   into account, ECMP attempts to distribute the traffic as equally as
   possible among the equal-cost paths independent of the congestion
   status of each path.  As a result, given two equal-cost paths, it is
   possible that one of the paths will be more congested than the other.
   Another drawback of ECMP is that load sharing cannot be done on
   multiple paths which have non-identical costs.


4.3 Overlay Model


   In the overlay model, a virtual-circuit network, such as ATM or frame



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 25]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   relay, provides virtual-circuit connectivity between routers that are
   located at the edges of the virtual-cirtuit network.  In this mode,
   two routers that are connected through a virtual circuit see a direct
   adjacency between themselves independent of the physical route taken
   by the virtual circuit through the ATM or frame relay network.  Thus,
   the overlay model essentially decouples the logical topology that
   routers see from the physical topology that the ATM or frame relay
   network manages.  The overlay model enables the network operator to
   perform traffic engineering by re-configuring the virtual circuits so
   that a virtual circuit on a congested physical path can be re-routed
   to a less congested one.

   The overlay model requires the management of two separate networks
   (e.g., IP and ATM) which results in increased operational complexity
   and cost.  In the fully-meshed overlay model, each router would peer
   to every other router in the network. Some of the issues with the
   overlay model are discussed in [AWD2].


4.4 Constrained-Based Routing


   Constrained-based routing pertains to a class of routing systems that
   compute routes through a network subject to satisfaction of a set of
   constraints and requirements. The constraints may be imposed by the
   network and/or by administrative policies. Constraints may include
   bandwidth, delay, and policy instruments such as resource class
   attributes. The concept of constraint-based routing in IP networks
   was first defined in [AWD1] within the context of MPLS traffic
   engineering requirements. Unlike QoS routing which generally deals
   with routing traffic flows in order to QoS prescribed QoS
   requirements, constraint-based routing is applicable to traffic
   aggregates as well as flow and may also take policy restrictions into
   account.


4.5 Overview of Recent IETF Projects Related to Traffic Engineering


   This subsection reviews a number of recent IETF activities that are
   pertinent to Internet traffic engineering.


4.5.1 Integrated Services


   The IETF developed the integrated services model which requires
   resources, such as bandwidth and buffers, to be reserved a priori for
   a given traffic flow to ensure that the quality of service requested
   by the traffic flow is satisfied. The integrated services model
   requires additional components beyond those used in the best-effort
   model such as packet classifiers, packet schedulers, and admission
   control.  A packet classifier is used to identify flows that are to
   receive a certain level of service. A packet scheduler handles the



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 26]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   service of different packet flows to ensure that QoS commitments are
   met.  Admission control is used to determine whether a router has the
   necessary resources to accept a new flow.

   Two services have been defined: guaranteed service [RFC-2212] and
   controlled-load service [RFC-2211]. The guaranteed service can be
   used for applications that require real-time delivery. For this type
   of application, data that is delivered to the application after a
   certain time is generally considered worthless. Thus guaranteed
   service has been designed to provide a firm bound on the end-to-end
   packet delay for a flow.

   The controlled-load service can be used for adaptive applications
   that can tolerate some delay but that are sensitive to traffic
   overload conditions. This type of applications typically function
   satisfactorily when the network is lightly loaded but degrade
   significantly when the network is heavily loaded. Thus, controlled-
   load service has been designed to provide approximately the same
   service as best-effort service in a lightly loaded network regardless
   of actual network conditions. Controlled-load service is described
   qualitatively in that no target values of delay or loss are
   specified.

   The main issue with the Integrated services model has been
   scalability, especially in large public IP networks which may
   potentially have millions of concurrent micro-flows.


4.5.2 RSVP


   RSVP, a soft state signaling protocol, was originally invented as a
   signaling protocol for applications to reserve network resources
   [RFC-2205]. Under RSVP, the sender sends a PATH Message to the
   receiver, specifying the characteristics of the traffic.  Every
   intermediate router along the path forwards the PATH Message to the
   next hop determined by the routing protocol. Upon receiving a PATH
   Message, the receiver responds with a RESV Message to request
   resources for the flow. The RESV message travels to the source in the
   opposite direction along the path through which the PATH message
   traversed. Every intermediate router along the path can reject or
   accept the request of the RESV Message.  If the request is rejected,
   the router will send an error message to the receiver, and the
   signaling process will terminate. If the request is accepted, link
   bandwidth and buffer space are allocated for the flow and the related
   flow state information will be installed in the router.

   One of the issues with the original RSVP specification was
   scalability, because reservations were required for micro-flows, so
   that the amount of state maintained on network increases linearly
   with the number of micro-flows. Recently, however, RSVP has been
   modified and extended in several ways to overcome the scaling
   problems and to enable it to become a versatile signaling protocol
   for IP networks. For example, RSVP has been extended to reserve



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 27]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   resources for aggregation of flows, to set up MPLS explicit label
   switched paths, and to perform other signaling functions within the
   Internet.


4.5.3 Differentiated Services


   The essence of the Differentiated Services (Diffserv) effort within
   the IETF is to allow traffic to be categorized and divided into
   classes, and subsequently to allow each class to be treated
   differently, especially during times when there is a shortage of
   resources such as link bandwidth and buffer space [RFC-2475].

   Diffserv defines the Differentiated Services field (DS field,
   formerly known as TOS octet) and uses it to indicate the forwarding
   treatment a packet should receive [RFC-2474]. Diffserv also
   standardizes a number of Per-Hop Behavior (PHB) groups. Using
   different classification, policing, shaping and scheduling rules,
   several classes of services can be defined.

   In order for a customer to receive Differentiated Services from its
   Internet Service Provider (ISP), it may be necessary for the customer
   to have a Service Level Agreement (SLA) with the ISP. An SLA may
   explicitly or implicitly specify a Traffic Conditioning Agreement
   (TCA) which defines classifier rules as well as metering, marking,
   discarding, and shaping rules.

   At the ingress to a Diffserv network, packets are classified,
   policed, and possibly shaped. When a packet traverses the boundary
   between different Diffserv domains, the DS field of the packet may be
   re-marked according to existing agreements between the domains.

   In Differentiated Services, there are only a finite and limited
   number of service classes that can be indicated by the DS field. The
   main advantage of the Diffserv approach is scalability: Since
   resources are allocated on a per-class basis, the amount of state
   information is proportional to the number of classes rather than to
   the number of application flows.

   It should be evident from the above discussion that the Diffserv
   model essentially deals with traffic management issues on a per hop
   basis. Thus, the Diffserv control model consists of a collection of
   micro-TE control mechanisms. Other traffic engineering capabilities
   such as capacity management, including routing control, are also
   required in Diffserv networks in order to deliver acceptable service
   quality.


4.5.4 MPLS


   MPLS is an advanced forwarding scheme which also includes extensions
   to conventional IP control plane protocols. MPLS extends the Internet



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 28]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   routing model, and enhances packet forwarding and path control
   [RoVC].

   Each MPLS packet has a fixed length label affixed to the header. In a
   non-ATM/FR environment, the header contains a 20-bit label, a 3-bit
   Experimental field (formerly known as Class-of-Service or CoS field),
   a 1-bit label stack indicator and an 8-bit TTL field. In an ATM (FR)
   environment, the header contains only a label encoded in the VCI/VPI
   (DLCI) field.  An MPLS capable router, termed Label Switching Router
   (LSR), examines the label and possibly the experimental field in
   forwarding a packet.

   At the ingress LSRs of an MPLS-capable domain, IP packets are
   classified into forwarding equivalence classes (FECs) and routed
   based on a variety of factors, including e.g. a combination of the
   information carried in the IP header of the packets and the local
   routing information maintained by the LSRs. An MPLS header is then
   appended to each packet according to the notion of forwarding
   equivalence classes. Within an MPLS-capable domain, an LSR will use
   the label prependend to packets as the index into a local next hop
   label forwarding entry (NHLFE). The packet is then processed as
   specified in NHLFE.. The incoming label may be replaced by an
   outgoing label and the packet may be switched to the next LSR. This
   label-switching process is very similar to the label (VCI/VPI)
   swapping process in ATM networks.  Before a packet leaves an MPLS
   domain, its MPLS header is removed. The path through which a FEC
   traverses between an ingress LSRs and an egress LSRs is called a
   Label Switched Path (LSP). The path of an explicit LSP is defined at
   the originating (ingress) node of the LSP. MPLS can use a signaling
   protocol such as RSVP or LDP to set up LSPs.

   MPLS is a very powerful technology for Internet traffic engineering
   because it supports explicit LSPs which allow constraint-based
   routing to be implemented efficiently in IP networks.


4.5.5 IP Performance Metrics


   The IPPM WG has been developing a set of standard metrics that can be
   applied to the quality, performance, and reliability of Internet
   services by network operators, end users, or independent testing
   groups [RFC2330], so that users and service providers have accurate
   common understanding of the performance and reliability of the
   Internet component 'clouds' that they use/provide.  Examples of
   performance metrics include one-way packet loss [RFC2680], one-way
   delay [RFC2679], and connectivity measures between two nodes
   [RFC2678]. Other metrics include second-order measures of packet loss
   and delay.

   Performance metrics are useful for specifying Service Level
   Agreements (SLAs), which are sets of service level objectives
   negotiated between users and service providers, where each objective
   is a combination of one or more performance metrics subject to



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 29]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   constraints.


4.5.6 Flow Measurement


   A flow measurement system enables network traffic flows to be
   measured and analyzed at the flow level for a variety of purposes.
   RTMF has produced an architecture document that defines a method to
   specify traffic flows, and a number of components (meters, meter
   readers, manager) to measure the traffic flows [RFC-2722].  A meter
   observes packets passing through a measurement point, classifies them
   into certain groups, accumulates certain usage data such as the
   number of packets and bytes for each group, and stores the usage data
   in a flow table.  For this purpose, a group may represent a user
   application, a host, a network, a group of networks, any combination
   of the above, etc.  A meter reader gathers usage data from various
   meters so that it can be made available for analysis.  A manager is
   responsible for configuring and controlling meters and meter readers.
   The instructions received by a meter from a manager include flow
   specification, meter control parameters, and sampling techniques.
   The instructions received by a meter reader from a manager include
   the meter's address whose data is to be collected, the frequency of
   data collection, and the types of flows to be collected.


4.5.7 Endpoint Congestion Management


   The work in endpoint congestion management is intended to catalog a
   set of congestion control mechanisms that transport protocols can
   use, and to develop a unified congestion control mechanism across a
   subset of an endpoint's active unicast connections called a
   congestion group.  A congestion manager continuously monitors the
   state of the path for each congestion group under its control, and
   uses that information to instruct a scheduler on how to partition
   bandwidth among the connections of that congestion group.


4.6 Overview of ITU Activities Related to Traffic Engineering


   This section provides an overview of prior work within the ITU-T
   pertaining to traffic engineering in traditional telecommunications
   networks.

   ITU-T Recommendations E.600 [itu-e600], E.701 [itu-e701], and E.801
   [itu-e801] address traffic engineering issues in traditional
   telecommunications networks. Recommendation E.600 provides a
   vocabulary for describing traffic engineering concepts, while E.701
   defines reference connections, Grade of Service (GOS), and traffic
   parameters for ISDN.  Recommendation E.701 uses the concept of a
   reference connection to identify representative cases of different
   types of connections without describing the specifics of their actual



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 30]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   realizations by different physical means. As defined in
   Recommendation E.600, "a connection is an association of resources
   providing means for communication between two or more devices in, or
   attached to, a telecommunication network."  Also, E.600 defines "a
   resource as any set of physically or conceptually identifiable
   entities within a telecommunication network, the use of which can be
   unambiguously determined" [itu-e600].  There can be different types
   of connections as the number and types of resources in a connection
   may vary.

   Typically, different network segments are involved in the path of a
   connection.  For example, a connection may be local, national, or
   international.  The purposes of reference connections are to clarify
   and specify traffic performance issues at various interfaces between
   different network domains.  Each domain may consist of one or more
   service provider networks.

   Reference connections provide a basis to define grade of service
   (GoS) parameters related to traffic engineering within the ITU-T
   framework.  As defined in E.600, "GoS refers to a number of traffic
   engineering variables which are used to provide a measure of the
   adequacy of a group of resources under specified conditions."  These
   GoS variables may be probability of loss, dial tone delay, etc.  They
   are essential for network internal design and operation, as well as
   component performance specification.

   In the ITU framework, GoS is different from quality of service (QoS).
   QoS is the performance perceivable by a user of a telecommunication
   service and expresses the user's degree of satisfaction of the
   service.  GoS is a set of network oriented measures which
   characterize the adequacy of a group of resources under specified
   conditions. On the other hand, QoS parameters focus on performance
   aspects which are observable at the service access points and network
   interfaces, rather than their causes within the network.  For a
   network to be effective in serving its users, the values of both GoS
   and QoS parameters must be related, with GoS parameters typically
   making a major contribution to the QoS.

   To assist the network provider in the goal of improving efficiency
   and effectiveness of the network, E.600 stipulates that a set of GoS
   parameters must be selected and defined on an end-to-end basis for
   each major service category provided by a network.  Based on a
   selected set of reference connections, suitable target values are
   then assigned to the selected GoS parameters, under normal and high
   load conditions.  These end-to-end GoS target values are then
   apportioned to individual resource components of the reference
   connections for dimensioning purposes.


5.0 Taxonomy of Traffic Engineering Systems


   This section presents a short taxonomy of traffic engineering
   systems. A taxonomy of traffic engineering systems can be constructed



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 31]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   based on traffic engineering styles and traffic engineering views.
   Such a classification system is shown below:

    - Time-dependent vs State-dependent
    - Offline vs Online
    - Centralized vs Distributed
    - Local vs Global Information
    - Prescriptive vs Descriptive
    - Open Loop vs Closed Loop

   In the following subsections, these classification systems are
   described in greater detail.


5.1 Time-Dependent Versus State-Dependent


   TE methodologies can be classified into two basic types:  time-
   dependent or state-dependent.  In this framework, all TE schemes are
   considered to be dynamic.  Static TE implies that no traffic
   engineering methodology or algorithm is being applied.

   In the time-dependent TE, historical information based on seasonal
   variations in traffic is used to pre-program routing plans.
   Additionally, customer subscription or traffic projection may be
   used.  Pre-programmed routing plans typically change on a relatively
   long time scale (e.g., diurnal). Time-dependent algorithms make no
   attempt to adapt to random variations in traffic or changing network
   conditions. An example of time-dependent algorithm is a global
   centralized optimizer where the input to the system is traffic matrix
   and multiclass QoS requirements described [MR99].

   State-dependent or adaptive TE adapts the routing plans for packets
   based on the current state of the network. The current state of the
   network gives additional information on variations in actual traffic
   (i.e., perturbations from regular variations ) that could not be
   predicted by using historical information.  An example of state-
   dependent TE that operates in a relatively long time scale is
   constraint-based routing, and an example that operates in a
   relatively short time scale is a load-balancing algorithm described
   in [OMP] and [MATE].

   The state of the network can be based on various parameters such as
   utilization, packet delay, packet loss, etc. These parameters in turn
   can be obtained in several ways. For example, each router may flood
   these parameters periodically or by means of some kind of trigger to
   other routers.  An alternative approach is to have a particular
   router that wants to perform adaptive TE to send probe packets along
   a path to gather the state of that path. Yet, another approach is to
   have some management system to gather MIB information from the
   interfaces.  Because of the dynamic nature of the network conditions,
   expeditious and accurate gathering of state information is typically
   critical to adaptive TE.  State- dependent algorithms may be applied
   to increase network efficiency and resilience. While time-dependent



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 32]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   algorithms are more suitable for predictable traffic variations,
   state-dependent algorithms are more suitable for adapting to the
   prevailing state of the network.


5.2 Offline Versus Online


   Traffic engineering requires the computation of routing plans.  The
   computation itself may be done offline or online.  For the scenarios
   where the routing plans do not need to be executed in real-time, then
   the computation can be done offline.  As an example, routing plans
   computed from forecast information may be computed offline.
   Typically, offline computation is also used to perform extensive
   search on multi-dimensional space.

   Online computation is required when the routing plans need to adapt
   to changing network conditions as in state-dependent algorithms.
   Unlike offline computation which can be computationally demanding,
   online computation is geared toward simple calculations to fine-tune
   the allocations of resources such as load balancing.


5.3 Centralized Versus Distributed


   With centralized control, there is a central authority which
   determines routing plans on behalf of each router.  The central
   authority collects the network-state information from all routers,
   and returns the routing information to the routers periodically.  The
   routing update cycle is a critical parameter which directly impacts
   the performance of the network being controlled.  Centralized control
   may need high processing power and high bandwidth control channels.

   With distributed control, route selection is determined by each
   router autonomously based on the state of the network.  The network
   state may be obtained by the router using some probing method, or
   distributed by other by routers on a periodic basis.


5.4 Local Versus Global


   TE algorithms may require local or global network-state information.
   It is to be noted that the scope network-state information does refer
   to the scope of the optimization. In other words, it is possible for
   a TE algorithm to perform global optimization based on local state
   information. Similarly, a TE algorithm may arrive at a local optimum
   solution even if it relies on global state information.

   Global information pertains to the state of the entire domain that is
   being traffic engineered. Examples include traffic matrix, or loading
   information on each link.  Global state information is typically
   required with centralized control.  In some cases, distributed-



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 33]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   controlled TEs may also need global information.

   Local information pertains to the state of a portion of the domain.
   Examples include the bandwidth and packet loss rate of a particular
   path.  Local state information may be sufficient for distributed-
   controlled TEs.


5.5 Prescriptive Versus Descriptive


   Prescriptive traffic engineering evaluates alternatives and
   recommends a course of action. Prescriptive traffic engineering can
   be further categorized as either corrective or perfective. Corrective
   TE prescribes a course of action to address an existing or predicted
   anomaly. Perfective TE prescribes a course of action to evolve and
   improve network performance even when no anomalies are evident.

   Descriptive traffic engineering characterizes the state of the
   network and assesses the impact of various policies without
   recommending any particular course of action.


5.6 Open-Loop Versus Closed-Loop


   Open-loop control is where control action does not use any feedback
   information from the current network state. The control action may,
   however, use its own on local information for accounting purposes.

   Closed-loop control is where control action utilizes feedback
   information from the network state. The feedback information may be
   in the form historical information or current measurement.


6.0 Requirements for Internet Traffic Engineering


   This section describes the some high level requirements and
   recommendations for traffic engineering in the Internet. Because this
   is a framework document, these requirements are presented in very
   general terms. Additional documents to follow may elaborate on
   specific aspects of these requirements in greater detail.

   [NOTE: THIS SECTION IS AN INITIAL VERSION OF THE HIGH LEVEL TE
   REQUIREMENTS. IT WILL BE REVISED OVER TIME TO EXTEND AND REFINE IT.]


6.1 Generic Requirements


   Usability:  In general, it is desirable to have a TE system that can
   be readily deployed in an existing network. It is also desirable to
   have a TE system that is easy to operate and maintain.



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 34]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   Automation:  Whenever feasible, a TE system should automate the
   traffic engineering functions to minimize operator intervention in
   the control of operational networks.

   Scalability:  Contemporary public networks are growing very fast with
   respect to network size and traffic volume.  Therefore, a TE system
   SHOULD be scalable to remain applicable as the network evolves. In
   particular, a TE system SHOULD remain functional as the network
   expands with regard to the number of routers and links, and with
   respect  to the traffic volume.  A TE system SHOULD have a scalable
   architecture, SHOULD not adversely impair other functions and
   processes in a network element, and SHOULD not consume too much
   network resources when collecting and distributing state information
   or when exerting control.

   Stability:  Stability is a very important consideration in traffic
   engineering systems that respond to changes in the state of the
   network.  State-dependent traffic engineering methodologies typically
   mandate a tradeoff between responsiveness and stability.  It is
   strongly RECOMMENDED that when tradeoffs are warranted between
   responsiveness and stability, that the tradeoff should be made in
   favor of stability (especially in public IP backbone networks).

   Flexibility:  A TE system SHOULD be flexible to allow for changes in
   optimization policy. In particular, a TE system SHOULD provide
   sufficient configuration options so that a network administrator can
   tailor the TE system to a particular environment.  It may also be
   desirable to have both online and offline TE subsystems which can be
   independently enabled and disabled.  In multiclass networks, TE
   systems SHOULD also have options that support class based performance
   optimization.

   Observability:  As part of the TE system, mechanisms SHOULD exist to
   collect statistics from the network and to analyze them to determine
   how well the network is functioning.  Derived statistics such as
   traffic matrices, link utilization, latency, packet loss, and other
   performance measures or interest which are derived from network
   measurements can be used as indicators of prevailing network
   conditions.  Other examples of status information which should be
   observed include existing functional routes, and e.g. in the context
   of MPLS existing LSP routes, etc.

   Simplicity:  Generally, a TE system should be as simple as possible
   consistent with the intended applications. More importantly, the TE
   system should be relatively easy to use (i.e., clean, convenient, and
   intuitive user interfaces).  Simplicity in user interface does not
   necessarily imply that the TE system will use naive algorithms. Even
   when complex algorithms and internal structures are used, such
   complexities should be hidden as much as possible from the network
   administrator through the user interface.

   Congestion management: A TE system SHOULD map the traffic onto the
   network to minimize congestion.  If the total traffic load cannot be
   accommodated, then a TE system may rely on short time scale



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 35]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   congestion control mechanisms to mitigate congestion.  A TE system
   SHOULD be compatible with and complement existing congestion control
   mechanisms.  It is generally desirable to minimize the maximum
   resource utilization per service in an operational network.  The use
   of trunk reservation technique may also be useful in some situations.

   Survivability: It is critical for an operational network to recover
   promptly from network failures and to maintain the required QoS for
   existing services.  Survivability generally mandates introducing
   redundancy into the architecture, design, and operation of networks.
   There is a tradeoff between the level of survivability that can be
   attained and the cost required to attain it. The time required to
   restore a network service from a failure depends on several factors,
   including the particular context in which the failure occurred, the
   architecture and design of network, the characteristics of the
   network elements and network protocols, the applications and services
   that were impacted by the failure, etc. The extent and impact of
   service disruptions due to a network failure or outage can vary
   depending on the length of the outage, the part of the network where
   the failure occurred, the type and criticallity of the network
   resources that were impaired by the failure, the types of services
   that were impacted by the failure (e.g., voice quality degradation
   may be tolerable for an inexpensive VoIP service, but not be
   tolerable for a toll-quality VoIP service). Survivability can be
   addressed at the device level by developing network elements that are
   more reliable; and at the network level by incorporating redundancy
   into the architecture, design, and operation of networks.  It is
   recommended that a philosophy of robustness and survivability should
   be adopted in the architecture, design, and operation of IP networks
   (expecially public IP networks) and network elements. At the same
   time, because different contexts may demand different levels of
   survivability, the mechanisms developed to support network
   survivability should be flexible so that they can be tailored to
   different needs.


6.2 Routing Requirements


   [NOTE: THIS SECTION IS STILL WORK IN PROGRESS]

   Routing control is one of the most significant aspects of Internet
   traffic engineering.  Traditional IGPs which are based on shortest
   path algorithms have limited control capabilities for traffic
   engineering.  These limitations include:

   1. The well know issues with shortest path protocols. Since IGPs
   always use the shortest paths to forward traffic, load sharing cannot
   be done among paths of different costs.  Using shortest paths to
   forward traffic conserves network resources, but it may cause the
   following problems: 1) If traffic from a source to a destination
   exceeds the capacity of the shortest path, the shortest path will
   become congested while a longer path between these two nodes is
   under-utilized; 2) the shortest paths from different sources can



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 36]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   overlap at some links. If the total traffic from different sources
   exceeds the capacity of any of these links, congestion will occur.
   Such problems occur because traffic demand changes over time but
   network topology cannot be changed as rapidly, causing the network
   architecture to become suboptimal over time.

   2. Equal-Cost Multi-Path (ECMP) supports sharing of traffic among
   equal cost paths between two nodes. However, ECMP attempst to divide
   the traffic as equally as possible among the equal cost shortest
   paths. Generally, ECMP does not support configurable load splitting
   ratios among equal cost paths.  The result is that in the aggregate,
   one of the paths may carry significantly more traffic than other
   paths because it also may also carry traffic from other sources.

   3. Modifying IGP metric to control traffic distribution tends to have
   network-wide effect. Consequently, undesirable and unanticipated
   traffic shifts can be triggered as a result.

   Because of these limitations, new capabilities are needed to control
   the routing function in IP networks.  Some of these capabilities are
   described below.

   Constraint-based routing is highly desirable in IP networks,
   especially public IP backbones with complex topologies [AWD1].
   Constraint-based routing computes routes that fulfil some
   requirements subject to constraints.  Constraints may include
   bandwidth, hop count, delay, and administrative policy instruments
   such as resource class attributes [AWD1, RFC-2386].  This makes it
   possible to select that satisfy a given set of requirements subject
   to network and administrative policy constraints. Routes computed
   through constraint-based routing are not necessarily the shortest
   paths. Constraint-based routing works best with path oriented
   technologies that support explicit routing such as MPLS.

   Constraint-based routing can also be used as a means to redistribute
   traffic onto the infrastructure, even for best effort traffic.  For
   example, is the bandwidth constraints are set the bandwidth
   constraint of the paths and reservable bandwidth of the link
   properly, the congestion caused by uneven traffic distribution as
   described above can be avoided.  The performance and resource
   efficiency of the network is thus improved.

   In order compute routes subject to constraints, a number of
   enhancements are needed to conventional link state IGPs such as OSPF
   and IS-IS. The basic extensions required are outlined in [Li-IGP].
   Specializations of these requirements to OSPF were described in
   [KATZ] and to IS-IS in [SMIT].  Essentially, these enhancements
   require the propagation of additional information in link state
   advertisements. Specifically, in addition to normal link-state
   information, an enhanced IGP is required to propagate a number of
   topology state information that are needed for constraint-based
   routing. Some of the additional topology state information include
   link attributes such as: 1) reservable bandwidth, and 2) link
   resource class attribute which is an administratively specified



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 37]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   property of the link. The resource class attribute concept was
   defined in [AWD1].  The additional topology state information is
   carried in new TLVs or sub-TLVs in IS-IS, or in the Opaque LSA in
   OSPF [SMIT, KATZ].

   An enhanced link-state IGP may flood information more frequently than
   a normal IGP. This is because even without changes in topology,
   changes in reservable bandwidth or link affinity can trigger the
   enhanced IGP to initiate flooding.  In order to avoid consuming
   excessive link bandwidth and computational resources, a tradeoff is
   typically required between the timeliness of the information flooded
   and the flooding frequency.

   In a TE system, it is also desirable for the routing subsystem to
   make load splitting ratio among multiple paths (with equal cost or
   different cost) configurable.  This capability gives network
   administrators more flexibility in controlling traffic distribution,
   and can be very useful for avoiding/relieving congestion in some
   situations. Examples can be found in [XIAO].

   Another desirable feature of the routing system is the capability to
   control the route of subsets of traffic without affecting the routes
   of other traffic; provided that sufficient resources exist for this
   purpose. This capability allows more refined control over the
   distribution of traffic accross the network.  For example, the
   capability to move traffic from a source to a destination away from
   its original path to another path without affecting the paths of
   other traffic allows traffic to moved from resource-poor network
   segments to resource-rich segments. Path oriented technologies such
   as MPLS support this capability naturally.

   If the network supports multiple classes of service, the routing
   subsystem SHOULD have the capability to select different paths for
   different classes of traffic.


6.3 Traffic Mapping Requirements


   Traffic mapping pertains to the assignment of the traffic to the
   network topology to meet certain requirements and optimize resource
   usage.  Traffic mapping can be performed by time-dependent or state-
   dependent mechanisms, as described in Section 5.1.  A TE system
   SHOULD support both time-dependent and state-dependent mechanisms.

   For the time-dependent mechanism:
       - a TE system SHOULD maintain traffic matrices.
       - a TE system SHOULD have an algorithm that generates a mapping
         plan for each traffic trunk.
       - In certain environments (e.g., MPLS) a TE system SHOULD be
         able to control the path from any source to any destination;
         e.g., with explicit routing.
       - a TE system SHOULD be able to setup multiple paths to forward
         traffic from any source to any destination, and distribute the



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 38]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


         traffic among them based on a configurable traffic split.
       - a TE system SHOULD provide a graceful migration from one
         mapping plan to another as the traffic matrix changes to
         minimize service disruption.

   For the state-dependent mechanism:
       - a TE system SHOULD be able to gather and maintain link state
         information, for example, by using enhanced OSPF or IS-IS.
       - for a given demand request, QoS requirements, and other
         constraints, a TE system SHOULD be able to compute and setup a
         path, for example, by using constraint-based routing.
       - a TE system SHOULD be able to perform load balancing among
         multiple paths. Load balancing SHOULD NOT compromise the
         stability of the network.

   In general, a TE system SHOULD support modification of IGP link
   metrics to induce changes in the traffic mapping patterns.


6.4 Measurement Requirements


   The importance of measurement in traffic engineering has been stated
   previously.  In order to support the traffic engineering function,
   mechanisms SHOULD be provided to measure and collect statistics from
   the network.  Additional capabilities may be provided to help in the
   analysis of the statistics.  The actions of these mechanisms SHOULD
   not adversely affect the accuracy and integrity of the statistics
   collected. The mechanisms for statistical data acquisition SHOULD
   also be able to scale as the network evolves.

   Traffic statistics may be classified according to time scales, which
   may be long-term or short-term.  Long-term traffic statistics are
   very useful for traffic engineering. Long-term time scale traffic
   statistics MAY capture or reflect seasonality network workload (e.g.,
   hourly, daily, and weekly variations in traffic profiles; etc.).  For
   a network that supports multiple classes of service, aspects of the
   monitored traffic statistics MAY also reflect class of service
   characteristics.  Analysis of the long-term traffic statistics MAY
   yield secondary statistics such as busy hour characteristics, traffic
   growth patterns, persistent congestion and hot-spot problems within
   the network, imbalances in link utilization caused by routing
   anomalies, etc.

   There SHOULD also be a mechanism for constructing traffic matrices
   for both long-term and short-term traffic statistics. In multiservice
   IP networks, the traffic matrices MAY also be constructed for
   different service classses.  Each element of a traffic matrix
   represents a statistic of traffic flow between a pair of abstract
   nodes.  An abstract node may represent a router, a collection of
   routers, or a site in a VPN.

   At the short-term time scale, traffic statistics SHOULD provide
   reasonable and reliable indicators of the current state of the



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 39]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   network.  In particular, some traffic statistics SHOULD reflect link
   utilization, and link and path congestion status. Examples of
   congestion indicators include excessive packet delay, packet loss,
   and high resource utilization.  Examples of mechanisms for
   distributing such information including SNMP, probing techniques,
   FTP, and IGP link state advertisements, etc.


6.5 Network Survivability


   Network survivability refers to the capability of the network to
   maintain service continuity in the presence of failures within the
   network. This can be accomplished by promptly recovering from network
   failures and maintaining the required QoS for existing services after
   recovery. Survivability has become an issue of great concern to the
   Internet community with the increasing demands to carry mission
   critical traffic, real-time traffic, and other high priority traffic
   over the Internet. As network technologies continue improve, failure
   protection and restoration capabilities have become available from
   multiple layers. At the bottom of the layered stack, optical networks
   are now capable of providing dynamic ring and mesh restoration
   functionality as well as traditional protection functionality. For
   instance, the SONET/SDH layer provides survivability capability with
   Automatic Protection Switching (APS), as well as self-healing ring
   and mesh architectures. Similar functionality are provided by layer 2
   technologies such as ATM (generally with slower mean restoration
   times). At the IP layer, rerouting is used to restore service
   continuity following link and node outages. Rerouting at the IP layer
   occurs after a period of routing convergence, which may require
   seconds to minutes to complete. In order to support advanced
   survivability requirements, path-oriented technologies such a MPLS
   can be used to enhance the survivability of IP networks; in a
   potentially cost effective manner. The advantages of path oriented
   technologies such as MPLS for IP restoration becomes even more
   evident when class based protection and restoration capabilities are
   required.

   Recently, a common suite of control plane protocols has been proposed
   for both MPLS and optical transport networks under the acronym
   Multiprotocol Lambda Switching [AWD5]. This new paradigm of
   Multiprotocol Lambda Switching will support even more sophisticated
   mesh restoration capabilities at the optical layer for the emerging
   IP over WDM network architectures.

   Another important aspect regarding multi-layer survivability is that
   various technologies at different layers provide protection and
   restoration capabilities at different temporal granularities (i.e.,
   in terms of time scales) and at different bandwidth granularity (from
   packet-level to wavelength level). Protection and restoration
   capabilities can also be aware or unaware of different service
   classes.

   As noted previously, the impact of service outages varies



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 40]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   significantly for different service classes depending on the
   effective duration of the outage.  The duration of an outage can vary
   from milliseconds (with minor service impact), to seconds (with
   possible call drops for IP telephony and session time-outs), to
   minutes and hours (with potentially considerable social and business
   impact).

   Generally, it is a challenging task to cordinate different protection
   and restoration capabilities across multiple layers in a cohesive
   manner so as to ensure that network survivability is maintained at
   reasonable cost. Protection and restoration coordination across
   layers may not always be feasible, because, for example, networks at
   different layers might belong to different administrative domains.

   In the following paragraphs, some of the general requirements for
   protection and restoration coordination are highlighted.

   - Protection/restoration capabilities from different layers SHOULD be
   coordinated whenever feasible and appropriate in order to provide
   network survivability in a flexible and cost effective manner. One
   way to achieve the coordination is to minimize function duplication
   across layers. Escalation of alarms and other fault indicators from
   lower layers to higher layers may also be performed in a coordinated.
   A temporal order of restoration triger timing at different layers is
   another way to coordinate multi-layer protection/restoration.

   - Spare capacity at higher layers is often regarded as working
   traffic at lower layers. Placing protection/restoration functions in
   many layers may increase redundancy and robustness, but it SHOULD not
   result in significant and avoidable inneficiencies in network
   resource utilization.

   - It is generally desirable to have a protection/restoration scheme
   that is bandwidth efficient.

   - Failure notification throughout the network SHOULD be timely and
   reliable.

   - Alarms and other fault monitoring and reporting capabilities SHOULD
   be provided at appropriate layers.


6.5.1 Survivability in MPLS Based Networks


   MPLS is an important emerging technology that enhances IP networks in
   terms of features and services. Because MPLS is path-oriented it can
   potentially provide faster and more predictable protection and
   restoration capabilities than conventional IP systems. This
   subsection provides an outline of some of the basic features and
   requirements of MPLS networks regarding protection and restoration. A
   number of Internet drafts also discuss protection and restoration
   issues in MPLS networks (see e.g., [ACJ99], [MSOH99], and [Shew99]).




Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 41]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   Protection types for MPLS networks can be categorized into link
   protection, node protection, path protection, and segment protection,
   as discussed below.

   - Link Protection: The goal of link protection is to protect an LSP
   from a given link failure. Under link protection, the path of the
   protect or backup LSP (also called secondary LSP) is disjoint from
   the path of the working or operational LSP at the particular link
   over which protection is required. When the protected link fails,
   traffic on the working LSP is switched over to the protect LSP at the
   head-end of the failed link. This is a local repair method which can
   be potentially fast. It might be more appropriate in situations where
   some network elements along a given path are less reliable than
   others.

   - Node Protection: The goal of LSP node protection is to protect an
   LSP from a given node failiure. Under node protection, the path of
   the protect LSP is disjoint from the path of the working LSP at
   particular node that is to be protected. The secondary path is also
   disjoint from the primary path at all links associated with the node
   to be protected. When the node fails, traffic on the working LSP is
   switched over to the protect LSP at the upstream LSR that directly
   connects to the failed node.

   - Path Protection: The goal of LSP path protection is to protect an
   LSP from failure at any point along its routed path. Under path
   protection, the path of the protect LSP is completely disjoint from
   the path of the working LSP. The advantage of path protection is that
   the protect LSP protects the working LSP from all possible link and
   node failures along the path, except for failures that might occur at
   the ingress and egress LSRs. Additionally, since the path selection
   is end-to-end, path protection mign yield more efficient in terms of
   resource usage than link or node protection.  However, in general,
   path protection may be slower than link and node protection.

   - Segment Protection: In some cases, an MPLS domain may be
   partitioned into multiple protection domains whereby a failure in a
   protection domain is rectified with that domain.  In cases where an
   LSP traverses multiple protection domains, a protection mechanism
   within a domain only needs to protect the segment of the LSP that
   lies within the domain. Segment protection will generally be faster
   than path protection because recovery generally occurs closer to the
   fault.


   Protection option: Anoter issue to consider is the concept of
   protection options. It can be described in general using the notation
   m:n protection where m is the number of protect LSPs used to protect
   n working LSPs. In the following, some feasible protection options
   are described.

   - 1:1: one working LSP is protected/restored by one protect LSP;

   - n:1: one working LSP is protected/restored by n protect LSPs,



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 42]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   perhaps with configurable load splitting ratio. In situations where
   more than one protect LSP is used, it may be desirable to share the
   traffic accross the protect LSPs when the working LSP fails in so as
   to satisfy the bandwidth requirement of the traffic trunk associated
   with the working LSP, especially when it may not be feasible to find
   one path that can satisfy the the bandwidth requirement of the
   primary LSP;

   - 1:n: one protection LSP is used to protect/restore n working LSPs;

   - 1+1: traffic is sent cocurrently on both the working LSP and the
   protect LSP. In this case, the egress LSR selects one of the two LSPs
   based on some location traffic integrity decision process.  This
   option would probably not be used pervasively in IP networks due to
   its inefficiency in terms of resource utilization.


   Resilience Attributes:

   - Basic attribute: reroute using IGP or protection LSP(s) when a
   segment of the working path fails, or no rerouting at all.

   -  Extended attributes:

   1. Protection LSP establishment attribute: the protection LSP is i)
   pre-established, or ii) established-on-demand after receiving failure
   notification. Pre-established protection LSP can be faster while
   established-on-demand one can potentially find a more optimal path
   and with more efficient resource usage.

   2. Constraint attribute under failure condition: the protection LSP
   requires certain constraint(s) to be satisfied, which can be the same
   or less than the ones under normal condition, e.g., bandwidth
   requirement, or choose to use 0-bandwidth requirement under any
   failure condition.

   3. Protection LSP resource reservation attribute: resource allocation
   of a pre-established protection LSP is, i) pre-reserved, or ii)
   reserved-on-demand after receiving failure notification;

   A pre-established and pre-reserved protection LSP can guarantee that
   the QoS of existing services is maintained upon failure while a pre-
   established and reserve-on-demand one or an established-on-demand one
   may not be able to. In addition, it is the fastest among the three.
   It can switch packets on the protection LSP once the ingress LSR
   receives the failure notification message without experiencing any
   delay for resource availability checking and protection LSP
   establishment. However, a pre-established protection LSP may not be
   able to adapt to any new change in the network since its
   establishment if there could be a better path due to the change. In
   addition, the bandwidth being reserved on the protection LSP is
   subtracted from the available bandwidth pool on all associated links,
   hence, not available for admitting new LSPs in the future. On the
   other hand, it differs from SONET protection in terms that the



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 43]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   reserved bandwidth does not sit idle, instead it can be used by any
   traffic presents on those links.  Now, comparing a pre-established
   protection LSP and an established-on-demand one, the former is
   potentially faster since it only needs to wait to check if the
   requested bandwidth is available on the pre-established path without
   waiting for the path to be set up. However, if the requested
   bandwidth is not available on the pre-established path, it may choose
   to use an established-on-demand one as a second option.

   Failure Notification:

   Failure notification SHOULD be reliable and fast enough, i.e., at
   least in the same order as IGP notification, which is through LSA
   flooding, if not faster.


6.6 Content Distribution (Webserver) Requirements


   The Internet is dominated by client-server interactions, especially
   Web traffic. The location of major information servers has a
   significant impact on the traffic patterns within the Internet, and
   on the perception of service quality by end users.

   A number of dynamic load balancing techniques have been devised to
   improve the performance of replicated Web servers. The impact of
   these techniques is that the traffic becomes more dynamic in the
   Internet, because Web servers can be dynamically picked based on the
   locations of the clients, and the relative performance of different
   networks or different parts of a network.  This process can be called
   Traffic Directing (TD).  It is similar to Traffic Engineering but is
   at the application layer.

   Scheduling systems in TD that allocate servers to in replicated,
   geographically dispersed information distribution systems may require
   performance parameters of the network to make effective decisions.
   It is desirable that the TE system provide such information.  The
   exact parameters needed are to be defined. When there is congestion
   in the network, the TD and TE systems SHOULD act in a coordinated
   manner. This topic is for further study.

   Because TD can introduce more traffic dynamics into a network,
   network planning SHOULD take this into consideration.  It can be
   desirable to reserve a certain amount of extra capacity for the links
   to accommodate this additional traffic fluctuation.


6.7 Offline Traffic Engineering Support Systems


   If optimal link efficiency is desired, an offline and centralized
   traffic engineering support system MAY be provided as an integral
   part of an overall TE system.  An offline and centralized traffic
   engineering support system can be used to compute the paths for the



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 44]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   traffic trunks. By taking all the trunk requirements, link attributes
   and network topology information into consideration, an offline TE
   support system can typically find a better trunk placement than an
   online TE system, where every router in the network finds paths
   originated from it in a distributed manner based on its own
   information.  An offline TE support system may compute paths for
   trunks periodically, e.g., daily, for the purpose of re-optimization.
   The computed paths can then be downloaded into the routers. An online
   TE  support system is still needed, so that routers can adapt to
   changes promptly.


6.8 Traffic Engineering in Diffserv Environments


   [NOTE: THIS SECTION IS WORK IN PROGRESS AND WILL BE UPDATED IN THE
   NEXT VERSION OF  DRAFT]

   Traffic engineering will be very important in Diffserv environments.
   This section describes the traffic engineering features and
   requirements that are specifically pertinent to Differentiated
   Services (Diffserv) capable IP networks.



7.0 Multicast Considerations


   For further study.


8.0 Inter-Domain Considerations


   Inter-domain traffic engineering is concerned with the performance
   optimization for traffic that originates in one administrative domain
   and terminates in a different one.

   Traffic exchange between autonomous occurs through exterior gateway
   protocols. Currently, BGP-4 [bgp4] is the defacto EGP standard.

   Traditionally, in the public Internet, BGP based policies are used to
   control import and export policies for inter-domain traffic.  BGP
   policies are also used to determine exit and entrance points to and
   from peer networks.

   Inter-domain TE is inherently more difficult than intra-domain TE.
   The reasons for this are both technical and administrative.
   Technically, the current version of BGP does not propagate topology
   and link state information outside accross domain boundaries.
   Administratively, there are differences in operating costs and
   network capacities between domains, and what may be considered a good
   solution in one domain may not necessarily be a good in another
   domain. Moreover, it would generally be considered imprudent for one



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 45]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   domain to permit another domain to influence the routing and control
   of traffic in its network.

   When Diffserv becomes widely deployed, inter-domain TE will become
   even more important, but more challenging to address.

   MPLS TE-tunnels (explicit LSPs) add a degree of flexibility in terms
   of selection of exit points for inter-domain routing.  The concept of
   relative and absolute metrics were defined in [SHEN]. If the BGP
   attributes are defined such that the BGP decision process depends on
   IGP metrics to select exit points for Inter-domain traffic, then some
   inter-domain traffic destined to a given peer network can be made to
   prefer a given exit point by establishing a TE-tunnel between the
   router making the selection to the peering point via a TE-tunnel and
   assigning the TE-tunnel a metric which is smaller than the IGP cost
   to all other peering points. If a peer accepts and processes MEDs,
   then a similar MPLS TE-tunnel based scheme can be applied to cause
   certain entrance point to be preferred by setting MED to be the IGP
   cost, which has been modified by the tunnel metric.

   Similar to intra-domain TE, Inter-domain TE is best accomplished when
   a traffic matrix can be derived.  traffic matrix for inter-domain
   traffic.

   Generally, redistribution of inter-domain traffic requires
   coordination between peering partners. Any export policy in one
   domain that results load redistribution across peer points can
   significantly affect the traffic distribution inside the domain of
   the peering partner. This, in turn, will affect the intra-domain TE
   due to changes in the intra-domain traffic matrix. Therefore, it is
   critical for peering partners to negotiate and coordinate with each
   other before attemping any policy changes that may result in
   significant shifts in inter-domain traffic. In practice, this
   coordination can be quite challenging for technical and non-technical
   reasons.

   It is a matter of speculation as to whether MPLS, or similar
   technologies, can be extended to allow selection of constrained-paths
   across domain boundaries.


9.0 Conclusion


   This document described a framework for traffic engineering in the
   Internet.  It presented an overview of some of the basic issues
   surrounding traffic engineering in IP networks. The context of TE was
   described, a TE process models and a taxonomy of TE styles were
   presented.  A brief historical review of pertinent developments
   related to traffic engineering was provided. Finally, the document
   specified a set of generic requirements, recommendations, and options
   for Internet traffic engineering.





Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 46]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


10.0 Security Considerations


   This document does not introduce new security issues.


11.0 Acknowledgments


   The authors would like to thank Jim Boyle for inputs on the
   requirements section, Francois Le Faucheur for inputs on class-type,
   and Gerald Ash for inputs on routing in telephone networks.  The
   subsection describing an "Overview of ITU Activities Related to
   Traffic Engineering" was adapted from a contribution by Waisum Lai.


12.0 References


   [ACJ99] L. Anderson, B. Cain, and B. Jamoussi, "Requirement Framework
   for Fast Re-route with MPLS", Work in progress, October 1999.

   [ASH1] J. Ash, M. Girish, E. Gray, B. Jamoussi, G. Wright,
   "Applicability Statement for CR-LDP," Work in Progress, 1999.

   [ASH2] J. Ash, Dynamic Routing in Telecommunications Networks, McGraw
   Hill, 1998

   [AWD1] D. Awduche, J. Malcolm, J. Agogbua, M. O'Dell, J. McManus,
   "Requirements for Traffic Engineering over MPLS," RFC 2702, September
   1999.

   [AWD2] D. Awduche, "MPLS and Traffic Engineering in IP Networks,"
   IEEE Communications Magazine, December 1999.

   [AWD3] D. Awduche, L. Berger, D. Gan, T. Li, G. Swallow, and V.
   Srinivasan "Extensions to RSVP for LSP Tunnels," Work in Progress,
   1999.

   [AWD4] D. Awduche, A. Hannan, X. Xiao, " Applicability Statement for
   Extensions to RSVP for LSP-Tunnels" Work in Progress, 1999.

   [AWD5] D. Awduche et al, "An Approach to Optimal Peering Between
   Autonomous Systems in the Internet," International Conference on
   Computer Communications and Networks (ICCCN'98), October 1998.

   [AWD6] D. Awduche, Y. Rekhter, J. Drake, R. Coltun, "Multiprotocol
   Lambda Switching: Combining MPLS Traffic Engineering Control with
   Optical Crossconnects," Work in Progress, 1999.

   [CAL] R. Callon, P. Doolan, N. Feldman, A. Fredette, G. Swallow, A.
   Viswanathan, A Framework for Multiprotocol Label Switching," Work in
   Progress, 1999.




Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 47]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   [FGLR] A. Feldmann, A. Greenberg, C. Lund, N. Reingold, and J.
   Rexford, "NetScope: Traffic Engineering for IP Networks," to appear
   in IEEE Network Magazine, 2000.

   [FlJa93] S. Floyd and V. Jacobson, "Random Early Detection Gateways
   for Congestion Avoidance", IEEE/ACM Transactions on Networking, Vol.
   1 Nov. 4., August 1993, p. 387-413.

   [Floy94] S. Floyd, "TCP and Explicit Congestion Notification", ACM
   Computer Communication Review, V. 24, No. 5, October 1994, p. 10-23.

   [HuSS87] B.R. Hurley, C.J.R. Seidl and W.F. Sewel, "A Survey of
   Dynamic Routing Methods for Circuit-Switched Traffic", IEEE
   Communication Magazine, Sep 1987.

   [itu-e600] ITU-T Recommendation E.600, "Terms and Definitions of
   Traffic Engineering", March 1993.

   [itu-e701] ITU-T Recommendation E.701 "Reference Connections for
   Traffic Engineering", October 1993.

   [JAM] B. Jamoussi, "Constraint-Based LSP Setup using LDP," Work in
   Progress, 1999.

   [Li-IGP] T. Li, G. Swallow, and D. Awduche, "IGP Requirements for
   Traffic Engineering with MPLS," Work in Progress, 1999

   [LNO96] T. Lakshman, A. Neidhardt, and T. Ott, "The Drop from Front
   Strategy in TCP over ATM and its Interworking with other Control
   Features", Proc. INFOCOM'96, p. 1242-1250.

   [MATE] I. Widjaja and A. Elwalid, "MATE: MPLS Adaptive Traffic
   Engineering," Work in Progress, 1999.

   [McQ80] J.M. McQuillan, I. Richer, and E.C. Rosen, "The New Routing
   Algorithm for the ARPANET", IEEE. Trans. on Communications, vol. 28,
   no. 5, pp. 711-719, May 1980.

   [MR99] D. Mitra and K.G. Ramakrishnan, "A Case Study of Multiservice,
   Multipriority Traffic Engineering Design for Data Networks, Proc.
   Globecom'99, Dec 1999.

   [MSOH99] S. Makam, V. Sharma, K. Owens, C. Huang,
   "Protection/Restoration of MPLS Networks", Work in Progress, October,
   1999.

   [OMP] C. Villamizar, "MPLS Optimized OMP", Work in Progress, 1999.

   [RFC-1349] P. Almquist, "Type of Service in the Internet Protocol
   Suite", RFC 1349, Jul 1992.

   [RFC-1458] R. Braudes, S. Zabele, "Requirements for Multicast
   Protocols," RFC 1458, May 1993.




Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 48]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   [RFC-1771] Y. Rekhter and T. Li, "A Border Gateway Protocol 4 (BGP-
   4), RFC 1771, March 195.

   [RFC-1812] F. Baker (Editor), "Requirements for IP Version 4
   Routers," RFC 1812, June 1995.

   [RFC-1997] R. Chandra, P. Traina, and T. Li, "BGP Community
   Attributes" RFC 1997, August 1996.

   [RFC-1998] E. Chen and T. Bates, "An Application of the BGP Community
   Attribute in Multi-home Routing," RFC 1998, August 1996.

   [RFC-2178] J. Moy, "OSPF Version 2", RFC 2178, July 1997.

   [RFC-2205] R. Braden, et. al., "Resource Reservation Protocol (RSVP)
   - Version 1 Functional Specification", RFC 2205, September 1997.

   [RFC-2211] J. Wroclawski, "Specification of the Controlled-Load
   Network Element Service", RFC 2211, Sep 1997.

   [RFC-2212] S. Shenker, C. Partridge, R. Guerin, "Specification of
   Guaranteed Quality of Service," RFC 2212, September 1997

   [RFC-2215] S. Shenker, and J. Wroclawski, "General Characterization
   Parameters for Integrated Service Network Elements", RFC 2215,
   September 1997.

   [RFC-2216] S. Shenker,  and J. Wroclawski, "Network Element Service
   Specification Template", RFC 2216, September 1997.

   [RFC-2330] V. Paxson et al., "Framework for IP Performance Metrics",
   RFC 2330, May 1998.

   [RFC-2386] E. Crawley, R. Nair, B. Rajagopalan, and H. Sandick, "A
   Framework for QoS-based Routing in the Internet", RFC 2386, Aug.
   1998.

   [RFC-2475] S. Blake et al., "An Architecture for Differentiated
   Services", RFC 2475, Dec 1998.

   [RFC-2597] J. Heinanen, F. Baker, W. Weiss, and J. Wroclawski,
   "Assured Forwarding PHB Group", RFC 2597, June 1999.

   [RFC-2678] J. Mahdavi and V. Paxson, "IPPM Metrics for Measuring
   Connectivity", RFC 2678, Sep 1999.

   [RFC-2679] G. Almes, S. Kalidindi, and M. Zekauskas, "A One-way Delay
   Metric for IPPM", RFC 2679, Sep 1999.

   [RFC-2680] G. Almes, S. Kalidindi, and M. Zekauskas, "A One-way
   Packet Loss Metric for IPPM", RFC 2680, Sep 1999.

   [RFC-2722] N. Brownlee, C. Mills, and G. Ruth, "Traffic Flow
   Measurement: Architecture", RFC 2722, Oct 1999.



Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 49]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


   [RoVC] E. Rosen, A. Viswanathan, R. Callon, "Multiprotocol Label
   Switching Architecture," Work in Progress, 1999.

   [Shew99] S. Shew, "Fast Restoration of MPLS Label Switched Paths",
   draft-shew-lsp-restoration-00.txt, October 1999.

   [SLDC98] B. Suter, T. Lakshman, D. Stiliadis, and A. Choudhury,
   "Design Considerations for Supporting TCP with Per-flow Queueing",
   Proc. INFOCOM'99, 1998, p. 299-306.

   [XIAO] X. Xiao, A. Hannan, B. Bailey, L. Ni, "Traffic Engineering
   with MPLS in the Internet", IEEE Network magazine, March 2000.

   [YaRe95] C. Yang and A. Reddy, "A Taxonomy for Congestion Control
   Algorithms in Packet Switching Networks", IEEE Network Magazine, 1995
   p. 34-45.

   [SMIT] H. Smit and T. Li, "IS-IS extensions for Traffic
   Engineering,"Internet Draft, Work in Progress, 1999

   [KATZ] D. Katz, D. Yeung, "Traffic Engineering Extensions to
   OSPF,"Internet Draft, Work in Progress, 1999

   [SHEN] N. Shen and H. Smit, "Calculating IGP routes over Traffic
   Engineering tunnels" Internet Draft, Work in Progress, 1999.
































Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 50]


Internet Draft      draft-ietf-tewg-framework-00.txt   Expires July 2000


13.0 Authors' Addresses:


      Daniel O. Awduche
      UUNET (MCI Worldcom)
      22001 Loudoun County Parkway
      Ashburn, VA 20147
      Phone: 703-886-5277
      Email: awduche@uu.net

      Angela Chiu
      AT&T Labs
      Room C4-3A22
      200 Laurel Ave.
      Middletown, NJ 07748
      Phone: (732) 420-2290
      Email: alchiu@att.com

      Anwar Elwalid
      Lucent Technologies
      Murray Hill, NJ 07974, USA
      Phone: 908 582-7589
      Email: anwar@lucent.com

      Indra Widjaja
      Fujitsu Network Communications
      Two Blue Hill Plaza
      Pearl River, NY 10965, USA
      Phone: 914-731-2244
      Email: indra.widjaja@fnc.fujitsu.com

      Xipeng Xiao
      Global Crossing
      141 Caspian Court,
      Sunnyvale, CA 94089
      Email: xipeng@globalcenter.net
      Voice: +1 408-543-4801




















Awduche/Chiu/Elwalid/Widjaja/Xiao                              [Page 51]