Computer Networks 40 (2002) 131-148

www.elsevier.com/locate/comnet
Engineering a multiservice IP backbone to support tight SLAs
Clarence Filsfils a'*9 John Evans b
a Cisco Systems, Brussels, Belgium b Cisco Systems Ltd., 10 New Square Park, Bedfont Lakes, Feltham, Middlesex, UK W14 8HA
Abstract
This paper describes technologies that enable IP service providers to offer tighter service level agreements for IP performance, in order to create competitive advantage and better serve their customers. The SLA parameters that need to be tightened are defined and then the technologies that should be considered are described, together with the decision criteria on where each technology shouldbe used. This paper is basedupon current best practise andincludes results from both lab testing and deployment experience. The specific technologies discussed are differentiated services, fast IGP convergence, and traffic engineering. Consideration is given to how these technologies should be deployed and operated.
© 2002 Elsevier Science B.V. All rights reserved.
IDT: Differentiatedservices; MPLS; Traffic engineering; IGP convergence
1. Introduction
When using public transport, the traveller may benefit from contractual commitments from the transport service provider, for example that 95% of journeys will arrive within 5 min of the sched-uledtime. The commitments may include other parameters or metrics such as number of stops en route andany meals included. The more competitive the market for the particular service, the more comprehensive andthe tighter the commitments or service level agreements that are offered. In the same way, the increase of competition between IP service providers (SPs) together with the height-enedimportance of IP to business operations has
Corresponding author.
E-mail addresses: cfilsfil@cisco.com, cf@cisco.com (C. Filsfils), joevans@cisco.com (J. Evans).
ledto an increaseddemand andconsequent supply of IP services with tighter service level agreements (SLAs) for IP performance.
The IP technical community has developed a set of technologies that enable IP networks to be en-gineeredto support tight SLA commitments:
• Differentiated services (Diffserv). The Diffserv architecture allows differentiated delay, jitter andloss commitments to be supportedon the same IP backbone for different types or classes of service.
• Faster IGP convergence. New developments in Interior Gateway routing protocols (IGPs) allow for faster convergence upon link or node failure, hence enabling higher service availability to be offered.
• MPLS traffic engineering. MPLS traffic engineering (Diffserv-aware or not) introduces con-straint-basedrouting andadmission control to
IP backbones. This allows optimum use to be made of the installed backbone bandwidth capacity, or conversely allows the same level of service to be offeredfor less capacity. It can also be usedto ensure that the amount of low-jitter traffic per link does not exceed a specified maximum.
• MPLS traffic engineering fast reroute. MPLS traffic engineering fast reroute is an IP protection technique that enables connectivity to be restoredaroundlink andnode failures in a few tens of milliseconds.
For an SP IP service, the SLA commitments are basedon delay, jitter, packet loss rate, throughput andavailability. This paper focuses on defining these SLA parameters, anddescribing why and how the above technologies shouldbe usedin order for these SLA parameters to be tightened. Consideration is also given to how networks using these technologies shouldbe operated. There are other factors that will affect a service provider's ability to offer tight SLAs in addition to the technologies described, but which are not covered in this paper; these include: network security and BGP convergence.
In focussing on service provider IP backbone networks, it is notedthat the mechanisms employed at the edge of the network to deliver tight SLAs may be different from those used in the core. In the backbone, where traffic is aggregated, SLA requirements for a traffic class can be translated into the appropriate bandwidth requirements, and the problem of SLA assurance can effectively be reduced to that of bandwidth provisioning. At the network edge, other considerations, such as serialisation delay, become significant. The mechanisms employedat the edge of the network are not considered further in this paper.
2. Customer requirements—what an SLA should commit to
The following metrics are highlightedas the most important for specifying the quality of an IP backbone service [15], For each metric, typical values are given for different types or classes of service.
2.1. Network one-way delay
In IP backbone terms, network one-way delay characterizes the time difference between the reception of an IP packet at an ingress point of presence (POP) andits transmission at an egress POP. Network one-way delay is made up of four components:
• Propagation delay. Propagation delay is con-strainedby the speedof light in a medium and for optical fibre is around5 ms per 1000 km. Propagation delay can vary as network topology changes, when a link fails, for example, or when an underlying network (e.g. SDH/SONET) reroutes its circuit paths.
• Switching delay. Switching or processing delay is the time difference between receiving a packet on an incoming router interface andthe enqueuing of the packet in the scheduler of its outbound interface. Switching delays on today's high-performance routers are negligible, typically in the order of 10-20 is per packet.
• Scheduling delay. Scheduling (or queuing) delay is defined as the time difference between the enqueuing of a packet on the outboundinterface scheduler, and the start of clocking the packet onto the outboundlink. This is a function of the scheduling algorithm used and of the scheduler queue utilization, which is in turn a function of the queue capacity andthe offered traffic loadandprofile. This effect is analysed in more detail later in this paper.
• Serialisation delay. Serialisation delay is the time taken to clock a packet onto a link andis dependent upon the link speed and the packet size. Serialisation delay is considered negligible at link speeds above STM-1/OC3, such as backbone links: a 1500 byte packet is clockedat STM-1/ OC3 rate (155 Mbps) in 80 is, at STM-16/ OC48 rate (2.5 Gbps) in 5 is andat STM-64/ OC192 rate (10 Gbps) in 1.25 is.
The goal commonly usedin designing networks to support voice over IP (VoIP) is a delay budget of 150 ms from mouth to ear. A design should apportion this budget to the various components of network delay (propagation delay through the backbone, scheduling delay due to congestion, and the access link serialisation delay) and service delay (due to VoIP gateway codec and de-jitter buffer). Propagation delay is often budgeted by using the widest diameter in the network, which for example, in a national network in the US would give a worst case (coast-to-coast) of 6000 km or 30 ms of one-way propagation delay.
Interactive data applications often require a roundtrip time (RTT) of less than 250 ms to allow for a smooth interaction between the human and the application server.
SPs today typically commit on monthly average one-way delay over all their POP-to-POP pairs.
2.2. Network delay-jitter
Network delay-jitter characterizes the variation of network delay; it is generally computed as the variation of the delay for two consecutive packets. Jitter is causedby the variation in the components of delay previously described:
• Propagation delay. Propagation delay can vary as network topology changes, when a link fails, for example, or when the topology of a lower layer network (e.g. SDH/SONET) changes, causing a sudden peak of jitter. Current IP backbone experience wouldsuggest that these occurrences are more common than is generally believed[6].
• Switching delay. Switching delay can vary as some pacakets might require more processing than others. This effect is becoming less of a consideration as packet switching is imple-mentedusing hardware pipelines whose switching delay characteristics are deterministic.
• Scheduling delay. Variation in scheduling delay is causedas scheduling queues oscillate between empty andfull.
ITU standard G.114 states that 150 ms of end-to-end oneway delay does not cause a perceivable degradation in voice quality for most use of telephony.
Jitters buffers (also known as play-out buffers) are usedto remove delay variation by turning variable network delays into constant delays at the destination end systems. Consequently, in networks that are engineeredto support low-delay services such as VoIP it is important that they are also engineeredfor low jitter. IP backbones that are engineeredto support high-quality VoIP services typically budget for 5-10 ms of jitter in the backbone; assuming 10 backbone hops, this gives a jitter budget per hop of 500-1000 is. Schedulers that implement a priority queuing mechanism such as the ''modified'' 2 Deficit roundrobin (mDRR) scheduler [25] implemented on high-performance routers have jitter characteristics 5-10 times better than this. Data applications do not generally require specific constraints on jitter.
2.3. Loss
Loss characterizes the packet drops that occur between the ingress link of the ingress POP andthe egress link of the egress POP. It is observedthat US backbone service providers usually offer an average monthly loss rate of less than 1%. Backbones engineeredfor high-quality VoIP services typically plan for a loss rate of less than 0.25%. The same target range is also usedfor high-quality data services; TCP throughput has been shown to decrease as an inverse of the square root of the probability of packet loss [19].
2.4. Bandwidth and throughput
IP services are commonly soldwith a defined bandwidth, where the bandwidth reflects the access link capacity provisionedfor the service. Defined bandwidth may not be the same as achieved throughput, however. Throughput characterises the available user bandwidth between an ingress POP andegress POP. The requirement for this SLA parameter is obvious for point-to-point services such as virtual wires, as being defined by the Pseudo Wire Emulation Edge to Edge (PWE3)
2 The DRR algorithm usedon Cisco routers has been modified by Cisco to add support for a strict priority queue.

Fig. 1. TCP throughput as a function of packet loss andRTT with MSS = 1460 bytes.
Working Group [22] within the IETF. For multipoint-to-multipoint services, such as VPNs, the SLA definition will need to exclude cases where the loss of throughput is due to customer-based aggregation. For example, where 10 sites each with STM-1/OC3 access links are all sending traffic at full rate to a single site with only an STM-4/OC12 access link [9].
For TCP/IP traffic, achievedthroughput is largely dependent upon the probability of packet loss andthe achievedRTT. Consequently, contracted bandwidth may not relate to achieved throughput. The graph in Fig. 1 uses the relationship defined by Mathis [19] to show how TCP throughput varies as a function of packet loss andRTT.
2.5. Per-flow sequence preservation
Per-flow sequence preservation is not yet a common component of IP service SLA commitments even though it is acceptedbest practise in IP network design. Due to the impact that packet reordering can have on some applications, however, it is starting to be offeredby SPs for high-quality data and video services. To prevent out of order packet delivery, it is important that any load balancing is achievedat a per-flow level rather than at a per packet level, andscheduling algorithms assure that packets from the same flow are servicedin order andfrom the same queue.
Real-time video is commonly impacted by reordering as the end-user applications either do not have the time to reorder the received frames or simply do not support this functionality; in both cases, reordering means a higher packet loss. Most TCP implementations interpret the receipt of three consecutive similar acknowledgements as an indication of packet loss and hence retransmit the next packet and slow down their rate; reordering therefore has a significant impact on TCP throughput [18]. Whilst the reordering magnitude would needto be very significant in order to affect a VoIP flow whose inter-packet gap is a multiple of 10 ms, the reordering might cause significant jitter due to variation in the propagation delay (for example, US east coast to west coast via a north or the south route).
2.6. Availability
Availability for IP services is generally defined in one of two ways; in terms of either network availability or service availability. Network availability is defined as the fraction of time that network connectivity is available between a specified ingress point anda specified egress point. Service availability is defined as the fraction of time the service is available between a specifiedingress point anda specified egress point within the bounds of the defined SLAs. There can be overlap between the definition of network or service availability andthe definition of other SLA parameters, for example, 100% network availability implies 0% packet loss.
2.7. Classes of service
Amongst the QoS enabledIP services offeredby service providers today there is common support for classes of service designed to meet the needs of three traffic types: real-time traffic, business data traffic andbest-effort data traffic. Typical SLA commitments andservice characteristics for these classes are as follows:
• Real-time. This class targets applications such as VoIP and video. It is defined in terms of low loss (less than 0.25%), low delay, and low jitter (typically 5 ms within the backbone) andhas a specified bandwidth and availability. Attainable throughput is derived from bandwidth and loss. The class may support a commitment for per-flow sequence preservation.
• Business data. This class represents business critical interactive applications such as SNA, SAP R/3, Telnet, andpossibly intranet Web applications to selectedURLs. It is definedin terms of delay (RTT should be less than 250 ms), and loss (less than 1% loss rate is typical, with targets of less than 0.1% also available), with a specified bandwidth and availability. Attainable throughput is derivedfrom loss andRTT. Jitter is not important for this service class andis not defined. The class may support a commitment for per-flow sequence preservation.
• Best-effort data. This class represents all other traffic that has not been classifiedas real-time or business. It is defined in terms of a loss rate with a specified bandwidth and availability. Attainable throughput is derived from loss. Delay andjitter are not important for this service and are not defined. The class may support a commitment for per-flow sequence preservation.
2.8. Design objectives
In designing an IP backbone network with support for different classes of traffic, service providers have three key design objectives:
• Committing to the different per class SLA requirements.
• Making optimal use of available bandwidth.
• Keeping the design as simple as possible.
In the remainder of this paper, we describe the different technologies that are required to satisfy these requirements.
3. The differentiated services architecture
3.1. IP backbone diffserv overview
The Diffserv architecture [3] is the preferred technology for large-scale IP QoS deployments today, such as service provider backbone networks.
Diffserv achieves scalability through performing complex QoS functions such as classification, marking, and conditioning operations at the edges of the network. Traffic is classifiedandthen markedusing the Diffserv code point (DSCP) [21] into a limitednumber of traffic aggregates or classes. Within the core of the network, scheduling andqueuing control mechanisms are appliedto the traffic classes basedupon the DS fieldmarking; all traffic conditioning and dropping is handled intelligently at the network layer using IP Diffserv quality of service mechanisms. Diffserv is not prescriptive in defining the scheduling and queuing control algorithms that shouldbe implementedat each hop, but rather, uses a level of abstraction in defining the externally observable forwarding behaviours, termedper-hop behaviours (PHBs), that can be appliedto traffic at each hop. Currently, three PHBs are defined:
• The expedited forwarding (EF) PHB. The EF PHB [8,16] is usedto support traffic with low loss, low delay, low jitter, assured bandwidth requirements, such as VoIP.
• The assured forwarding (AF) PHB. The AF PHB [14] is usedto support data traffic with assured bandwidth requirements.
• The default PHB. This PHB [21] represents the
default forwarding behaviour. Packets, which are not identified as belonging to another class, belong to this aggregate.
A typical Diffserv backbone implementation, that couldbe usedto support the three classes of service previously defined, is described as follows:
• Real-time. The real-time traffic class is treated with an EF per-hop behaviour. This is typically implementedusing a strict priority queue within a scheduling mechanism, 3 which services the strict priority queue with priority above all other queues. Such a queuing mechanism assures that the real-time class is isolatedfrom the impact of the other classes.
3 Such as the modified Deficit Round Robin scheduler used on high-performance Cisco routers.
The forwarding resources for the class are al-locatedto ensure that with the expectedload there will be no congestion within the class, such that it receives gooddeterministic service with low loss, low delay and low jitter. • Business data. Business data class traffic is treatedwith an AF per-hop behaviour. This is typically allocatedto a queue within a weighted-fair-queuing-like scheduling mechanism. Such a queuing mechanism assures that the class receives a bandwidth allocation relative to the best-effort data class and that it can use all available forwarding resources after the VoIP andbest-effort data class queues have been serviced.
The forwarding resources for the class are allo-catedto ensure that for the majority of the time, with the expectedloadthere is no congestion within the class such that it receives goodde-terministic service with low loss. The assumption is made that the majority of the business data class traffic is TCP/IP [20], and hence the Random Early Detection (RED) [13] congestion control mechanism is usedwithin the class queue rather than tail drop to ensure that TCP throughput within the class is maximised when congestion occurs. If a service is defined with in-andout-of-contract capabilities (similar to the concept of discard eligibility [DE] set/unset within frame-relay), WeightedRED (WRED) is usedto have two different RED profiles 4 per queue: an aggressive profile for out-of-contract traffic, anda gentler profile for in-contract traffic to make sure that in case ofclass congestion, out-of-contract packets are dropped first. 5
4 The characteristics of RED are defined by a minimum queue threshold(minth), maximum queue threshold(maxth), andprobability of discardat maxth(maxp). WRED allows multiple redprofiles to be supported in the same queue with separately defined minth, maxth andmaxp per profile. This results in different drop characteristics (and consequently probability of drop) per profile.
5 This is achievedby choosing a maximum threshold for out-of-contract traffic smaller or equal to the minimum threshold for in-contract traffic.
• Best-effort data. The best-effort class traffic uses an AF per-hop behaviour. This wouldtypically be allocatedto a queue within a weightedfair-queuing scheduling mechanism. Such a queuing mechanism assures that the class receives a bandwidth allocation relative to the business data class and that it will be able to use all available forwarding resources after the VoIP andbusiness data class queues have been serviced.
As with the business data class, the assumption is made that the majority of the best-effort data class traffic is TCP/IP, therefore the RED congestion control mechanism is usedwithin the class queue rather than tail drop to ensure that TCP goodput within the class is maximised when congestion occurs.
This paper does not consider the classification, conditioning and marking functions performed at the edges of the network, but assumes that such mechanisms are used. An edge design may offer more classes of service; for example, two or three different classes for business data traffic to either isolate applications and/or ease the budgeting of the bandwidth cost between different customer departments. In this case, the capability to map several edge classes into a single aggregate backbone class is requiredon the SP's edge router. Several edge business data classes may be mapped into the backbone business data class or two distinct edge VoIP and video classes may be mapped into the aggregate backbone real-time class. This mapping can be realizedin two ways:
• A backbone class can match several DSCPs. For example, if at the edge of the network, a DSCP value of 40 represents VoIP at the edge anda value 32 represents Video, the backbone aggregate real-time class wouldmatch both
DSCP 40 andDSCP 32.
• When MPLS is usedin the backbone, the edge SP router can set the 3 bit MPLS experimental (EXP) fieldas a function of the receivedDSCP. For example, if an EXP value of 5 is usedfor the aggregate real-time backbone class, the SP's edge router will impose MPLS labels headers with an EXP value of 5 for packets received with DSCP 40, representing the edge VoIP class, or with DSCP 32, representing the edge Video class.
3.2. Diffserv backbone design for low loss and low jitter
Designing an IP backbone network for low loss, low delay and low jitter can be relatively simple: one simply needs to overprovision the bandwidth comparedto the actual load[4,6,7]. Ref. [6] shows that for a best-effort IP backbone, worst case jitter was measuredat less than 1 ms for probes sent at 1Mbps during a seven-day period between the east andwest coast POPs of a Tier-1 Internet Service Provider (ISP). During this period, the loss rate was zero. These measurements demonstrate the excellent performance that can be achievedin an IP backbone when designed with high-speed links (from STM-4/OC12 to STM-64/OC192) andwith conservative capacity provisioning rules where links are upgraded when utilisation reaches 4050%, aiming to ensure that there is at least twice as much capacity as actual load.
Such a simple design rule allows tight SLAs to be achievedfor delay, jitter andloss. Unfortunately, however, this does not satisfy our initial requirements as this represents an expensive option for the SP. If, for example, between two POPs there is 150 Mbps of VoIP traffic and1.5 Gbps of best-effort data traffic, then by the above rule, twice the sum of the VoIP andbest-effort traffic wouldbe neededto assure low jitter andloss of the VoIP class. In this case, the sum of the VoIP and data traffic is 1.65 Gbps and hence 3.3 Gbps of capacity needbe provisioned. In practice, this wouldtypically be supportedusing two STM-16/ OC48 links, resulting in 5 Gbps of bandwidth being deployed to support 1.65 Gbps of aggregate loadwith 150 Mbps of low delay, low jitter, low loss traffic.
Diffserv provides a solution to this problem, in that it allows per class virtual backbones to be built on a single physical backbone. This gives SPs the flexibility to have different under- or overbooking ratios (the ratio of offeredloadto available capacity) for each service class. Using the previous example, this couldallow the VoIP class capacity to be over provisionedby a factor of 4 hence ensuring that the class receives goodservice (with low delay, low jitter and low loss), whilst the data class capacity could be over provisioned by a factor of 1.2 (a realistic figure still giving good service). This wouldresult in 2.1 Gbps of bandwidth required in total, or rounded up to a single STM-1 /OC48 link, which represents a potential saving of 1x STM-16/OC48 link over the non-Diffserv case. To explain how this calculation was derived, the VoIP class traffic is assumed to be servicedfrom a strict priority queue andthus will effectively have access to all bandwidth on the
physical link; for an STM-16/OC48 link this would
result in effective over provisioning by a factor of 2.5 Gbps/150 Mbps = ~ 16. The best-effort class traffic being servicedfrom a weighted-fair queue would, however, have access to all available bandwidth on the physical link once the VoIP class traffic has been serviced; for an STM-16/OC48 link this would result in effective bandwidth over provisioning by a factor of [(2.5 Gbps-150 Mbps)/1.5 Gbps] = ~ 1.6.
This example is intended to highlight two key points: that backbone Diffserv deployment is conceptually simple andthat the concepts involved have been proven by experience with deployed best-effort IP networks. It is simple in that it allows different service levels to be supported merely by using different under- or over-provisioning ratios per class; the higher the available capacity com-paredto the offered load, the tighter the SLA (lower delay, jitter, and loss rates) that can be supported. 7 It is proven in that empirical evidence
6 In practise, an overprovision factor of 4 is used, based upon the simple assumption that single-link or node failure conditions can result in a doubling of the per link load. Consequently, using as overprovision factor of 2*2* the maximum expectedloadin non-failure conditions aims to ensure that even in failure conditions, there is twice as much capacity as traffic load andhence low delay, jitter andloss service is maintained.
7 Assuming two classes, A andB, such that A's overprovi-sioning ratio is higher than B. A tighter delay SLA for A can be expressedeither by a smaller delay boundfor A than B but with the same availability, or by the same delay bound but with an higher availability for A than for B, or a combination of both schemes.
from high-speedbest-effort (single class of service) SP IP backbones indicates that extremely good jitter andloss targets can be achievedwith simple rules of over provisioning.
3.3. Diffserv backbone deployment considerations
Consideration obviously needs to be given to whether the cost of deploying Diffserv outweighs the benefits it provides. There is no generic answer to this question, andthe benefits that will be gained will vary deployment by deployment. In the example above, if the Diffserv deployment cost exceeds the cost of an additional STM-16/OC48 link (andthe router ports which terminate it) then there is clearly no sense in deploying Diffserv. We consider some of the most significant factors that impact the economic viability of deploying Diffserv:
• Economic benefit. The maximum potential economic benefit stands to be gained from Diffserv deployments where the traffic requiring the highest SLA targets represents a minor proportion of the overall capacity. As the previous example demonstrates, the absence of Diffserv leads the designer to provision capacity equal to the aggregate loadacross all classes multipliedby the over dimensioning ratio of the tightest-SLA class. This can be extremely expensive when the tightest-SLA class represents a low proportion of the aggregate traffic. Conversely, when all classes require the same level of service, and hence the same overbooking ratio, there is no benefit to be gainedfrom Diffserv.
• Impact on router performance. If Diffserv EF/ AF forwarding behaviours have an impact on router forwarding performance, the less aggregate throughput the router can support with Diffserv enabled, and consequently, the higher the per port cost of the network deployment. Today's high-performance routers typically implement the EF/AF forwarding behaviours in ASICs, ensuring that there is no forwarding penalty associatedwith the support of the Diff-serv functionality
• Simplicity of deployment. Backbone Diffserv deployments generally require relatively minor and simple changes to existing router configurations. A typical Diffserv design consists in defining three queues: an EF class queue for real-time traffic such as VoIP, andtwo AF class queues, one for the business data class and the other for the best-effort class. The real-time class traffic is servicedwith a priority queuing treatment, andthe expectedloadfor the class is below 25% (using an overprovision factor of 4) of the available link capacity in most designs. The bandwidth remaining once the real-time class has been servicedis allocatedwith 90% to the business data class and 10% to the best-effort class. An example router configuration, which could be used to implement the described Diffserv design, is shown in Fig. 2; additional lines of configuration requiredto implement the Diffserv policy are shown in boldtype in the figure.
As can be seen from Fig. 2, only 12 additional lines of configuration are requiredto implement the Diffserv policy for a single interface. Only a single additional line of configuration is required for each additional interface where this Diffserv template will be configured. Typically, in backbone Diffserv deployments these configurations are appliedonce, andthen remain static andare never changed.
Furthermore, migrating a backbone to Diff-serv can be achievedseamlessly: the backbone configuration can be undertaken independently of the configuration requiredat the network edge to ensure that traffic is appropriately conditioned and marked on ingress to the network. The benefits of Diffserv, however, will not be realiseduntil both edge andbackbone components are complete. • NMS/OSS requirements.  Backbone network management systems (NMS) andoperational support systems (OSS) typically needenhancing to support Diffserv deployments: o The NMS system needs to be enhanced to retrieve bytes/packets transmittedanddropped per class rather than per interface. In response to the Diffserv deployments that have occurred during the 24 months preceding the publication of this paper, NMS applications now typically provide support for such statistics.
class-map REALTIME
match ip dscp 40 class-map BUSINESS
match Ip dscp 32 class-map BEST_EFFORT
match any
i
policy-map DIFFSERVPOLICY class REAL_TIME
priority class BUSINESS
bandwidth remaining percent 90 class BEST_EFFORT
bandwidth remaining percent 10
1 j
interface pos 1/0
service-policy outbound DIFFSERV_POLICY
This section defines the classification criteria used to determine which traffic goes in which queues, tn this case traffic marked DSCP 40 and 32 will be classified as Real-time and Business Data classes respectively, whilst all other traffic will be classified as the Best-effort class
This section is a template, which defines the actual queuing treatment that each class will receive. A strict priority queue is defined for the Real-time class, whilst the Business data and Best-effort classes are allocated to queues, which are guaranteed a minimum of 90% and 10% of the remaining bandwidth respectively. The RED configuration for the data classes is not shown.
In this section, the queuing template defined above is attached to an actually interface
Fig. 2. Example backbone router Diffserv configuration.
o The deployment of an active SLA probing system is highly advisable [15] in order to be able to monitor (andreport) delay andjitter. Some router vendors implement software agents in their routers 8 that sendandreceive probes with user-definable DSCP and protocol identities (e.g. FTP, HTTP, DNS). Leveraging the installedbase of routers in each POP allows rapiddeployment of an SLA active monitoring system without any major rollout of new network equipment. • Capacity planning. In terms of operational process, the capacity planning of a Diffserv backbone is similar to a single-class best-effort IP network: loadstatistics are collectedon a per-class of service basis andwhen loadthresholds are reached, an addition of network bandwidth is triggered. The accuracy of this capacity planning can be tunedbasedupon the active SLA probing results which allows correlation between per class loadandSLA parameter reports of delay and jitter.
8 For example, service assurance agent (SAA) functionality in Cisco IOS.
• Training. Diffserv is a new technology and therefore it is inevitable that training of design andoperational staff will be requiredto support a backbone Diffserv deployment.
3.4. Diffserv backbone performance characteristics
We conclude the discussion on Diffserv by presenting the results of router basedtesting, which illustrate both the tight-latency, jitter, andloss capabilities of today's router technology and the potential benefits andcharacteristics that can be achieved with backbone Diffserv deployment de-scribedabove.
The testing was undertaken using a Cisco 12416 router. This router has a distributed architecture that supports the EF andAF per-hop behaviours implementedin ASICs on each line cardusing a deficit round robin (DRR) [25] scheduling algorithm, which has been modified by Cisco to add support for a strict priority queue for EF class traffic.
The Diffserv configuration shown in Fig. 2 was usedfor all tests. The packet size usedfor real-time traffic during testing was 200 bytes, whilst the business data and best-effort traffic followed an
Internet-mix packet size distribution. The router under test has three ingress STM-16/OC-48 ports receiving traffic from a traffic generator. The router aggregates this traffic andforwards it onto the single-hop link under test which is an STM-16/OC-
48 Packet over SDH/SONET (POS) link.
Three characteristics of the router EF andAF implementation, which are key to the successful deployment of the Diffserv design described above, form the basis of the tests andresults presented:
• Latency of EF class. The first test measuredthe one-way delay of the real-time class in the presence of interface congestion. A worst-case delay success criterion of 500 (is was set in order to ensure that the 500 is delay-jitter target was never exceeded. 10
• Latency ofAF class. The secondtest measured the delay of the business data class traffic under increasing loadwithin that class.
• Accuracy of AF bandwidth allocation. The final test measured the bandwidth allocation accuracy of the AF class traffic (business data and best-effort), for different relative bandwidth allocations. Successful Diffserv deployment depends upon being able to manage the relative under- and over-booking ratios between the classes, which is in turn dependent upon the accuracy of the scheduler implementation in terms of AF bandwidth allocation.
3.4.1. Latency for EF traffic
The first results demonstrate the low delay that can be achievedusing an EF compliant priority queue scheduler. Fig. 3 charts the recorded per-centile distribution for the real-time class (EF) traffic delay through a single-hop link running at STM-16/OC48 andbeing congested with 30% of VoIP, 45% of business traffic and150% of best-effort traffic.
Fifty eight percentage of the packets are small packets (40 bytes), 33% of the packets are medium sized packets (552 bytes) and9% of the packets are large packets (1500 bytes).
10 The delay-jitter can never exceed the difference between the best-case andworst-case measuredone-way delay.

Fig. 3. Latency for EF class.
Fig. 3 clearly illustrates the low-jitter service provided by the priority queuing mechanisms to the VoIP (EF) traffic. The percentile 100 delay of the VoIP packets is 125 (is. This result was independent of the load of the other classes even under 225% congestion of the outboundport. These results are significantly better than the target of 500 is.
3.4.2. Latency for business class
This test demonstrates that very good maximum delay can be achieved for a well-provisioned AF class queue, such as the business data class queue in our case. In this test, the link under test was loaded with bursty traffic in the PQ up to 30% of the STM-16/OC48 link rate, and150% of best-effort class traffic. The business data class load was then variedfrom 0% to 200% (of the configured business class capacity) andthe maximum delay of the business class traffic was measured. Fig. 4 illustrates the test results.
With business class loadless than business class capacity, no packet loss is experiencedfor the class andthe latency remains extremely low at ~160 is until ^85% business class load. As the business loadincreases to 100% the latency increases to ~1 ms at 100% load. Above 100% business class load packet loss occurs andthe average delay mea-suredincreases to ^100 ms at 220% load, which in the case of our testing was the expectedresult

Fig. 4. Latency for an AF class.

Fig. 5. Bandwidth accuracy for AF queues.
determined by the RED configuration that we used. 11
3.4.3. Bandwidth allocation between AF queues
The final test demonstrates the accuracy of the bandwidth allocation that can be achieved between different AF classes. In this test, the link under test was loaded with bursty traffic in the PQ up to 30% of the STM-16/OC48 link rate, and variable equal rates of business andbest-effort class traffic giving an aggregate interface loadof greater than 200% of the STM-16/OC48 link rate. The ratio of the configuredbandwidth allocations between the business andbest-effort classes was then varied, and the actual bandwidth allocation ratio was measuredandcomparedto the expected result. Fig. 5 graphs the difference between the measuredbusiness bandwidth allocation andthe expectedbandwidth allocation, as a percentage of the STM-16/OC48 link rate.
Fig. 5 shows different test cases, which correspond to different relative bandwidth allocations between the business andbest-effort classes (50/50, 66/34, 75/25 etc.). In all tests, the bandwidth allocation accuracy is well within 1% of the expected result.
4. Fast IGP convergence
Link or node failures in an IP backbone cause packet losses until the network has reconverged aroundthe failedlink or node. These packet losses directly impact the availability that can be offered for SLAs across all classes. To assess the significance of this one can compute the amount of downtime corresponding to different availability commitments: the often-quoted 99.999% or ''five-nines''-target figure for network availability potentially equates to less than 1 s of downtime per day.
Another way to illustrate the significance of downtime due to a link or node failure is to consider the impact on applications and end-users. For VoIP calls, the end-users will perceive a glitch in the call as soon as a few samples are lost. For example, with one sample every 20 ms, a loss in connectivity of 100-150 ms will be perceptible to the human ear. If the connectivity loss is for 1-2 s the call can be dropped. Consequently, networks supporting high-quality VoIP services are being engineeredfor network convergence upon link/ node failure of 1 s or less. The time taken for an IP network to reconverge is dependent upon the size of the network, 12 the Interior Gateway Routing protocol (IGP) usedandits specific configuration. For high-SLA availability targets to be offered, it is important that the routing protocol is tunedfor rapidconvergence. A key component of tuning IGP convergence is the tuning of the timers, which determine how frequently the main routing protocol events can occur. Historically, this has re-sultedin a trade-off between rapidconvergence
andincreasedrouting protocol stability: short timers resultedin rapidconvergence but with more potential for instability, where longer timers re-sultedin increasedstability but slower convergence. The pragmatic result of this trade-off was that routing protocol timers were generally set conservatively andIP network convergence was typically a few tens of seconds.
Experience gainedfrom large-scale service provider deployment, however, indicated that such IGP implementations were very stable andhence that more emphasis couldbe placedon faster convergence. Further, recent developments to ISIS andOSPF link state IP IGPs have focussedon combining the best of both worlds leading to significant reductions in the convergence that can be achievedwhilst still maintaining stability. Whereas previously IGP timers were static andlong, now with the introduction of dynamic timers they can adapt their responsiveness depending upon the stability of the network. This allows IGPs to be tunedsuch that when the network is stable, their timers will be short andthey will react within a few milliseconds to any network topology changes. In times of network instability (e.g. causedby a flapping link), however, the IGP timers will increase in order to throttle the rate of response to network events. This scheme ensures fast convergence when the network is stable andmoderate routing protocol overhead(e.g. CPU cycles consumed) when the network is unstable.
In addition to the advancements in the tuning of routing protocol timers, a number of other developments have improvedthe IGP convergence that can be achieved:
• Partial routing calculation and incremental SPF [23,24]. IGP implementations have been en-hancedto react more optimally to various topology changes. For example, with the introduction of partial routing calculations for IS-IS and OSPF, if only an IP leaf subnetwork changes without a topological change to the shortest path tree, then the router does not need to recompute the shortest-path tree, it just recomputes its routing table basedupon the existing tree. On the other hand, when only a part of the graph has changed, incremental SPF optimises the calculation by only recomputing the part that has changedinsteadof the whole graph. Such optimisation leads to faster routing computations, hence to faster convergence. Early empirical data from large-scale network deployments suggests that the average IGP routing table computation time can be reduced by 90% due to such optimisations.
• Design practice. Best practise for service provider IGP design today aims to reduce the number of routes that are carriedin the IGP; all Internet andcustomer routes are carriedby the Border Gateway protocol (BGP); in some cases even the interior link prefixes are carried in BGP. This practise significantly reduces IGP routing table computation times andhence results in faster IGP convergence. Obviously, the routing table for a network of 100 routers and 1000 IP prefixes can be computedfaster than one with 100 routers and 10,000 prefixes.
• Rapid layer 2 failure detection. Most service provider backbones today are built using Packet Over SDH/SONET (POS) links where signalling inherent in SDH/SONET is designed to detect link or node failures in less than 10 ms. This rapid failure detection can considerably speed up convergence, when comparedto other media such as Ethernet, which rely on hellos at the IP layer to detect failure.
The combination of these optimisations has re-sultedin a reduction of IGP convergence times from several 10 s of seconds, to 1-2 s being pragmatically achievable today. This has been con-firmedin testing in a 1000 router network with IP 4000 prefixes in the IGP (work in progress). Further, as we learn from additional deployment experience, and with additional tuning and enhancements, sub-secondIGP convergence may become a realistic possibility.
5. MPLS traffic engineering and diffserv-aware traffic engineering
5.1. MPLS traffic engineering
In conventional service provider IP networks routing protocols such as OSPF andIS-IS forwardIP packets on the shortest cost path to the destination IP address of each IP packet. The computation of the shortest cost path is based upon a simple additive metric, where each link has an appliedmetric, andthe cost for a path is the sum of the link metrics in the path. Availability of network resources, such as bandwidth, is not taken into account and, consequently, traffic can aggregate on the shortest path, potentially causing links on the shortest path to be congestedwhile links on alternative paths are under-utilised.
This property of conventional IP routing protocols, of traffic aggregation on the shortest path, can cause sub-optimal use of network resources, andcan consequently impact the SLAs that can be offered(or require more network capacity than is optimally required).
MPLS traffic engineering (TE) [2] uses the implicit MPLS characteristic of separation between the data plane (also known as the forwarding plane) andcontrol plane to allow routing decisions to be made on criteria other than the destination IP address in the IP header, such as available link bandwidth. MPLS TE effectively provides an explicit routing capability at Layer 3, allowing paths to be usedother than the shortest cost path to a destination, thereby avoiding traffic aggregation on the shortest path andproviding more optimal use of available bandwidth.
MPLS TE uses the following mechanisms:
• Information on available network resources, including a pool of available bandwidth main-tainedper link, are flooded by means of extensions to link-state basedIP routing protocols
such as IS-IS [26] andOSPF [17].
• A constraint-basedrouting (CBR) algorithm is usedto compute the traffic path basedupon a fit between the available network resources (advertised via IS-IS or OSPF) and the resources required, i.e. a requested amount of bandwidth.
• The Resource ReSerVation Protocol (RSVP) [5], with enhancements for MPLS TE [1], is used to signal andmaintain an explicit route (termed a ''traffic engineeredtunnel''), from head-end to tail-end, in the form of an MPLS Label Switched
Path (LSP). This LSP follows the path determinedby the constrain-basedrouting algorithm. In signalling the tunnel, admission control is performedat every hop.
• Traffic routedonto these LSPs or tunnels will then follow the traffic engineeredexplicit route to the destination, rather than the conventional IGP shortest path.
The following conditions can all be drivers for the deployment of MPLS traffic engineering
[28]:
• Network asymmetry. Asymmetrical network topologies can often leadto traffic being aggre-gatedon the shortest path whilst other viable paths are under-utilised. Network designers will often try to ensure that networks are symmetrical such that where parallel paths exist, they are of equal cost andhence the loadcan be bal-ancedacross them using conventional IGPs. Ensuring network symmetry, however, is not always possible due to economic or topological constraints. Traffic engineering offers obvious benefits in these cases.
• Unexpected demand. In the presence of unexpected traffic demand (e.g. due to some new popular content), there may not be enough capacity on the shortest path (or paths) to satisfy the demand. There may be capacity available on non-shortest paths, however, andhence traffic engineering can provide benefit.
• Long bandwidth lead-times. There may be instances when new traffic demands are expected andnew capacity is requiredto satisfy the demand, but is not available in suitable timescales. In these cases, traffic engineering can be usedto make use of available bandwidth on non-shortest path links.
The use of TE gives the service provider flexibility in how to manage their backbone bandwidth in order to achieve their SLAs. The more effective use of bandwidth potentially allows higher service availability targets to be offeredwith the existing backbone bandwidth. Alternatively, it offers the potential of achieving the existing service availability targets with less backbone bandwidth.
5.2. Diffserv and TE
MPLS TE andDiffserv can be deployedcon-currently in an IP backbone, with TE determining the path that traffic takes on aggregate basedupon aggregate bandwidth constraints, and Diffserv being usedon each link for differential scheduling of packets on a per class basis. Whilst TE and Diffserv are orthogonal technologies they can be usedin concert for combinedbenefit: TE allows distribution of traffic on non-shortest paths for more efficient use of available bandwidth, whilst Diffserv allows over/under-provisioning ratios to be determined on a per class basis.
MPLS TE, however, computes tunnel paths for aggregates across all traffic classes andtraffic from different classes may use the same TE tunnels. MPLS TE is aware of only a single aggregate global pool of available bandwidth per link and is unaware of what specific link bandwidth resources are allocatedto which queues, andhence to which class.
Consequently, MPLS TE has been extended with Diffserv-aware traffic engineering (DS-TE) [10], which introduces the concept of an additional andmore restrictive pool of available bandwidth on every link. This more restrictive bandwidth pool is termedthe sub-pool, while the regular TE bandwidth is called the global pool (the sub-pool is a portion of the global pool). The sub-pool may be usedfor constraint-basedrouting andadmission control of tunnels for ''guaranteed'' or EF class traffic andthe global pool usedfor regular (non-"guaranteed") or AF class traffic.
In supporting DS-TE, extensions have been added to IS-IS and OSPF [11] to advertise the available sub-pool bandwidth per link as well as the available global-pool bandwidth. In addition, the TE constraint-basedrouting algorithms have been enhancedfor DS-TE in order to take into account the constraint of available sub-pool bandwidth in computing the path of sub-pool tunnels. RSVP has also been extended [11] to indicate if it is signalling a sub-pool or global-pool tunnel.
It is understood that setting an upper bound on the EF class (e.g. VoIP) effective utilization per link allows a way to restrict the effects of delay and

Fig. 6. DS-TE deployment example.
jitter due to accumulated burst [4,7]. DS-TE can be usedto assure that this upper boundis not exceeded. For example, consider the network in Fig. 6 DS-TE couldbe usedto ensure that traffic is routedover the network so that, on every link, there is never more than 25% (or any assigned percentage) of the link capacity for EF class traffic, whilst there can be up to 100% of the link capacity for EF andAF class traffic in total. Each link in Fig. 6 is 100 Mbps. R1 is sending an aggregate of 40 Mbps of traffic to R8, andR2 is also sending an aggregate of 40 Mbps of traffic to R8. An IGP and TE metric value of one is appliedto each link.
In this case, both the IGP andnon-Diffserv aware TE wouldpick the same route. The IGP wouldpick the top route (R1/R2 ! R3 ! R4 ! R5 ! R8) because it is the shortest path (metric of 4), whilst TE wouldpick the same path because it is the shortest path that has sufficient bandwidth available (metric of 4, 100 Mbps bandwidth available, 80 Mbps required). The decision to route both traffic aggregates via the top path may not seem appropriate ifwe examine the composition of the aggregate traffic flows.
If each of the aggregate flows is comprisedof 5 Mbps of VoIP traffic and35 Mbps of business data traffic, then in this case the total VoIP traffic loadon the top links will be 10 Mbps, which is within our EF class boundof 25%. If, however, each traffic aggregate is comprisedof 20 Mbps of VoIP and20 Mbps of business data traffic then such routing wouldaggregate 40 Mbps of VoIP traffic on the R3 ! R4 ! R5 links, thereby exceeding our EF class bound of 25%. DT-TE can be usedto overcome this problem: each link is con-figuredwith an available global pool bandwidth of 100 Mbps, andan available sub-pool bandwidth of 25 Mbps (i.e. 25% of 100 Mbps). A global-pool tunnel of 20 Mbps is then configuredfrom R1 to R8 for business data traffic, and a sub-pool tunnel of 20 Mbps for VoIP traffic. Similarly, from R2 to R8 a global-pool tunnel of 20 Mbps is configured for business data traffic, and a sub-pool tunnel of 20 Mbps for VoIP class traffic. The DS-TE constraint basedrouting algorithm wouldthen route the sub-pool tunnels to ensure that the 25 Mbps boundis not exceededon any link, andof the tunnels from R1 andR2 to R8, one sub-pool tunnel wouldbe routedvia the top path (R1/ R2 ! R3 ! R4 ! R5 ! R8) andthe other via the bottom path (R1/R2 ! R6 ! R7 ! R5 ! R8). 13 In this particular case, there wouldbe enough available bandwidth for both global pool tunnels to be routedvia the top path (R1/R2 ! R3 ! R4 ! R5 ! R8), which has the shortest metric.
DS-TE enables service providers to perform separate route computation andadmission control for different classes of traffic. This enables the distribution of EF andAF class loadover all available EF andAF class capacity making optimal use of available capacity. It also provides a tool for constraining the EF class utilization per link to a specifiedmaximum thus providing a mechanism to help bound the delay and jitter. In order to provide these benefits, however, the configured bandwidth for the sub-pool andglobal pool must represent queuing resources, which are only available for traffic-engineeredtraffic, and hence non-traffic en-gineeredtraffic shouldbe queuedseparately on each link. By combining DS-TE with Diffserv queuing mechanisms on each link, the service provider can offer tight-SLAs for EF class traffic (such as VoIP) with admission control without large over-provisioning of capacity.
6. MPLS traffic engineering fast reroute
In Section 4, we highlightedthat link or node failures in an IP backbone can significantly impact the availability that can be offeredfor SLAs across all classes. Whilst sub-secondconvergence for IP routing protocols is a realistic prospect, it is ex-pectedthat IGP convergence will not be able to match the capabilities of SDH/SONET networks, which use the capabilities of Multiplexer Section Protection (MSP) andAutomatic Protection Switching (APS) respectively to recover around failures in tens of milliseconds. This is because the functions are performed in fundamentally different ways: IGP convergence is basedon a distributed computation 14 whereas SDH/SONET restoration is basedupon local detection andpre-computed local protection aroundthe failure.
MPLS traffic engineering fast reroute (FRR) extends the concepts of local failure detection and protection to MPLS TE in order to provide very rapidrecovery aroundfailures (e.g. a few tens of milliseconds) prior to any distributed convergence/ reoptimisation. Without FRR, under failure conditions, the head-end of a TE tunnel determines a new route for the tunnel LSP. Recovery at the head-end provides for the optimal use of resources, however, due to messaging delays, the head-end cannot recover as fast as is possible by making a repair at the point of failure.
MPLS TE Fast Reroute adds additional capabilities to MPLS TE in that it provides local protection of tunnel LSPs in the presence of link failure. This enables all traffic carriedby tunnel LSPs that traverse a failedlink to be rerouted aroundthe failure. The reroute decision is completely controlledlocally by the router interfacing the failedlink.
MPLS TE FRR uses the following mechanisms:
• Hierarchical LSPs. FRR uses the concept of hierarchical LSPs; the protectedtunnel LSPs are switchedrapidly into backup tunnel LSP at the point of failure (also known as the point of local repair). The backup tunnel-LSP provides an explicit route aroundthe failedlink or node. To support FRR, extensions have been added to
A propagation-delay constraint can also be specified for the sub-pool tunnels to ensure that the chosen path exhibits a propagation delay smaller or equal to the specified value [12].
14 The local node next to the failure distributes the information andthen it is up to all the other nodes to each compute their routing table to route aroundthe failure.
RSVP [27] to indicate which tunnels are fast reroutable andto carry additional label information requiredfor FRR. For maximum protection, the path for the backup tunnel LSP can be determined to use disjoint resources (such as optical channels, fibres or ducts) from the protectednode or link.
• Rapid failure detection. If POS links are used, SDH/SONET signalling allows link or node failures to be detected by directly connected nodes in less than 10 ms. Where POS links are not used, RSVP hellos can be used between adjacent nodes for failure detection.
• Rapid local protection. Rapidlocal protection aroundthe failure is possible because the switching entries for the backup tunnel-LSP are pre-computed. Once a failure is detected, the local node has only to copy the pre-computed backup tunnel-LSP switching entry into its switching table for recovery aroundthe failure to be achieved.
• Path reoptimisation. The head-end of the tunnel is also notifiedof the link failure through the IGP or through RSVP; the head-end may then attempt to establish a new, andpossibly more optimal, LSP that bypasses the failure, using make-before-break signalling with RSVP. This is a significant advantage compared to SDH/
SONET protection.
The local nature of FRR allows very rapid protection andrestoration aroundfailures in IP backbone networks. For SDH/ SONET links, detecting the failure of a link is typically done in less than 10 ms, andwith FRR, many hundreds of protectedtunnel-LSPs can be switchedaroundthe failure in less than 50 ms. This is equivalent to the level of protection provided by MSP and APS and in SDH andSONET networks respectively.
FRR is designed for backbone deployment where the number of network components is typically relatively low, but where the failure of those components can have severe impacts on services andSLAs. The determination of optimal routing for FRR backup tunnels in different failure scenarios is, however, a complex problem andneeds to take into account factors including the available bandwidth on potential backup paths, tunnel inter relationships andinterdependencies on the lower layer network topologies. This subject is currently the focus of further research anddevelopment efforts.
For the service provider, FRR provides the capability to significantly improve the perception of service quality for IP telephony users. For IP telephony, if IP connectivity is lost for a few hundreds of milliseconds, the users will perceive a glitch in their call. By deploying FRR to protect key network resources, service providers can ensure that link failures are imperceptible to IP telephony users andcan hence offer the highest availability of service for VoIP class traffic. Further, enabling this protection technique at the IP layer allows for better statistical multiplexing, better reoptimisation upon active protection and cost advantage due to the consolidated architecture, when comparedto providing this capability at lower layers, using SDH/SONET capabilities for example.
7. Conclusion
In the context of ever more competitive service provider offerings and ever more demanding requirements from their customers, this paper has analysedwhich SLA parameters are significant for IP service performance (delay, jitter, loss bandwidth/throughput, per-flow sequence preservation andavailability) andhas listedthe targets usually set for these parameters for the typical backbone aggregatedclasses of service.
We have reviewedthe technologies that can be usedto tighten these SLAs andhence serve as foundation for a multiservice IP backbone: Diffserv, fast IGP convergence, traffic engineering, Diffserv-aware traffic engineering andMPLS TE
FRR.
The Diffserv analysis showedthat there should be an important economical benefit in leveraging this technology, which enables under- or overbooking to be performedon a per class rather than aggregate basis. This per class under-/over-book-ing capability enables tight-SLAs to be offeredfor some classes of traffic without aggregate over-provisioning of capacity, leading to savings in terms of required bandwidth. It was noted that the typical Diffserv deployment strategy targeted by current designs is relatively simple and only requires a one-time router configuration. The results of router-basedtesting finally illustrates the tight-SLA capabilities of today's high-performance routers andconfirmed that the previously defined SLA targets couldeasily be met. For these reasons, a multiservice IP backbone design should leverage Diffserv technology.
Recent developments in IGP implementations have resultedin significant improvements in IP backbone IGP convergence with convergence times 1-2 s being pragmatically achievable today. This reduction in convergence times allows higher availability targets to be offeredfor SLAs across all service classes. Consequently, fast IGP convergence is also recommended as a foundation of multiservice IP backbone network designs
MPLS traffic engineering gives the service provider the capability to use available backbone bandwidth more effectively. Either it allows higher service availability targets to be offeredwith the existing backbone bandwidth or, alternatively, it offers the potential of achieving the existing service availability targets with less backbone bandwidth. Notwithstanding these benefits, it is noted that not all networks will benefit from the deployment of
TE
Diffserv-aware MPLS traffic engineering extends the base capabilities of TE to allow route computation andadmission control to be per-formedseparately for different classes of service. By combining DS-TE with Diffserv mechanisms on each link, the service provider can offer tight-SLAs for EF class traffic (such as VoIP) with admission control without large over-provisioning of capacity.
Finally, for networks seeking sub-100 ms convergence, MPLS TE fast reroute provides the capability for protection aroundfailures at the IP Layer. By deploying FRR to protect key network resources, service providers can ensure that link failures are imperceptible to IP telephony users andcan hence offer the highest availability of service for VoIP class traffic.
In summary, the foundation for a multiservice tight-SLA IP network consists in a Diffserv and
Fast-IGP design. This design is complemented by MPLS technologies such as traffic engineering, Diffserv-aware traffic engineering andfast reroute depending on the specific context and requirements of the considered network.
Acknowledgements
The authors wouldlike to thank Olivier Bona-venture, Thomas Telkamp andBruce Thompson for their valuable feedback and constructive comments.
References
[1] D.O. Awduche et al., RSVP-TE: extensions to RSVP for
LSP tunnels, RFC3209, December 2001. [2] D. Awduche et al., Requirements for traffic engineering
over MPLS, RFC 2702, September 1999. [3] S. Blake et al., RFC2475, An Architecture for Differenti-
atedService, December 1998. [4] T. Bonald, A. Proutiere, J. Roberts, Statistical guarantees
for streaming flows using expedited forwarding, INFO-
COM 2001.
[5] R. Braden et al., RFC2205, Resource Reservation Protocol (RSVP) Version 1 Functional Specification, September 1997.
[6] S. Casner, C. Alaettinoglu, C.-C. Kuan, A fine-grained view of high-performance networking, Packet Design, NANOG 22, May 20-22, 2001.
[7] A. Charny, J.-Y. Le Boudec, Delay bounds in a network with aggregate scheduling, in: First International Workshop on Quality of Future Internet Services, Berlin, Germany, 2000.
[8] B. Davie (Ed.) et al., An expedited forwarding PHB, Internet draft, April 2001 Charny et al., An expedited forwarding PHB, Internet Draft.
[9] N.G. Duffieldet al., A flexible model for resource management in virtual private networks, SIGCOMM'99. [10] Le Faucheur et al., Requirements for support of Diff-Serv-
aware MPLS Traffic Engineering, Internet Draft. [11] Le Faucheur et al., Protocol extensions for support of Diff-Serv-aware   MPLS   Traffic   Engineering,   Internet Draft.
[12] Le Faucheur et al., Use of IGP Metric as a secondTE
Metric, Internet Draft. [13] S. Floyd, V. Jacobson, Random early detection gateways
for congestion avoidance, IEEE/ACM Transactions on
Networking 1 (4) (1993) 397-U3. [14] J. Heinanen et al., RFC2597, AssuredForwarding PHB
Group, June 1999. [15] http://www.ietf.org/html.charters/ippm-charter.html.
[16] V. Jacobson et al., RFC2598, An expedited forwarding
PHB, June 1999. [17] D. Katz et al., Traffic engineering extensions to OSPF,
Internet Draft.
[18] M. Laor, L. Gendel, Effect of packet reordering in a backbone link on applications throughput, submittedto IEEE Network Magazine.
[19] M. Mathis, The macroscopic behavior of the TCP Congestion Avoidance Algorithm, Computer Communication Review, July 1997.
[20] S. McCreary, K. Claffy, Trends in wide area IP traffic patterns—a view from Ames Internet Exchange, in: Proceedings of 13th ITC Specialist Seminar on Internet Traffic Measurement andModelling, Monterey, CA, 18-20 September, 2000.
[21] K. Nichols et al., RFC2474, Definition of the Differenti-atedServices Field(DS Field) in the IPv4 andIPv6 Headers, December 1998.
[22] http://www.ietf.org/html.charters/pwe3-charter.html.
[23] J.M. McQuillan, I. Richer, E.C. Rosen BBN Report 3803, ARPANET Routing Algorithm Improvements, First Semiannual Technical Report, April 1978.
[24] J.M. McQuillan, I. Richer, E.C. Rosen, The New Routing Algorithm for the ARPANET, IEEE Transactions on Communications, May 1980.
[25] M. Shreedhar, G. Varghese, Efficient fair queuing using deficit round robin, SIGCOMM 1995.
[26] H. Smit, T. Li, ISIS extensions for traffic engineering, Internet Draft.
[27] G. Swallow, R. Goguen, RSVP Label allocation for
backup tunnels, Internet Draft.
[28] T. Telkamp, Hot Interconnect, Standford 2001.
Clarence Filsfils has an Engineering Degree in Computer Sciences from the Institute Montefiore of the University of Liege, Belgium anda Business Degree from the Solvay Business School, Brussels Belgium. He joinedCisco in 1996 andas a Distinguished Engineer; he focuses on IP Core Routing and Capacity Management (IP QoS/Traffic Engineering) designs.
John Evans receiveda B.Eng. (Hons) degree in Electronic Engineering from the University of Manchester Institute of Science andTechnology (UMIST),
UK in 1991 andan M.Sc. in Communications Engineering from UMIST in 1996. He joinedCisco Systems in 1998 andas a Consulting Engineer; he focuses on IP network design and development with special interests in core routing andtraffic management, including IP quality of service and traffic engineering. Prior to joining Cisco, he worked on the design and development of large-scale networks for the financial community andfor the military.