CCAMP Working Group D. Papadimitriou
M. Vigoureux
Internet Draft (Alcatel)
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt
K. Shiomoto
Expiration Date: August 2004 (NTT)
D. Brungard
(ATT)
J.L. Le Roux
(FT)
February 2004
Generalized MPLS Architecture for Multi-Region Networks
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet- Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as "work in progress".
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
Abstract
Most of the initial efforts on Generalized MPLS (GMPLS) have been
related to environments of single switching capability devices e.g.
one data plane layer, as such, the complexity raised by the control
of such data planes is similar to the one expected in classical
IP/MPLS networks. The fundamental reason being that an IP-based
control plane protocol suite for Label Switch Routers (LSR) or
Optical Cross-Connects (OXC) has exactly the same Level (i.e. single
data plane layer) complexity.
Vigoureux, Shiomoto et al. - Expires August 2004 [Page 1]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
The present document analyses the various GMPLS signaling and routing
aspects when considering network environments consisting of multiple
switching data layers e.g. supporting combined Packet/Layer-2
Switching - OXC devices. The examples provide an overview of the
tradeoffs in using a GMPLS control plane for combined Ethernet MAC -
opaque, service transparent, and/or fully transparent data planes.
The intent of this memo is also to demonstrate that these aspects may
not be as straightforward as they may first appear.
Table of Contents
Conventions used in this document.................................2
1. Introduction...................................................3
1.1 Context and Motivations....................................3
1.2 Rationales for Multi-Region Networks:......................4
1.3 Problem statement..........................................5
2. Routing over Forwarding Adjacencies............................5
2.1 Scalability of Single Region Networks......................6
2.2 Scalability of Multi-Region Networks.......................7
2.3 Emulating Data Plane Overlays using FAs....................7
2.4 FA Attributes Inheritance..................................8
2.5 FA Abstraction for Recovery................................9
3. Cross Region Considerations....................................9
3.1 Interface adaptation capability descriptor................10
3.2 Regeneration capability...................................15
3.3 Dedicated Traffic Parameters..............................16
3.4 Applications..............................................16
4. Extended Scope of Switching Capabilities......................17
4.1 L2SC Switching............................................17
4.2 Example...................................................19
4.3 Waveband switching........................................20
5. Conclusion....................................................20
Security Considerations..........................................21
References.......................................................21
Acknowledgments..................................................23
Authors Addresses...............................................23
Contributors.....................................................24
Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119.
In addition the reader is assumed to be familiar with the concepts
developed in [GMPLS-ARCH], [RFC-3471], and [GMPLS-RTG] as well as
[MPLS-HIER] and [MPLS-BDL].
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 2]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
1. Introduction
Generalized Multi-Protocol Label Switching (GMPLS) facilitates the
realization of seamless control integration of IP/MPLS networks with
SONET/SDH (see [T1.105]/[G.707]) or G.709 (see [G.709]) optical
transport networks. In particular, the applicability of GMPLS to both
packet/frame and circuit switching technologies (i.e. unified control
plane architecture) provides a unified control management approach
for both provisioning resources and restoration techniques.
One of the additional advantages driving the construction of a
unified GMPLS control plane is the desire to support multi LSP-
region [MPLS-HIER] routing and traffic engineering capability. This
would enable effective network resource utilization of both the
Packet/Layer2 LSP regions and the Time Division Multiplexing (TDM) or
Lambda (Optical Channels) LSP regions in high capacity networks.
1.1 Context and Motivations
Vertical integration refers (see [TE-WG]) to the definition of
collaborative mechanisms within a single control plane instance
driving multiple (but at least two) data planes (also referred in the
scope of GMPLS as switching layers). Horizontal integration is
defined when each entity constituting the network environment
includes at least one common (data plane) switching capability and
the control plane topology extends over several partitions being
either areas or autonomous systems (see [INTER-AREA-AS]). In this
latter case, the integration is thus defined between nodes hosting
the same switching capability. For instance, the control plane
interconnection between lambda switching capable routing areas
defines an horizontal integration. On the other hand, an environment
in which at least two different switching capabilities are present
and where these capabilities are both hosted by the same device
and/or hosted by different devices involves a vertical integration
within the GMPLS control plane. Such multi-switching layer capable
networks are referred to as Multi LSP-Region Networks or simply
Multi-Region Networks (MRN).
Note here that, the CCAMP Working Group is currently actively working
on extensions to this horizontal integration (the initial iteration
being the single area context worked out over the past few years) by
considering common multi-area and multi-AS traffic engineering
techniques and protocol extensions [INTER-AREA-AS]. As a first phase
vertical integration, as proposed in this document, we focus on
single area only environments.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 3]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
From the control plane viewpoint (as defined in [MPLS-HIER]) a data
plane layer is mapped to an LSP region. Following this approach, a
Traffic Engineering link or simply TE Link (at the control plane
level) maps exactly the definition proposed in the canonical layered
model (see [G.805]) where a link is defined as a link bundle (using
the IETF terminology). More generically, the TE link notion is now
recursively defined and accepted implying that the link bundle term
will be used only when referring to a set of component links or
ports. Therefore, the TE Link concept opens the door for a clear
separation between the routing adjacencies and the data plane bearer
links (or channels). Furthermore, TE Links have been extended to non
adjacent devices by introducing the Forwarding Adjacency (FA) concept
enabling in turn to decrease the number of control plane instances to
control N transport layers. Last, the bundling of FA is also defined
in [MPLS-BDL] allowing for additional flexibility in controlling
large scale backbone networks.
Using the Forwarding Adjacency, a node may (under its local control
policy configuration) advertise an LSP as a TE link into the same
OSPF/ISIS instance as the one that induces this LSP. Such a link is
referred to as a "Forwarding Adjacency" (FA) and the corresponding
LSP to as a "Forwarding Adjacency LSP", or simply FA-LSP. Since the
advertised entity appears as a TE link in OSPF/ISIS, both end-point
nodes of the FA-LSP must belong to the same OSPF area/ISIS level
(intra-area improvement of scalability). Afterwards, OSPF/ISIS floods
the link-state information about FAs just as it floods the
information about any other TE Link. This allows other nodes to use
FAs as any other TE Links for path computation purposes.
1.2 Rationales for Multi-Region Networks:
The rationales for investigating vertical integration (and thus
multi-region networks) in the GMPLS distributed control plane context
can be summarized as follows:
- The maintenance of multiple instances of the control plane on
devices hosting more than one switching capability not only (and
obviously) increases the complexity of their interactions but also
increases the total amount of processing individual instances would
handle.
- The merge of both data and control plane addressing spaces helps in
avoiding multiple identification for the same object (a link for
instance or more generally any network resource), on the other hand
such aggregation does not impact the separation between the control
and the data plane.
- The collaboration between associated control planes (packet/framed
data planes) and non-associated control planes (SONET/SDH, G.709,
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 4]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
etc.) is facilitated due to the capability of hooking the associated
in-band signaling to the IP terminating interfaces of the control
plane.
- Resource management and policies to be applied at the edges of such
environment would be facilitated (less control to management
interactions) and more scalable (through the use of aggregated
information).
In this context, Hybrid Photonic Networks (HPN) can be differentiated
from Multi-Region Networks (MRN). The main difference between nodes
included in an HPN environment and nodes included in an MRN
environment can be expressed as follows: some of the former MUST
include at least for some (but at least two) of their interfaces an
LSC switching capability with "lambda" (photonic) encoding.
1.3 Problem statement
The control by a single GMPLS instance of at least two different
switching capabilities rises some issues with regards to the control
plane scalability as well as inter-working issues between these
switching capabilities. Typically, devices present in an MRN will
have the information about all the TE-Links corresponding to the
different switching capabilities present in the environment. This
will lead, in turn, to the maintenance of large LSDB resulting in
CSPF computation time possibly exceeding reasonable value.
Scalability also concerns the maintenance of a very large number of
signaling sessions. Section 4 addresses these types of issues while
section 5 covers issues resulting from devices hosting at least two
different switching capabilities, or, more broadly, cross layer
considerations.
2. Routing over Forwarding Adjacencies
In order to extend MPLS to support non-packet TE attributes within
the scope of an integrated (routing) model encompassing several data
planes, GMPLS needs to support control of several data plane layers
(or switching layers) using the same protocol instance.
Forwarding Adjacencies (FAs) as described in [MPLS-HIER] are a useful
and powerful tool for improving the scalability of Generalized MPLS
(GMPLS) Traffic Engineering (TE) capable networks.
Through the aggregation of TE Label Switched Paths (LSPs) this
concept enables the creation of a vertical (nested) TE-LSP Hierarchy.
Forwarding Adjacency LSPs (FA-LSP) may be advertised as TE link (or
simply FA) into the same instance of OSPF/ISIS as the one that was
used to create, initiate or trigger this LSP, allowing other LSRs to
use FAs as TE links for their path computation. As such, forwarding
adjacency LSPs have characteristics of both TE links and LSPs.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 5]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
While this definition is in perfect alignment for non-packet LSP
regions and boundaries, the same concept(s) can also be re-used in
the MPLS LSP context but with a major difference. The mapping goes in
the opposite direction i.e. from the control to the IP/MPLS
forwarding plane, since in the packet domain FA-LSPs are purely
abstract objects that would, if well tailored, provide additional
scalability within a routing plane instance (i.e. define virtual TE
links without increasing the number of routing adjacencies). Indeed,
the use of FAs provides a mechanism for improving bandwidth (or more
generally any resource) utilization when its dynamic allocation can
be performed in discrete units; it also enables aggregating
forwarding state, and in turn, reducing significantly the number of
required labels as well as the number of signaling sessions.
Therefore, FAs may significantly improve the scalability of GMPLS TE-
capable control planes, and allow the creation of a TE-LSP hierarchy.
From this, and when combining multi-region environments, the
challenges that arise are related to the combination of both types of
mappings (and in particular their control) for both super-classes of
LSPs i.e. packet LSPs and circuit-oriented LSPs (a.k.a. non-packet
LSPs) from or to the same control plane instance.
2.1 Scalability of Single Region Networks
The main issue arising with FAs is related to the mapping
directionality (from the data to the control plane). FAs allow
avoiding the well-known O(N^2) at the control plane level by using
the mechanisms defined in [MPLS-HIER] but requires a specific
policing at the LSP region edges (or boundaries) in order to avoid
full meshes both at the data plane level and control plane level.
Currently, and as specified in [MPLS-HIER], it is expected that FAs
will not be used for establishing OSPF/ISIS peering relation between
the routers at the ends of the adjacency thus clearly avoiding N
square problem at the control plane level. On the other hand,
specific policies would be required to avoid a full mesh of FAs. A
full mesh of FAs would lead, at the control plane level, to a full
mesh of signaling sessions while, at the data plane, it would lead to
poor resource utilization. Avoiding full meshes can be accomplished
by setting the default metric of the FA to max[1, (the TE metric of
the FA-LSP path - 1)] so that it attracts traffic in preference to
setting up a new LSP. Such policing clearly states that FA-LSPs are
driven by a most fit approach: do not create new FA-LSPs as long as
existing ones are not filled up. The main issue with this approach is
definitely related to "what to advertise and originate at LSP region
boundaries". For instance, fully filled FA-LSPs should not be
advertised (if preemption is not allowed), while, attracting traffic
should be provided in some coordinated manner in order to avoid sub-
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 6]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
optimal FA-LSP setup. Moreover, nothing precludes the maintenance of
several partially filled FA-LSPs that are kept unused indefinitely
(even if their metric is set optimally) in particular when the
bandwidth of the FA-LSP can not (due to its discrete bandwidths
units) be exactly set to the incoming LSP request.
Note: the latter suggests filtering of the corresponding LSAs at the
regions' boundaries. In OSPF this can be accomplished by not
advertising the link as a regular LSA, but only as a TE opaque LSA
(see [RFC-2370]).
2.2 Scalability of Multi-Region Networks
The Shortest Path First (SPF) computation complexity is, in classical
cases, proportional to L Log(N) where L is the number of links (here,
more precisely TE links) and N the number of address prefixes. As
such, the full mesh scalability issues can be solved in two ways
either by improving the computational capabilities of the (C-)SPF
algorithms or simply by keeping existing Log(N) complexity but then
by improving collaboration between data planes.
The former can be achieved for instance by using Fibonacci heaps with
Log(Log(N)) complexity for instance, which in turn, allows for a
number of TE links greater than 1E6 (versus 1E3 with classical
implementations). The latter can be achieved by considering M
regions, over the same control plane topology and without using any
heuristics, one increases this complexity to M x L (Log (M x N)).
However, since TE Links can advertise multiple Interface Switching
Capabilities (ISC), the number of links can be limited (by
combination) by using specific topological maps referred to as
Virtual Network Topologies (VNT). The introduction of virtual
topological maps leads us to consider the concept of emulation of
data plane overlays [MAMLTE]. Therefore, the expectation here is to
reduce the overall computational complexity to L Log(M+1) x Log
(Log(M+1) x N) (note: with M = 1 it brings L Log(N)).
2.3 Emulating Data Plane Overlays using FAs
According to [MPLS-HIER] ISC ordering, we can use the following
terminology: FA-LSP(1) corresponds to TE Links for which the ISC is
equal to PSC-1, FA-LSP(2) to PSC-2, FA-LSP(3) to PSC-3, FA-LSP(4) =
PSC-4, FA-LSP(5) to LS2SC, FA-LSP(6) to TDM, FA-LSP(7) to LSC and FA-
LSP(8) to FSC.
FA-LSP(i) is routed over the FA-LSP(i+j) with j >= 1. In other words
a set of FA-LSPs(i+j) with j fixed provides a Virtual Network
Topology (VNT) for routing FA-LSPs(i). The virtual network topology
offered by a set of FA-LSPs(i) is denoted by VNT(i) in this document.
The virtual network topology is changed by setting up and/or tearing
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 7]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
down one (or more) FA-LSP(i). The change of the VNT(i) affects the
routing of FA-LSPs(i-j). It is expected that boundary LSRs of VNT(i)
will behave consistently with respect to any internal (LSP/link
recovery) or external (LSP/link provisioning) triggering event.
Routing is dependent on the network topology and associated link
state. Routing stability may be impaired if the Virtual Network
Topology frequently changes and/or if the status of links in the
Virtual Network Topology frequently changes. In this context,
robustness of the Virtual Network Topology is defined as the
capability to smooth changes that may occur and avoid their
subsequent propagation. Changes of the Virtual Network Topology may
be caused by the creation and/or deletion of several LSPs. Creation
and deletion of LSPs may be triggered by adjacent regions or through
operational actions to meet change of traffic demand. Routing
robustness should be traded with adaptability with respect to the
change of incoming traffic requests.
2.4 FA Attributes Inheritance
Several FA-LSPs(i) between LSRs over LSP region(i+1) are already
established, and several FA-LSPs(i-1) have been setup over this
topology and are partially filled up. One of the latter LSR regions
sees an incoming LSP request. It can be demonstrated that the TE
metric (in addition to the Maximum LSP Bandwidth and Unreserved
Bandwidth see [GMPLS-RTG]) alone is not a sufficient metric to
compute the best path between these regions. This suggests that the
inheritance process over which the TE-Metric of the FA is not as
straightforward as proposed in [MPLS-HIER].
The best example is a packet LSP (PSC-1) to be multiplexed into PSC-
2 region that lies over an LSC region. The metric MUST not take only
into account packet boundaries interface features, properties and
traffic engineering attributes such as delay or bit-rate but also for
instance the distance over the LSP region that this LSP will have to
travel. As such, the TE Metric for the Lambda LSP (in this example,
FA-LSP(i+1)) must be at least defined as a combination of the bit-
rate and the distance, classically the bit-rate times the distance
with some weighting factors. The main issue from this perspective is
that joined Path TE Metric wouldn't in such a case tackle
simultaneously both packet and optical specifics.
This suggests the definition of more flexible TE Metric, at least the
definition of a TE Metric per ISC. Simply adjust the TE Metric to the
(TE Metric of the FA-LSP path - 1) is a valid approach between LSP
over the same region class (PSC-1, PSC-2, ... , PSC-N, for instance)
but not necessarily between PSC and LSC region.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 8]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
Other TE attributes that need a specific processing during
inheritance are the Shared Risk Link Groups (SRLG) (see for instance
[SRLG]) Resource Class/Color (i.e. Administrative Groups) and
Protection (see Section 2.5).
The next section will describe the specific TE attributes to be
considered for devices hosting at least two switching capabilities,
in particular the interface switching adaptation capabilities.
2.5 FA Abstraction for Recovery
In multi switching environments the inheritance of protection and
restoration related TE link attributes must also be considered.
1) Considering a 1:1 end-to-end LSP recovery scheme, two FA-LSPs may
be set up to form a single FA. To enhance the availability of the FA,
the primary and the secondary LSPs are set up. The primary LSP is
used to carry the normal traffic (see [TERM] and [E2E-RECOVERY]).
Once the failure occurs affecting the primary LSP, the normal traffic
is carried over the secondary LSP. From the routing perspective,
there is no topological change to carry the traffic. These two LSPs
should, therefore, be advertised within the scope of a single FA TE
link advertisement with link protection type of 1:1. This FA will be
processed by an upper layer as a span protected link.
2) Considering now a single FA-LSP, span protected over each link
(i.e. at the underlying layer).
The question that arises is how should this span protected FA-LSP be
advertised in the IGP. A link protection type of 1:1 could also be
used for this advertisement but then, an upper layer would have no
means to differentiate the two cases. However, these two recovery
schemes (end-to-end and span) have major differences in terms of
recovery delay and robustness [RECOVERY].
Therefore, abstraction and summarization must be performed when
advertising FA-LSPs as TE links (to an upper layer) but using the
Link Protection Type flags and applying simple attribute inheritance
might not be sufficient to distinguish different recovery schemes.
3. Cross Region Considerations
In an MRN, as described here above, some LSR could contain, under the
control of a single GMPLS instance, multiple interface switching
capabilities such as PSC and Time-Division-Multiplexing (TDM) or PSC
and Lambda Switching Capability (LSC) or LSC and Waveband Switching
Capability WBSC).
These LSRs, hosting multiple Interface Switching Capabilities (ISC),
are required to hold and advertise resource information on link
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 9]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
states and topology. They also may have to consider certain portions
of internal LSR resources to terminate hierarchical label switched
paths (LSPs), since circuit switch capable units such as TDMs, LSCs,
and FSCs require rigid resources.
For example, an LSR with PSC+LSC switching capability can switch a
Lambda LSP but can never terminate the Lambda LSP if there is no
unused adaptation capability between the LSC and the PSC layers.
Therefore, within multi-region LSR networks, the advertisement the
so-called adaptation capability to terminate LSPs provides critical
information to take into account when performing multi-region path
computation. This concept enables a local LSR to discriminate remote
LSRs (and thus allows their selection during path computation) with
respect to their adaptation capability e.g. to terminate Lambda LSPs
at the PSC level.
Hence, here we introduce the idea of discriminating the (internal)
adaptation capability from the (interface) switching capability by
considering an interface adaptation capability descriptor.
3.1 Interface adaptation capability descriptor
The interface adaptation capability descriptor can be interpreted
either as the adaptation capability information from an incoming
interface or as the adaptation capability to an outgoing interface
for a given interface switching capability. By introducing such an
additional descriptor (as a sub-object of the ISC sub-TLV, for
instance), the local GMPLS control plane can swiftly search which
LSRs can terminate a certain encoding type of LSP and successfully
establish an LSP tunnel between two PSCs.
As an example, consider for instance a multiple LSP-region domain
comprising simultaneously PSC LSRs, LSC LSRs, PSC+LSC LSRs and
PSC+TDM+LSC LSRs. The LSRs discriminate the type of the links
connecting these LSRs by interpreting the interface switching
capability descriptor included in the TE Link TLV of the TE opaque
LSAs [LSP-HIER].
The interface switching capability at both ends of a TE link between
LSRs for which individual resources (lambdas) are represented by
wavelength labels shall be [LSC, LSC], [{TDM|PSC}, LSC], or [LSC,
{TDM|PSC}]. On the other hand, the interface switching capability at
both ends of a TE link shall be [PSC,PSC] for LSPs "carrying" a shim
header label, or shall be [TDM, TDM], [TDM,PSC] or [PSC,TDM] for TE
links whose individual resources (timeslots) are represented by TDM
labels. Thus, based on the interface switching capability descriptor,
the LSRs can impose proper constraints in order to compute the paths
of the LSPs. For example, LSRs can understand that a remote TDM LSR
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 10]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
with [LSC,TDM] link cannot be a lambda LSP intermediate link with the
exception that it can initiate or terminate lambda LSPs and switch
"TDM timeslots".
However, LSRs cannot infer the internal LSP switching capability of
remote LSRs, especially if the LSRs have a multi-switching capability
architecture such as a PSC+TDM+LSC as shown below or more generally,
more than two ISC capabilities. In the LSR, LSC may have a direct
inner interface not only to TDM but also to PSC. The LSP can be
interfaced at both TDM or PSC. This kind of multi-switching
architecture may become very common in optical networks.
..........................
. .
. -------- .
. | | .
. | ISC2 | .
. -<->--| | .
. | | | .
. | -------- .
. | .
. | -------- .
. | | | .
. -<->--| ISC1 | .
. | | .
-----<---------| | .
----->---------| | .
. -------- .
..........................
In the above figure, the switching capabilities ISC1 and ISC2 can be
grouped in a single TE link, and the bandwidth information defined as
follows:
Let X be the initial Unreserved Bandwidth of the TE link then the Max
LSP bandwidth can be equal to X for the ISC1 (as advertised in the
ISC1 sub-TLV) and equal to Y12 for the ISC2 (as advertised in the
ISC2 sub-TLV). Y12 represents the link bandwidth between the two
ISCs. The bandwidth accounting/updating is then dependent of the
inner architecture. In this case no specific adaptation capability
descriptor is required.
The following cases, however, highlight the limitations of such
procedure and the need for an enhanced switching adaptation
description.
..........................
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 11]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
. .
TE2 . -------- .
----->---------| | .
-----<---------| ISC2 | .
. --->--| | .
. | --<--| | .
. || -------- .
. || .
. || -------- .
. | -->--| | .
. ---<--| ISC1 | .
TE1 . | | .
----->---------| | .
-----<---------| | .
. -------- .
..........................
For the above picture, two cases can be considered regarding the
switching capability configuration. Note that both TE1 and TE2 belong
to the same physical link.
Let the triplet <TE, ISC1, ISC2> represent respectively the
Unreserved Bandwidth of the TE link, the Maximum LSP Bandwidth of
ISC1 and the Maximum LSP Bandwidth of ISC2.
In a first scheme the switching capabilities can be declared as two
separate TE links: for TE link 1 (TE_1) and TE link 2 (TE_2): <X1,
X1, Y12> and <X2, X2, Y21>
In a second scheme, the capabilities are described as part of a
single TE Link: <X, X+Y12, X+Y21>.
While the first case rises some issues concerning bandwidth
accounting coordination between the two TE Links, the later is
confronted to an over-provisioning issue being, in addition, highly
dependent on the Minimum LSP bandwidth value. Also, these approaches
are limited 1) by the number of switching capabilities hosted by a
single system and 2) by the number of ways these switching
capabilities interacts (i.e. the number of ways data can be
encapsulated/decapsulated when passing from one switching capability
to another).
1) Number of Switching Capabilities:
-------
------| |------
| | PSC | |
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 12]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
| --| |-- |
| | ------- | |
| \|/ /|\ |
| | ------- | |
| --| |-- |
\|/ | TDM | /|\
| --| |-- |
| | ------- | |
| \|/ /|\ |
| | ------- | |
| --| |--_ |
------| |------
| |
/|---| |---|\ Fiber #1
========| |---| LSC |---| |========
========| |---| |---| |========
\|---| |---|/ Fiber #N
-------
Referring to this figure, the problem with the use of the interface
switching capability descriptor at the PSC+TDM+LSC LSR, is the
shortage of LSP termination capability information. The PSC+TDM+LSC
LSR provides only switching capability information by abstracting the
internal node capabilities. Therefore, remote LSRs cannot accurately
determine which switching capability can be switched and/or
terminated at the PSC+TDM+LSC LSR. For such a node supporting LSC,
TDM and PSC switching capability, the determination of the resource
made available to cross for instance the LSC to PSC region boundary
(LSC -> PSC) may involve one of the following region cross- over: LSC
-> PSC and LSC -> TDM -> PSC. This can be represented as follows:
-------
| |
----| PSC |----
| | | |
----- ------- -----
| | | |
----- ------- -----
| | | | | |
| ---| TDM |--- |
| ---| |--- |
| | | | | |
----- ------- -----
| | | |
----- ------- -----
| | | | _| |
| ---| LSC |--- |
-----| |-----
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 13]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
2) Number of adaptation capabilities:
In addition, the LSP Encoding Type (representing the nature of the
link that the LSP traverses) is "lambda". Therefore, as depicted in
the following figure, this issue become more complex once each
switching capability supports multiple framing, for instance, at PSC,
Ethernet-MAC framing and PPP framing.
-------
| |
------------| PSC |------------
| ----| |---- |
| | | | | |
----- ----- ------- ----- -----
| ETH | | PPP | | PPP | | ETH |
----- ----- ------- ----- -----
| | | | | | | | | |
| | | ---| TDM |--- | | |
| -----------| |----------- |
| | | | | |
Another example occurs when L2SC (Ethernet) switching can be adapted
in LAPS X.86 and GFP for instance before reaching the TDM switching
matrix:
-------
| |
------------| L2SC |------------
| ----| |---- |
| | | | | |
----- ----- ------- ----- -----
| X86 | | GFP | | GFP | | X86 |
----- ----- ------- ----- -----
| | | | | | | | | |
| | | ---| TDM |--- | | |
| -----------| |----------- |
| | | | | |
Similar circumstances can occur, if a switching fabric that supports
both PSC and L2SC functionalities is assembled with LSC interfaces
enabling "lambda" (photonic) encoding. In the switching fabric, some
interfaces can terminate Lambda LSPs and perform frame (or cell)
switching whilst other interfaces can terminate Lambda LSPs and
perform packet switching.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 14]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
Thus, the interface switching capability descriptor provides the
information for the forwarding (or switching) capability only. In
order for remote LSRs to understand properly the termination
capability of the other LSRs, additional information to the existing
interface switching capability descriptor is essential in achieving
seamless multi-region routing. In turn, adequate processing of this
additional information will allow the signaling of packet LSP set- up
combined with an automated triggering of new Lambda LSPs between LSRs
that do not yet have a preferred Lambda LSP to carry the Packet LSP.
(see [MLRT]).
Note that in the context of Hybrid Photonic Networks, additional
constraints such as the regeneration capability drive even more the
need for an adaptation switching capability descriptor.
3.2 Regeneration capability
In an HPN context, the lower LSP region provides to the upper LSP
region a regeneration/conversion function (using for instance opto-
electronic interfaces). More precisely a regeneration function can
deliver conversion (within a given pre- determined range or not)
while conversion may be delivered independently of the existence of
any regeneration capability.
The following classification applies from the definition of the
regeneration function:
1. If the regeneration function is defined as an Interface Switching
Capability (or simply ISC see [GMPLS-RTG] and [MPLS-HIER]), then if
this ISC value is lower or equal to the incoming LSP switching type,
the request may be processed by the network. Otherwise if the LSP
Switching Type > ISC value of the region, the LSP request can not be
processed and is simply rejected (see [MPLS-HIER] for a definition of
the relationship between ISC values).
2. If the regeneration function is not defined as an interface
switching capability (pure regeneration without any connection
function defined) then the following alternative applies depending on
the encoding type defined at its entry points. If the regeneration
depends on the encoding type of the incoming LSP request the latter
must be the same as the one provided by the regeneration function.
Otherwise the LSP request is simply rejected or tunneled toward the
next hop (if feasible). Notice here that forwarding an LSP request to
the next hop and expecting the latter would provide enough
regeneration capacity for this incoming LSP is a complex problem,
since one can not, with the currently available GMPLS tools,
guarantee that this request will not itself be forwarded to the next
hop, and so on.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 15]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
Moreover, by extending the knowledge of the interface capability to
terminate (adapt) a given signal, it would be possible for instance
to characterize more precisely the interfaces (physical) distance
coverage. This may be achieved by considering information such as the
transmission distance range (Short Haul, Long Haul, Ultra Long Haul,
etc.) or even the signal modulation format. This would provide
dynamic interface resource management (versus the current Network
Management techniques). In turn, this would decrease the time needed
for selecting resources during path computation.
3.3 Dedicated Traffic Parameters
This point is related to whether or not dedicated traffic parameters
should be defined for LSPs established in MRN environments such as
the ones defined for Sonet/SDH (see [SONET-SDH] and G.709 (see
[GMPLS-G709]).
With respect to spatial routing the LSP Encoding Type, Switching Type
and G-PID (see [RFC-3471] for the corresponding definitions) provides
the required information to pertinently setup such LSPs. It is
nevertheless expected here to see some additional capability allowing
for intermediate states, in particular when the regeneration function
is defined as a switching layer (see also Section 6.2).
With respect to spectral routing the main issue raises from the
passing of external physical constraints between conversion points.
In addition to the Multiplier usage that may help in establishing/
deleting parallel LSPs, additional information concerning the
physical constraint each sub-path MUST fulfill should be considered
e.g. maximum distance and BER per (sub-path). A parameter equivalent
to the Transparency level may also help in providing a hop-by-hop
negotiation of the regeneration capability to be used.
3.4 Applications
In multi-region environments, crossing LSP regions during
provisioning can occur for two main reasons: grooming or regeneration
(when delivered by a switching capable layer).
1. Grooming
LSP grooming deals with the optimization of network resource
utilization. Multi-region environments are particularly well adapted
for this feature as they may provide different switching
granularities allowing for the tunnelling of several finer grained
LSPs into a coarser grained LSP. In this context, it can be useful
from the control plane viewpoint not to terminate the multiplexed LSP
and simply tunnel this LSP into a lower-region LSP viewed as a common
segment for each incoming LSPs.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 16]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
However, this raises the problem of the representation of the newly
established LSP at the control plane level. In particular, concerning
the maintenance of the two LSPs (head-end and tail-end LSPs) that
forms the newly spliced LSPs. Further consideration on grooming are
left for further study as it includes aspects leading to the
definition of multipoint-to-point LSPs (beyond the scope of this
document).
2. Regeneration
Due to the constraints of optical transmission, the optical signal
may have to be regenerated along the LSP path. Some multi-region
network may require to cross a region boundary to access the
regeneration function. This rises the question of the so-called LSP
integrity when crossing region boundaries.
Consider for instance a Lambda LSP in a LSC+PSC multi-region network.
For a given reason the LSP needs to be regenerated at an intermediate
node. It will thus use the O/E/O interfaces present in the PSC
region. From the control plane viewpoint either two Lambda LSPs are
seen (ingress to intermediate and intermediate to egress) or a single
one (ingress to egress).
Keeping a single Lambda LSP would prevent from maintaining, at the
control plane level, several entities for a single connection. It
should be also noted here that one assumes that regeneration is
delivered between LSPs (from ingress to intermediate and intermediate
to egress) defined within regions of the same switching capability
(i.e. LSC-PSC-LSC). This would in turn facilitate the processing of
both the regenerated entities and the (pool of) regeneration
resources that would need to be marked.
4. Extended Scope of Switching Capabilities
When considering multi-region environments, two common examples of
multi-switching combinations are:
- Packet(LSR)/Layer-2(Switch) with TDM (SONET/SDH) or LSC (OXC)
- Multi-Granularity OXC (including opaque and transparent switching
capabilities at different granularity levels)
The first implies some considerations with respect to Layer-2
Switching Capable interfaces and L2SC environments. The latter
implies further considerations on Waveband Switching aspects.
4.1 L2SC Switching
Layer 2 Switching capable interfaces and Layer 2 LSPs are in the
scope of GMPLS (see [GMPLS-ARCH], [GMPLS-RTG] and [RFC-3471]). Such
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 17]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
interfaces are defined as capable to recognize frame/cell boundaries
and can forward data based on the content of the frame/cell header.
They include mainly interfaces on Ethernet bridges that forward data
based on the content of the MAC header. This section provides an
overview of the issues to be considered when introducing GMPLS in
Ethernet MAC-based networks.
In this context, the possible development of a GMPLS signaling
profile for Ethernet networks, involves the definition of a label
space. From this perspective, two questions arise: 1) what the label
value space represents and is the corresponding label value space
semantic-full (see [GMPLS-SONET-SDH]) or semantic-less (see [RFC-
3471]) and 2) how is the label value space implemented (i.e.
associated with data plane or non-associated and therefore exchanged
over dedicated signaling channels or even a combination of both). A
contiguous problem arises that the set of potential solutions must be
backward compatible meaning that non-GMPLS controlled Ethernet
interfaces should be capable to inter-work with GMPLS controlled
Ethernet interfaces.
In addition to the label considerations, an additional problem
appears due to the type of environment in which these Ethernet
interfaces are considered. These interfaces may be either so-called
LAN PHY's (thus implying a broadcast capable environment) or WAN
PHY's (thus implying point-to-point links). On the other side, one
has to consider MAC-based capable interfaces over Non-Broadcast
Multiple Access (NBMA) technologies such as MPLS (Ethernet-over-
MPLS) and over circuit-oriented technologies such as SDH and OTN
(through different adaptation technologies such as LAPS X.86 and
GFP). This by taking into account that the MAC Address space is by
definition non-hierarchical. The latter implies the definition of an
identification space translating the topological location of the
Ethernet end-points from an IP-based perspective and this optimally
independently of the underlying bearer technology of the Ethernet
frames.
The ideal situation would be to define a "one size fits all"
solution. However, it is clear that inferring label value space from
the bearer technology implies the development of so-called snooping
approaches, while on the other side LAN PHY's would not scale such a
solution implying the transformation of Broadcast Access (BA)
environment into a NBMA one (using star, hub-and-spoke, or multi-
tree approaches). Therefore, a heuristic has to be provided to solve
these problems while avoiding introduction of complex address
resolution mechanisms for such environments. Broadcasts are mainly
used in LAN environments for address resolution (ARP) and
bootstrapping (DHCP) reasons. Thus a potential solution would be to
let the network operate in a BA mode for such operations and bring
its operational mode back to an NBMA mode for unicast/multicast frame
processing. The same would apply for unknown unicast frames.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 18]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
Therefore, a first step towards a solution would be reached, if one
can guarantee a dual operational mode for these environments: 1)
first mode being backward compatible with the broadcast exchanges as
defined by the IEEE (using IEEE 802.1d and related, thus using an
associated control plane) and 2) the second mode being GMPLS
compatible (thus using a non-associated IP-based distributed control
plane) for the unicast operations
The next issue relates to the realization of resource reservation
over Ethernet interfaces using GMPLS signaling techniques and its
applicability. For more detailed considerations see [L2SC-LSP].
4.2 Example
The following example details the usage of the concepts presented in
the previous sections of this document in delivering a virtual
topology for L2SC-over-LSC nodes.
Consider the following network topology:
1 2
| |
3---A---B---C---D---5
| | | |
| E---F |
| | | |
4---G---H---I---J---6
| |
7 8
In this topology each node identified with a letter is a dual
switching capable node (L2SC/LSC or L2SC/WBSC) and nodes identified
with a number refers to L2SC capable devices.
An Lambda LSP is established covering all dual-switching nodes [A-B-
C-D-J-I-F-E-H-G].
This FA-LSP constitutes the virtual topology for the dual switching
nodes. This is viewed from the L2SC level as a L2SC capable multi-
access link that may be accessed (upon local policy basis) from each
node constituting the topology. Another example, would be, for
instance, a Lambda LSP routed over [A-B-C-D-J] but precluding access
to node C.
Afterwards, each node (more precisely the L2SC region) may trigger
the establishment of L2SC LSPs on top of this multi-access FA-LSP
that would allow setting up multi-partitioning of the bandwidth
capacity made available by the "fat pipe" having a higher ISC value.
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 19]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
These L2SC LSP's may be for instance, using the above example, [A-B-
C-D], [A-B-D-J-I-G] or [J-I-F-E], even if the latter wouldn't be
usable by any incoming LSP. Each of these L2SC LSP's are simply L2SC
FA-LSP's forming a L2SC-capable virtual topology. This topology can
be subsequently used by external devices to establish L2SC LSP's
using these FA's as links.
Bandwidth accounting is performed on a per FA basis, translating into
intermediate node bandwidth aggregation accounted on a per priority
basis. In turn, this accounting translates into restriction over the
accessibility of each of the links constituting the Lambda LSP.
The above example implies that currently defined ISCs (see [GMPLS-
RTG]) such as L2SC might be extended to more than one value with the
following relationship L2SC (=L2SC-1) < L2SC-2 < L2SC-3 < L2SC-4 <
TDM. The (data plane) flow aggregation mechanisms for L2SC LSPs being
out of scope of the present document.
4.3 Waveband switching
The GMPLS protocol suite, as currently defined, supports waveband
switching through inverse multiplexing or switching of individual
(contiguous) wavelength components. It may be thus appropriate to
integrate wavebands in the switching hierarchy in order to reflect,
at the control plane level, waveband physical components
(multiplexer/demultiplexer) availability at the data plane [WBEXT].
Also, depending on the (passive/active) components used in an optical
network, wavelength spacing in the optical multiplex can vary. Some
components like multiplexer/demultiplexer impose or depend on that
spacing. Therefore, it may be appropriate to detail the component
capability with respect to spacing, and/or to indicate the number of
supported wavelengths per waveband. Moreover, one may also expect in
case of (standardized) waveband nominal frequency values some
simplification during the corresponding wavelength assignment.
In the MRN context, the main issue with Waveband Switching can be
viewed as follows. If the LSRs support in addition to waveband
switching an ISC in the set {PSC, L2SC, TDM, FSC} then waveband
switching can be assumed (from the control plane processing
viewpoint) as being equivalent to Lambda Switching, if one considers
labels as described here above. However if the additional switching
capability within a single device, or even network, includes
interfaces with LSC capability then either links should have a
specific resource class assigned or dedicated values should be
considered for the LSP Encoding Type, Switching Type and G-PID (when
bands are carried over fibers).
5. Conclusion
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 20]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
In this draft, we address the issues when using the GMPLS protocol
suite as a unified control plane for MRN environments. Several
proposals for enhancing the current GMPLS mechanisms are presented.
The proposals are based on current GMPLS mechanisms and in alignment
with GMPLS architecture (see [GMPLS-ARCH]). This memo analyzes the
suitability of the GMPLS protocol suite for the MRN environment,
keeping a strict and full alignment with the current and preferred
suite of signaling and routing protocols (in particular, OSPF, IS-IS,
RSVP-TE and LMP).
By starting from a single area context, the expectations coming out
from the first release of this memo, are clearly intended to open the
field to a more detailed description of the collaborative processes
within the GMPLS protocol suite.
The main guideline of this work is backward compatibility with the
current GMPLS protocols suite. The second guideline is limiting and
efficiently handling the complexity introduced. This memo provides an
introduction to MRNs and aspects to be considered. We invite the
CCAMP community to collaborate on progressing this critical GMPLS
topic: an integrated control plane supporting multiple data layers.
Security Considerations
In its current version, this memo does not introduce new security
consideration from the ones already detailed in the GMPLS protocol
suite.
References
[G.707] ITU-T, "Network node interface for the Synchronous
Digital Hierarchy", Recommendation G.707
[G.709] ITU-T, "Interfaces for the Optical Transport Network"
Recommendation G.709
[G.805] ITU-T, "Generic functional architecture of transport
networks", Recommendation G.805
[GMPLS-RTG] K. Kompella (Editor), Y. Rekhter (Editor) et al.
"Routing Extensions in Support of Generalized MPLS",
Internet Draft, Work in Progress,
draft-ietf-ccamp-gmpls-routing-09.txt
[GMPLS-G709] D. Papadimitriou (Editor) et al. "Generalized MPLS
Signaling Extensions for G.709 Optical Transport
Networks Control", Internet Draft, Work in Progress,
draft-ietf-ccamp-gmpls-g709-06.txt
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 21]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
[LSP-HIER] K. Kompella and Y. Rekhter, "LSP Hierarchy with
Generalized MPLS TE", Internet Draft, Work in Progress,
draft-ietf-mpls-lsp-hierarchy-08.txt
[RECOVERY] CCAMP P&R Design Team, Analysis of Generalized Multi-
Protocol Label Switching (GMPLS)-based Recovery
Mechanisms (including Protection and Restoration),
Work in Progress,
draft-ietf-ccamp-gmpls-recovery-analysis-02.txt
[INTER-AREA-AS] A. Ayyangar, J. Vasseur, " Inter-area and Inter-AS
MPLS Traffic Engineering", Internet Draft, Work in
Progress,
draft-vasseur-ayyangar-ccamp-inter-area-AS-TE-00.txt
[L2SC-LSP] D. Papadimitriou, et. Al., "Generalized MPLS Signaling
for Layer-2 Label Switched Paths (LSP)", Internet Draft,
Work in Progress,
draft-papadimitriou-ccamp-gmpls-l2sc-lsp-01.txt
[MAMLTE] K. Shiomoto et al., "Multi-area multi-layer traffic
engineering using hierarchical LSPs in GMPLS networks",
Internet Draft, Work in Progress
draft-shiomoto-multiarea-te-01.txt.
[MLRT] W. Imajuku et al., "Multilayer routing using multilayer
switch capable LSRs, Internet Draft, Work in Progress,
draft-imajuku-ml-routing-02.txt.
[MPLS-BDL] K. Kompella, Y. Rekhter and Lou Berger, "Link Bundling
in MPLS Traffic Engineering", Internet Draft, Work in
Progress
draft-ietf-mpls-bundle-04.txt
[RFC-2370] R. Coltun, "The OSPF Opaque LSA Option",
IETF RFC 2370
[RFC-3471] L. Berger et al., "Generalized Multi-Protocol Label
Switching (GMPLS) Signaling Functional Description",
IETF RFC 3471
[SONET-SDH] E. Mannie and D. Papadimitriou et al., "Generalized
Multi-Protocol Label Switching Extensions for SONET and
SDH Control", Internet Draft, Work in Progress,
draft-ietf-ccamp-gmpls-sonet-sdh-08.txt
[SRLG] D. Papadimitriou et al. "Shared Risk Link Groups
Inference and Processing", Internet Draft, Work in
Progress,
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 22]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
draft-papadimitriou-ccamp-srlg-processing-02.txt
[SURVEY] L. Berger, Y. Rekhter et al., "Generalized MPLS
Signaling - Implementation Survey",
Internet Draft, Work in Progress,
draft-ietf-ccamp-gmpls-signaling-survey-03.txt
[WBEXT] R. Douville et al., "Extensions to Generalized MPLS for
Waveband Switching", Internet Draft, Work in Progress
draft-douville-ccamp-gmpls-waveband-extensions-03.txt
Acknowledgments
We would like to thank here, Sven Van Den Bosch, Richard Douville,
Olivier Audouin, Amaury Jourdan, Emmanuel Desmet and Bernard sales.
The authors would like to thank Mr. Wataru Imajuku for the
discussions on adaptation between regions [MLRT].
Authors Addresses
Dimitri Papadimitriou (Alcatel)
Francis Wellensplein 1,
B-2018 Antwerpen, Belgium
Phone : +32 3 240 8491
E-mail: dimitri.papadimitriou@alcatel.be
Martin Vigoureux (Alcatel)
Route de Nozay,
91461 Marcoussis cedex, France
Phone: +33 (0)1 69 63 18 52
E-mail: martin.vigoureux@alcatel.fr
Kohei Shiomoto (NTT Network Innovation Laboratories)
3-9-11 Midori-cho
Musashino-shi, Tokyo 180-8585, Japan
Phone: +81 422 59 4402
E-mail: shiomoto.kohei@lab.ntt.co.jp
Deborah Brungard (AT&T)
Rm. D1-3C22 - 200 S. Laurel Ave.
Middletown, NJ 07748, USA
Phone: +1 732 420 1573
E-mail: dbrungard@att.com
Jean-Louis Le Roux (FTRD/DAC/LAN)
Avenue Pierre Marzin
22300 Lannion, France
Phone: +33 (0)2 96 05 30 20
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 23]
draft-vigoureux-shiomoto-ccamp-gmpls-mrn-04.txt March 2004
E-mail:jean-louis.leroux@rd.francetelecom.com
Contributors
Eiji Oki (NTT Network Innovation Laboratories)
3-9-11 Midori-cho
Musashino-shi, Tokyo 180-8585, Japan
Phone : +81 422 59 3441
E-mail: oki.eiji@lab.ntt.co.jp
Nobuaki Matsuura (NTT Network Service Systems Laboratories)
3-9-11 Midori-cho
Musashino-shi, Tokyo 180-8585, Japan
Phone : +81 422 59 3758
E-mail: matsuura.nobuaki@lab.ntt.co.jp
Emmanuel Dotaro (Alcatel)
Route de Nozay,
91461 Marcoussis cedex, France
Phone : +33 1 6963 4723
E-mail: emmanuel.dotaro@alcatel.fr
Vigoureux, Shiomoto et al. - Expires April 2004 [Page 24]