Individual R. White
Internet-Draft R. Adams
Expires: April 4, 2005 Cisco Systems
V. Manral
SiNett Corp.
October 4, 2004
Considerations in Benchmarking Routing Protocol Network Convergence
draft-white-network-benchmark-01
Status of this Memo
This document is an Internet-Draft and is subject to all provisions
of section 3 of RFC 3667. By submitting this Internet-Draft, each
author represents that any applicable patent or other IPR claims of
which he or she is aware have been or will be disclosed, and any of
which he or she become aware will be disclosed, in accordance with
RFC 3668.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on April 4, 2005.
Copyright Notice
Copyright (C) The Internet Society (2004).
Abstract
This document attempts to discuss some of the definitions required to
undertake the specifications of benchmarks measuring routing
protocols convergence within a network, and also to discuss some of
the possible ways to benchmark a routing protocol performance within
a network, and some of the implications of those benchmarks. The
White, et al. Expires April 4, 2005 [Page 1]
Internet-Draft Considerations in Benchmarking RPs October 2004
definition of convergence is discussed first, then polling network
devices. Several tests which are commonly used to measure network
convergence are examined.
This draft does not attempt to define what techniques should be used
to benchmark network convergence, but only to provide considerations
testers should consider when attempting to measure network
convergence using various methods.
1. Motivation
As the ability to benchmark components within a network appears to be
coming under greater scrutiny, and specifications are being written
to standardize ways to measure the performance of individual
components within given frameworks, the next level of benchmarking
has not been approached, that of measuring the performance of
networks. But what is meant when we say the performance of a
network, from the perspective of routing protocols? Various tests
have been used in the past to measure the convergence of a network,
some of which actually measure completely different things than
others.
It's important to attempt to examine the measurement of network
convergence in a way that exposes these differences, and helps
vendors, end users, and those in the research community have some
common ground when discussing network convergence.
2. A Problem of Definitions
As we examine the issues and concepts surrounding the measurement of
network performance in terms of convergence, we find that most of the
basic problems we face surround defining the terms in use. For
instance, what is convergence, exactly? What is a network? In the
following sections, we discuss each of these concepts, and attempt to
address each one.
2.1 Networks
In its most nominal form, a network is composed of a group of devices
interconnected in some way, which send data over these
interconnections for various purposes. But, when we discuss the
concept of routing protocol convergence within a network, the
definition needs to be more precise. For instance, since hosts do
not, generally, participate in routing, should they be considered a
part of the network when benchmarking the performance of a routing
protocol? The obvious answer appears to be a resounding no, but, in
some possible tests types, hosts, acting as testing devices, play a
large part in the test itself.
White, et al. Expires April 4, 2005 [Page 2]
Internet-Draft Considerations in Benchmarking RPs October 2004
When considering tests in which hosts participate as traffic or route
generators, or other testing devices, we must consider the impact
these hosts have on the test results, although we may not consider
them a part of the network.
2.2 Convergence
Convergence is probably one of the hardest words in networking to
define. Just about everyone who has worked on networks for a period
of time knows what it means, but no-one can explain it sufficiently
to someone who doesn't understand how a network works for it to be
understood. In fact, this is because there are several different
meanings attributed to convergence, and which meaning is intended
depends on the context in which the word is set. Convergence can
mean:
o The time at which all the routing protocol processes running on
devices which participate in routing in the network agree on the
best path to each reachable destination in the network.
o The time at which the best path to each reachable destination in
the network has been loaded into some local table which may then
be used to forward packets (the routing information base, or RIB).
o The time at which each router in the network has built the tables
necessary to actually forward packets through the net- work, so
that a packet transmitted from one part of the network would
actually reach any given reachable destination within the network.
For instance, on a Cisco router, show ip ospf stats would allow the
tester to see the time of the last completed SPF, show ip route would
allow the tester to see what routes are installed in the RIB, and
show ip cef would allow the tester to see the forwarding information
which has been built from the RIB. Each test designed to measure the
performance of routing protocols within a network must determine
which type of convergence is being measured, if that measurement is
acceptable to the information being gathered, and which test will
actually measure the desired type of convergence.
2.3 Polling Devices in a Network
One common way to measure network convergence is to poll the devices
in the network, using some command supplied within the routing
software, to determine when particular events have occurred, or par-
ticular pieces of information have reached all the routers in the
network. Polling eliminates the need for the clock of each device
within the network to be synchronized for the test to have meaningful
results. However, there are some issues with the rate of polling
dev- ices within the network which need to be addressed in any test
which polls devices for this information; the first is the rate at
which polling takes place.
White, et al. Expires April 4, 2005 [Page 3]
Internet-Draft Considerations in Benchmarking RPs October 2004
If, in a test, you are attempting to measure some parameter to within
one second of its occurrence, then you would need to poll at a rate
much higher than once per second.
test starts here
|
| event occurs here
| |
v v
-+----------+----------+----------+---
^ ^ ^ ^
| | | |
0 seconds 1 second 2 seconds 3 seconds
For instance, in this time line, suppose a polling event is set up to
take place every second. An event is started just after some polling
event takes place, but the polling process doesn't recognize the test
as starting until the 1 second poll. An event occurs just before the
2 second poll, and the polling process detects this at the 2 second
poll. The polling process would indicate that from the time the
event started until the time the event has finished, one second has
elapsed. In reality, closer to two seconds has elapsed.
The interval of the polling process can be reduced until the
measurement is felt to be accurate, but it should be at least half of
the desired accuracy. Common practice actually shows that it should
be about one tenth of the desired accuracy.
A second consideration when polling for network events is the
performance of the device running the polling process. If the
process cannot poll each device at the scheduled interval, or the
polling is "jittered," the time between each actual poll varies by
some amount, the accuracy of the tests will be called into question.
The amount of jitter introduced by the polling device, and the rate
at which the device can effectively poll, should be measured in some
way, and this measurement should be taken into account when designing
tests which rely on polling.
Finally, when polling devices to determine when a network event
occurs, issues with serialization must be considered. Most devices
used for polling will not be able to poll several devices within the
network at once, and will thus serialize the polling of devices.
White, et al. Expires April 4, 2005 [Page 4]
Internet-Draft Considerations in Benchmarking RPs October 2004
p1 p3 p5 p7 p9
| p2| p4| p6| p8| p10
| | | | | | | | | |
v v v v v v v v v v
-+----------+----------+----------+---
^ ^ ^ ^
| | | |
0 seconds 1 second 2 seconds 3 seconds
Suppose, for instance, a single device is polling ten devices in the
network. If it can poll five devices per second, it will take a full
two seconds for it to detect any event on all ten devices, giving an
effective accuracy of about four seconds. The amount of time
required for a polling device to serialize through all the devices it
is polling needs to be considered when polling a very large number of
devices.
2.4 Passing Traffic Through the Network to Determine Convergence
One of the most widely used tests for determining network convergence
is starting some traffic stream at one end of a network, disrupting
or completing the network, and determining how long the traffic
stream is either not delivered, or takes to be delivered. For
instance:
Source----R1----R2----Sink
A traffic stream is generated on Source, and the link between R1 and
R2 is connected in some way. The time between the connection of this
link and the arrival of the traffic at the Sink is measured as
network convergence. This type of test is extremely useful in
testing real the response of a network to changing conditions. There
are some considerations which should be examined when using this sort
of test, or examining the results of this sort of test
2.4.1 The Various Elements of Performance Cannot Be Separated
Using this sort of testing, there is no way to separate the
performance of a routing protocol from the performance the
interaction between the routing protocol and the forwarding engine,
nor from the performance of the forwarding engine itself. In many
tests, this is acceptable, since these are all elements of the
network in total, but if specific elements of routing protocol
performance are being measured, such tests can be problematic when
attempting to analyze the results.
White, et al. Expires April 4, 2005 [Page 5]
Internet-Draft Considerations in Benchmarking RPs October 2004
2.4.2 The Total Convergence of the Network May Not Be Measured
If you have the following topology:
Source-----R1----R2-----Sink
| |
R3 R4
| |
R5----R6
Suppose a traffic stream is sourced from Source, and then all the
devices in the network are brought up (R1 through R6). The time from
the device startup to the traffic stream reaching the sink is
measured as network convergence.
As soon as the path Source, R1, R2, Sink converges, the Sink will
begin receiving traffic, and the network will be considered to be
converged by the test. However, without polling the remaining
routers, R3 through R6, there is no way to know if those routers have
also converged on the best path to the Sink and the Source. While
this example may be considered extreme, there are many complex
topologies where:
o The path chosen by the traffic stream may not be the path
expected.
o The path chosen by the traffic stream may switch during network
convergence, with the stream taking some secondary path at first,
and the successively better paths converging over the life of the
test.
o The path chosen by the traffic stream switches so quickly that no
traffic is lost, while the routing protocols still take some time
to converge.
Tests which rely on traffic passing through the network to determine
network convergence times should thoroughly examine the way in which
the test topology converges, and examine the consistency of that
convergence, with enough test runs to get a good feel for the range
of possible results. Examining the same test sequence with slight
changes in the network topology may help to provide an understanding
of how the network under test converges, and also may help to provide
more insight into the factors impacting convergence in the test
network.
It's also possible that if the test network does not converge
completely for some time after the test traffic successfully passes
through the topology, the continuing convergence could impact the
results of a second test run, if the test runs are placed too closely
together. If a first test is run, and a second test is started
immediately on traffic making it through the test topology, the
White, et al. Expires April 4, 2005 [Page 6]
Internet-Draft Considerations in Benchmarking RPs October 2004
results of the second test may be skewed by convergence which is
still taking place from the first test run.
These are important considerations which should be noted when
examining or performing tests which rely on the presence of a data
stream within a routing system to measure convergence.
3 Informative References
[OSPF-BENCH]
Manral, V., White, R. and A. Shaikh, "Benchmarking Basic
OSPF Single Router Control Plane Convergence",
draft-ietf-bmwg-ospfconv-intraarea-10 (work in progress),
July 2004.
Authors' Addresses
Russ White
Cisco Systems
riw@cisco.com
Robert Adams
Cisco Systems
robeadam@cisco.com
Vishwas Manral
SiNett Corp.
vishwas@sinett.com
White, et al. Expires April 4, 2005 [Page 7]
Internet-Draft Considerations in Benchmarking RPs October 2004
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2004). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
White, et al. Expires April 4, 2005 [Page 8]