Benchmarking Methodology for Network Interconnect Devices
RFC 1944
Document | Type |
RFC - Informational
(May 1996; No errata)
Obsoleted by RFC 2544
|
|
---|---|---|---|
Authors | Jim McQuaid , Scott Bradner | ||
Last updated | 2013-03-02 | ||
Stream | Internent Engineering Task Force (IETF) | ||
Formats | plain text html pdf htmlized (tools) htmlized bibtex | ||
Stream | WG state | (None) | |
Document shepherd | No shepherd assigned | ||
IESG | IESG state | RFC 1944 (Informational) | |
Consensus Boilerplate | Unknown | ||
Telechat date | |||
Responsible AD | (None) | ||
Send notices to | (None) |
Network Working Group S. Bradner Request for Comments: 1944 Harvard University Category: Informational J. McQuaid Bay Networks May 1996 Benchmarking Methodology for Network Interconnect Devices Status of This Memo This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Abstract This document discusses and defines a number of tests that may be used to describe the performance characteristics of a network interconnecting device. In addition to defining the tests this document also describes specific formats for reporting the results of the tests. Appendix A lists the tests and conditions that we believe should be included for specific cases and gives additional information about testing practices. Appendix B is a reference listing of maximum frame rates to be used with specific frame sizes on various media and Appendix C gives some examples of frame formats to be used in testing. 1. Introduction Vendors often engage in "specsmanship" in an attempt to give their products a better position in the marketplace. This often involves "smoke & mirrors" to confuse the potential users of the products. This document defines a specific set of tests that vendors can use to measure and report the performance characteristics of network devices. The results of these tests will provide the user comparable data from different vendors with which to evaluate these devices. A previous document, "Benchmarking Terminology for Network Interconnect Devices" (RFC 1242), defined many of the terms that are used in this document. The terminology document should be consulted before attempting to make use of this document. Bradner & McQuaid Informational [Page 1] RFC 1944 Benchmarking Methodology May 1996 2. Real world In producing this document the authors attempted to keep in mind the requirement that apparatus to perform the described tests must actually be built. We do not know of "off the shelf" equipment available to implement all of the tests but it is our opinion that such equipment can be constructed. 3. Tests to be run There are a number of tests described in this document. Not all of the tests apply to all types of devices under test (DUTs). Vendors should perform all of the tests that can be supported by a specific type of product. The authors understand that it will take a considerable period of time to perform all of the recommended tests nder all of the recommended conditions. We believe that the results are worth the effort. Appendix A lists some of the tests and conditions that we believe should be included for specific cases. 4. Evaluating the results Performing all of the recommended tests will result in a great deal of data. Much of this data will not apply to the evaluation of the devices under each circumstance. For example, the rate at which a router forwards IPX frames will be of little use in selecting a router for an environment that does not (and will not) support that protocol. Evaluating even that data which is relevant to a particular network installation will require experience which may not be readily available. Furthermore, selection of the tests to be run and evaluation of the test data must be done with an understanding of generally accepted testing practices regarding repeatability, variance and statistical significance of small numbers of trials. 5. Requirements In this document, the words that are used to define the significance of each particular requirement are capitalized. These words are: * "MUST" This word, or the words "REQUIRED" and "SHALL" mean that the item is an absolute requirement of the specification. * "SHOULD" This word or the adjective "RECOMMENDED" means that there may exist valid reasons in particular circumstances to ignore this item, but the full implications should be understood and the case carefully weighed before choosing a different course. * "MAY" This word or the adjective "OPTIONAL" means that this item is truly optional. One vendor may choose to include the item because Bradner & McQuaid Informational [Page 2] RFC 1944 Benchmarking Methodology May 1996 a particular marketplace requires it or because it enhances the product, for example; another vendor may omit the same item. An implementation is not compliant if it fails to satisfy one or more of the MUST requirements for the protocols it implements. AnShow full document text