Skip to main content

Methodology for Benchmarking SIP Networking Devices
draft-ietf-bmwg-sip-bench-meth-07

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft that was ultimately published as RFC 7502.
Authors Carol Davids , Vijay K. Gurbani , Scott Poretsky
Last updated 2013-01-07 (Latest revision 2013-01-06)
Replaces draft-poretsky-bmwg-sip-bench-meth
RFC stream Internet Engineering Task Force (IETF)
Formats
Reviews
Additional resources Mailing list discussion
Stream WG state In WG Last Call
Awaiting Expert Review/Resolution of Issues Raised, Other - see Comment Log
Document shepherd (None)
IESG IESG state Became RFC 7502 (Informational)
Consensus boilerplate Unknown
Telechat date (None)
Responsible AD Ron Bonica
IESG note ** No value found for 'doc.notedoc.note' **
Send notices to bmwg-chairs@tools.ietf.org, draft-ietf-bmwg-sip-bench-meth@tools.ietf.org
draft-ietf-bmwg-sip-bench-meth-07
Benchmarking Methodology Working Group                         C. Davids
Internet-Draft                          Illinois Institute of Technology
Intended status: Informational                                V. Gurbani
Expires: July 10, 2013                                Bell Laboratories,
                                                          Alcatel-Lucent
                                                             S. Poretsky
                                                    Allot Communications
                                                         January 6, 2013

          Methodology for Benchmarking SIP Networking Devices
                   draft-ietf-bmwg-sip-bench-meth-07

Abstract

   This document describes the methodology for benchmarking Session
   Initiation Protocol (SIP) performance as described in SIP
   benchmarking terminology document.  The methodology and terminology
   are to be used for benchmarking signaling plane performance with
   varying signaling and media load.  Both scale and establishment rate
   are measured by signaling plane performance.  The SIP Devices to be
   benchmarked may be a single device under test (DUT) or a system under
   test (SUT).  Benchmarks can be obtained and compared for different
   types of devices such as SIP Proxy Server, SBC, and server paired
   with a media relay or Firewall/NAT device.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on July 10, 2013.

Copyright Notice

   Copyright (c) 2013 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

Davids, et al.            Expires July 10, 2013                 [Page 1]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Davids, et al.            Expires July 10, 2013                 [Page 2]
Internet-Draft        SIP Benchmarking Methodology          January 2013

Table of Contents

   1.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Benchmarking Topologies  . . . . . . . . . . . . . . . . . . .  5
   4.  Test Setup Parameters  . . . . . . . . . . . . . . . . . . . .  5
     4.1.  Selection of SIP Transport Protocol  . . . . . . . . . . .  5
     4.2.  Signaling Server . . . . . . . . . . . . . . . . . . . . .  5
     4.3.  Associated Media . . . . . . . . . . . . . . . . . . . . .  5
     4.4.  Selection of Associated Media Protocol . . . . . . . . . .  6
     4.5.  Number of Associated Media Streams per SIP Session . . . .  6
     4.6.  Session Duration . . . . . . . . . . . . . . . . . . . . .  6
     4.7.  Attempted Sessions per Second  . . . . . . . . . . . . . .  6
     4.8.  Stress Testing . . . . . . . . . . . . . . . . . . . . . .  6
     4.9.  Benchmarking algorithm . . . . . . . . . . . . . . . . . .  6
   5.  Reporting Format . . . . . . . . . . . . . . . . . . . . . . .  9
     5.1.  Test Setup Report  . . . . . . . . . . . . . . . . . . . .  9
     5.2.  Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10
     5.3.  Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10
   6.  Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10
     6.1.  Baseline Session Establishment Rate of the test bed  . . . 10
     6.2.  Session Establishment Rate without media . . . . . . . . . 11
     6.3.  Session Establishment Rate with Media on DUT/SUT . . . . . 11
     6.4.  Session Establishment Rate with Media not on DUT/SUT . . . 12
     6.5.  Session Establishment Rate with Loop Detection Enabled . . 13
     6.6.  Session Establishment Rate with Forking  . . . . . . . . . 13
     6.7.  Session Establishment Rate with Forking and Loop
           Detection  . . . . . . . . . . . . . . . . . . . . . . . . 14
     6.8.  Session Establishment Rate with TLS Encrypted SIP  . . . . 14
     6.9.  Session Establishment Rate with IPsec Encrypted SIP  . . . 15
     6.10. Session Establishment Rate with SIP Flooding . . . . . . . 16
     6.11. Maximum Registration Rate  . . . . . . . . . . . . . . . . 16
     6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16
     6.13. Maximum IM Rate  . . . . . . . . . . . . . . . . . . . . . 17
     6.14. Session Capacity without Media . . . . . . . . . . . . . . 17
     6.15. Session Capacity with Media  . . . . . . . . . . . . . . . 18
     6.16. Session Capacity with Media and a Media Relay/NAT
           and/or Firewall  . . . . . . . . . . . . . . . . . . . . . 19
   7.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 19
   8.  Security Considerations  . . . . . . . . . . . . . . . . . . . 19
   9.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 19
   10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20
     10.1. Normative References . . . . . . . . . . . . . . . . . . . 20
     10.2. Informative References . . . . . . . . . . . . . . . . . . 20
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20

Davids, et al.            Expires July 10, 2013                 [Page 3]
Internet-Draft        SIP Benchmarking Methodology          January 2013

1.  Terminology

   In this document, the key words "MUST", "MUST NOT", "REQUIRED",
   "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
   RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as
   described in BCP 14, conforming to [RFC2119] and indicate requirement
   levels for compliant implementations.

   Terms specific to SIP [RFC3261] performance benchmarking are defined
   in [I-D.sip-bench-term].

   RFC 2119 defines the use of these key words to help make the intent
   of standards track documents as clear as possible.  While this
   document uses these keywords, this document is not a standards track
   document.  The term Throughput is defined in [RFC2544].

2.  Introduction

   This document describes the methodology for benchmarking Session
   Initiation Protocol (SIP) performance as described in Terminology
   document [I-D.sip-bench-term].  The methodology and terminology are
   to be used for benchmarking signaling plane performance with varying
   signaling and media load.  Both scale and establishment rate are
   measured by signaling plane performance.

   The SIP Devices to be benchmarked may be a single device under test
   (DUT) or a system under test (SUT).  The DUT is a SIP Server, which
   may be any [RFC3261] conforming device.  The SUT can be any device or
   group of devices containing RFC 3261 conforming functionality along
   with Firewall and/or NAT functionality.  This enables benchmarks to
   be obtained and compared for different types of devices such as SIP
   Proxy Server, SBC, SIP proxy server paired with a media relay or
   Firewall/NAT device.  SIP Associated Media benchmarks can also be
   made when testing SUTs.

   The test cases covered in this methodology document provide
   benchmarks metrics of Registration Rate, SIP Session Establishment
   Rate, Session Capacity, and IM Rate.  These can be benchmarked with
   or without associated Media.  Some cases are also included to cover
   Forking, Loop detection, Encrypted SIP, and SIP Flooding.  The test
   topologies that can be used are described in the Test Setup section.
   Topologies are provided for benchmarking of a DUT or SUT.
   Benchmarking with Associated Media can be performed when using a SUT.

   SIP permits a wide range of configuration options that are also
   explained in the Test Setup section.  Benchmark metrics could
   possibly be impacted by Associated Media.  The selected values for

Davids, et al.            Expires July 10, 2013                 [Page 4]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Session Duration and Media Streams Per Session enable benchmark
   metrics to be benchmarked without Associated Media.  Session Setup
   Rate could possibly be impacted by the selected value for Maximum
   Sessions Attempted.  The benchmark for Session Establishment Rate is
   measured with a fixed value for maximum Session Attempts.

   Finally, the overall value of these tests is to serve as a comparison
   function between multiple SIP implementations.  One way to use these
   tests is to derive benchmarks with SIP devices from Vendor-A, derive
   a new set of benchmarks with similar SIP devices from Vendor-B and
   perform a comparison on the results of Vendor-A and Vendor-B.  This
   document does not make any claims on the interpretation of such
   results.

3.  Benchmarking Topologies

   Familiarity with the benchmarking models in Section 2.2 of
   [I-D.sip-bench-term] is assumed.  Figures 1 through 10 in
   [I-D.sip-bench-term] contain the canonical topologies that can be
   used to perform the benchmarking tests listed in this document.

4.  Test Setup Parameters

4.1.  Selection of SIP Transport Protocol

   Test cases may be performed with any transport protocol supported by
   SIP.  This includes, but is not limited to, SIP TCP, SIP UDP, and
   TLS.  The protocol used for the SIP transport protocol must be
   reported with benchmarking results.

4.2.  Signaling Server

   The Signaling Server is defined in the companion terminology
   document, ([I-D.sip-bench-term], Section 3.2.2) It is a SIP-speaking
   device that complies with RFC 3261.  Conformance to [RFC3261] is
   assumed for all tests.  The Signaling Server may be the DUT or a
   component of a SUT.  The Signaling Server may include Firewall and/or
   NAT functionality.  The components of the SUT may be a single
   physical device or separate devices.

4.3.  Associated Media

   Some tests require Associated Media to be present for each SIP
   session.  The test topologies to be used when benchmarking SUT
   performance for Associated Media are shown in [I-D.sip-bench-term],
   Figures 4 and 5.

Davids, et al.            Expires July 10, 2013                 [Page 5]
Internet-Draft        SIP Benchmarking Methodology          January 2013

4.4.  Selection of Associated Media Protocol

   The test cases specified in this document provide SIP performance
   independent of the protocol used for the media stream.  Any media
   protocol supported by SIP may be used.  This includes, but is not
   limited to, RTP, RTSP, and SRTP.  The protocol used for Associated
   Media MUST be reported with benchmarking results.

4.5.  Number of Associated Media Streams per SIP Session

   Benchmarking results may vary with the number of media streams per
   SIP session.  When benchmarking a SUT for voice, a single media
   stream is used.  When benchmarking a SUT for voice and video, two
   media streams are used.  The number of Associated Media Streams MUST
   be reported with benchmarking results.

4.6.  Session Duration

   SUT performance benchmarks may vary with the duration of SIP
   sessions.  Session Duration MUST be reported with benchmarking
   results.  A Session Duration of zero seconds indicates transmission
   of a BYE immediately following successful SIP establishment indicate
   by receipt of a 200 OK.  An infinite Session Duration indicates that
   a BYE is never transmitted.

4.7.  Attempted Sessions per Second

   DUT and SUT performance benchmarks may vary with the the rate of
   attempted sessions offered by the Tester.  Attempted Sessions per
   Second MUST be reported with benchmarking results.

4.8.  Stress Testing

   The purpose of this document is to benchmark SIP performance; this
   document does not benchmark stability of SIP systems under stressful
   conditions such as a high rate of Attempted Sessions per Second.

4.9.  Benchmarking algorithm

   In order to benchmark the test cases uniformly in Section 6, the
   algorithm described in this section should be used.  Both, a prosaic
   description of the algorithm and a pseudo-code description are
   provided.

   The goal is to find the largest value of a SIP session-request-rate,
   measured in sessions-per-second, which the DUT/SUT can process with
   zero errors.  To discover that number, an iterative process (defined
   below) is used to find a candidate for this rate.  Once the candidate

Davids, et al.            Expires July 10, 2013                 [Page 6]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   rate has been found, the DUT/SUT is subjected to an offered load
   whose arrival rate is set to that of the candidate rate.  This test
   is run for an extended period of time, which is referred to as
   infinity, and which is, itself, a parameter of the test labeled T in
   the pseudo-code.  This latter phase of testing is called the steady-
   state phase.  If errors are encountered during this steady-state
   phase, then the candidate rate is reduced by a defined percent, also
   a parameter of test, and the steady-state phase is entered again
   until a final (new) steady-state rate is achieved.

   The iterative process itself is defined as follows: a starting rate
   of 100 sessions per second (sps) is selected.  The test is executed
   for the time period identified by t in the pseudo-code below.  If no
   failures occur, the rate is increased to 150 sps and again tested for
   time period t.  The attempt rate is continuously ramped up until a
   failure is encountered before the end of the test time t.  Then an
   attempt rate is calculated that is higher than the last successful
   attempt rate by a quantity equal to half the difference between the
   rate at which failures occurred and the last successful rate.  If
   this new attempt rate also results in errors, a new attempt rate is
   tried that is higher than the last successful attempt rate by a
   quantity equal to half the difference between the rate at which
   failures occurred and the last successful rate.  Continuing in this
   way, an attempt rate without errors is found.  The operator can
   specify margin of error using the parameter G, measured in units of
   sessions per second.

   The pseudo-code corresponding to the description above follows.

   ; ---- Parameters of test, adjust as needed
   t  := 5000   ; local maximum; used to figure out largest
                ; value
   T  := 50000  ; global maximum; once largest value has been
                ; figured out, pump this many requests before calling
                ; the test a success
   m  := {...}  ; other attributes that affect testing, such
                ; as media streams, etc.
   s  := 100    ; Initial session attempt rate (in sessions/sec)
   G  := 5      ; granularity of results - the margin of error in sps
   C  := 0.05   ; caliberation amount: How much to back down if we
                ; have found candidate s but cannot send at rate s for
          ; time T without failures

   ; ---- End of parameters of test
   ; ---- Initialization of flags, candidate values and upper bounds

   f  := false  ; indicates that you had a success after the upper limit

Davids, et al.            Expires July 10, 2013                 [Page 7]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   F  := false  ; indicates that test is done
   c  := 0      ; indicates that we have found an upper limit

   proc main
      find_largest_value  ; First, figure out the largest value.

      ; Now that the largest value (saved in s) has been figured out,
      ; use it for sending out s requests/s and send out T requests.

      do {
         send_traffic(s, m, T)    ; send_traffic not shown
         if (all requests succeeded) {
            F := true ; test is done
         } else if (one or more requests fail) {
            s := s - (C * s)  ; Reduce s by calibration amount
            steady_state
         }
      } while (F == false)
   end proc

   proc find_largest_value
      ; Iterative process to figure out the largest value we can
      ; handle with no failures
      do  {
          send_traffic(s, m, t) ; Send s request/sec with m
                                ; characteristics until t requests have
                                ; been sent
          if (all requests succeeded) {
             s' := s ; save candidate value of metric

             if ( c == 0 ) {
             s  := s + (0.5 * s)

             }else if ((c == 1) &&  (s??-s?)) > 2*G ) {
                 s := s + ( 0.5 * (s?? ? s );

             }else if ((c == 1) &&  ((s??-s?) <= 2*G ) {
                 f := true;

          }
             else if (one or more requests fail)  {
             c  := 1     ; we have found an upper bound for the metric
             s?? := s    ; save new upper bound
             s  := s - (0.5 * (s ? s?))
          }
      } while (f == false)
   end proc

Davids, et al.            Expires July 10, 2013                 [Page 8]
Internet-Draft        SIP Benchmarking Methodology          January 2013

5.  Reporting Format

5.1.  Test Setup Report

     SIP Transport Protocol = ___________________________
     (valid values: TCP|UDP|TLS|SCTP|specify-other)
     Session Attempt Rate = _____________________________
     (session attempts/sec)
     IS Media Attempt Rate = ____________________________
     (IS media attempts/sec)
     Total Sessions Attempted = _________________________
     (total sessions to be created over duration of test)
     Media Streams Per Session =  _______________________
     (number of streams per session)
     Associated Media Protocol =  _______________________
     (RTP|RTSP|specify-other)
     Media Packet Size =  _______________________________
     (bytes)
     Media Offered Load =  ______________________________
     (packets per second)
     Media Session Hold Time =  _________________________
     (seconds)
     Establishment Threshold time =  ____________________
     (seconds)
     Loop Detecting Option =  ___________________________
     (on|off)
     Forking Option
        Number of endpoints request sent to = ___________
     (1, means forking is not enabled)
        Type of forking = _______________________________
     (serial|parallel)
     Authentication option = ___________________________________
       (on|off; if on, please see Notes 2 and 3 below).

   Note 1: Total Sessions Attempted is used in the calculation of the
   Session Establishment Performance ([I-D.sip-bench-term], Section
   3.4.5).  It is the number of session attempts ([I-D.sip-bench-term],
   Section 3.1.6) that will be made over the duration of the test.

   Note 2: When the Authentication Option is "on" the test tool must be
   set to ignore 401 and 407 failure responses in any test described as
   a "test to failure."  If this is not done, all such tests will yield
   trivial benchmarks, as all attempt rates will lead to a failure after
   the first attempt.

   Note 3: When the Authentication Option is "on" the DUT/SUT uses two

Davids, et al.            Expires July 10, 2013                 [Page 9]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   transactions instead of one when it is establishing a session or
   accomplishing a registration.  The first transaction ends with the
   401 or 407.  The second ends with the 200 OK or another failure
   message.  The Test Organization interested in knowing how many times
   the EA was intended to send a REGISTER as distinct from how many
   times the EA wound up actually sending a REGISTER may wish to record
   the following data as well: Number of responses of the following
   type:

   401: _____________ (if authentication turned on; N/A otherwise)
   407: _____________ (if authentication turned on; N/A otherwise)

5.2.  Device Benchmarks for IS

     Registration Rate =  _______________________________
     (registrations per second)
     Re-registration Rate =  ____________________________
     (registrations per second)
     Session Capacity = _________________________________
     (sessions)
     Session Overload Capacity = ________________________
     (sessions)
     Session Establishment Rate =  ______________________
     (sessions per second)
     Session Establishment Performance =  _______________
     (total established sessions/total sessions attempted)(no units)
     Session Attempt Delay =  ___________________________
     (seconds)

5.3.  Device Benchmarks for NS

     IM Rate =  _______________________________ (IM messages per second)

6.  Test Cases

6.1.  Baseline Session Establishment Rate of the test bed

   Objective:
      To benchmark the Session Establishment Rate of the Emulated Agent
      (EA) with zero failures.

Davids, et al.            Expires July 10, 2013                [Page 10]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 in
          [I-D.sip-bench-term].
      2.  Set media streams per session to 0.
      3.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the baseline session establishment rate.  This rate MUST
          be recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.

   Expected Results:  This is the scenario to obtain the maximum Session
      Establishment Rate of the EA and the test bed when no DUT/SUT is
      present.  The results of this test might be used to normalize test
      results performed on different test beds or simply to better
      understand the impact of the DUT/SUT on the test bed in question.

6.2.  Session Establishment Rate without media

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with no
      associated media and zero failures.

   Procedure:
      1.  If the DUT/SUT is being benchmarked as a user agent client or
          a user agent server, configure the DUT in the test topology
          shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
          Alternatively, if the DUT is being benchmarked as a proxy or a
          B2BUA, configure the DUT in the test topology shown in Figure
          5 in [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          7 in [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate.  This rate MUST be
          recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.

   Expected Results:  This is the scenario to obtain the maximum Session
      Establishment Rate of the DUT/SUT.

6.3.  Session Establishment Rate with Media on DUT/SUT

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when Associated Media is included in the benchmark
      test and the media is running through the DUT/SUT.

Davids, et al.            Expires July 10, 2013                [Page 11]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Procedure:
      1.  If the DUT is being benchmarked as a user agent client or a
          user agent server, configure the DUT in the test topology
          shown in Figure 3 or Figure 4 of [I-D.sip-bench-term].
          Alternatively, if the DUT is being benchmarked as a B2BUA,
          configure the DUT in the test topology shown in Figure 6 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          9 in [I-D.sip-bench-term].
      3.  Set media streams per session to 1.
      4.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with media.  This rate MUST
          be recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      Associated Media with any number of media streams per SIP session
      are expected to be identical to the Session Establishment Rate
      results obtained without media in the case where the server is
      running on a platform separate from the platform on which the
      Media Relay, NAT or Firewall is running.  Session Establishment
      Rate results obtained with Associated Media may be lower than
      those obtained without media in the case where the server and the
      NAT, Firewall or Media Relay are running on the same platform.

6.4.  Session Establishment Rate with Media not on DUT/SUT

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when Associated Media is included in the benchmark
      test but the media is not running through the DUT/SUT.

   Procedure:
      1.  If the DUT is being benchmarked as proxy or B2BUA, configure
          the DUT in the test topology shown in Figure 7 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 in [I-D.sip-bench-term].
      3.  Set media streams per session to 1.
      4.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with media.  This rate MUST
          be recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.

Davids, et al.            Expires July 10, 2013                [Page 12]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Expected Results:  Session Establishment Rate results obtained with
      Associated Media with any number of media streams per SIP session
      are expected to be identical to the Session Establishment Rate
      results obtained without media in the case where the server is
      running on a platform separate from the platform on which the
      Media Relay, NAT or Firewall is running.  Session Establishment
      Rate results obtained with Associated Media may be lower than
      those obtained without media in the case where the server and the
      NAT, Firewall or Media Relay are running on the same platform.

6.5.  Session Establishment Rate with Loop Detection Enabled

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when the Loop Detection option is enabled and no
      media streams are present.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, and loop
          detection is supported in the DUT, then configure the DUT in
          the test topology shown in Figure 5 in [I-D.sip-bench-term].
          If the DUT does not support loop detection, then this step can
          be skipped.
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Turn on the Loop Detection option in the DUT or SUT.
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with loop detection
          enabled.  This rate MUST be recorded using any pertinent
          parameters as shown in the reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      Loop Detection may be lower than those obtained without Loop
      Detection enabled.

6.6.  Session Establishment Rate with Forking

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when the Forking Option is enabled.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, and
          forking is supported in the DUT, then configure the DUT in the
          test topology shown in Figure 5 in [I-D.sip-bench-term].  If
          the DUT does not support forking, then this step can be

Davids, et al.            Expires July 10, 2013                [Page 13]
Internet-Draft        SIP Benchmarking Methodology          January 2013

          skipped.
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Set the number of endpoints that will receive the forked
          invitation to a value of 2 or more (subsequent tests may
          increase this value at the discretion of the tester.)
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with forking.  This rate
          MUST be recorded using any pertinent parameters as shown in
          the reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      Forking may be lower than those obtained without Forking enabled.

6.7.  Session Establishment Rate with Forking and Loop Detection

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when both the Forking and Loop Detection Options are
      enabled.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, then
          configure the DUT in the test topology shown in Figure 5 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Enable the Loop Detection Options on the DUT.
      5.  Set the number of endpoints that will receive the forked
          invitation to a value of 2 or more (subsequent tests may
          increase this value at the discretion of the tester.)
      6.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with forking and loop
          detection.  This rate MUST be recorded using any pertinent
          parameters as shown in the reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      Forking and Loop Detection may be lower than those obtained with
      only Forking or Loop Detection enabled.

6.8.  Session Establishment Rate with TLS Encrypted SIP

Davids, et al.            Expires July 10, 2013                [Page 14]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when using TLS encrypted SIP.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, then
          configure the DUT in the test topology shown in Figure 5 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Configure Tester to enable TLS over the transport being
          benchmarked.  Make a note the transport when compiling
          results.  May need to run for each transport of interest.
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with encryption.  This rate
          MUST be recorded using any pertinent parameters as shown in
          the reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      TLS Encrypted SIP may be lower than those obtained with plaintext
      SIP.

6.9.  Session Establishment Rate with IPsec Encrypted SIP

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when using IPsec Encryoted SIP.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, then
          configure the DUT in the test topology shown in Figure 5 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Configure Tester for IPSec.
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with encryption.  This rate
          MUST be recorded using any pertinent parameters as shown in
          the reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      IPSec Encrypted SIP may be lower than those obtained with
      plaintext SIP.

Davids, et al.            Expires July 10, 2013                [Page 15]
Internet-Draft        SIP Benchmarking Methodology          January 2013

6.10.  Session Establishment Rate with SIP Flooding

   Objective:
      To benchmark the Session Establishment Rate of the SUT with zero
      failures when SIP Flooding is occurring.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, then
          configure the DUT in the test topology shown in Figure 5 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Set s = 500 (c.f.  Section 4.9).
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with flooding.  This rate
          MUST be recorded using any pertinent parameters as shown in
          the reporting format of Section 5.1.

   Expected Results:  Session Establishment Rate results obtained with
      SIP Flooding may be degraded.

6.11.  Maximum Registration Rate

   Objective:
      To benchmark the maximum registration rate of the DUT/SUT with
      zero failures.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, then
          configure the DUT in the test topology shown in Figure 5 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  Set media streams per session to 0.
      4.  Set the registration timeout value to at least 3600 seconds.
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the maximum registration rate.  This rate MUST be recorded
          using any pertinent parameters as shown in the reporting
          format of Section 5.1.
   Expected Results:

6.12.  Maximum Re-Registration Rate

Davids, et al.            Expires July 10, 2013                [Page 16]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Objective:
      To benchmark the maximum re-registration rate of the DUT/SUT with
      zero failures.

   Procedure:
      1.  If the DUT is being benchmarked as a proxy or B2BUA, then
          configure the DUT in the test topology shown in Figure 5 in
          [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          8 of [I-D.sip-bench-term].
      3.  First, execute test detailed in Section 6.11 to register the
          endpoints with the registrar.
      4.  After at least 5 mintes of Step 2, but no more than 10 minutes
          after Step 2 has been performed, execute test detailed in
          Section 6.11 again (this will count as a re-registration).
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the maximum re-registration rate.  This rate MUST be
          recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.
   Expected Results:  The rate should be at least equal to but not more
      than the result of Section 6.11.

6.13.  Maximum IM Rate

   Objective:
      To benchmark the maximum IM rate of the SUT with zero failures.

   Procedure:
      1.  If the DUT/SUT is being benchmarked as a user agent client or
          a user agent server, configure the DUT in the test topology
          shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
          Alternatively, if the DUT is being benchmarked as a proxy or a
          B2BUA, configure the DUT in the test topology shown in Figure
          5 in [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          5 in [I-D.sip-bench-term].
      3.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the maximum IM rate.  This rate MUST be recorded using any
          pertinent parameters as shown in the reporting format of
          Section 5.1.

   Expected Results:

6.14.  Session Capacity without Media

Davids, et al.            Expires July 10, 2013                [Page 17]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Objective:
      To benchmark the Session Capacity of the SUT without Associated
      Media.
   Procedure:
      1.  If the DUT/SUT is being benchmarked as a user agent client or
          a user agent server, configure the DUT in the test topology
          shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
          Alternatively, if the DUT is being benchmarked as a proxy or a
          B2BUA, configure the DUT in the test topology shown in Figure
          5 in [I-D.sip-bench-term].
      2.  Configure a SUT according to the test topology shown in Figure
          7 in [I-D.sip-bench-term].
      3.  Set the media treams per session to be 0.
      4.  Set the Session Duration to be a value greater than T.
      5.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the baseline session establishment rate.  This rate MUST
          be recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.
      6.  The Session Capacity is the product of T and the Session
          Establishment Rate.
   Expected Results:  The maximum rate at which the DUT/SUT can handle
      session establishment requests with no media for an infinitely
      long period with no errors.  This is the SIP "throughput" of the
      system with no media.

6.15.  Session Capacity with Media

   Objective:
      To benchmark the session capacity of the DUT/SUT with Associated
      Media.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 3 or
          Figure 4 of [I-D.sip-bench-term] depending on whether the DUT
          is being benchmarked as a user agent client or user agent
          server.  Alternatively, configure the DUT in the test topology
          shown in Figure 6 or Figure 7 in [I-D.sip-bench-term]
          depending on whether the DUT is being benchmarked as a B2BUA
          or as a proxy.  If a SUT is being benchmarked, configure the
          SUT as shown in Figure 9 of [I-D.sip-bench-term].
      2.  Set the media streams per session to 1.
      3.  Set the Session Duration to be a value greater than T.
      4.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the baseline session establishment rate.  This rate MUST
          be recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.
      5.  The Session Capacity is the product of T and the Session
          Establishment Rate.

Davids, et al.            Expires July 10, 2013                [Page 18]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Expected Results:  Session Capacity results obtained with Associated
      Media with any number of media streams per SIP session will be
      identical to the Session Capacity results obtained without media.

6.16.  Session Capacity with Media and a Media Relay/NAT
       and/or Firewall

   Objective:
      To benchmark the Session Establishment Rate of the SUT with
      Associated Media.
   Procedure:
      1.  Configure the SUT as shown in Figure 7 or Figure 10 in
          [I-D.sip-bench-term].
      2.  Set media streams per session to 1.
      3.  Execute benchmarking algorithm as defined in Section 4.9 to
          get the session establishment rate with media.  This rate MUST
          be recorded using any pertinent parameters as shown in the
          reporting format of Section 5.1.

   Expected Results:  Session Capacity results obtained with Associated
      Media with any number of media streams per SIP session may be
      lower than the Session Capacity without Media result if the Media
      Relay, NAT or Firewall is sharing a platform with the server.

7.  IANA Considerations

   This document does not requires any IANA considerations.

8.  Security Considerations

   Documents of this type do not directly affect the security of
   Internet or corporate networks as long as benchmarking is not
   performed on devices or systems connected to production networks.
   Security threats and how to counter these in SIP and the media layer
   is discussed in RFC3261, RFC3550, and RFC3711 and various other
   drafts.  This document attempts to formalize a set of common
   methodology for benchmarking performance of SIP devices in a lab
   environment.

9.  Acknowledgments

   The authors would like to thank Keith Drage and Daryl Malas for their
   contributions to this document.  Dale Worley provided an extensive
   review that lead to improvements in the documents.

Davids, et al.            Expires July 10, 2013                [Page 19]
Internet-Draft        SIP Benchmarking Methodology          January 2013

10.  References

10.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

   [I-D.sip-bench-term]
              Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance
              Benchmarking Terminology",
              draft-ietf-bmwg-sip-bench-term-07 (work in progress),
              March 2012.

10.2.  Informative References

   [RFC3261]  Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
              A., Peterson, J., Sparks, R., Handley, M., and E.
              Schooler, "SIP: Session Initiation Protocol", RFC 3261,
              June 2002.

Authors' Addresses

   Carol Davids
   Illinois Institute of Technology
   201 East Loop Road
   Wheaton, IL  60187
   USA

   Phone: +1 630 682 6024
   Email: davids@iit.edu

   Vijay K. Gurbani
   Bell Laboratories, Alcatel-Lucent
   1960 Lucent Lane
   Rm 9C-533
   Naperville, IL  60566
   USA

   Phone: +1 630 224 0216
   Email: vkg@bell-labs.com

Davids, et al.            Expires July 10, 2013                [Page 20]
Internet-Draft        SIP Benchmarking Methodology          January 2013

   Scott Poretsky
   Allot Communications
   300 TradeCenter, Suite 4680
   Woburn, MA  08101
   USA

   Phone: +1 508 309 2179
   Email: sporetsky@allot.com

Davids, et al.            Expires July 10, 2013                [Page 21]