Skip to main content

Evaluation Test Cases for Interactive Real-Time Media over Cellular Networks
draft-sarker-rmcat-cellular-eval-test-cases-00

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft whose latest revision state is "Replaced".
Authors Zaheduzzaman Sarker , Ingemar Johansson
Last updated 2014-02-14
Replaced by draft-ietf-rmcat-wireless-tests, RFC 8869
RFC stream (None)
Formats
Additional resources
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-sarker-rmcat-cellular-eval-test-cases-00
Network Working Group                                          Z. Sarker
Internet-Draft                                              I. Johansson
Intended status: Informational                               Ericsson AB
Expires: August 18, 2014                               February 14, 2014

  Evaluation Test Cases for Interactive Real-Time Media over Cellular
                                Networks
             draft-sarker-rmcat-cellular-eval-test-cases-00

Abstract

   It is evident that to ensure seamless and robust user experience
   across all type of access networks multimedia communication suits
   should adapt to the changing network conditions.  There is an ongoing
   effort in IETF RMCAT working group to standardize rate adaptive
   algorithm(s) to be used in the real-time interactive communication.
   In this document test cases are described to evaluate the
   performances of the proposed endpoint adaptation solutions in a
   cellular network such as LTE network.  It is aimed that the proposed
   solutions should be evaluated using the test cases defines in this
   document to select most optimal solutions.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on August 18, 2014.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of

Sarker & Johansson       Expires August 18, 2014                [Page 1]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminologies . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Cellular Network Specific Test Cases  . . . . . . . . . . . .   4
     3.1.  Varying Network Load  . . . . . . . . . . . . . . . . . .   5
       3.1.1.  Network Connection  . . . . . . . . . . . . . . . . .   5
       3.1.2.  Simulation Setup  . . . . . . . . . . . . . . . . . .   6
     3.2.  Bad Radio Coverage  . . . . . . . . . . . . . . . . . . .   7
       3.2.1.  Network connection  . . . . . . . . . . . . . . . . .   8
       3.2.2.  Simulation Setup  . . . . . . . . . . . . . . . . . .   8
   4.  Desired Evaluation Metrics  . . . . . . . . . . . . . . . . .   8
   5.  Conclusion  . . . . . . . . . . . . . . . . . . . . . . . . .   9
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   9
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   9
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  10
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  10

1.  Introduction

   Cellular networks are an integral part of the Internet.  Mobile
   devices connected to the cellular networks produces huge amount of
   media traffic in the Internet.  It is important to evaluate the
   performance of the proposed RMCAT candidates in the cellular network.

   A cellular environment is more complicated than a wireline ditto
   since it seeks to provide services in the context of variable
   available bandwidth, location dependencies and user mobilities at
   different speeds.  In a cellular network the user may reach the cell
   edge which may lead to a significant amount of retransmissions to
   deliver the data from the base station to the destination and vice
   versa.  These network links or radio links will often act as a
   bottleneck for the rest of the network which will eventually lead to
   excessive delays or packet drops.  An efficient retransmission or
   link adaptation mechanism can reduce the packet loss probability but
   there will still be some packet losses and delay variations.
   Moreover, with increased cell load or handover to a congested cell,
   transport network level congestion will become even worse.  Besides,

Sarker & Johansson       Expires August 18, 2014                [Page 2]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   there are certain characteristics which make the cellular network
   different and challenging than other types of access network such as
   Wi-Fi and wired network.  In a cellular network -

   o  The bottleneck is often a shared link with relatively few users.

      *  The cost per bit over the shared link varies over time and is
         different for different users.

      *  Left over/ unused resource can be grabbed by other greedy
         users.

   o  Queues are always per radio bearer hence each user can have many
      of such queues.

   o  Users can experience both Inter and Intra Radio Access Technology
      (RAT) handovers ("handover" definition in [HO-def-3GPP] ).

   o  Handover between cells, or change of serving cells (see in
      [HO-LTE-3GPP] and [HO-UMTS-3GPP] ) might cause user plane
      interruptions which can lead to bursts of packet losses, delay and
      /or jitter.  The exact behavior depends on the type of radio
      bearer.  Typically, the default best effort bearers do not
      generate packet loss, instead packets are queued up and
      transmitted once the handover is completed.

   o  The network part decides how much the user can transmit.

   o  The cellular network has variable link capacity per user

      *  Can vary as fast as a period of milliseconds.

      *  Depends on lots of facts (such as distance, speed,
         interference, different flows).

      *  Uses complex and smart link adaptation which makes the link
         behavior ever more dynamic.

      *  The scheduling priority depends on the estimated throughput.

   o  Both Quality of Service (QoS) and non-QoS radio bearers can be
      used.

   Hence, a real-time communication application operating in such a
   cellular network need to cope with shared bottleneck link and
   variable link capacity, event likes handover, non-congestion related
   loss, abrupt change in bandwidth (both short term and long term) due
   to handover, network load and bad radio coverage.  Even though 3GPP

Sarker & Johansson       Expires August 18, 2014                [Page 3]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   define QoS bearers [QoS-3GPP] to ensure high quality user experience,
   adaptive real-time applications are desired.

   Different mobile operators deploy their own cellular network with
   their own set of network functionalities and policies.  Usually, a
   mobile operator network includes 2G, EDGE, 3G and 4G radio access
   technologies.  Looking at the specifications of such radio
   technologies it is evident that only 3G and 4G radio technologies can
   support the high bandwidth requirements from real-time interactive
   video applications.  The future real-time interactive application
   will impose even greater demand on cellular network performance which
   makes 4G (and beyond radio technologies) more suitable access
   technology for such genre of application.

   RMCAT evaluation criteria [I-D.ietf-rmcat-eval-criteria] document
   provides the guideline to perform the evaluation on candidate
   algorithms and recognize cellular networks to be important access
   link, however, it does not provides particular test cases to evaluate
   the performance of the candidate algorithm.  In this document we
   device test cases specifically targeting cellular networks such as
   LTE networks.

2.  Terminologies

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC2119 [RFC2119]

3.  Cellular Network Specific Test Cases

   The key factors to define test cases for cellular network are

   o  Shared and varying link capacity

   o  Mobility

   o  Handover

   However, for cellular network it is very hard to separate such events
   from one another as these events are heavily related.  Hence instead
   of devising separate test cases for all those important events we
   have divided the test case in two categories.  It should be noted
   that in the following test cases the goal is to evaluate the
   performance of candidate algorithms over radio interface of the
   cellular network.  Hence it is assumed that the radio interface is
   the bottleneck link between the communicating peers and that the core
   network does not add any extra congestion in the path.  Also the
   combination of multiple access technologies such as one user has LTE

Sarker & Johansson       Expires August 18, 2014                [Page 4]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   connection and another has Wi-Fi connection is kept out of the scope
   of this document.  However, later those additional scenarios can also
   be added in this list of test cases.  While defining the test cases
   we assumed a typical real-time telephony scenario over cellular
   networks where one real-time session consists of one voice stream and
   one video stream.  We recommend that an LTE network simulator is used
   for the test cases defined in this document.

3.1.  Varying Network Load

   The goal of this test is to evaluate the performance of candidate
   congestion control algorithm under varying network load.  The network
   load variation is created by adding and removing network users a.k.a.
   User Equipments (UEs) during the simulation.  In this test case each
   of the user/UE in the media session is an RMCAT compliant endpoint.
   The arrival of users follows a Poisson distribution, which is
   proportional to the length of the call, so that the number of users
   per cell is kept fairly constant during the evaluation period.  At
   the beginning of the simulation there should be enough amount of time
   to warm-up the network.  This is to avoid running the evaluation in
   an empty network where network nodes are having empty buffers, low
   interference at the beginning of the simulation.  This network
   initialization period is therefore excluded from the evaluation
   period.

   This test case also includes user mobility and competing traffic.
   The competing traffics includes both same kind of flows (with same
   adaptation algorithms) and different kind of flows (with different
   service and congestion control).  The investigated congestion control
   algorithms should show maximum possible network utilization and
   stability in terms of rate variations, lowest possible end to end
   frame latency, network latency and Packet Loss Rate (PLR) at
   different cell load level.

3.1.1.  Network Connection

   Each mobile user is connected to a fixed user.  The connection
   between the mobile user and fixed user consists of a LTE radio
   access, an Evolved Packet Core (EPC) and an Internet connection.  The
   mobile user is connected to the EPC using LTE radio access technology
   which is further connected to the Internet.  The fixed user is
   connected to the Internet via wired connection with no bottleneck
   (practically infinite bandwidth).  The Internet and wired connection
   in this setup does not add any network impairments to the test, it
   only adds 10ms of one-way transport propagation delay.

Sarker & Johansson       Expires August 18, 2014                [Page 5]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

3.1.2.  Simulation Setup

   The desired simulation setup as follows-

   1.  Radio environment

       A.  Deployment : 3GPP case 1[Deployment]

       B.  Antenna: Multiple-Input and Multiple-Output (MIMO)

       C.  Mobility: [3km/h, 30km/h]

       D.  Transmission bandwidth: 10Mhz

       E.  Number of cells: multi cell deployment (3 Cells per Base
           Station (BS) * 7 BS) = 21 cells

       F.  Cell radius: 166.666 Meters

       G.  Scheduler: Proportional fair with no priority

       H.  Bearer: Default bearer for all traffic.

       I.  Active Queue Management (AQM) settings: AQM [on,off]

   2.  End to end Round Trip Time (RTT): on avg. {40ms,....,150ms}

   3.  User arrival model: Poisson arrival model

   4.  User intensity = {0.7,.....,10.5}

   5.  Simulation duration: 91s

   6.  Evaluation period : 30s-60s

   7.  Media traffic model

       1.  Media Type: Video

           a.  Number of Media source per user: One (1)

           b.  Media direction: [Uplink, Downlink]

           c.  Media duration per user: 30s

           d.  Media codec : Variable BitRate (VBR)

           e.  Bitrate :

Sarker & Johansson       Expires August 18, 2014                [Page 6]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

               -  Maximum bitrate: nominal 1.5 Mbps

               -  Minimum bitrate: nominal 150kbps

           f.  Adaptation:

               -  Bitrate: 150kbps-1.5mbps

               -  Resolution: 144p-720p(1080p)

               -  Framerate: 10fps-30fps

       2.  Media Type : Voice

           a.  Number of Media source per user: One (1)

           b.  Media direction: {Uplink, Downlink}

           c.  Media duration per user: 30s

           d.  Media codec: Constant BitRate (CBR)

           e.  Bitrate :

               -  Maximum bitrate: 20 kbps

           f.  Adaptation: off

   8.  Other traffic model: Maximum of 2Mbps/cell (web browsing or FTP
       traffic)

3.2.  Bad Radio Coverage

   The goal of this test is to evaluate the performance of candidate
   congestion control algorithm when users visit part of the network
   with bad radio coverage.  The scenario is created by using larger
   cell radius than previous test case.  In this test case each of the
   user/UE in the media session is an RMCAT compliant endpoint.  The
   arrival of users follows a Poisson distribution, which is
   proportional to the length of the call, so that the number of users
   per cell is kept fairly constant during the evaluation period.  At
   the beginning of the simulation there should be enough amount of time
   to warm-up the network.  This is to avoid running the evaluation in
   an empty network where network nodes are having empty buffers, low
   interference at the beginning of the simulation.  This network
   initialization period is therefore excluded from the evaluation
   period.

Sarker & Johansson       Expires August 18, 2014                [Page 7]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   This test case also includes user mobility and competing traffic.
   The competing traffics includes same kind of flows (with same
   adaptation algorithms) . The investigated congestion control
   algorithms should show maximum possible network utilization and
   stability in terms of rate variations, lowest possible end to end
   frame latency, network latency and Packet Loss Rate (PLR) at
   different cell load level.

3.2.1.  Network connection

   Same as defined in Section 3.1.1

3.2.2.  Simulation Setup

   The desired simulation setup is same as Varying Network Load test
   case defined in Section 3.1 except following changes-

   1.  Radio environment : Same as defined in Section 3.1.2 except
       followings

       A.  Deployment : 3GPP case 3[Deployment]

       B.  Cell radius: 577.3333 Meters

   2.  User intensity = {0.7,.....,7.0}

   3.  Media traffic model: Same as defined in Section 3.1.2

   4.  Other traffic model: None

4.  Desired Evaluation Metrics

   RMCAT evaluation criteria document [I-D.ietf-rmcat-eval-criteria]
   defines metrics to be used to evaluate candidate algorithms.
   However, looking at the nature and distinction of cellular networks
   we recommend at minimum following metrics to be used to evaluate the
   performance of the candidate algorithms for the test cases defined in
   this document.

   The desired metrics are-

   o  Average cell throughput (for all cells), shows cell utilizations

   o  Application sending and receiving bitrate, goodput

   o  Packet Loss Rate (PLR)

   o  End to end Media frame delay

Sarker & Johansson       Expires August 18, 2014                [Page 8]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   o  Transport delay

   o  Algorithm stability in terms of rate variation

   [Editor's note: This document needs to define the metrics clearly (or
   provide references to the respective definition) and the procedure of
   measuring those.]

5.  Conclusion

   This document defines two test cases that are considered important
   for cellular networks.  Moreover, this document also provides a
   framework to define more additional test cases for cellular network.

6.  Acknowledgements

   We would like to thank Tomas Frankkila, Magnus Westerlund, Kristofer
   Kristofer Sandlund for their valuable comments while writing this
   draft.

7.  IANA Considerations

   This memo includes no request to IANA.

8.  Security Considerations

   Security issues have not been discussed in this memo.

9.  References

9.1.  Normative References

   [Deployment]
              TS 25.814, 3GPP., "Physical layer aspects for evolved
              Universal Terrestrial Radio Access (UTRA)", October 2006,
              <http://www.3gpp.org/ftp/specs/archive/25_series/25.814/
              25814-710.zip>.

   [HO-LTE-3GPP]
              TS 36.331, 3GPP., "E-UTRA- Radio Resource Control (RRC);
              Protocol specification", December 2011,
              <http://www.3gpp.org/ftp/specs/archive/36_series/36.331/
              36331-990.zip>.

   [HO-UMTS-3GPP]
              TS 25.331, 3GPP., "Radio Resource Control (RRC); Protocol
              specification", December 2011, <http://www.3gpp.org/ftp/
              specs/archive/25_series/25.331/25331-990.zip>.

Sarker & Johansson       Expires August 18, 2014                [Page 9]
Internet-Draft          RMCAT Cellular Test Cases          February 2014

   [HO-def-3GPP]
              TR 21.905, 3GPP., "Vocabulary for 3GPP Specifications",
              December 2009, <http://www.3gpp.org/ftp/specs/archive/
              21_series/21.905/21905-940.zip>.

   [I-D.ietf-rmcat-eval-criteria]
              Singh, V. and J. Ott, "Evaluating Congestion Control for
              Interactive Real-time Media", draft-ietf-rmcat-eval-
              criteria-00 (work in progress), January 2014.

   [QoS-3GPP]
              TS 23.203, 3GPP., "Policy and charging control
              architecture", June 2011, <http://www.3gpp.org/ftp/specs/
              archive/23_series/23.203/23203-990.zip>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

9.2.  Informative References

   [I-D.ietf-rmcat-cc-requirements]
              Jesup, R., "Congestion Control Requirements For RMCAT",
              draft-ietf-rmcat-cc-requirements-02 (work in progress),
              February 2014.

Authors' Addresses

   Zaheduzzaman Sarker
   Ericsson AB
   Laboratoriegraend 11
   Luleae  97753
   Sweden

   Phone: +46 107173743
   Email: zaheduzzaman.sarker@ericsson.com

   Ingemar Johansson
   Ericsson AB
   Laboratoriegraend 11
   Luleae  97753
   Sweden

   Phone: +46 10 7143042
   Email: ingemar.s.johansson@ericsson.com

Sarker & Johansson       Expires August 18, 2014               [Page 10]