Evaluating Congestion Control for Interactive Real-time Media
draft-ietf-rmcat-eval-criteria-05

2 - Two-sided: Truncated Gaussian PDV Distribution.  Four quantities
   to define: the appropriate x_min and x_max for test (e.g., +/- two
   sigma values), the standard deviation, and the mean.

   3 - One Sided: Truncated Gaussian PDV Distribution.  Quantities to
   define: three sigma value, the standard deviation, and the mean]

5.  WiFi or Cellular Links

   [I-D.ietf-rmcat-wireless-tests] describes the test cases to simulate
   networks with wireless links.  The document describes mechanism to
   simulate both cellular and WiFi networks.

6.  Traffic Models

6.1.  TCP taffic model

   Long-lived TCP flows will download data throughout the session and
   are expected to have infinite amount of data to send or receive.  For
   example, to

   Each short TCP flow is modeled as a sequence of file downloads
   interleaved with idle periods.  Not all short TCPs start at the same
   time, i.e., some start in the ON state while others start in the OFF
   state.

   The short TCP flows can be modelled as follows: 30 connections start
   simultaneously fetching small (30-50 KB) amounts of data.  This
   covers the case where the short TCP flows are not fetching a video
   file.

   The idle period between bursts of starting a group of TCP flows is
   typically derived from an exponential distribution with the mean
   value of 10 seconds.

   [These values were picked based on the data available at
   http://httparchive.org/interesting.php as of October 2015].

6.2.  RTP Video model

   [I-D.ietf-rmcat-video-traffic-model] describes two types of video
   traffic models for evaluating RMCAT candidate algorithms.  The first
   model statistically characterizes the behavior of a video encoder.
   Whereas the second model uses video traces.

   For example, test sequences are available at: [xiph-seq] and
   [HEVC-seq].

Singh, et al.          Expires September 22, 2016               [Page 9]
Internet-Draft   Evaluating Congestion Control for RMCAT      March 2016

   [Open issue: Which sequences are used?  All?  Some subset?]

6.3.  Background UDP

   [Open issue: Background UDP flow is modeled as a constant bit rate
   (CBR) flow.  It will download data at a particular CBR rate for the
   complete session, or will change to particular CBR rate at predefined
   intervals.  They parameters are still TBD.  e.g., packet size, packet
   spacing interval, etc.]

7.  Security Considerations

   Security issues have not been discussed in this memo.

8.  IANA Considerations

   There are no IANA impacts in this memo.

9.  Contributors

   The content and concepts within this document are a product of the
   discussion carried out in the Design Team.

   Michael Ramalho provided the text for the Jitter model.

10.  Acknowledgements

   Much of this document is derived from previous work on congestion
   control at the IETF.

   The authors would like to thank Harald Alvestrand, Anna Brunstrom,
   Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde,
   Stefan Holmer, Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers
   O'Hanlon, Colin Perkins, Michael Ramalho, Zaheduzzaman Sarker,
   Timothy B.  Terriberry, Michael Welzl, and Mo Zanaty for providing
   valuable feedback on earlier versions of this draft.  Additionally,
   also thank the participants of the design team for their comments and
   discussion related to the evaluation criteria.

11.  References

11.1.  Normative References

   [RFC3550]  Schulzrinne, H., Casner, S., Frederick, R., and V.
              Jacobson, "RTP: A Transport Protocol for Real-Time
              Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
              July 2003, <http://www.rfc-editor.org/info/rfc3550>.

Singh, et al.          Expires September 22, 2016              [Page 10]
Internet-Draft   Evaluating Congestion Control for RMCAT      March 2016

   [RFC3551]  Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
              Video Conferences with Minimal Control", STD 65, RFC 3551,
              DOI 10.17487/RFC3551, July 2003,
              <http://www.rfc-editor.org/info/rfc3551>.

   [RFC3611]  Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed.,
              "RTP Control Protocol Extended Reports (RTCP XR)", RFC
              3611, DOI 10.17487/RFC3611, November 2003,
              <http://www.rfc-editor.org/info/rfc3611>.

   [RFC4585]  Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
              "Extended RTP Profile for Real-time Transport Control
              Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI
              10.17487/RFC4585, July 2006,
              <http://www.rfc-editor.org/info/rfc4585>.

   [RFC5506]  Johansson, I. and M. Westerlund, "Support for Reduced-Size
              Real-Time Transport Control Protocol (RTCP): Opportunities
              and Consequences", RFC 5506, DOI 10.17487/RFC5506, April
              2009, <http://www.rfc-editor.org/info/rfc5506>.

   [I-D.ietf-rmcat-cc-requirements]
              Jesup, R. and Z. Sarker, "Congestion Control Requirements
              for Interactive Real-Time Media", draft-ietf-rmcat-cc-
              requirements-09 (work in progress), December 2014.

   [I-D.ietf-avtcore-rtp-circuit-breakers]
              Perkins, C. and V. Varun, "Multimedia Congestion Control:
              Circuit Breakers for Unicast RTP Sessions", draft-ietf-
              avtcore-rtp-circuit-breakers-14 (work in progress), March
              2016.

   [I-D.ietf-rmcat-wireless-tests]
              Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
              M. Ramalho, "Evaluation Test Cases for Interactive Real-
              Time Media over Wireless Networks", draft-ietf-rmcat-
              wireless-tests-01 (work in progress), November 2015.

11.2.  Informative References

   [RFC5033]  Floyd, S. and M. Allman, "Specifying New Congestion
              Control Algorithms", BCP 133, RFC 5033, DOI 10.17487/
              RFC5033, August 2007,
              <http://www.rfc-editor.org/info/rfc5033>.

   [RFC5166]  Floyd, S., Ed., "Metrics for the Evaluation of Congestion
              Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March
              2008, <http://www.rfc-editor.org/info/rfc5166>.

Singh, et al.          Expires September 22, 2016              [Page 11]
Internet-Draft   Evaluating Congestion Control for RMCAT      March 2016

   [RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
              Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
              <http://www.rfc-editor.org/info/rfc5681>.

   [I-D.ietf-rmcat-eval-test]
              Sarker, Z., Varun, V., Zhu, X., and M. Ramalho, "Test
              Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat-
              eval-test-03 (work in progress), March 2016.

   [I-D.ietf-rmcat-video-traffic-model]
              Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic
              Sources for RMCAT Evaluations", draft-ietf-rmcat-video-
              traffic-model-00 (work in progress), January 2016.

   [SA4-EVAL]
              R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4
              Evaluation Framework", 3GPP R1-081955, 5 2008.

   [SA4-LR]   S4-050560, 3GPP., "Error Patterns for MBMS Streaming over
              UTRAN and GERAN", 3GPP S4-050560, 5 2008.

   [TCP-eval-suite]
              Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier,
              R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a
              Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August
              2008.

   [xiph-seq]
              Xiph.org, , "Video Test Media",
              http://media.xiph.org/video/derf/ , .

   [HEVC-seq]
              HEVC, , "Test Sequences",
              http://www.netlab.tkk.fi/~varun/test_sequences/ , .

Appendix A.  Application Trade-off

   Application trade-off is yet to be defined. see RMCAT requirements
   [I-D.ietf-rmcat-cc-requirements] document.  Perhaps each experiment
   should define the application's expectation or trade-off.

A.1.  Measuring Quality

   No quality metric is defined for performance evaluation, it is
   currently an open issue.  However, there is consensus that congestion
   control algorithm should be able to show that it is useful for
   interactive video by performing analysis using a real codec and video
   sequences.

Singh, et al.          Expires September 22, 2016              [Page 12]
Internet-Draft   Evaluating Congestion Control for RMCAT      March 2016

Appendix B.  Change Log

   Note to the RFC-Editor: please remove this section prior to
   publication as an RFC.

B.1.  Changes in draft-ietf-rmcat-eval-criteria-05

   o  Improved text surrounding wireless tests, video sequences, and
      short-TCP model.

B.2.  Changes in draft-ietf-rmcat-eval-criteria-04

   o  Removed the guidelines section, as most of the sections are now
      covered: wireless tests, video model, etc.

   o  Improved Short TCP model based on the suggestion to use
      httparchive.org.

B.3.  Changes in draft-ietf-rmcat-eval-criteria-03

   o  Keep-alive version.

   o  Moved link parameters and traffic models from eval-test

B.4.  Changes in draft-ietf-rmcat-eval-criteria-02

   o  Incorporated fairness test as a working test.

   o  Updated text on mimimum evaluation requirements.

B.5.  Changes in draft-ietf-rmcat-eval-criteria-01

   o  Removed Appendix B.

   o  Removed Section on Evaluation Parameters.

B.6.  Changes in draft-ietf-rmcat-eval-criteria-00

   o  Updated references.

   o  Resubmitted as WG draft.

B.7.  Changes in draft-singh-rmcat-cc-eval-04

   o  Incorporate feedback from IETF 87, Berlin.

   o  Clarified metrics: convergence time, bandwidth utilization.

Singh, et al.          Expires September 22, 2016              [Page 13]
Internet-Draft   Evaluating Congestion Control for RMCAT      March 2016

   o  Changed fairness criteria to fairness test.

   o  Added measuring pre- and post-repair loss.

   o  Added open issue of measuring video quality to appendix.

   o  clarified use of DropTail and AQM.

   o  Updated text in "Minimum Requirements for Evaluation"

B.8.  Changes in draft-singh-rmcat-cc-eval-03

   o  Incorporate the discussion within the design team.

   o  Added a section on evaluation parameters, it describes the flow
      and network characteristics.

   o  Added Appendix with self-fairness experiment.

   o  Changed bottleneck parameters from a proposal to an example set.

   o

B.9.  Changes in draft-singh-rmcat-cc-eval-02

   o  Added scenario descriptions.

B.10.  Changes in draft-singh-rmcat-cc-eval-01

   o  Removed QoE metrics.

   o  Changed stability to steady-state.

   o  Added measuring impact against few and many flows.

   o  Added guideline for idle and data-limited periods.

   o  Added reference to TCP evaluation suite in example evaluation
      scenarios.

Authors' Addresses

Singh, et al.          Expires September 22, 2016              [Page 14]
Internet-Draft   Evaluating Congestion Control for RMCAT      March 2016

   Varun Singh
   Nemu Dialogue Systems Oy
   Runeberginkatu 4c A 4
   Helsinki  00100
   Finland

   Email: varun.singh@iki.fi
   URI:   http://www.callstats.io/

   Joerg Ott
   Technical University of Munich
   Faculty of Informatics
   Boltzmannstrasse 3
   Garching bei Muenchen, DE  85748
   Germany

   Email: ott@in.tum.de

   Stefan Holmer
   Google
   Kungsbron 2
   Stockholm  11122
   Sweden

   Email: holmer@google.com

Singh, et al.          Expires September 22, 2016              [Page 15]