Skip to main content

AQM Characterization Guidelines
draft-ietf-aqm-eval-guidelines-01

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft that was ultimately published as RFC 7928.
Authors Nicolas Kuhn , Preethi Natarajan , Naeem Khademi , David Ros
Last updated 2015-03-05
Replaces draft-kuhn-aqm-eval-guidelines
RFC stream Internet Engineering Task Force (IETF)
Formats
Reviews
Additional resources Mailing list discussion
Stream WG state WG Document
Document shepherd (None)
IESG IESG state Became RFC 7928 (Informational)
Consensus boilerplate Unknown
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-ietf-aqm-eval-guidelines-01
Internet Engineering Task Force                             N. Kuhn, Ed.
Internet-Draft                                          Telecom Bretagne
Intended status: Informational                         P. Natarajan, Ed.
Expires: September 6, 2015                                 Cisco Systems
                                                         N. Khademi, Ed.
                                                      University of Oslo
                                                                  D. Ros
                                           Simula Research Laboratory AS
                                                           March 5, 2015

                    AQM Characterization Guidelines
                   draft-ietf-aqm-eval-guidelines-01

Abstract

   Unmanaged large buffers in today's networks have given rise to a slew
   of performance issues.  These performance issues can be addressed by
   some form of Active Queue Management (AQM) mechanism, optionally in
   combination with a packet scheduling scheme such as fair queuing.
   The IETF Active Queue Management and Packet Scheduling working group
   was formed to standardize AQM schemes that are robust, easily
   implementable, and successfully deployable in today's networks.  This
   document describes various criteria for performing precautionary
   characterizations of AQM proposals.  This document also helps in
   ascertaining whether any given AQM proposal should be taken up for
   standardization by the AQM WG.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on September 6, 2015.

Kuhn, et al.            Expires September 6, 2015               [Page 1]
Internet-Draft       AQM Characterization Guidelines          March 2015

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Guidelines for AQM designers  . . . . . . . . . . . . . .   5
     1.2.  Reducing the latency and maximizing the goodput . . . . .   5
     1.3.  Glossary  . . . . . . . . . . . . . . . . . . . . . . . .   6
     1.4.  Requirements Language . . . . . . . . . . . . . . . . . .   6
   2.  End-to-end metrics  . . . . . . . . . . . . . . . . . . . . .   6
     2.1.  Flow completion time  . . . . . . . . . . . . . . . . . .   6
     2.2.  Packet loss . . . . . . . . . . . . . . . . . . . . . . .   7
     2.3.  Packet loss synchronization . . . . . . . . . . . . . . .   7
     2.4.  Goodput . . . . . . . . . . . . . . . . . . . . . . . . .   8
     2.5.  Latency and jitter  . . . . . . . . . . . . . . . . . . .   9
     2.6.  Discussion on the trade-off between latency and goodput .   9
   3.  Generic set up for evaluations  . . . . . . . . . . . . . . .  10
     3.1.  Topology and notations  . . . . . . . . . . . . . . . . .  10
     3.2.  Buffer size . . . . . . . . . . . . . . . . . . . . . . .  11
     3.3.  Congestion controls . . . . . . . . . . . . . . . . . . .  11
   4.  Various TCP variants  . . . . . . . . . . . . . . . . . . . .  12
     4.1.  TCP-friendly Sender . . . . . . . . . . . . . . . . . . .  12
     4.2.  Aggressive Transport Sender . . . . . . . . . . . . . . .  13
     4.3.  Unresponsive Transport Sender . . . . . . . . . . . . . .  13
     4.4.  TCP initial congestion window . . . . . . . . . . . . . .  14
     4.5.  Traffic Mix . . . . . . . . . . . . . . . . . . . . . . .  14
   5.  RTT fairness  . . . . . . . . . . . . . . . . . . . . . . . .  15
     5.1.  Motivation  . . . . . . . . . . . . . . . . . . . . . . .  15
     5.2.  Required tests  . . . . . . . . . . . . . . . . . . . . .  15
     5.3.  Metrics to evaluate the RTT fairness  . . . . . . . . . .  16
   6.  Burst absorption  . . . . . . . . . . . . . . . . . . . . . .  16
     6.1.  Motivation  . . . . . . . . . . . . . . . . . . . . . . .  16
     6.2.  Required tests  . . . . . . . . . . . . . . . . . . . . .  17
   7.  Stability . . . . . . . . . . . . . . . . . . . . . . . . . .  17
     7.1.  Motivation  . . . . . . . . . . . . . . . . . . . . . . .  18

Kuhn, et al.            Expires September 6, 2015               [Page 2]
Internet-Draft       AQM Characterization Guidelines          March 2015

     7.2.  Required tests  . . . . . . . . . . . . . . . . . . . . .  18
       7.2.1.  Definition of the congestion Level  . . . . . . . . .  18
       7.2.2.  Mild Congestion . . . . . . . . . . . . . . . . . . .  19
       7.2.3.  Medium Congestion . . . . . . . . . . . . . . . . . .  19
       7.2.4.  Heavy Congestion  . . . . . . . . . . . . . . . . . .  19
       7.2.5.  Varying congestion levels . . . . . . . . . . . . . .  19
       7.2.6.  Varying Available Bandwidth . . . . . . . . . . . . .  19
     7.3.  Parameter sensitivity and stability analysis  . . . . . .  20
   8.  Implementation cost . . . . . . . . . . . . . . . . . . . . .  21
     8.1.  Motivation  . . . . . . . . . . . . . . . . . . . . . . .  21
     8.2.  Required discussion . . . . . . . . . . . . . . . . . . .  21
   9.  Operator control knobs and auto-tuning  . . . . . . . . . . .  21
   10. Interaction with ECN  . . . . . . . . . . . . . . . . . . . .  22
     10.1.  Motivation . . . . . . . . . . . . . . . . . . . . . . .  22
     10.2.  Required discussion  . . . . . . . . . . . . . . . . . .  22
   11. Interaction with scheduling . . . . . . . . . . . . . . . . .  22
     11.1.  Motivation . . . . . . . . . . . . . . . . . . . . . . .  22
     11.2.  Required discussion  . . . . . . . . . . . . . . . . . .  23
   12. Discussion on methodology, metrics, AQM comparisons and
       packet sizes  . . . . . . . . . . . . . . . . . . . . . . . .  23
     12.1.  Methodology  . . . . . . . . . . . . . . . . . . . . . .  23
     12.2.  Comments on metrics measurement  . . . . . . . . . . . .  23
     12.3.  Comparing AQM schemes  . . . . . . . . . . . . . . . . .  23
       12.3.1.  Performance comparison . . . . . . . . . . . . . . .  24
       12.3.2.  Deployment comparison  . . . . . . . . . . . . . . .  25
     12.4.  Packet sizes and congestion notification . . . . . . . .  25
   13. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  25
   14. Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  25
   15. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  25
   16. Security Considerations . . . . . . . . . . . . . . . . . . .  25
   17. References  . . . . . . . . . . . . . . . . . . . . . . . . .  26
     17.1.  Normative References . . . . . . . . . . . . . . . . . .  26
     17.2.  Informative References . . . . . . . . . . . . . . . . .  26
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  27

1.  Introduction

   Active Queue Management (AQM) addresses the concerns arising from
   using unnecessarily large and unmanaged buffers, in order to improve
   network and application performance.  Several AQM algorithms have
   been proposed in the past years, most notably Random Early Detection
   (RED), BLUE, and Proportional Integral controller (PI), and more
   recently CoDel [CODEL] and PIE [PIE].  In general, these algorithms
   actively interact with the Transmission Control Protocol (TCP) and
   any other transport protocol that deploys a congestion control scheme
   to manage the amount of data they keep in the network.  The available
   buffer space in the routers and switches should be large enough to
   accommodate the short-term buffering requirements.  AQM schemes aim

Kuhn, et al.            Expires September 6, 2015               [Page 3]
Internet-Draft       AQM Characterization Guidelines          March 2015

   at reducing mean buffer occupancy, and therefore both end-to-end
   delay and jitter.  Some of these algorithms, notably RED, have also
   been widely implemented in some network devices.  However, the
   potential benefits of the RED scheme have not been realized since RED
   is reported to be usually turned off.  The main reason of this
   reluctance to use RED in today's deployments comes from its
   sensitivity to the operating conditions in the network and the
   difficulty of tuning its parameters.

   A buffer is a physical volume of memory in which a queue or set of
   queues are stored.  In real implementations of switches, a global
   memory is shared between the available devices: the size of the
   buffer for a given communication does not make sense, as its
   dedicated memory may vary over the time and real-world buffering
   architectures are complex.  For the sake of simplicity, when speaking
   of a specific queue in this document, "buffer size" refers to the
   maximum amount of data the buffer may store, which can be measured in
   bytes or packets.  The rest of this memo therefore refers to the
   maximum queue depth as the size of the buffer for a given
   communication.

   In order to meet mostly throughput-based Service-Level Agreement
   (SLA) requirements and to avoid packet drops, many home gateway
   manufacturers resort to increasing the available memory beyond
   "reasonable values".  This increase is also referred to as
   Bufferbloat [BB2011].  Deploying large unmanaged buffers on the
   Internet has lead to the increase in end-to-end delay, resulting in
   poor performance for latency-sensitive applications such as real-time
   multimedia (e.g., voice, video, gaming, etc).  The degree to which
   this affects modern networking equipment, especially consumer-grade
   equipment's, produces problems even with commonly used web services.
   Active queue management is thus essential to control queuing delay
   and decrease network latency.

   The Active Queue Management and Packet Scheduling Working Group (AQM
   WG) was recently formed within the TSV area to address the problems
   with large unmanaged buffers in the Internet.  Specifically, the AQM
   WG is tasked with standardizing AQM schemes that not only address
   concerns with such buffers, but also are robust under a wide variety
   of operating conditions.  In order to ascertain whether the WG should
   undertake standardizing an AQM proposal, the WG requires guidelines
   for assessing AQM proposals.  This document provides the necessary
   characterization guidelines.

Kuhn, et al.            Expires September 6, 2015               [Page 4]
Internet-Draft       AQM Characterization Guidelines          March 2015

1.1.  Guidelines for AQM designers

   One of the key objectives behind formulating the guidelines is to
   help ascertain whether a specific AQM is not only better than drop-
   tail but also safe to deploy.  The guidelines help to quantify AQM
   schemes' performance in terms of latency reduction, goodput
   maximization and the trade-off between these two.  The guidelines
   also help to discuss AQM's safe deployment, including self-
   adaptation, stability analysis, fairness, design and implementation
   complexity and robustness to different operating conditions.

   This memo details generic characterization scenarios that any AQM
   proposal MUST be evaluated against.  Irrespective of whether or not
   an AQM is standardized by the WG, we RECOMMEND the relevant scenarios
   and metrics discussed in this document to be considered.  This
   document presents central aspects of an AQM algorithm that MUST be
   considered whatever the context is such as, burst absorption
   capacity, RTT fairness or resilience to fluctuating network
   conditions.  These guidelines do not cover every possible aspect of a
   particular algorithm.  In addition, it is worth noting that the
   proposed criteria are not bound to a particular evaluation toolset.
   These guidelines do not present context dependent scenarios (such as
   802.11 WLANs, data-centers or rural broadband networks).

   This document details how an AQM designer can rate the feasibility of
   their proposal in different types of network devices (switches,
   routers, firewalls, hosts, drivers, etc) where an AQM may be
   implemented.

1.2.  Reducing the latency and maximizing the goodput

   The trade-off between reducing the latency and maximizing the goodput
   is intrinsically linked to each AQM scheme and is key to evaluating
   its performance.  This trade-off MUST be considered in various
   scenarios to ensure the safety of an AQM deployment.  Whenever
   possible, solutions should aim at both maximizing goodput and
   minimizing latency.  This document proposes guidelines that enable
   the reader to quantify (1) reduction of latency, (2) maximization of
   goodput and (3) the trade-off between the two.

   Testers SHOULD discuss in a reference document the performance of
   their proposal in terms of performance and deployment in regards with
   those of drop-tail: basically, these guidelines provide the tools to
   understand the deployment costs versus the potential gain in
   performance due to the introduction of the proposed scheme.

Kuhn, et al.            Expires September 6, 2015               [Page 5]
Internet-Draft       AQM Characterization Guidelines          March 2015

1.3.  Glossary

   o  AQM: there may be a debate on whether a scheduling scheme is
      additional to an AQM mechanism or is a part of an AQM scheme.  The
      rest of this memo refers to AQM as a dropping/marking policy that
      does not feature a scheduling scheme.

   o  buffer: a physical volume of memory in which a queue or set of
      queues are stored.

   o  buffer size: the maximum amount of data that may be stored in a
      buffer, measured in bytes or packets.

1.4.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.  End-to-end metrics

   End-to-end delay is the result of propagation delay, serialization
   delay, service delay in a switch, medium-access delay and queuing
   delay, summed over the network elements along the path.  AQM
   algorithms may reduce the queuing delay by providing signals to the
   sender on the emergence of congestion, but any impact on the goodput
   must be carefully considered.  This section presents the metrics that
   MAY be used to better quantify (1) the reduction of latency, (2)
   maximization of goodput and (3) the trade-off between these two.
   These metrics MAY be considered to better assess the performance of
   an AQM scheme.

   The metrics listed in this section are not necessarily suited to
   every type of traffic detailed in the rest of this document.  It is
   therefore NOT REQUIRED to measure all of the following metrics in
   every scenario discussed in this document necessarily, if the chosen
   metric is not relevant to the context of the evaluation scenario
   (e.g. latency vs. gooodput trade-off in application-limited traffic
   scenarios).  The tester SHOULD however measure and report on all the
   metrics relevant to the context of the evaluation scenario.

2.1.  Flow completion time

   The flow completion time is an important performance metric for the
   end-user when the flow size is finite.  Considering the fact that an
   AQM scheme may drop/mark packets, the flow completion time is
   directly linked to the dropping/marking policy of the AQM scheme.
   This metric helps to better assess the performance of an AQM

Kuhn, et al.            Expires September 6, 2015               [Page 6]
Internet-Draft       AQM Characterization Guidelines          March 2015

   depending on the flow size.  The Flow Completion Time (FCT) is
   related to the flow size (Fs) and the Goodput for the flow (G) as
   follows:

   FCT [s] = Fs [B] / ( G [Mbps] / 8 )

2.2.  Packet loss

   Packet losses, that may occur in a queue, impact on the end-to-end
   performance at the receiver's side.

   The tester MUST evaluate, at the receiver:

   o  the packet loss probability: this metric should also be frequently
      measured during the experiment, since the long-term loss
      probability is only of interest for steady-state scenarios.

   o  the interval between consecutive losses: the time between two
      losses should be measured.

   The packet loss probability can be assessed by simply evaluating the
   loss ratio as a function of the number of lost packets and the total
   number of packets sent.  This might not be easily done in laboratory
   testing, for which these guidelines advice the tester:

   o  to check that for every packet, a corresponding packet was
      received within a reasonable time, as explained in [RFC2679].

   o  to keep a count of all packets sent, and a count of the non-
      dupplicate packets received, as explained in the section 10 of
      [RFC2544].

   The interval between consecutive losses, which is also called a gap,
   is a metric of interest for VoIP traffic and, as a result, has been
   further specified in [RFC3611].

2.3.  Packet loss synchronization

   One goal of an AQM algorithm should be to help with avoiding global
   synchronization of flows sharing the bottleneck buffer on which the
   AQM operates ([RFC2309]).  It is therefore important to assess the
   "degree" of packet-loss synchronization between flows, with and
   without the AQM under consideration.

   As discussed e.g. in [LOSS-SYNCH-MET-08], loss synchronization among
   flows may be quantified by several slightly different metrics that
   capture different aspects of the same issue.  However, in real-world
   measurements the choice of metric may be imposed by practical

Kuhn, et al.            Expires September 6, 2015               [Page 7]
Internet-Draft       AQM Characterization Guidelines          March 2015

   considerations -- e.g. whether fine-grained information on packet
   losses in the bottleneck available or not.  For the purpose of AQM
   characterization, a good candidate metric is the global
   synchronization ratio, measuring the proportion of flows losing
   packets during a loss event.  [YU06] used this metric in real-world
   experiments to characterize synchronization along arbitrary Internet
   paths; the full methodology is described in [YU06].

   If an AQM scheme is evaluated using real-life network environments,
   it is worth pointing out that some network events, such as failed
   link restoration may cause synchronized losses between active flows
   and thus confuse the meaning of this metric.

2.4.  Goodput

   The goodput has been defined in the section 3.17 of [RFC2647] as the
   number of bits per unit of time forwarded to the correct destination
   interface of the Device Under Test (DUT) or the System Under Test
   (SUT), minus any bits lost or retransmitted.  This definition induces
   that the test setup needs to be qualified to assure that it is not
   generating losses on its own.

   Measuring the end-to-end goodput enables an appreciation of how well
   the AQM improves transport and application performance.  The measured
   end-to-end goodput is linked to the AQM scheme's dropping/marking
   policy -- e.g. the smaller the number of packet drops, the fewer
   packets need retransmission, minimizing AQM's impact on transport and
   application performance.  Additionally, an AQM scheme may resort to
   Explicit Congestion Notification (ECN) marking as an initial means to
   control delay.  Again, marking packets instead of dropping them
   reduces the number of packet retransmissions and increases goodput.
   End-to-end goodput values help to evaluate the AQM scheme's
   effectiveness in minimizing packet drops that impact application
   performance and to estimate how well the AQM scheme works with ECN.

   The measurement of the goodput let the tester evaluate to which
   extent the AQM is able to maintain a high link utilization.  This
   metric should be also obtained frequently during the experiment as
   the long-term goodput is relevant for steady-state scenarios only and
   may not necessarily reflect how the introduction of an AQM actually
   impacts the link utilization during at a certain period of time.  It
   is worth pointing out that the fluctuations in the values obtained
   from these measurements may depend on other factors than the
   introduction of an AQM, such as link layer losses due to external
   noise or corruption, fluctuating bandwidths (802.11 WLANs), heavy
   congestion levels or transport layer's rate reduction by congestion
   control mechanism.

Kuhn, et al.            Expires September 6, 2015               [Page 8]
Internet-Draft       AQM Characterization Guidelines          March 2015

2.5.  Latency and jitter

   The latency, or the one-way delay metric, is discussed in [RFC2679].
   There is a consensus on a adequate metric for the jitter, that
   represents the one-way delay variations for packets from the same
   flow: the Packet Delay Variation (PDV), detailed in [RFC5481], serves
   well all use cases.

   The end-to-end latency differs from the queuing delay: it is linked
   to the network topology and the path characteristics.  Moreover, the
   jitter strongly depends on the traffic pattern and the topology as
   well.  The introduction of an AQM scheme would impact on these
   metrics and therefore they SHOULD be considered in the end-to-end
   evaluation of performance.

   The guidelines advice that the tester SHOULD measure the minimum,
   average and maximum as well as the coefficient of variation of the
   average values for these metrics.

2.6.  Discussion on the trade-off between latency and goodput

   The metrics presented in this section MAY be considered, in order to
   discuss and quantify the trade-off between latency and goodput.

   This trade-off can also be illustrated with figures following the
   recommendations of the section 5 of [TCPEVAL2013].  Each of the end-
   to-end delay and the goodput should be measured frequently for every
   fixed time interval.

   With regards to the goodput, and in addition to the long-term
   stationary goodput value, it is RECOMMENDED to take measurements
   every multiple of RTTs.  We suggest a minimum value of 10 x RTT (to
   smooth out the fluctuations) but higher values are encouraged
   whenever appropriate for the presentation depending on the network's
   path characteristics.  The measurement period MUST be disclosed for
   each experiment and when results/values are compared across different
   AQM schemes, the comparisons SHOULD use exactly the same measurement
   periods.

   With regards to latency, it is highly RECOMMENDED to take the samples
   on per-packet basis whenever possible depending on the features
   provided by hardware/software and the impact of sampling itself on
   the hardware performance.  It is generally RECOMMENDED to provide at
   least 10 samples per RTT.

   From each of these sets of measurements, the 10th and 90th
   percentiles and the median value should be computed.  For each
   scenario, a graph can be generated, with the x-axis showing the end-

Kuhn, et al.            Expires September 6, 2015               [Page 9]
Internet-Draft       AQM Characterization Guidelines          March 2015

   to-end delay and the y-axis the goodput.  This graph provides part of
   a better understanding of (1) the delay/goodput trade-off for a given
   congestion control mechanism, and (2) how the goodput and average
   queue size vary as a function of the traffic load.

3.  Generic set up for evaluations

   This section presents the topology that can be used for each of the
   following scenarios, the corresponding notations and discusses
   various assumptions that have been made in the document.

3.1.  Topology and notations

       +---------+                                        +-----------+
       |senders A|                                        |receivers B|
       +---------+                                        +-----------+

       +--------------+                                +--------------+
       |traffic class1|                                |traffic class1|
       |--------------|                                |--------------|
       | SEN.Flow1.1 +---------+            +-----------+ REC.Flow1.1 |
       |        +     |        |            |          |        +     |
       |        |     |        |            |          |        |     |
       |        +     |        |            |          |        +     |
       | SEN.Flow1.X +-----+   |            |  +--------+ REC.Flow1.X |
       +--------------+    |   |            |  |       +--------------+
            +            +-+---+---+     +--+--+---+            +
            |            |Router L |     |Router R |            |
            |            |---------|     |---------|            |
            |            | AQM     |     |         |            |
            |            | BuffSize|     |         |            |
            |            | (Bsize) +-----+         |            |
            |            +-----+--++     ++-+------+            |
            +                  |  |       | |                   +
       +--------------+        |  |       | |          +--------------+
       |traffic classN|        |  |       | |          |traffic classN|
       |--------------|        |  |       | |          |--------------|
       | SEN.FlowN.1 +---------+  |       | +-----------+ REC.FlowN.1 |
       |        +     |           |       |            |        +     |
       |        |     |           |       |            |        |     |
       |        +     |           |       |            |        +     |
       | SEN.FlowN.Y +------------+       +-------------+ REC.FlowN.Y |
       +--------------+                                +--------------+

                     Figure 1: Topology and notations

Kuhn, et al.            Expires September 6, 2015              [Page 10]
Internet-Draft       AQM Characterization Guidelines          March 2015

   Figure 1 is a generic topology where:

   o  various classes of traffic can be introduced;

   o  the timing of each flow (i.e., when does each flow start and stop)
      may be different;

   o  each class of traffic can comprise various number of flows;

   o  each link is characterized by a couple (RTT,capacity);

   o  Flows are generated between A and B, sharing a bottleneck (Routers
      L and R);

   o  The bottleneck link SHOULD be asymmetric in terms of bandwidth:
      the capacity from senders to receivers is higher than the one from
      receivers to senders;

   o  The traffic SHOULD be bi-directional between A and B (downlink and
      uplink).  The Tester MAY additionally evaluate uni-directional
      traffic scenarios as well (downlink-only or uplink-only).

   This topology may not perfectly reflect actual topologies, however,
   this simple topology is commonly used in the world of simulations and
   small testbeds.  This topology can be considered as adequate to
   evaluate AQM proposals, similarly to the topology proposed in
   [TCPEVAL2013].  The tester should carefully choose the topology that
   is going to be used to evaluate the AQM scheme.

3.2.  Buffer size

   The size of the buffers should be carefully chosen, and MAY be set to
   the bandwidth-delay product.  However, if the context or the
   application requires a specific buffer size, the tester MUST justify
   and detail the way the maximum queue size is set.  Indeed, the
   maximum size of the buffer may affect the AQM's performance and its
   choice SHOULD be elaborated for a fair comparison between AQM
   proposals.  While comparing AQM schemes the buffer size SHOULD remain
   the same across the tests.

3.3.  Congestion controls

   This memo features three kind of congestion controls:

   o  Standard TCP congestion control: the base-line congestion control
      is TCP NewReno with SACK, as explained in [RFC5681].

Kuhn, et al.            Expires September 6, 2015              [Page 11]
Internet-Draft       AQM Characterization Guidelines          March 2015

   o  Aggressive congestion controls: a base-line congestion control for
      this category is TCP Cubic.

   o  Less-than Best Effort (LBE) congestion controls: an LBE congestion
      control 'results in smaller bandwidth and/or delay impact on
      standard TCP than standard TCP itself, when sharing a bottleneck
      with it.'  [RFC6297]

   Recent transport layer protocols are not mentioned in the following
   sections, for the sake of simplicity.

4.  Various TCP variants

   Network and end-devices need to be configured with a reasonable
   amount of maximum available buffer space in order to absorb transient
   bursts.  In some situations, network providers tend to configure
   devices with large buffers in order to avoid packet drops triggered
   by a full buffer and to maximize the link utilization for standard
   loss-based TCP traffic.  Loss-based TCP congestion controls
   (including standard NewReno TCP) fill up these unmanaged buffers
   until the TCP sender receives a signal (packet drop) to decrease the
   sending rate.  The larger the buffer is, the higher the buffer
   occupancy, and therefore the queuing delay.  On the other hand, an
   efficient AQM scheme SHOULD convey early congestion signals to TCP
   senders so that the average queuing delay is brought under control.

   Not all applications run over the same flavor of TCP or even
   necessarily use TCP.  Variety of applications generate different
   classes of traffic which may not react to congestion signals (a.k.a
   unresponsive flows) or may not decrease their sending rate as
   expected (a.k.a aggressive flows); AQM schemes aim at maintaining the
   queuing delay under control, which is challenged if aggressive or
   unresponsive traffics are present.

   This section provides guidelines to assess the performance of an AQM
   proposal for various traffic profiles -- different types of senders
   (with different TCP congestion control variants, unresponsive,
   aggressive), traffic mix with different applications, etc.

4.1.  TCP-friendly Sender

   This scenario helps to evaluate how an AQM scheme reacts to a TCP-
   friendly transport sender.  A single long-lived, non application-
   limited, TCP NewReno flow transfers data between sender A and
   receiver B.  Other TCP friendly congestion control schemes such as
   TCP-friendly rate control [RFC5348] etc MAY also be considered.

Kuhn, et al.            Expires September 6, 2015              [Page 12]
Internet-Draft       AQM Characterization Guidelines          March 2015

   For each TCP-friendly transport considered, the graph described in
   Section 2.6 could be generated.

4.2.  Aggressive Transport Sender

   This scenario helps to evaluate how an AQM scheme reacts to a
   transport sender that is more aggressive than a single TCP-friendly
   sender.  We define 'aggressiveness' as a higher increase factor than
   standard upon a successful transmission and/or a lower than standard
   decrease factor upon a unsuccessful transmission (e.g. in case of
   congestion controls with Additive-Increase Multiplicative-Decrease
   (AIMD) principle, a larger AI and/or MD factors).  A single long-
   lived, non application-limited, TCP Cubic flow transfers data between
   sender A and receiver B.  Other aggressive congestion control schemes
   MAY also be considered.

   For each flavor of aggressive transports, the graph described in
   Section 2.6 could be generated.

4.3.  Unresponsive Transport Sender

   This scenario helps to evaluate how an AQM scheme reacts to a
   transport sender that is not responsive to congestion signals (ECN
   marks and/or packet drops) from the AQM scheme.  Note that faulty
   transport implementations on end-hosts and/or faulty network elements
   on the path that modify congestion signals in packet headers (e.g.
   modifying the ECN-related bitsets) [I-D.ietf-aqm-recommendation] may
   also lead to a similar situation, such that the AQM scheme needs to
   adapt to the unresponsive traffic.  To this end, these guidelines
   propose the two following scenarios.

   The first scenario aims at creating a test environment that results
   in constant queue build up; we consider unresponsive flow(s) with an
   overall sending rate that is greater than the bottleneck's link
   capacity between routers L and R.  This scenario consists of a long-
   lived non application-limited UDP flow that transfers data between
   sender A and receiver B.  Graphs described in Section 2.6 could be
   generated.

   The second scenario aims to test to which extent the AQM scheme is
   able to keep the responsive fraction of overall traffic load under
   control, this scenario considers a mixture of TCP-friendly and
   unresponsive traffics.  It consists of a long-lived non application-
   limited UDP flow and a single long-lived, non application-limited,
   TCP NewReno flow that transfer data between sender A and receiver B.
   As opposed to the first scenario, the rate of the UDP traffic should
   be less than or equal to half of the bottleneck capacity.  For each

Kuhn, et al.            Expires September 6, 2015              [Page 13]
Internet-Draft       AQM Characterization Guidelines          March 2015

   type of traffic, the graph described in Section 2.6 could be
   generated.

4.4.  TCP initial congestion window

   This scenario helps to evaluate how an AQM scheme adapts to a traffic
   mix consisting of TCP flows with different values of the Initial
   congestion Window (IW).

   For this scenario, we consider two types of flows that MUST be
   generated between sender A and receiver B:

   o  A single long-lived non application-limited TCP NewReno flow;

   o  A single long-lived application-limited TCP NewReno flow, with an
      IW set to 3 or 10 packets.  The size of the data transferred MUST
      be strictly higher than 10 packets and should be lower than 100
      packets.

   The transmission of the non application-limited flow MUST start
   before the transmission of the application-limited flow and only
   after the steady state has been reached by non application-limited
   flow.

   For each of these scenarios, the graph described in Section 2.6 could
   be generated for each class of traffic (application-limited and non
   application-limited).  The completion time of the application-limited
   TCP flow could be measured.

4.5.  Traffic Mix

   This scenario helps to evaluate how an AQM scheme reacts to a traffic
   mix consisting of different applications such as:

   o  Bulk TCP transfer

   o  Web traffic

   o  VoIP

   o  Constant Bit Rate (CBR) UDP traffic

   o  Adaptive video streaming

   Various trafic mixes can be considered.  These guidelines RECOMMEND
   to examine at least the following example: 1 bi-directionnal VoIP; 6
   Webs; 1 CBR; 1 Adaptive Video; 5 bulk TCP.  Any other combinations
   could be considered and should be carefully documented.

Kuhn, et al.            Expires September 6, 2015              [Page 14]
Internet-Draft       AQM Characterization Guidelines          March 2015

   For each scenario, the graph described in Section 2.6 could be
   generated for each class of traffic.  In addition, other metrics such
   as end-to-end latency, jitter and flow completion time MUST be
   reported.

5.  RTT fairness

5.1.  Motivation

   The capability of AQM schemes to control the queuing delay highly
   depends on the way end-to-end transport protocols react to congestion
   signals.  When network path's intrinsic RTT varies, the behaviour of
   congestion control is impacted and so the capability of AQM schemes
   to control the queueing level.  It is therefore important to assess
   the AQM schemes against a set of intrinsic RTTs common in the
   Internet transfers (e.g. from 5 ms to 500 ms).

   Also, asymmetry in terms of difference in intrinsic RTT between
   various paths sharing the same bottleneck SHOULD be considered and
   the fairness between the flows SHOULD be discussed since in this
   scenario, a flow traversing on shorter RTT path may react faster to
   congestion and recover faster from it compared to another flow on a
   longer RTT path.  The introduction of AQM schemes may potentially
   improve this type of fairness.

   Moreover, introducing an AQM scheme may cause the unfairness between
   the flows, even if the RTTs are identical.  This potential unfairness
   SHOULD be investigated as well.

5.2.  Required tests

   The topology that SHOULD be used is presented in Figure 1:

   o  To evaluate the inter-RTT fairness, for each run, two flows
      divided into two categories.  Category I which RTT between sender
      A and Router L SHOULD be 100ms.  Category II which RTT between
      sender A and Router L SHOULD be in [5ms;560ms].  The maximum value
      for the RTT represents the RTT of a satellite link that, according
      to the section 2 of [RFC2488] should be at least 558ms.

   o  To evaluate the impact of the RTT value on the AQM performance and
      the intra-protocol fairness (the fairness for the flows using the
      same paths/congestion control), for each run, two flows (Flow1 and
      Flow2) SHOULD be introduced.  For each experiment, the set of RTT
      SHOULD be the same for the two flows and in [5ms;560ms].

   These flows MUST use the same congestion control algorithm.

Kuhn, et al.            Expires September 6, 2015              [Page 15]
Internet-Draft       AQM Characterization Guidelines          March 2015

5.3.  Metrics to evaluate the RTT fairness

   The output that MUST be measured is:

   o  for the inter-RTT fairness: (1) the cumulative average goodput of
      the flow from Category I, goodput_Cat_I (Section 2.4); (2) the
      cumulative average goodput of the flow from Category II,
      goodput_Cat_II (Section 2.4); (3) the ratio goodput_Cat_II/
      goodput_Cat_I; (4) the average packet drop rate for each category
      (Section 2.2).

   o  for the intra-protocol RTT fairness: (1) the cumulative average
      goodput of the two flows (Section 2.4); (2) the average packet
      drop rate for the two flows (Section 2.2).

6.  Burst absorption

6.1.  Motivation

   Packet arrivals can be bursty due to various reasons.  A packet burst
   can push the AQM schemes to drop/mark packets momentarily even though
   the average queue length may still be below the AQM's target queuing
   thresholds.  Dropping/marking one or more packets within a burst may
   result in performance penalties for the corresponding flows since the
   dropped/marked packets cause unnecessary rate reduction by congestion
   control as well as retransmission in case of drop only.  Performance
   penalties may turn into unmet SLAs and become disincentives for the
   AQM adoption.  Therefore, an AQM scheme SHOULD be designed to
   accommodate transient bursts.  AQM schemes do not present the same
   tolerance to packet bursts arriving at the buffer, therefore this
   tolerance MUST be quantified.

   Note that accommodating bursts translates to higher queue length and
   queuing delay.  Naturally, it is important that the AQM scheme brings
   bursty traffic under control quickly.  On the other hand, spiking
   packet drops in order to bring packet bursts quickly under control
   could result in multiple drops per flow and severely impact transport
   and application performance.  Therefore, an AQM scheme SHOULD bring
   bursts under control by balancing both aspects -- (1) queuing delay
   spikes are minimized and (2) performance penalties for ongoing flows
   in terms of packet drops are minimized.

   An AQM scheme maintains short average queues to allow the remaining
   space in the queue for temporary bursts of packets.  The tolerance to
   packet bursts depends on the number of packets in the queue, which is
   directly linked to the AQM algorithm.  Moreover, one AQM scheme may
   implement a feature controlling the maximum size of accepted bursts,
   that may depend on the buffer occupancy or the currently estimated

Kuhn, et al.            Expires September 6, 2015              [Page 16]
Internet-Draft       AQM Characterization Guidelines          March 2015

   queuing delay.  Also, the impact of the buffer size on such feature
   (a.k.a burst allowance) MAY be evaluated.

6.2.  Required tests

   For this scenario, the following traffic MUST be generated from
   sender A to receiver B:

   o  Web traffic with IW10: Web transfer of 100 packets with initial
      congestion window set to 10;

   o  Bursty video frames;

   o  Constant bit rate UDP traffic.

   o  A single bulk TCP flow as background traffic.

   Figure 2 presents the various cases for the traffic that MUST be
   generated between sender A and receiver B.

                   +-------------------------------------------------+
                   |Case| Traffic Type                               |
                   |    +-----+------------+----+--------------------+
                   |    |Video|Webs (IW 10)| CBR| Bulk TCP Traffic   |
                   +----|-----|------------|----|--------------------|
                   |I   |  0  |     1      |  1 |         0          |
                   +----|-----|------------|----|--------------------|
                   |II  |  0  |     1      |  1 |         1          |
                   |----|-----|------------|----|--------------------|
                   |III |  1  |     1      |  1 |         0          |
                   +----|-----|------------|----|--------------------|
                   |IV  |  1  |     1      |  1 |         1          |
                   +----+-----+------------+----+--------------------+

                    Figure 2: Bursty traffic scenarios

   For each of these scenarios, the graph described in Section 2.6 could
   be generated.  In addition, other metrics such as end-to-end latency,
   jitter, flow completion time MUST be generated.  For the cases of
   frame generation of bursty video traffic as well as the choice of web
   traffic pattern, we leave these details and their presentation to the
   testers.

7.  Stability

Kuhn, et al.            Expires September 6, 2015              [Page 17]
Internet-Draft       AQM Characterization Guidelines          March 2015

7.1.  Motivation

   Network devices experience varying operating conditions depending on
   factors such as time of the day, deployment scenario, etc.  For
   example:

   o  Traffic and congestion levels are higher during peak hours than
      off-peak hours.

   o  In the presence of scheduler, a queue's draining rate may vary
      depending on other queues: a low load on a high priority queue
      implies higher draining rate for lower priority queues.

   o  The available capacity on the physical layer may vary over time
      such as in the context of lossy channels.

   Whether the target context is a not stable environment, the
   capability of an AQM scheme to maintain its control over the queuing
   delay and buffer occupancy is challenged.  This document proposes
   guidelines to assess the behaviour of AQM schemes under varying
   congestion levels and varying draining rates.

7.2.  Required tests

   Note that the traffic profiles explained below comprises non
   application-limited TCP flows.  For each of the below scenarios, the
   results described in Section 2.6 SHOULD be generated.  For
   Section 7.2.5 and Section 7.2.6 they SHOULD incorporate the results
   in per-phase basis as well.

   Wherever the notion of time has explicitly mentioned in this
   subsection, time 0 starts from the moment all TCP flows have already
   reached their congestion avoidance phase.

7.2.1.  Definition of the congestion Level

   In these guidelines, the congestion levels are represented by the
   projected packet drop rate, had a drop-tail queue was chosen instead
   of an AQM scheme.  When the bottleneck is shared among non-
   application-limited TCP flows. l_r, the loss rate projection can be
   expressed as a function of N, the number of bulk TCP flows, and S,
   the sum of capacity and maximum buffer size based on Eq. 3 of
   [SCL-TCP]:

   l_r = 0.76 * N^2 / S^2

   N = S * sqrt(1/0.76) * sqrt (l_r)

Kuhn, et al.            Expires September 6, 2015              [Page 18]
Internet-Draft       AQM Characterization Guidelines          March 2015

7.2.2.  Mild Congestion

   This scenario helps to evaluate how an AQM scheme reacts to a light
   load of incoming traffic resulting in mild congestion -- packet drop
   rates around 1%. The number of bulk flows required to achieve this
   congestion level, N_mild, is then:

   N_mild = round(0.114*S)

7.2.3.  Medium Congestion

   This scenario helps to evaluate how an AQM scheme reacts to incoming
   traffic resulting in medium congestion -- packet drop rates around
   5%. The number of bulk flows required to achieve this congestion
   level, N_med, is then:

   N_med = round (0.256*S)

7.2.4.  Heavy Congestion

   This scenario helps to evaluate how an AQM scheme reacts to incoming
   traffic resulting in heavy congestion -- packet drop rates around
   10%. The number of bulk flows required to achieve this congestion
   level, N_heavy, is then:

   N_heavy = round (0.363*S)

7.2.5.  Varying congestion levels

   This scenario helps to evaluate how an AQM scheme reacts to incoming
   traffic resulting in various level of congestions during the
   experiment.  In this scenario, the congestion level varies within a
   large time-scale.  The following phases may be considered: phase I -
   mild congestion during 0-20s; phase II - medium congestion during
   20-40s; phase III - heavy congestion during 40-60s; phase I again,
   and so on.

7.2.6.  Varying Available Bandwidth

   This scenario helps to evaluate how an AQM scheme adapts to varying
   available bandwidth on the outgoing link.

   To emulate varying draining rates, the bottleneck bandwidth between
   nodes 'Router L' and 'Router R' varies over the course of the
   experiment as follows:

   o  Experiment 1: the capacity varies between two values within a
      large time-scale.  As an example, the following phases may be

Kuhn, et al.            Expires September 6, 2015              [Page 19]
Internet-Draft       AQM Characterization Guidelines          March 2015

      considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps
      during 20-40s; phase I again, and so on.

   o  Experiment 2: the capacity varies between two values within a
      short time-scale.  As an example, the following phases may be
      considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps
      during 100-200ms; phase I again, and so on.

   The tester MAY choose a phase time-interval value different than what
   is stated above, if the network's path conditions (such as bandwidth-
   delay product) necessitate.  In this case the choice of such time-
   interval value SHOULD be stated and elaborated.

   The tester MAY additionally evaluate the two mentioned scenarios
   (short-term and long-term capacity variations), during and/or
   including TCP slow-start phase.

   More realistic fluctuating bandwidth patterns MAY be considered.  The
   tester MAY choose to incorporate realistic scenarios with regards to
   common fluctuation of bandwidth in state-of-the-art technologies.

   The scenario consists of TCP NewReno flows between sender A and
   receiver B.  In order to better assess the impact of draining rates
   on the AQM behavior, the tester MUST compare its performance with
   those of drop-tail.

7.3.  Parameter sensitivity and stability analysis

   An AQM scheme's control law is the primary means by which the queuing
   delay is controlled.  Hence understanding the control law is critical
   to understanding the AQM scheme's behavior.  The control law may
   include several input parameters whose values affect the AQM scheme's
   output behavior and its stability.  Additionally, AQM schemes may
   auto-tune parameter values in order to maintain stability under
   different network conditions (such as different congestion levels,
   draining rates or network environments).  The stability of these
   auto-tuning techniques is also important to understand.

   AQM proposals SHOULD provide background material showing control
   theoretic analysis of the control law and the input parameter space
   within which the control law operates as expected; or could use other
   ways to discuss its stability.  For parameters that are auto-tuned,
   the material SHOULD include stability analysis of the auto-tuning
   mechanism(s) as well.  Such analysis helps to understand an AQM
   scheme's control law better and the network conditions/deployments
   under which the AQM scheme is performing stably.

Kuhn, et al.            Expires September 6, 2015              [Page 20]
Internet-Draft       AQM Characterization Guidelines          March 2015

8.  Implementation cost

8.1.  Motivation

   An AQM scheme's successful deployment is directly related to its cost
   of implementation.  Network devices may need hardware or software
   implementations of the AQM mechanism.  Depending on a device's
   capabilities and limitations, the device may or may not be able to
   implement some or all parts of the AQM logic.

   AQM proposals SHOULD provide pseudo-code for the complete AQM scheme,
   highlighting generic implementation-specific aspects of the scheme
   such as "drop-tail" vs. "drop-head", inputs (e.g. current queuing
   delay, queue length), computations involved, need for timers, etc.
   This helps to identify costs associated with implementing the AQM
   scheme on a particular hardware or software device.  Also, it helps
   the WG to understand which kind of devices can easily support the AQM
   and which cannot.

8.2.  Required discussion

   AQM proposals SHOULD highlight parts of AQM logic that are device
   dependent and discuss if and how AQM behavior could be impacted by
   the device.  For example, a queueing-delay based AQM scheme requires
   current queuing delay as input from the device.  If the device
   already maintains this value, then it is trivial to implement the AQM
   logic on the device.  On the other hand, if the device provides
   indirect means to estimate the queuing delay (for example:
   timestamps, dequeing rate etc), then the AQM behavior is sensitive to
   how accurate enough the queuing delay estimations are on that device.
   Highlighting the AQM scheme's sensitivity to queuing delay
   estimations helps implementers to identify optimal means of
   implementing the mechanism on the device.

9.  Operator control knobs and auto-tuning

   One of the biggest hurdles of RED deployment was/is its parameter
   sensitivity to operating conditions -- how difficult it is to tune
   RED parameters for a deployment in order to get maximum benefit from
   the RED implementation.  Fluctuating congestion levels and network
   conditions add to the complexity.  Incorrect parameter values lead to
   poor performance.  This is one reason why RED is reported to be
   usually turned off by the network operators.

   Any AQM scheme is likely to have parameters whose values affect the
   AQM's control law and behavior.  Exposing all these parameters as
   control knobs to a network operator (or user) can easily result in a
   unsafe AQM deployment.  Unexpected AQM behavior ensues when parameter

Kuhn, et al.            Expires September 6, 2015              [Page 21]
Internet-Draft       AQM Characterization Guidelines          March 2015

   values are set improperly.  A minimal number of control knobs
   minimizes the number of ways a possibly naive user can break a system
   where an AQM scheme is deployed at.  Fewer control knobs make the AQM
   scheme more user-friendly and easier to deploy and debug.

   We highly recommend that an AQM scheme SHOULD minimize the number of
   control knobs exposed for the operator's tuning.  An AQM scheme
   SHOULD expose only those knobs that control the macroscopic AQM
   behavior such as queue delay threshold or queue length threshold and
   so on.

   Additionally, an AQM scheme's safety is directly related to its
   stability under varying operating conditions such as varying traffic
   profiles and fluctuating network conditions, as described in
   Section 7.  Operating conditions vary often and hence it is necessary
   that the AQM scheme MUST remain stable under these conditions without
   the need for additional external tuning.  If AQM parameters require
   tuning under these conditions, then the AQM MUST self-adapt necessary
   parameter values by employing auto-tuning techniques.

10.  Interaction with ECN

10.1.  Motivation

   Apart from packet drops, Explicit Congestion Notification (ECN) is an
   alternative mean to signal the data senders about network congestion.
   The AQM recommendation document [I-D.ietf-aqm-recommendation]
   describes some of the benefits of using ECN coupled with an AQM
   mechanism.

10.2.  Required discussion

   An AQM scheme SHOULD support ECN and the testers MUST discuss and
   describe the support of ECN.

11.  Interaction with scheduling

11.1.  Motivation

   Coupled with an AQM scheme, a router may schedule the transmission of
   packets in a specific manner by introducing a scheduling scheme.
   This algorithm may create sub-queues and integrate a dropping policy
   on each of these sub-queues.  Another scheduling policy may modify
   the way packets are sequenced, modifying the timestamp of each
   packet.

Kuhn, et al.            Expires September 6, 2015              [Page 22]
Internet-Draft       AQM Characterization Guidelines          March 2015

11.2.  Required discussion

   The scheduling and the AQM conjointly impact on the end-to-end
   performance.  During the characterization process of a dropping
   policy, the tester MAY discuss the feasibility to add scheduling to
   its algorithm.  This discussion as an instance, MAY explain whether
   the dropping policy is applied when packets are being enqueued or
   dequeued.

12.  Discussion on methodology, metrics, AQM comparisons and packet
     sizes

12.1.  Methodology

   A sufficiently detailed description of the test setup MUST be
   provided which facilitates other testers to replicate the tests if
   required.  The test setup MAY include software and hardware
   specifications and versions.  The tester is encouraged to make the
   detailed test setup and the results publicly available.

   The proposals SHOULD be experimented on real-life systems, or they
   MAY be evaluated with event-driven simulations (such as ns-2, ns-3,
   OMNET, etc).  The proposed scenarios are not bound to a particular
   evaluation toolset.

12.2.  Comments on metrics measurement

   In this document, we presented the end-to-end metrics that SHOULD be
   used to evaluate the trade-off between latency and goodput in
   Section 2.  In addition to the end-to-end metrics, the queue-level
   metrics (normally collected at the device operating the AQM) provide
   a better understanding of the AQM behavior under study and the impact
   of its internal parameters.  Whenever it is possible (e.g. depending
   on the features provided by the hardware/software), these guidelines
   RECOMMEND to collect queue-level metrics, such as link utilization,
   queuing delay, queue size or packet drop/mark statistics in addition
   to the AQM-specific parameters.  However, the evaluation MUST be
   primarily based on externally observed end-to-end metrics.

   These guidelines do not aim to detail on the way these metrics can be
   measured, since they highly depend on the evaluation toolset and/or
   hadrware.

12.3.  Comparing AQM schemes

   This memo recognizes that the guidelines mentioned above may be used
   for comparing AQM schemes.  It recommends that AQM schemes MUST be
   compared against both performance and deployment categories.  In

Kuhn, et al.            Expires September 6, 2015              [Page 23]
Internet-Draft       AQM Characterization Guidelines          March 2015

   addition, this section details how best to achieve a fair comparison
   of AQM schemes by avoiding certain pitfalls.

12.3.1.  Performance comparison

   AQM schemes MUST be compared against all the generic scenarios
   presented in this memo.  AQM schemes MAY be compared for specific
   network environments such as data centers, home networks, etc.  If an
   AQM scheme's parameter(s) were externally tuned for optimization or
   other purposes, these values MUST be disclosed.

   Note that AQM schemes belong to different varieties such as queue-
   length based schemes such as RED or queueing-delay based scheme such
   as CoDel and PIE.  Also, AQM schemes expose different control knobs
   associated with different semantics.  For example, while both PIE and
   CoDel are queueing-delay based schemes and each expose a knob to
   control the queueing delay -- PIE's "queueing delay reference" vs.
   CoDel's "queueing delay target", the two schemes' knobs have
   different semantics resulting in different control points.  Such
   differences in AQM schemes SHOULD not be overlooked while making
   comparisons.

   This document recommends the following procedures for a fair
   performance comparison between the AQM schemes:

   1.  Comparable control parameters and comparable input values:
       carefully identify the set of parameters that control similar
       behavior between the two AQM schemes and ensure these parameters
       have comparable input values.  For example, while comparing how
       well a queue-length based AQM scheme controls queueing delay vs.
       a queueing-delay based AQM scheme, identify the two schemes'
       parameters that control queueing delay and ensure that their
       input values are comparable.  Similarly, to compare two AQM
       schemes on how well they accommodate packet bursts, identify
       burst-related control parameters and ensure they are configured
       with similar values.

   2.  Compare over a range of input configurations: there could be
       situations when the set of control parameters that affect a
       specific behavior have different semantics between the two AQM
       schemes.  As mentioned above, PIE's knob to control queueing
       delay has different semantics from CoDel's.  In such situations,
       these schemes MUST be compared over a range of input
       configurations.  For example, compare PIE vs. CoDel over the
       range of target delay input configurations.

Kuhn, et al.            Expires September 6, 2015              [Page 24]
Internet-Draft       AQM Characterization Guidelines          March 2015

12.3.2.  Deployment comparison

   AQM schemes MUST be compared against deployment criteria such as the
   parameter sensitivity (Section 7.3), auto-tuning (Section 9) or
   implementation cost (Section 8).

12.4.  Packet sizes and congestion notification

   An AQM scheme may be considering packet sizes while generating
   congestion signals.  [RFC7141] discusses the motivations behind this.
   For example, control packets such as DNS requests/responses, TCP
   SYNs/ACKs are small, but their loss can severely impact the
   application performance.  An AQM scheme may therefore be biased
   towards small packets by dropping them with smaller probability
   compared to larger packets.  However, such an AQM scheme is unfair to
   data senders generating larger packets.  Data senders, malicious or
   otherwise, are motivated to take advantage of such AQM scheme by
   transmitting smaller packets, and could result in unsafe deployments
   and unhealthy transport and/or application designs.

   An AQM scheme SHOULD adhere to the recommendations outlined in
   [RFC7141], and SHOULD NOT provide disproportionate advantage to flows
   with smaller packets.

13.  Acknowledgements

   This work has been partially supported by the European Community
   under its Seventh Framework Programme through the Reducing Internet
   Transport Latency (RITE) project (ICT-317700).

14.  Contributors

   Many thanks to S.  Akhtar, A.B.  Bagayoko, F.  Baker, D.  Collier-
   Brown, G.  Fairhurst, T.  Hoiland-Jorgensen, C.  Kulatunga, W.
   Lautenschlager, A.C.  Morton, R.  Pan, D.  Taht and M.  Welzl for
   detailed and wise feedback on this document.

15.  IANA Considerations

   This memo includes no request to IANA.

16.  Security Considerations

   This document, by itself, presents no new privacy nor security
   issues.

Kuhn, et al.            Expires September 6, 2015              [Page 25]
Internet-Draft       AQM Characterization Guidelines          March 2015

17.  References

17.1.  Normative References

   [I-D.ietf-aqm-recommendation]
              Baker, F. and G. Fairhurst, "IETF Recommendations
              Regarding Active Queue Management", draft-ietf-aqm-
              recommendation-11 (work in progress), February 2015.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", RFC 2119, 1997.

   [RFC7141]  Briscoe, B. and J. Manner, "Byte and Packet Congestion
              Notification", RFC 7141, 2014.

17.2.  Informative References

   [BB2011]   "BufferBloat: what's wrong with the internet?", ACM Queue
              vol. 9, 2011.

   [CODEL]    Nichols, K. and V. Jacobson, "Controlling Queue Delay",
              ACM Queue , 2012.

   [LOSS-SYNCH-MET-08]
              Hassayoun, S. and D. Ros, "Loss Synchronization and Router
              Buffer Sizing with High-Speed Versions of TCP", IEEE
              INFOCOM Workshops , 2008.

   [PIE]      Pan, R., Natarajan, P., Piglione, C., Prabhu, MS.,
              Subramanian, V., Baker, F., and B. VerSteeg, "PIE: A
              lightweight control scheme to address the bufferbloat
              problem", IEEE HPSR , 2013.

   [RFC2309]  Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering,
              S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G.,
              Partridge, C., Peterson, L., Ramakrishnan, K., Shenker,
              S., Wroclawski, J., and L. Zhang, "Recommendations on
              Queue Management and Congestion Avoidance in the
              Internet", RFC 2309, April 1998.

   [RFC2488]  Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP
              Over Satellite Channels using Standard Mechanisms", BCP
              28, RFC 2488, January 1999.

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

Kuhn, et al.            Expires September 6, 2015              [Page 26]
Internet-Draft       AQM Characterization Guidelines          March 2015

   [RFC2647]  Newman, D., "Benchmarking Terminology for Firewall
              Performance", RFC 2647, August 1999.

   [RFC2679]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Delay Metric for IPPM", RFC 2679, September 1999.

   [RFC3611]  Friedman, T., Caceres, R., and A. Clark, "RTP Control
              Protocol Extended Reports (RTCP XR)", RFC 3611, November
              2003.

   [RFC5348]  Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
              Friendly Rate Control (TFRC): Protocol Specification", RFC
              5348, September 2008.

   [RFC5481]  Morton, A. and B. Claise, "Packet Delay Variation
              Applicability Statement", RFC 5481, March 2009.

   [RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
              Control", RFC 5681, September 2009.

   [RFC6297]  Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort
              Transport Protocols", RFC 6297, June 2011.

   [SCL-TCP]  Morris, R., "Scalable TCP congestion control", IEEE
              INFOCOM , 2000.

   [TCPEVAL2013]
              Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP
              Evaluation Suite", IRTF ICCRG , 2013.

   [YU06]     Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis
              of loss synchronisation between concurrent TCP flows",
              Australian Telecommunication Networks and Application
              Conference (ATNAC) , 2006.

Authors' Addresses

   Nicolas Kuhn (editor)
   Telecom Bretagne
   2 rue de la Chataigneraie
   Cesson-Sevigne  35510
   France

   Phone: +33 2 99 12 70 46
   Email: nicolas.kuhn@telecom-bretagne.eu

Kuhn, et al.            Expires September 6, 2015              [Page 27]
Internet-Draft       AQM Characterization Guidelines          March 2015

   Preethi Natarajan (editor)
   Cisco Systems
   510 McCarthy Blvd
   Milpitas, California
   United States

   Email: prenatar@cisco.com

   Naeem Khademi (editor)
   University of Oslo
   Department of Informatics, PO Box 1080 Blindern
   N-0316 Oslo
   Norway

   Phone: +47 2285 24 93
   Email: naeemk@ifi.uio.no

   David Ros
   Simula Research Laboratory AS
   P.O. Box 134
   Lysaker, 1325
   Norway

   Phone: +33 299 25 21 21
   Email: dros@simula.no

Kuhn, et al.            Expires September 6, 2015              [Page 28]