Skip to main content

Characterization Guidelines for Active Queue Management (AQM)
draft-ietf-aqm-eval-guidelines-13

Revision differences

Document history

Date Rev. By Action
2016-07-19
13 (System) RFC Editor state changed to AUTH48-DONE from AUTH48
2016-06-30
13 (System) RFC Editor state changed to AUTH48 from RFC-EDITOR
2016-06-23
13 (System) RFC Editor state changed to RFC-EDITOR from EDIT
2016-06-14
13 (System) RFC Editor state changed to EDIT
2016-06-14
13 (System) IESG state changed to RFC Ed Queue from Approved-announcement sent
2016-06-14
13 (System) Announcement was received by RFC Editor
2016-06-14
13 (System) IANA Action state changed to No IC from In Progress
2016-06-14
13 (System) IANA Action state changed to In Progress
2016-06-14
13 Cindy Morgan IESG state changed to Approved-announcement sent from IESG Evaluation::AD Followup
2016-06-14
13 Cindy Morgan IESG has approved the document
2016-06-14
13 Cindy Morgan Closed "Approve" ballot
2016-06-14
13 Cindy Morgan Ballot approval text was generated
2016-06-14
13 Cindy Morgan Ballot writeup was changed
2016-06-14
13 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-13.txt
2016-06-13
12 Benoît Claise [Ballot comment]
- Random Early Detection (RED), BLUE, and Proportional Integral controller (PI)
Would you have references?
2016-06-13
12 Benoît Claise [Ballot Position Update] Position for Benoit Claise has been changed to No Objection from Discuss
2016-06-10
12 (System) Sub state has been changed to AD Followup from Revised ID Needed
2016-06-10
12 Nicolas Kuhn IANA Review state changed to Version Changed - Review Needed from IANA OK - No Actions Needed
2016-06-10
12 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-12.txt
2016-06-09
11 Ralph Droms Request for Telechat review by GENART Completed: Ready. Reviewer: Ralph Droms.
2016-05-19
11 Amy Vezza IESG state changed to IESG Evaluation::Revised I-D Needed from IESG Evaluation
2016-05-19
11 Joel Jaeggli
[Ballot comment]
On 5/19/16 6:50 AM, João Taveira wrote:
> Hi,
>
> Overall the document looks good, if somewhat strenuous to implement
> fully …
[Ballot comment]
On 5/19/16 6:50 AM, João Taveira wrote:
> Hi,
>
> Overall the document looks good, if somewhat strenuous to implement
> fully in practice. Nitpicks inline.
>
> # Section 2.4
>
> s/in the bottleneck available/at the bottleneck are available
>
> # Section 2.5
>
> s/allows the tester evaluate/allows the tester to evaluate
> s/This metric should be also/This metric should also be
>
> # Section 2.7
>
> The initial paragraph of this section is confusing since it asks the
> reader to consider the metrics "as explained in the rest of this
> document", which obviously the reader will have not been exposed to.
>
> Some of the implementation details were ambiguous to me. For example:
> "It is suggested to take measurements at least every K x minRTT (to
> smooth out the fluctuations), with K=10. Higher values for K are
> encouraged whenever it is more appropriate for the presentation of the
> results.". There seems to be some contradiction between _suggesting_
> K=10, and then _encouraging_ a higher K, which would reduce the sampling
> frequency in practice. Finally the paragraph suggests "It is generally
> RECOMMENDED to provide at least 10 samples per RTT.", which would
> suggest K = 1/10. I believe my misinterpretation stems from the fact a
> measurement period is introduced (K x minRTT), but all subsequent
> context refers to a measurement frequency.
>
> At the end of the section, section references could be enclosed in
> parentheses since they are not part of the text: i.e. "This graph
> provides part of a better understanding of (1) the delay/goodput
> trade-off for a given congestion control mechanism (Section 5) ..."
>
> # Section 3.1
>
> It is not obvious how the asymmetrical topology characterizes the AQM
> system in any useful manner. If the objective is to see how a particular
> AQM algorithm performs under TCP ACKs, it would be more useful to study
> how the drop rate varies for streams of small packets. My main concern
> here is that testing reverse path performance will be tightly coupled
> with transport protocol behaviour, since you are reliant on a feedback loop.
>
> If you do decide that an asymmetrical test case is useful, it would be
> helpful to provide more guidance on how asymmetrical a topology should
> be. For TCP, is setting the capacity for the reverse path to 50% of the
> forwarding path any different from the symmetrical case in practice?
>
> # Section 3.3
>
> There is no baseline congestion control provided for LBE. Given LEDBAT
> is mentioned in section 5.4, it would probably be wise to just define it
> as a baseline in order to minimize implementation cost.
>
> # Section 4.4
>
> s/impact the application performance/impact application performance
> s/smaller probability/lower probability
>
> I'm somewhat surprised the topic of packet sizes is not covered more.
> This section highlights that attackers can gain undue advantage by using
> smaller packets, but if an AQM scheme does not take packet size into
> account, the converse could be true. In datacenter environments in
> particular, jumbo frames are relatively common, and so it would be
> valuable to understand whether an AQM scheme is biased towards a
> particular end of the size spectrum.
>
> # Section 5.4
>
> s/any of those listed/such as any of those listed
>
> # Section 6.1
>
> The first sentence of the second paragraph spans 6 lines, making it hard
> to read. I'd suggest breaking it at "... can be discussed. In this
> scenario ..."
>
> # Section 7.2
>
> It is unclear what the intent of describing what a web page download
> "could" be is. The document suggests that certain combinations of
> patterns MUST be tested, but then leaves most of the details up to the
> implementer, which makes comparisons between results trickier.
>
> This section would like benefit from being more agnostic to what
> applications it is trying to mimic, and instead describe traffic mixes
> in more general terms. As it stands it is unclear whether the intent of
> the section is to approximate existing application patterns (which will
> be out of date by the time the document is published), or underlying
> traffic characteristics (bursty, inelastic, parallel, etc).
>
> # Section 13
>
> This table would have been helpful sooner in the document as a summary
> of implementation requirements. Independently of where it ends up
> "summary" may be a better header than "conclusion", since it can be
> picked up by a hasty implementer as a reference to the rest of the document.
>
> One nitpick in this table is the "Operator Control" row, which suggests
> "Tuning SHOULD NOT be required.". This is more implementation guidance
> than characterization guideline, and should therefore be out of scope
> for this document (and indeed the recommendation is already part of
> RFC7567). In terms of characterization it would be more appropriate for
> this document to suggest that any tunable parameters MUST be explicitly
> declared and their impact discussed, in the form of a YANG definition or
> otherwise.
>
> Cheers,
2016-05-19
11 Joel Jaeggli [Ballot Position Update] New position, No Objection, has been recorded for Joel Jaeggli
2016-05-19
11 Benoît Claise
[Ballot discuss]
Has a RFC6390 performance directorate review done for the 2.X metrics? It should.
See http://www.ietf.org/iesg/directorate/performance-metrics.html
I guess that the metrics will be recorded …
[Ballot discuss]
Has a RFC6390 performance directorate review done for the 2.X metrics? It should.
See http://www.ietf.org/iesg/directorate/performance-metrics.html
I guess that the metrics will be recorded in the future (See draft-ietf-ippm-metric-registry-06
), right?
For example, Flow Completion Time and Packet Loss Synchronization are new, I believe.
And some other metrics are already documented in RFC6390 compliant documents. Pointers should be provided.
See https://tools.ietf.org/html/draft-ietf-xrblock-independent-burst-gap-discard-01#appendix-A for an example
2016-05-19
11 Benoît Claise
[Ballot comment]
- Random Early Detection (RED), BLUE, and Proportional Integral controller (PI)
Would you have references?

- BDP is mentioned a few times. Please …
[Ballot comment]
- Random Early Detection (RED), BLUE, and Proportional Integral controller (PI)
Would you have references?

- BDP is mentioned a few times. Please expand.

- Glossary section = terminology section, right? If we want to be consistent across documents

- section 12.2. Why not a MUST below?
  In order to understand an AQM's deployment considerations and
  performance under a specific environment, AQM proposals SHOULD
  describe the parameters that control the macroscopic AQM behavior,
  and identify any parameters that require tuning to operational
  conditions.
2016-05-19
11 Benoît Claise [Ballot Position Update] New position, Discuss, has been recorded for Benoit Claise
2016-05-18
11 Jari Arkko
[Ballot comment]
Several issues from the Gen-ART review by Ralph Droms need to be addressed, in my opinion.

In particular, Figure 1 or text under …
[Ballot comment]
Several issues from the Gen-ART review by Ralph Droms need to be addressed, in my opinion.

In particular, Figure 1 or text under needs clarification, the advise on ECN seems superfluous, and section 10 example needs to be labeled as an example.
2016-05-18
11 Jari Arkko [Ballot Position Update] New position, No Objection, has been recorded for Jari Arkko
2016-05-18
11 Alia Atlas [Ballot Position Update] New position, No Objection, has been recorded for Alia Atlas
2016-05-18
11 Suresh Krishnan [Ballot Position Update] New position, No Objection, has been recorded for Suresh Krishnan
2016-05-18
11 Ben Campbell [Ballot Position Update] New position, No Objection, has been recorded for Ben Campbell
2016-05-18
11 Kathleen Moriarty [Ballot Position Update] New position, No Objection, has been recorded for Kathleen Moriarty
2016-05-18
11 Spencer Dawkins
[Ballot comment]
I'm pretty sure I know what "steady state" means in this text

  The transmission of the non application-limited flow must start
  …
[Ballot comment]
I'm pretty sure I know what "steady state" means in this text

  The transmission of the non application-limited flow must start
  before the transmission of the application-limited flow and only
  after the steady state has been reached by non application-limited
  flow.
 
but I'm not sure how someone using this specification knows what it means, and it's asking the user to do something specific during evaluation. Is there a reference or definition you could provide?

(There are other uses of the phrase "steady state" in the document, and they would also benefit, but this is the use that needs the precision)
2016-05-18
11 Spencer Dawkins [Ballot Position Update] New position, Yes, has been recorded for Spencer Dawkins
2016-05-18
11 Alissa Cooper
[Ballot comment]
(1) In the abstract, I'm not sure "precautionary" is really the right adjective.

(2) Sec. 4.2 says: "However, the evaluation MUST be primarily …
[Ballot comment]
(1) In the abstract, I'm not sure "precautionary" is really the right adjective.

(2) Sec. 4.2 says: "However, the evaluation MUST be primarily based on externally observed end-to-end metrics." This seems like a bit of a weird use of normative language. It makes sense to me that you would normatively recommend how to setup an evaluation environment to be able to compare AQMs, but if people want to weigh their evaluations in the wrong the direction, there is not much to be done about it. It seems like this would be more useful if it focused on explaining why primarily focusing on e2e measurements is preferable.

(3) Overall I would recommend a pass through the whole document to check for the consistency with which normative language is used -- it seems like in some cases it isn't really necessary but it's there anyway, or its use is uneven (e.g., 4.3.2, 5.2, 6.1, 6.2, 6.3, 7.2 to name a few). There also seems to be a mix of normative statements about what proposals should say and how evaluation environments should be setup. It would help if each time a normative statement was made it clearly distinguished who the recommendation is directed towards. This became especially confusing in Sec. 13, where it is unclear whether "Scenario MUST be considered" means that an AQM spec needs to discuss this, or an AQM evaluation environment needs to test for this.

(4) Since this document provides guidelines for how to characterize AQM proposals, it seems inappropriate for it to be making normative recommendations about how AQM proposals should work, even if they are just re-statements of things from RFC 7567. In particular, these statements seem out of place:

- Sec. 4.4: "An AQM scheme SHOULD adhere to the recommendations outlined in
  [RFC7141], and SHOULD NOT provide undue advantage to flows with
  smaller packets [RFC7567].
 
    - Sec. 4.5: "Deployed AQM algorithms SHOULD implement Explicit Congestion
  Notification (ECN) as well as loss to signal congestion to endpoints
  [RFC7567]."
 
  - Sec. 13: "Tuning SHOULD NOT be required"

(5) Sec. 4.6 says: "This discussion as an instance, MAY explain whether the dropping policy is applied when packets are being enqueued or dequeued." I don't understand what "this discussion as an instance" means.

(6) In Sec. 7.2, is there some justification that could be provided for choosing the particular parameters listed to simulate web traffic? E.g., are these parameters considered typical or is there a reference to some standard way of simulating web traffic that could be referenced?

(7) Also in Sec. 7.2, why is the detail for the web flow given but the detail for bursty video is not? Surely there is more than one way to generate or simulate bursty video -- does it not matter what the bursts look like or how they are timed for this measurement purpose?

(8) In 9.1, I would suggest that bi-directional video is prevalent and important enough to be listed in the traffic mix analysis as well.

(9) I think the statement in Sec. 17 assumes that the tests envisioned by this document will be based on active measurements of simulated traffic. If that is a real scope limitation, and evaluation of AQMs based on passive measurement of traffic is out of scope, I think that should be stated somewhere in the document.

(10) Just curious: has anyone thought about using LMAP, or the LMAP information model at least, to configure the kinds of tests envisioned in this document?
2016-05-18
11 Alissa Cooper [Ballot Position Update] New position, No Objection, has been recorded for Alissa Cooper
2016-05-18
11 Alvaro Retana [Ballot comment]
Too bad there are no implementations... :-(
2016-05-18
11 Alvaro Retana [Ballot Position Update] New position, No Objection, has been recorded for Alvaro Retana
2016-05-18
11 Stephen Farrell [Ballot Position Update] New position, No Objection, has been recorded for Stephen Farrell
2016-05-17
11 Terry Manderson [Ballot Position Update] New position, No Objection, has been recorded for Terry Manderson
2016-05-16
11 Deborah Brungard [Ballot Position Update] New position, No Objection, has been recorded for Deborah Brungard
2016-05-13
11 Mirja Kühlewind Notification list changed to wes@mti-systems.com
2016-05-13
11 Mirja Kühlewind IESG state changed to IESG Evaluation from Waiting for Writeup
2016-05-13
11 Mirja Kühlewind Ballot has been issued
2016-05-13
11 Mirja Kühlewind [Ballot Position Update] New position, Yes, has been recorded for Mirja Kühlewind
2016-05-13
11 Mirja Kühlewind Created "Approve" ballot
2016-05-13
11 Mirja Kühlewind Ballot writeup was changed
2016-05-13
11 Mirja Kühlewind Changed consensus to Yes from Unknown
2016-05-13
11 Peter Yee Request for Telechat review by GENART is assigned to Ralph Droms
2016-05-13
11 Peter Yee Request for Telechat review by GENART is assigned to Ralph Droms
2016-05-09
11 Gunter Van de Velde Request for Last Call review by OPSDIR Completed: Has Issues. Reviewer: Linda Dunbar.
2016-05-04
11 (System) IESG state changed to Waiting for Writeup from In Last Call
2016-04-28
11 Ralph Droms Request for Last Call review by GENART Completed: On the Right Track. Reviewer: Ralph Droms.
2016-04-28
11 Tero Kivinen Request for Last Call review by SECDIR Completed: Has Nits. Reviewer: Tero Kivinen.
2016-04-27
11 (System) IANA Review state changed to IANA OK - No Actions Needed from IANA - Review Needed
2016-04-27
11 Sabrina Tanamal
(Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-aqm-eval-guidelines-11.txt, which is currently in Last Call, and has the following comments:

We understand that this …
(Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-aqm-eval-guidelines-11.txt, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any IANA actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, IANA does not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
IANA Specialist
ICANN
2016-04-23
11 Gunter Van de Velde Request for Last Call review by OPSDIR is assigned to Linda Dunbar
2016-04-23
11 Gunter Van de Velde Request for Last Call review by OPSDIR is assigned to Linda Dunbar
2016-04-22
11 Mirja Kühlewind Placed on agenda for telechat - 2016-05-19
2016-04-21
11 Jean Mahoney Request for Last Call review by GENART is assigned to Ralph Droms
2016-04-21
11 Jean Mahoney Request for Last Call review by GENART is assigned to Ralph Droms
2016-04-21
11 Tero Kivinen Request for Last Call review by SECDIR is assigned to Tero Kivinen
2016-04-21
11 Tero Kivinen Request for Last Call review by SECDIR is assigned to Tero Kivinen
2016-04-21
11 Mirja Kühlewind
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up.

Changes are expected over time. This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up.

Changes are expected over time. This version is dated 24 February 2012.

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?


Informational; this is appropriate.


(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  Relevant content can frequently be found in the abstract
  and/or introduction of the document. If not, this may be
  an indication that there are deficiencies in the abstract
  or introduction.


Abstract:

  Unmanaged large buffers in today's networks have given rise to a slew
  of performance issues.  These performance issues can be addressed by
  some form of Active Queue Management (AQM) mechanism, optionally in
  combination with a packet scheduling scheme such as fair queuing.
  This document describes various criteria for performing precautionary
  characterizations of AQM schemes.


Working Group Summary

  Was there anything in WG process that is worth noting? For
  example, was there controversy about particular points or
  were there decisions where the consensus was particularly
  rough?


Over it's lifetime, this document changed scope a couple of times.  It currently contains guidance for characterizing AQM algorithm performance, though it had started out to prescribe evaluation methods for AQM algorithms being considered in the working group.  As time went on, the focus shifted, and the WG did not use the guidelines in a prescriptive way.  The current guidelines are not fully parameterized, and it is assumed that someone performing a characterization would consider how to parameterize configurations relevant to their own networks or scenarios of interest.

Ideally there would be running code (as mentioned by Dave, among others) for performing the characterizations, e.g. implemented using some tool such as ns-3, mininet, etc, however, this has not come about yet.

Where it sits now, the document is suitable as Informational guidance, but it is not a prescriptive and fully defined spec, like for example, an IPPM metric would be.  It simply collects guidance and description of the set of things to look at and the measurements of interest in common AQM performance testing situations.



Document Quality

  Are there existing implementations of the protocol? Have a
  significant number of vendors indicated their plan to
  implement the specification? Are there any reviewers that
  merit special mention as having done a thorough review,
  e.g., one that resulted in important changes or a
  conclusion that the document had no substantive issues? If
  there was a MIB Doctor, Media Type or other expert review,
  what was its course (briefly)? In the case of a Media Type
  review, on what date was the request posted?


The document is of sufficient quality that someone (e.g. a graduate student) could read it and implement proper tests.


Personnel

  Who is the Document Shepherd? Who is the Responsible Area
  Director?


Wesley Eddy (wes@mti-systems.com) is the shepherd, and Mirja Kühlewind (ietf@kuehlewind.net) is the AD.


(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.


I have reviewed it multiple times; it is ready for publication.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?


No concerns.


(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.


N/A


(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.


During last call in the WG there were concerns raised about the test descriptions not being fully specified, but the document at that point was inconsistent about its goals, and it seems that fixing that remedied the issue.  There was not any threat of appeal or other action though, and after announcing the intention to continue with the revised document following WGLC, there was no protest.


(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.


Yes.


(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.


No.


(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 


Reasonable consensus.


(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)


No.


(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.


There are a couple of warnings about obsolete references; this can be fixed by the RFC Editor, but does not impact the tests described in the document.


(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.


N/A.


(13) Have all references within this document been identified as
either normative or informative?


Yes.


(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?


No.


(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.


No.


(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.


N/A.



(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).


There are no IANA considerations.


(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.


N/A


(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.


N/A
2016-04-20
11 Amy Vezza IANA Review state changed to IANA - Review Needed
2016-04-20
11 Amy Vezza
The following Last Call announcement was sent out:

From: The IESG
To: "IETF-Announce"
CC: wes@mti-systems.com, ietf@kuehlewind.net, aqm-chairs@ietf.org, draft-ietf-aqm-eval-guidelines@ietf.org, aqm@ietf.org
Reply-To: ietf@ietf.org …
The following Last Call announcement was sent out:

From: The IESG
To: "IETF-Announce"
CC: wes@mti-systems.com, ietf@kuehlewind.net, aqm-chairs@ietf.org, draft-ietf-aqm-eval-guidelines@ietf.org, aqm@ietf.org
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (AQM Characterization Guidelines) to Informational RFC


The IESG has received a request from the Active Queue Management and
Packet Scheduling WG (aqm) to consider the following document:
- 'AQM Characterization Guidelines'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2016-05-04. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  Unmanaged large buffers in today's networks have given rise to a slew
  of performance issues.  These performance issues can be addressed by
  some form of Active Queue Management (AQM) mechanism, optionally in
  combination with a packet scheduling scheme such as fair queuing.
  This document describes various criteria for performing precautionary
  characterizations of AQM schemes.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ballot/


No IPR declarations have been submitted directly on this I-D.


2016-04-20
11 Amy Vezza IESG state changed to In Last Call from Last Call Requested
2016-04-20
11 Mirja Kühlewind Last call was requested
2016-04-20
11 Mirja Kühlewind Ballot approval text was generated
2016-04-20
11 Mirja Kühlewind IESG state changed to Last Call Requested from AD Evaluation
2016-04-20
11 Mirja Kühlewind Last call announcement was changed
2016-04-18
11 Mirja Kühlewind Ballot writeup was generated
2016-04-18
11 Mirja Kühlewind Last call announcement was generated
2016-04-06
11 Cindy Morgan Shepherding AD changed to Mirja Kühlewind
2016-03-21
11 Martin Stiemerling IESG state changed to AD Evaluation from Publication Requested
2016-03-14
11 Wesley Eddy
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up.

Changes are expected over time. This version is dated …
As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up.

Changes are expected over time. This version is dated 24 February 2012.

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?


Informational; this is appropriate.


(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  Relevant content can frequently be found in the abstract
  and/or introduction of the document. If not, this may be
  an indication that there are deficiencies in the abstract
  or introduction.


Abstract:

  Unmanaged large buffers in today's networks have given rise to a slew
  of performance issues.  These performance issues can be addressed by
  some form of Active Queue Management (AQM) mechanism, optionally in
  combination with a packet scheduling scheme such as fair queuing.
  This document describes various criteria for performing precautionary
  characterizations of AQM schemes.


Working Group Summary

  Was there anything in WG process that is worth noting? For
  example, was there controversy about particular points or
  were there decisions where the consensus was particularly
  rough?


Over it's lifetime, this document changed scope a couple of times.  It currently contains guidance for characterizing AQM algorithm performance, though it had started out to prescribe evaluation methods for AQM algorithms being considered in the working group.  As time went on, the focus shifted, and the WG did not use the guidelines in a prescriptive way.  The current guidelines are not fully parameterized, and it is assumed that someone performing a characterization would consider how to parameterize configurations relevant to their own networks or scenarios of interest.

Ideally there would be running code (as mentioned by Dave, among others) for performing the characterizations, e.g. implemented using some tool such as ns-3, mininet, etc, however, this has not come about yet.

Where it sits now, the document is suitable as Informational guidance, but it is not a prescriptive and fully defined spec, like for example, an IPPM metric would be.  It simply collects guidance and description of the set of things to look at and the measurements of interest in common AQM performance testing situations.



Document Quality

  Are there existing implementations of the protocol? Have a
  significant number of vendors indicated their plan to
  implement the specification? Are there any reviewers that
  merit special mention as having done a thorough review,
  e.g., one that resulted in important changes or a
  conclusion that the document had no substantive issues? If
  there was a MIB Doctor, Media Type or other expert review,
  what was its course (briefly)? In the case of a Media Type
  review, on what date was the request posted?


The document is of sufficient quality that someone (e.g. a graduate student) could read it and implement proper tests.


Personnel

  Who is the Document Shepherd? Who is the Responsible Area
  Director?


Wesley Eddy (wes@mti-systems.com) is the shepherd, and Martin Stiemerling (mls.ietf@gmail.com) is the AD.


(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.


I have reviewed it multiple times; it is ready for publication.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?


No concerns.


(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.


N/A


(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.


During last call in the WG there were concerns raised about the test descriptions not being fully specified, but the document at that point was inconsistent about its goals, and it seems that fixing that remedied the issue.  There was not any threat of appeal or other action though, and after announcing the intention to continue with the revised document following WGLC, there was no protest.


(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.


Yes.


(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.


No.


(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 


Reasonable consensus.


(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)


No.


(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.


There are a couple of warnings about obsolete references; this can be fixed by the RFC Editor, but does not impact the tests described in the document.


(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.


N/A.


(13) Have all references within this document been identified as
either normative or informative?


Yes.


(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?


No.


(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.


No.


(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.


N/A.



(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).


There are no IANA considerations.


(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.


N/A


(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.


N/A
2016-03-14
11 Wesley Eddy Responsible AD changed to Martin Stiemerling
2016-03-14
11 Wesley Eddy IETF WG state changed to Submitted to IESG for Publication from WG Consensus: Waiting for Write-Up
2016-03-14
11 Wesley Eddy IESG state changed to Publication Requested
2016-03-14
11 Wesley Eddy IESG process started in state Publication Requested
2016-03-14
11 Wesley Eddy Tag Doc Shepherd Follow-up Underway cleared.
2016-03-14
11 Wesley Eddy Changed document writeup
2016-03-07
11 Wesley Eddy Tag Doc Shepherd Follow-up Underway set.
2016-03-07
11 Wesley Eddy IETF WG state changed to WG Consensus: Waiting for Write-Up from In WG Last Call
2016-02-15
11 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-11.txt
2016-02-02
10 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-10.txt
2015-11-27
09 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-09.txt
2015-10-14
08 (System) Notify list changed from "Wesley Eddy"  to (None)
2015-09-21
08 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-08.txt
2015-08-10
07 Wesley Eddy Notification list changed to "Wesley Eddy" <wes@mti-systems.com>
2015-08-10
07 Wesley Eddy Document shepherd changed to Wesley Eddy
2015-08-10
07 Wesley Eddy IETF WG state changed to In WG Last Call from WG Document
2015-07-23
07 Richard Scheffenegger Intended Status changed to Informational from None
2015-07-06
07 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-07.txt
2015-06-30
06 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-06.txt
2015-06-29
05 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-05.txt
2015-06-19
04 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-04.txt
2015-05-21
03 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-03.txt
2015-03-23
02 Naeem Khademi New version available: draft-ietf-aqm-eval-guidelines-02.txt
2015-03-05
01 Naeem Khademi New version available: draft-ietf-aqm-eval-guidelines-01.txt
2015-03-05
01 Naeem Khademi New version available: draft-ietf-aqm-eval-guidelines-01.txt
2014-09-18
00 Wesley Eddy This document now replaces draft-kuhn-aqm-eval-guidelines instead of None
2014-09-18
00 Nicolas Kuhn New version available: draft-ietf-aqm-eval-guidelines-00.txt