Characterization Guidelines for Active Queue Management (AQM)
draft-ietf-aqm-eval-guidelines-13

(1) In the abstract, I'm not sure "precautionary" is really the right adjective.

(2) Sec. 4.2 says: "However, the evaluation MUST be primarily based on externally observed end-to-end metrics." This seems like a bit of a weird use of normative language. It makes sense to me that you would normatively recommend how to setup an evaluation environment to be able to compare AQMs, but if people want to weigh their evaluations in the wrong the direction, there is not much to be done about it. It seems like this would be more useful if it focused on explaining why primarily focusing on e2e measurements is preferable.

(3) Overall I would recommend a pass through the whole document to check for the consistency with which normative language is used -- it seems like in some cases it isn't really necessary but it's there anyway, or its use is uneven (e.g., 4.3.2, 5.2, 6.1, 6.2, 6.3, 7.2 to name a few). There also seems to be a mix of normative statements about what proposals should say and how evaluation environments should be setup. It would help if each time a normative statement was made it clearly distinguished who the recommendation is directed towards. This became especially confusing in Sec. 13, where it is unclear whether "Scenario MUST be considered" means that an AQM spec needs to discuss this, or an AQM evaluation environment needs to test for this.

(4) Since this document provides guidelines for how to characterize AQM proposals, it seems inappropriate for it to be making normative recommendations about how AQM proposals should work, even if they are just re-statements of things from RFC 7567. In particular, these statements seem out of place:

	- Sec. 4.4: "An AQM scheme SHOULD adhere to the recommendations outlined in
   [RFC7141], and SHOULD NOT provide undue advantage to flows with
   smaller packets [RFC7567].
   
    - Sec. 4.5: "Deployed AQM algorithms SHOULD implement Explicit Congestion
   Notification (ECN) as well as loss to signal congestion to endpoints
   [RFC7567]."
   
   - Sec. 13: "Tuning SHOULD NOT be required"

(5) Sec. 4.6 says: "This discussion as an instance, MAY explain whether the dropping policy is applied when packets are being enqueued or dequeued." I don't understand what "this discussion as an instance" means.

(6) In Sec. 7.2, is there some justification that could be provided for choosing the particular parameters listed to simulate web traffic? E.g., are these parameters considered typical or is there a reference to some standard way of simulating web traffic that could be referenced?

(7) Also in Sec. 7.2, why is the detail for the web flow given but the detail for bursty video is not? Surely there is more than one way to generate or simulate bursty video -- does it not matter what the bursts look like or how they are timed for this measurement purpose?

(8) In 9.1, I would suggest that bi-directional video is prevalent and important enough to be listed in the traffic mix analysis as well.

(9) I think the statement in Sec. 17 assumes that the tests envisioned by this document will be based on active measurements of simulated traffic. If that is a real scope limitation, and evaluation of AQMs based on passive measurement of traffic is out of scope, I think that should be stated somewhere in the document.

(10) Just curious: has anyone thought about using LMAP, or the LMAP information model at least, to configure the kinds of tests envisioned in this document?

Too bad there are no implementations... :-(

- Random Early Detection (RED), BLUE, and Proportional Integral controller (PI)
Would you have references?

Several issues from the Gen-ART review by Ralph Droms need to be addressed, in my opinion. 

In particular, Figure 1 or text under needs clarification, the advise on ECN seems superfluous, and section 10 example needs to be labeled as an example.

On 5/19/16 6:50 AM, João Taveira wrote:
> Hi,
> 
> Overall the document looks good, if somewhat strenuous to implement
> fully in practice. Nitpicks inline.
> 
> # Section 2.4
> 
> s/in the bottleneck available/at the bottleneck are available
> 
> # Section 2.5
> 
> s/allows the tester evaluate/allows the tester to evaluate
> s/This metric should be also/This metric should also be
> 
> # Section 2.7
> 
> The initial paragraph of this section is confusing since it asks the
> reader to consider the metrics "as explained in the rest of this
> document", which obviously the reader will have not been exposed to.
> 
> Some of the implementation details were ambiguous to me. For example:
> "It is suggested to take measurements at least every K x minRTT (to
> smooth out the fluctuations), with K=10. Higher values for K are
> encouraged whenever it is more appropriate for the presentation of the
> results.". There seems to be some contradiction between _suggesting_
> K=10, and then _encouraging_ a higher K, which would reduce the sampling
> frequency in practice. Finally the paragraph suggests "It is generally
> RECOMMENDED to provide at least 10 samples per RTT.", which would
> suggest K = 1/10. I believe my misinterpretation stems from the fact a
> measurement period is introduced (K x minRTT), but all subsequent
> context refers to a measurement frequency.
> 
> At the end of the section, section references could be enclosed in
> parentheses since they are not part of the text: i.e. "This graph
> provides part of a better understanding of (1) the delay/goodput
> trade-off for a given congestion control mechanism (Section 5) ..."
> 
> # Section 3.1
> 
> It is not obvious how the asymmetrical topology characterizes the AQM
> system in any useful manner. If the objective is to see how a particular
> AQM algorithm performs under TCP ACKs, it would be more useful to study
> how the drop rate varies for streams of small packets. My main concern
> here is that testing reverse path performance will be tightly coupled
> with transport protocol behaviour, since you are reliant on a feedback loop.
> 
> If you do decide that an asymmetrical test case is useful, it would be
> helpful to provide more guidance on how asymmetrical a topology should
> be. For TCP, is setting the capacity for the reverse path to 50% of the
> forwarding path any different from the symmetrical case in practice?
> 
> # Section 3.3
> 
> There is no baseline congestion control provided for LBE. Given LEDBAT
> is mentioned in section 5.4, it would probably be wise to just define it
> as a baseline in order to minimize implementation cost.
> 
> # Section 4.4
> 
> s/impact the application performance/impact application performance
> s/smaller probability/lower probability
> 
> I'm somewhat surprised the topic of packet sizes is not covered more. 
> This section highlights that attackers can gain undue advantage by using
> smaller packets, but if an AQM scheme does not take packet size into
> account, the converse could be true. In datacenter environments in
> particular, jumbo frames are relatively common, and so it would be
> valuable to understand whether an AQM scheme is biased towards a
> particular end of the size spectrum.
> 
> # Section 5.4
> 
> s/any of those listed/such as any of those listed
> 
> # Section 6.1
> 
> The first sentence of the second paragraph spans 6 lines, making it hard
> to read. I'd suggest breaking it at "... can be discussed. In this
> scenario ..."
> 
> # Section 7.2
> 
> It is unclear what the intent of describing what a web page download
> "could" be is. The document suggests that certain combinations of
> patterns MUST be tested, but then leaves most of the details up to the
> implementer, which makes comparisons between results trickier.
> 
> This section would like benefit from being more agnostic to what
> applications it is trying to mimic, and instead describe traffic mixes
> in more general terms. As it stands it is unclear whether the intent of
> the section is to approximate existing application patterns (which will
> be out of date by the time the document is published), or underlying
> traffic characteristics (bursty, inelastic, parallel, etc).
> 
> # Section 13
> 
> This table would have been helpful sooner in the document as a summary
> of implementation requirements. Independently of where it ends up
> "summary" may be a better header than "conclusion", since it can be
> picked up by a hasty implementer as a reference to the rest of the document.
> 
> One nitpick in this table is the "Operator Control" row, which suggests
> "Tuning SHOULD NOT be required.". This is more implementation guidance
> than characterization guideline, and should therefore be out of scope
> for this document (and indeed the recommendation is already part of
> RFC7567). In terms of characterization it would be more appropriate for
> this document to suggest that any tunable parameters MUST be explicitly
> declared and their impact discussed, in the form of a YANG definition or
> otherwise.
> 
> Cheers,