Evaluation Test Cases for Interactive Real-Time Media over Wireless Networks
draft-ietf-rmcat-wireless-tests-11

==[ Old Discuss ]==
Please let me know if I've misunderstood the test execution protocol incorrectly:

Section 6.  Per the paragraph “The evaluation of the test cases are intended to carry out in a controlled lab environment … It is important to take appropriate caution to avoid leaking non-responsive traffic from unproven congestion avoidance techniques onto the open Internet”, this is good guidance in general case.  However, in the case of this document how applicable is it?  Didn’t Section 3 (“We, therefore, recommend that a cellular network simulator is used for the test cases defined in this document …” and practically establish it can’t be done without simulation with the scenario of the underground mine) and Section 4 (“We recommend to carry out the test cases as defined in this document using a simulator, such as [NS-2] or [NS-3]).   If all the testing is supposed to be in a simulator how is it leaking out onto the internet?  As far as I can tell, this helpful text is common in RMCAT document, but in this case could it please be tailored for the proposed testing regime.  

Perhaps something on the order adding text on the order of “Given the difficulty of deterministic wireless testing, it is RECOMMENDED and expected that the tests described in this document would be done via simulation.  However, <in the case of not doing it that way> <leave the existing language>”

[Telechat discussion was that that using a simulator is only recommended, so there is the possibility of live testing which might leak onto live networks]

==[ end ]==

** Section 3.1.  Per “In this test case, each of the user/UE in the media session is an RMCAT compliant endpoint”, what is a “RMCAT compliant endpoint”?  Could this be cited please.

** Section 3.1.  Per “At the beginning of the simulation, there should be enough time to warm-up the network”, intuitively the notion of “warm[ing]-up the network” makes sense.  However, is more precision required to side-by-side analysis of test runs of when the network is “warm enough”?

** Section 4.  Per “Statistics collected from enterprise Wi-Fi networks show that the two dominant physical modes are 802.11n and 802.11ac, accounting for 41% and 58% of connected devices”, it is would be valuable to cite this value and provide a timestamp.  This distribution will certainly change as this document ages.

** Section 4.  Per “Unless otherwise mentioned, test cases in this section are described using the underlying PHY- and MAC-layer parameters based on the IEEE 802.11n Standard.”, this focus on only 802.11n is surprising, since the next sentence establishes that 802.11.n is already less than half (41%) of the Wi-Fi traffic (and likely will continue to shrink).  Why not ac?

** There appear to be some differences between the description of the Cellular (Section 3) and Wifi (Section 4) test.
-- Section 4.1.2 and 4.2.2 are called “Test setup”, but Section 3.1.2 and 3.2.2. are called “Simulation setup”.  Was this intentional?

-- Section 4.*.4 discusses the expected test behavior, but Section 3.* does not?  Was that explicit?

Thank you for the work put into this document. It is really easy to read.

Please find below some non-blocking COMMENTs and NITs. An answer will be appreciated.

I hope that this helps to improve the document,

Regards,

-éric

== COMMENTS ==

-- Section 1 --
Does the following assertion always stand ? "This contrasts with the wired network setting where traffic flows from all users share the same queue."

-- Section 3.1.2 --
As a frequent high speed train passenger, I would have appreciated a simulation with mobility of 300 km/h ;-)

== NITS ==

-- Section 3.1.2 --
s/10Mhz/10MHz/ ?

Thanks for the work that went into creating this document.

ID Nits correctly reports:

  == Unused Reference: 'I-D.ietf-rmcat-cc-requirements' is defined on line
     998, but no explicit reference was found in the text


As a note for the AD: I can’t find a justification in the ballot or shepherd’s writeup for why this document has six authors instead of a smaller number of editors and a list of contributors. It would be nice to capture the exception rationale in one of those two places for posterity.

If my reading of the mailing list is correct, it looks like this document was the result of merging draft-sarker-rmcat-cellular-eval-test-cases and draft-fu-rmcat-wifi-test-case.  It would be nice if this document was tagged as replacing them.

You include the BCP 14 boilerplate and references, but the only BCP 14 key word in the document is a MUST in Section 6 that seems like it doesnt need to be BCP 14 in this document (it seems to refer to a tenet, not a requirement specified in this document).  I suggest removing the boilerplate and the references and making the “must” lower case.

Thanks to Adam for noting the author-count already.

Section 3

   Different mobile operators deploy their own cellular networks with
   their own set of network functionalities and policies.  Usually, a
   mobile operator network includes 2G, EDGE, 3G and 4G radio access

Is 2G still "typical"?  I was given to understand it was getting phased
out.

Section 3.1, 3.2

How do I tell how much time is "enough time to warm-up the network"?

Section 4.2.3

   b.  Multiple RTP-based media flows sharing the wireless uplink:N = 16
       (all downlink); M = 0.  When multiple clients attempt to transmit

I'm not sure how to interpret the parenthetical; is it a copy/paste
issue from (a) that should read "all uplink"?