Liaison statement
LS on harmonization of IP Capacity and Latency Parameters: Consent of Draft Rec. Y.1540 on IP packet transfer performance parameters and New Annex A with Lab & Field Evaluation Plans

Further Progress on Evaluating IP Capacity Metrics and Methods for Revised
Recommendation Y.1540, “Internet protocol data communication service – IP
packet transfer and availability performance parameters”

As an update to our communications from the Q17/12 Interim meeting (Darmstadt,
16-17 October 2018), the Geneva, 27 November – 6 December 2018 meeting with the
full SG12, and the results from the subsequent Q17/12 Interim meeting (Berlin,
5-7 March 2019), we now share further results from the Q17/12 meeting with the
Full Study Group 12 (Geneva, 7-16 May 2019). We have completed all planned
tests using the Phase 1 Laboratory test bed that supports development our
revised Rec. Y.1540 and new Annex A/Y.1540 specifications of IP Capacity and
Latency methods of measurement. We have compared several existing measurement
methods based on TCP and UDP transport. The Phase 2 Network tests on live
networks are also sufficiently complete for our work to proceed.

Our key results and cumulative findings from the May 2019 Geneva meeting are:

•       The Survey of prior testing and academic publications is now complete,
finding that: o       The benchmark measurement of the maximum achievable
access speed is only obtained by flooding the access link with UDP traffic. A
large scale research survey and the test tool of a major content provider
prefer UDP to accurately measure access speed. o       Parallel TCP based
measurements can only be used to estimate Internet access bandwidth, and
several publications discuss specific factors reducing the accuracy of parallel
TCP based measurements. TCP based measurements fail to detect the presence of
background traffic, causing further issues and inaccuracies. o       In a
recent study, UDP was found to be a viable Internet transport protocol, based
on tests in North & South America, Europe, Asia and Oceania. UDP traffic was
blocked on some ports or entirely in rare cases. o       Material on WiFi
performance and Encrypted stream Network QoS parameters introduce a set of
KPIs/metrics prioritized through machine learning, and models of QoE. •      
The text of the IP Capacity definition and Methods of measurement, and the
Phase 1 and Phase 2 sections of the test and evaluation plan were edited and
agreed during the meeting. In the agreed text, the Phase 1 study was a Lab
evaluation, and Phase 2 study was carried-out in production networks. This
material constitutes revisions to Y.1540 sections and the new ANNEX A/Y.1540.
The Q17 meeting agreed to seek Consent on this material. •       The meeting
agreed on an expanded Summary of the Phase 1 and Phase 2 Testing as a new
informative APPENDIX to Y.1540, to provide the supporting test data for the
conclusions represented as requirements in the body and Annex A. Some
conclusions are: o       Phase 1/Lab: Even without correction factors, the UDP
measurements are considerably closer to the calibrated shaper rate than TCP,
despite using the most favourable circumstances for TCP (no added delay or
background traffic). With correction for headers that are viewed by the traffic
shaper, UDP measurements using iPerf 2 are within 200ppm of the configured
shaper rate. UDP-based measurements are the Benchmark for capacity, accurately
assessing the “ground truth” of the traffic shaper rate under all tested
conditions. o       Phase 1/Lab: TCP measurements using iPerf2 underestimate
the shaper rate with or without correction factors. Typical round-trip delay
and the presence of competing/background traffic tend to make TCP-based
estimates of available capacity appreciably worse, and introduce considerable
results variability that reduces confidence in any measurement. o       Phase
2/Field: The Phase 1/Lab conclusions on UDP-based testing as the Benchmark, and
TCP as underestimating capacity were supported by the Phase 2/Field
measurements. TCP measurements on 1 Gbps PON exhibit significant
underestimation of capacity. UDP-based methods are clearly the preferred
transport protocol for Capacity assessment. •       It is clear from on-going
measurements that Internet subscriber use of TCP protocol is declining. Video
and browser traffic has been shifting to UDP and higher-layer reliability
mechanisms for years, with the most significant growth in 2018 due to CDN
adoption.

SG12 continues to invite interested parties to validate testing results, and to
coordinate on the development of a new generation of harmonized specifications
of IP Capacity performance metrics and methods of measurement, and other key
performance parameters.

In one year’s time, Study Group 12 has completed many new areas of
investigation, Consented its key IP performance Recommendation (Y.1540) with
the newest Capacity metrics, and reached the point where development of
coordinated specifications can proceed among collaborating SDOs.

All Q17/12 participants continue to seek additional SDO and individual support,
participation and/or constructive review, so that together, we can proceed
forward.