Liaison statement
LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG11-LS52)

ITU-T Study Group 12 thanks you for your liaison, sharing your plan to discuss
Draft Q.3961, “Testing methodologies of Internet related performance
measurements including e2e bit rate within the fixed and mobile operator’s
networks”, during future workshops and Joint meetings. We are aware that text
similar to draft requirements in Q.3961 was completely deleted from the
corresponding ETSI INT specification as a result of extensive comments during
the approval process, and there are additional comments post-approval which
remain un-addressed and unresolved after several months. We have been notified
of new and urgent work in ETSI STQ to revise clause 6 of TS 103 222-1 (the
reference replacing almost all requirements in the ETSI INT specification). The
current text of TS 103 222-1 (3-2018) specification retains a completely
non-compatible scope to draft Q.3961, and suffers from the same shortcomings
raised in the ETSI INT review process. As part of the Harmonization efforts for
IP-network performance specifications between ETSI STQ and Q17/12, we have
reviewed the revised text of clause 6/103 222-1, and find it satisfactory. Our
conclusion on this topic is that we agree and support the request of ETSI STQ,
to avoid referencing the (current) text of TS 103 222-1 (3-2018) while this
work is in-progress. The work is proceeding toward a prompt resolution.

Study Group 12’s plan is to prepare relevant normative specifications for
IP-layer Capacity and Flow-related Parameters (throughput), and that SG 11 make
use of the new specifications when available, as communicated in May 2018.  We
are pleased SG11’s recent liaison did not object to this plan, and we hope SG11
will participate in Harmonization efforts, as ETSI STQ has agreed.

As stated in your liaison, ITU-T SG11 envisages that benchmarking testing is
one of the components of draft Recommendation ITU-T Q.3961. Since the term
“benchmarking” is not defined in the liaison, and has many uses throughout the
industry, we would like to understand what this form of benchmarking is
intended compare.

We are also aware of a list of methods used by Regulators to *survey* Internet
performance, published by the OECD http://www.oecd.org/internet/speed-tests.htm
. Some Regulators have made disclaimers about the accuracy of their survey
results. For example:

        The locations and technical features of the broadband connections have
        been provided by the service users themselves, who are entirely and
        exclusively liable for their accuracy. The speed resulting from the
        measurement as the final speed of the connection is also affected by a
        series of factors, such as the quality of cables, connections and
        equipment or electromagnetic interference.

This well-expresses our concerns with the methods described in current SG 11
draft of Q.3961.

At this time, we would like to inform you of our progress to develop the
aforementioned normative specifications in Study Group 12, especially our
progress at the October 2018 Q17 Interim meeting in Darmstadt.

•       AT&T’s contribution describing how network access, usage patterns, user
application requirements, and future measurement paths have changed in the past
five years set the tone for the meeting. There are likely to be “Gigabit
islands” at the edge of future networks. •       ETSI TR103501 has been
presented and discussed. Essential points were the requirement for full
transparency of the measurement and data processing methodology. The TR
attempts to provide a full view of today’s methods for application level
measurement including multi socket and crowd sourcing approaches. •      
Y.1540 IP capacity and flow related parameter clauses were edited by the group.
•       Y.1540 annex A was expanded to include two phases of testing. The first
phase is a test using shaper/policer capabilities of a general purpose
computing system, based on AT&T’s contribution and preliminary test results.
This phase is parallel to the BEREC evaluation of Internet access measurements,
and adding critical test path parameters. •       Y.1540 annex A was expanded
to include information on evaluation of different access speed measurement
methods, conducted in 2012 by Goga and Teixeira. This work parallels our plan
for phase 2. •       There is a clear need to enlist active participants in the
testing program, who are willing to contribute both resources and personnel
consistent with the plans under development. Those who choose not to
participate will have no influence on the outcome, or the schedule for
completion, so they are encouraged to participate or wait patiently. •       A
list of regulator Internet access survey methods was compiled by AT&T and
discussed during the meeting. This list is more up to date than the 2015 OECD
list, and concludes that there are many different methods currently in use, and
no International consensus can be derived to support standard development from
reading such lists. •       AT&T contributed a comparison of two methods of LTE
Internet access speed assessment. The results show wide variation and strong
dependence on mobile device type. Larger sample sizes are needed to achieve the
comparison goals. •       Q17 reviewed the ETSI-STQ’s revised clause 6 of
TS-103 222-1, from the rapporteur of this new work item. The text was found to
be agreeable, with the suggestion to update the introduction to indicate these
changes.

Study Group 12 has provided a detailed explanation of the clause 6.12
requirements in Appendix IX of Rec. Y.1540, titled “Explanation of TCP-based
measurement inadequacy to meet normative requirements”.  However, we have
concluded that Capacity measurements described in RFC 8337 meet the
requirements in clause 6.12 of ITU-T Recommendation Y.1540.

We invite all experts to participate in the evaluation plan’s development and
to subsequently participate in the evaluations. SG 11’s plans to develop the
text of the draft Q.3961 should now move forward after the evaluation and
approval of Study Group 12’s normative specifications are complete, according
to the contribution-driven schedule.  Certainly, SG 11 experts will be willing
to contribute their expertise and resources to this plan, consistent with their
desire to complete work on this topic as quickly as possible.

The current draft of revisions to Y.1540, and a new Annex A for the new
parameter definitions now contains the current text of the evaluation plan for
candidate definitions for the new normative parameters for IP-layer Capacity
and Flow-related Parameters (throughput). Annex A contains the latest version
of the evaluation plan, with key references to existing comparisons of methods.

Study Group 12 would like to invite Study Group 11 to schedule co-located
meetings in Geneva, November 2018, and to participate in the SG12 workshop
adjacent to the meeting dates, on November 26, 2018. We would like to use the
workshop to share our experience, to make the work of overseeing ITU-T SG 12’s
mandate and our coordination roles more efficient and effective.

Attachments:
TD619- Current text of Annex A and Y.1540 clauses.

URL to the Workshop:
https://www.itu.int/en/ITU-T/Workshops-and-Seminars/qos/201811/Pages/default.aspx