Liaison statement
LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG11-LS31)

ITU-T Study Group 12 thanks you for your liaison, sharing your plan to put
Draft Q.3961, “Testing methodologies of Internet related performance
measurements including e2e bit rate within the fixed and mobile operator’s
networks”, on hold, awaiting the progress of other standards.

We are aware that text similar to Q.3961 is under development in ETSI INT and
has received extensive comments during the attempted approval process, all of
which remain un-addressed and unresolved after many months.

We are also aware of similar text recently revised by ETSI STQ, which has been
prepared for a specification with completely non-compatible scope to draft
Q.3961, and which suffers from the same shortcomings raised in the ETSI INT
approval process. In summary, there is no available text to support Draft
Q.3961 moving forward at this time, in any form.

Study Group 12 has participated in the Joint meeting with SG11 and ETSI INT,
and our plan is to prepare relevant normative specifications for IP-layer
Capacity and Flow-related Parameters (throughput), and that SG11 make use of
the new specifications when available, as expressed during the meeting by SG12
Chairman, Kwame Baah-Acheamfuor.  We were pleased that there were no objections
to this cooperative plan, and will proceed as agreed.

At this time, we would like to inform you of our progress to develop the
aforementioned normative specifications in Study Group 12.

The IETF has approved and published RFC 8337, “Model-Based Metrics for Bulk
Transport Capacity”, in March 2018. The Model-Based Metrics (MBM) work was the
result of many years considering the problem of transport capacity measurement,
primarily in the IETF IP Performance Metrics (IPPM) working group. The
specification carefully describes the many issues and difficulties with
repeatability when testing with standard-compliant TCP (section 4), and solves
these problems principally by designing a method and a set of diagnostic tests
where TCP’s flow-control is disabled. The method involves the evaluation of a
target transport performance in terms of transmission rate and Round-trip time
(RTT).

Study Group 12 now informs you of our plans to resolve the need for
Recommendations in this area.  Study Group 12 will take the MBM work as a
starting-point, prepare normative specifications for IP-layer Capacity and
Flow-related Parameters (throughput), and evaluate multiple versions of
candidate definitions for these parameters through a comprehensive evaluation
plan. The approach is to compare measurements using implementations of
candidate parameters with the “ground-truth” of baseline path bandwidth,
characterized on a per-element basis. The testing campaign will be conducted
over different access types and path conditions, evaluating the errors observed
and determining where the candidates are sufficiently accurate, and where they
are not.

Study Group 12 has provided a detailed explanation of the clause 6.12
requirements in Appendix IX of Rec. Y.1540, titled “Explanation of TCP-based
measurement inadequacy to meet normative requirements”.  However, we have
concluded that Capacity measurements described in RFC 8337 meet the
requirements in clause 6.12 of ITU-T Recommendation Y.1540 (as described in
SG12 C160, see attached).

We invite all experts to participate in the evaluation plan’s development and
to subsequently participate in the evaluations. SG11’s plans to develop the
text of the draft Q.3961 should now move forward after the evaluation and
approval of Study Group 12’s normative specifications are complete, according
to the contribution-driven schedule.  Certainly, SG11 experts will be willing
to contribute their expertise and resources to this plan, consistent with their
desire to complete work on this topic as quickly as possible.

The current draft of revisions to Y.1540, and a new Annex A for the new
parameter definitions now contains the current text of the evaluation plan for
candidate definitions for the new normative parameters for IP-layer Capacity
and Flow-related Parameters (throughput).

Study Group 12 would like to invite Study Group 11 to schedule co-located
meetings in Geneva, as this would give us the additional opportunity to
schedule a workshop adjacent to the meeting dates. We would like to use the
workshop to share our experience, to make the work of overseeing our mandate
and our coordination roles more efficient and effective.

Attachments:
SG12-TD540R2 “Revisions to Y.1540 and text of new Annex A and Initial
Evaluation Plan” (Editors of Y.1540 and new Annex A/Y.1540) SG12-C160
“Development of Y.1540 Flow-related Parameters based on IETF RFC 8337” (AT&T,
Inc.)