RTP Control Protocol (RTCP) Extended Report (XR) Blocks for Mean Opinion Score (MOS) Metric Reporting
draft-ietf-xrblock-rtcp-xr-qoe-17

[Ballot comment]
Thanks for addressing my DISCUSSion points.

One last comment, which Gonzalo could simply change in an RFC Editor note if you decide to do so; you don't need to do a new draft:

In section 3:

  This block reports the media quality in the form of a 1-5 MOS range

That's not really precise. Probably better to say:

  This block reports the media quality in the form of a MOS range
  (e.g., 1-5, 0-10, or 0-100, as specified by the calculation
  algorithm)

[Ballot discuss]
It sounds like we've made progress in the discussion. Updating my ballot to suit:

3.2.1 says:

      A 1-5 MOS score is multiplied by 10 and then represented in the
      7:9 format.

3.2.2 says:

      The estimated MOS value is multiplied by 10 and expressed in 7:6
      format.
   
That seems like a recipe for an implementation mistake. First, it seems like putting 3-bits of padding in front of the one in 3.2.1 so that they can both be 7:6 would make it so that people wouldn't accidentally screw up the values if they transferred between. If the WG is agreed that this is not a problem, I will simply clear.

Asking the implementation to multiply by 10 also seems unnecessary and a potential for interoperability failure (i.e., something that will accidentally not get done by an implementation). I understand this was done in 3611, but I presume that's because it was putting fractional values into an integer field and wanted to get at least 1 decimal place into the integer portion. You've got fixed-point here, so no need to do the multiply.

(It was also not clear where the range was coming from, but the authors explained that this was from the Calculation Algorithm.)

So, I suggest in both cases:

      The estimated Mean Opinion Score (MOS) for multimedia application
      performance is defined as including the effects of delay, loss,
      discard, jitter and other effects that would affect media quality.
      This is a unsigned fixed-point 7:6 value representing the MOS,
      allowing scores up to 127 in the integer part. MOS ranges are
      defined as part of the specification of the MOS estimation
      algorithm (Calculation Algorithm in this document), and are
      normally ranges like 1-5, 0-10, or 0-100. Two values are reserved:
      A value of 0x1FFE indicates out of range and a value of 0x1FFF
      indicates that the measurement is unavailable. Values outside of
      the range defined by the Calculation Algorithm, other than the two
      reserved values, MUST NOT be sent and MUST be ignored by the
      receiving system.

If you want 7:9 for 3.2.1, then you also need to change the reserved values to 0xFFFE and 0xFFFF.

[Ballot comment]
After the most recent update to 4.2, the text is still slightly broken:

  understand the extension SHOULD NOT include it from the SDP answer

I believe that this should be "SHOULD NOT include it in the SDP answer" instead.  The same error repeats in the next paragraph.

---

Former DISCUSS, which has been addressed:

In section 4.2, third paragraph (not counting bullet items above it):

  Segment extensions, with their directions, MAY be signaled for an
  "inactive" stream.  It is an error to use an extension direction
  incompatible with the stream direction (e.g., a "sendonly" attribute
  for a "recvonly" stream).

How is this error handled?  You an address this DISCUSS point by adding text that says how, or pointing out why it isn't a problem.

Comments that have been addressed:

Page 6, first paragraph:
  measurement techniques allows much large scale measurements to

This is obviously a typo; could be much larger scale measurements or much large scale measurement.  Might want to disambiguate before the copyedit happens to avoid misunderstandings.

      In this document, MOS Metrics MAY be reported for intervals or
      for the duration of the media stream (cumulative) however it
      is not recommended that sampled values are used.

Section 3.2, just above the Reserved heading:
      In this document, MOS Metrics MAY be reported for intervals or
      for the duration of the media stream (cumulative) however it
      is not recommended that sampled values are used.

It might be better to forbid sampled values if they aren't useful, so that implementations don't have to handle them.  Just a suggestion—you know better than I.  No need to explain if you disagree.

[Ballot discuss]
[I'm still waiting for a reply on the discussion we've been having, but I am updating my ballot for the changes that occurred in the latest version; thanks for those.]

3.2.1 says:

      A 1-5 MOS score is multiplied by 10 and then represented in the
      7:9 format.

3.2.2 says:

      The estimated MOS value is multiplied by 10 and expressed in 7:6
      format.
   
That seems like a recipe for an implementation mistake. First, it seems like putting 3-bits of padding in front of the one in 3.2.1 so that they can both be 7:6 would make it so that people wouldn't accidentally screw up the values.

But I also don't understand why you want 7 bits of integer if the values are limited to 1-5; why not use 3:10? The authors claim in discussion that integer portion is larger because other ranges might be used (e.g., 0-11 or 1-100), but that seems completely non-interoperable. Shouldn't everything be scaled to a range of 1-5?

Asking the implementation to multiply by 10 also seems unnecessary and a potential for interoperability failure (i.e., something that will accidentally not get done by an implementation). I understand this was done in 3611, but I presume that's because it was putting fractional values into an integer field and wanted to get at least 1 decimal place into the integer portion. You've got fixed-point here, so no need to do the multiply.

[Ballot comment]
5.3: I still don't understand why we have individuals as contacts for IETF standards track registered items. The IETF or IESG should be the registrant, not an individual.

[Ballot discuss]
In section 4.2, third paragraph (not counting bullet items above it):

  Segment extensions, with their directions, MAY be signaled for an
  "inactive" stream.  It is an error to use an extension direction
  incompatible with the stream direction (e.g., a "sendonly" attribute
  for a "recvonly" stream).

How is this error handled?  You an address this DISCUSS point by adding text that says how, or pointing out why it isn't a problem.

[Ballot comment]
Page 6, first paragraph:
  measurement techniques allows much large scale measurements to

This is obviously a typo; could be much larger scale measurements or much large scale measurement.  Might want to disambiguate before the copyedit happens to avoid misunderstandings.

      In this document, MOS Metrics MAY be reported for intervals or
      for the duration of the media stream (cumulative) however it
      is not recommended that sampled values are used.

Section 3.2, just above the Reserved heading:
      In this document, MOS Metrics MAY be reported for intervals or
      for the duration of the media stream (cumulative) however it
      is not recommended that sampled values are used.

It might be better to forbid sampled values if they aren't useful, so that implementations don't have to handle them.  Just a suggestion—you know better than I.  No need to explain if you disagree.

[Ballot discuss]
The metric names in the registry are not specific enough: payload type, calculation identifier metric, segment type, and potentially MOS. I guess they should say something about RTP. Let me file this DISCUSS while I double-check with the performance metric directorate.

[Ballot discuss]
Also seem comments below regarding the fixed point representation description. But aside from that:

3.2.1 says:

      A 1-5 MOS score is multiplied by 10 and then represented in the
      8:8 format.

3.2.2 says:

      The estimated MOS value is multiplied by 10 and expressed in
      6:7 format.
   
That seems like a recipe for an implementation mistake. If the values are limited to 1-5, why is the format not simply 3:13 in the 3.2.1 and 3:10 in 3.2.2? Or even better, but some padding in front of the one in 3.2.1 so that they can both be 3:10. Asking the implementation to multiply by 10 seems like something that will accidentally not get done by an implementation. Is this really what you want? Is there some reason I don't understand that you want it this way?

4.1:

  xr-mos-block = "mos-metrics" ["=" extmap *("," calgextmap)]

extmap is undefined.

[Ballot comment]
2.1: Is this text about numeric formats taken from somewhere else, and therefore everyone already understands it? I found it very confusing and hard to read. (For example, if I understand correctly, S does not indicate a two's-complement representation; S simply indicates that the value is signed, and its absence indicates that it is unsigned. Everything in this format is a fixed-point two's-complement representation.) But in the document, this representation is only used in two places, 3.2.1 and 3.2.2, and neither of them is signed. Seems like you could simply describe the value much more clearly (e.g., "an unsigned fixed-point twos-complement value, 8 bits of integer and 8 bits of fraction") in each section.

3.2.1:

      If the measured value is over range, the value
      0xFFFE MUST be reported.  If the measurement is unavailable, the
      value 0xFFFF MUST be reported.
     
I don't think those MUSTs are appropriate. Those are definitions. Instead:

      A value of 0xFFFE is a flag indicating that the measured value is
      out of range. A value of 0xFFFF is a flag indicating that the
      measurement is unavailable.
     
Similarly in 3.2.2.

5.3: I still don't understand why we have individuals as contacts for IETF standards track registered items. The IETF or IESG should be the registrant, not an individual.

IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-xrblock-rtcp-xr-qoe-12.  Authors should review the comments and/or questions below.  Please report any inaccuracies and respond to any questions as soon as possible.

IANA understands that, upon approval of this document, there are four actions which IANA must complete.

First, in the RTCP XR Block Type subregistry of the RTP Control Protocol Extended Reports (RTCP XR) Block Type Registry located at:

http://www.iana.org/assignments/rtcp-xr-block-types/

a new block type will be registered as follows:

BT: [ TBD-at-registration ]
Name: MOS Metrics Block
Reference: [ RFC-to-be ]

Second, in the RTCP XR SDP Parameters subregistry of the RTP Control Protocol Extended Reports (RTCP XR) Session Description Protocol (SDP) Parameters Registry located at:

http://www.iana.org/assignments/rtcp-xr-sdp-parameters/

a new parameter will be registered as follows:

Parameter: mos-metrics
Reference: [ RFC-to-be ]

Third, in the att-field (both session and media level) subregistry of the Session Description Protocol (SDP) Parameters located at:

http://www.iana.org/assignments/sdp-parameters/

a new attribute will be reegistered as follows:

Type: att-field (both session and media level)
SDP Name: calgextmap
Reference: [ RFC-to-be ]

Fourth, a new registry called the "RTCP XR MOS Metric block - multimedia application Calculation Algorithm" registry will be created as a sub-registry of the "RTP Control Protocol Extended Reports (RTCP XR) Block Type Registry" located at:

http://www.iana.org/assignments/rtcp-xr-block-types/

This registry will be described as "Applies to the multimedia session where each type of media are sent in a separate RTP stream and also applies to the session where Multi-channel audios are carried in one RTP stream."

Maintenance of the new registry will be via Specification Required as defined by RFC 5226.

There are initial registrations in the new registry as follows:

Name Name Description Reference Type
Show quoted text
P564 ITU-T P.564 Compliant Algorithm [P.564] Voice
G107 ITU-T G.107 [G.107] Voice
TS101_329 ETSI TS 101 329-5 Annex E [ETSI] Voice
JJ201_1 TTC JJ201.1 [TTC] Voice
G107_1 ITU-T G.107.1 [G.107.1] Voice
P862 ITU-T P.862 [P.862] Voice
P862_2 ITU-T P.862.2 [P.862.2] Voice
P863 ITU-T P.863 [P.863] Voice
P1201_1 ITU-T P.1201.1 [P.1201.1] Multimedia
P1201_2 ITU-T P.1201.2 [P.1201.2] Multimedia
P1202_1 ITU-T P.1202.1 [P.1202.1] Video
P1202_2 ITU-T P.1202.2 [P.1202.2] Video

IANA understands that these four actions are the only ones required upon approval of this document.

Note:  The actions requested in this document will not be completed
until the document has been approved for publication as an RFC.
This message is only to confirm what actions will be performed.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (RTP Control Protocol (RTCP) Extended Report (XR) Blocks for MOS Metric Reporting) to Proposed Standard


The IESG has received a request from the Metric Blocks for use with
RTCP's Extended Report Framework WG (xrblock) to consider the following
document:
- 'RTP Control Protocol (RTCP) Extended Report (XR) Blocks for MOS Metric
  Reporting'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2013-11-27. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document defines an RTP Control Protocol (RTCP) Extended Report
  (XR) Block including two new segment types and associated SDP
  parameters that allow the reporting of MOS Metrics for use in a range
  of RTP applications.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcp-xr-qoe/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcp-xr-qoe/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)? Why
is this the proper type of RFC? Is this type of RFC indicated in the
title page header?

Standards Track. This status is used by all documents that define RTCP-XR blocks issued by the XRBLOCK WG. The type is indicated on the title page header.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

This document defines an RTP Control Protocol (RTCP) Extended Report
(XR) Block including two new segment types and associated SDP
parameters that allow the reporting of MOS Metrics for use in a range
of RTP applications.

Working Group Summary

The WG reviewed carefully the document several times and there is
good support behind it.

Document Quality

Two vendors indicated that they have similar functionality in their
code and intentions to implement the standard block when approved.
There was one important change to notice which resulted from the SDP
Expert Review which led to the change of the title of the document from
QoE to MOS Metric reporting, which defines better the scope of the document.

Personnel

Dan Romascanu is the Document Shepherd. Gonzalo Camarillo is the Responsible
Area Director.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd. If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

I reviewed the document and I believe that it is ready for publication.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

SDP and PM-DIR expert reviews were performed.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

All the authors confirmed either by answering explicitly my query on
this subject, or by referring to the boilerplate text in the submitted
I-Ds that state that the documents are submitted in full conformance with
the provisions of BCP 78 and BCP 79.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No IPR disclosures where submitted on this I-D.

One of the authors (Alan Clark) mentioned that while they (his company) do
not have IPR directly related to the draft, i.e. to the reporting of a MOS
score in an RTCP XR payload, they do have extensive IPR related to the
calculation of voice, audio and video MOS scores and have declared IPR to
both ITU and ETSI in relation to many of the objective MOS estimation
algorithms listed in the draft. His understanding of IETF IPR policy is
that the request for disclosures relates to IPR required to implement the
RFC and not to the computation of values that may be carried by the protocol.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it?

The XRBLOCK is not attended by a large number of people nowadays, there are
about ten active contributors, these were in strong consensus in favor
of the document.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

There is one warning about unused reference [RFC5234] which is because the
mention to 5234 is embedded in the code. Another warning about [ATSC] being
a possible downref can be ignored.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

SDP and PM-DIR reviews were performed, and the comments of the experts
were addressed.

(13) Have all references within this document been identified as
either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

No.


(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No.


(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

The IANA Considerations Section includes a request to add a new RTCP XR
Block Type value, a request to add a new RTCP XR SDP Parameter, and a request
to create new registry to be called "RTCP XR MOS Metric block - multimedia
application Calculation Algorithm" as a sub-registry of the "RTP Control
Protocol Extended Reports (RTCP XR) Block Type Registry". These requests
are clear and they include all needed information that will allow to IANA
to perform their actions.


(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

The review process for the registry is "Specification Required"

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

N/A