Video Codec Requirements and Evaluation Methodology
draft-ietf-netvc-requirements-10

[Ballot comment]
Clearing my Discuss as we had a brief discussion on the IESG telechat, and
the responsible AD will follow the discussion with the tsvart reviewer.

Original Discuss portion:

I support Roman's Discuss.

I am sympathetic to the tsv-art reviewer's concerns that this document
is focused on video technology of 5 years ago and may lack relevant in
the current world.  I don't intend to hold a Discuss point for any
specific resolution, but I do think the IESG should discuss whether this
concern affects the value of publishing this document as an RFC.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Original Comment portion (unchanged)

Section 2.1

What do "PAM" and "RA" mean?  Moving Appendix A earlier (before Section
2) or referring to it from the Introduction would be helpful.  Note that
RFC style is to expand on first use...

      . High Dynamic Range (HDR), Wide Color Gamut (WCG), high
          resolution (currently, up to 4K), high frame-rate content are
          important use cases, the codec should be able to encode such
          content efficiently.

nits: missing "and" in serial list, and the last comma is a comma splice.

Section 2.5

[Google didn't help me find reference [9].]

Section 2.6

The (long) list in Section 2.5 includes "cloud gaming"; how much overlap
does that have with this service?

Section 3.1, 3.2

What is the difference between "General Requirements" and "Basic
Requirements"?

ection 3.2.1

Is "Exemplary input source formats" supposed to just be an example, or
an indication of the pinnacle of possible values?

Section 4.1

  Initially, for the codec selected as a reference one (e.g., HEVC or
  VP9), a set of 10 QP (quantization parameter) values should be
  specified (in a separate document on Internet video codec testing)
  and corresponding quality values should be calculated. [...]

This seems to suggest ("Initially", "for the codec selected") that the
evaulation requirements are not yet complete.  Are they intended to be a
single set of requirements for the codec's development, or customized to
some per-application requirements?  (Is there a reference needed to
ongoing work to solidify these requirements?)

  QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) },
          i in R

I would suggest defining the argmin function.

It's surprising to see no reference to draft-ietf-netvc-testing from
this document.

Section 6

I don't see much need for a "Conclusions" section of this nature, in
this document.

Appendix B

Defining (e.g.) "high dynamic range" and "wide color gamut" with respect
to "normal" or "conventional" mechanisms does not really provide a
stable and archival reference for comparison.

[Ballot discuss]
I support Roman's Discuss.

I am sympathetic to the tsv-art reviewer's concerns that this document
is focused on video technology of 5 years ago and may lack relevant in
the current world.  I don't intend to hold a Discuss point for any
specific resolution, but I do think the IESG should discuss whether this
concern affects the value of publishing this document as an RFC.

[Ballot comment]
Section 2.1

What do "PAM" and "RA" mean?  Moving Appendix A earlier (before Section
2) or referring to it from the Introduction would be helpful.  Note that
RFC style is to expand on first use...

      . High Dynamic Range (HDR), Wide Color Gamut (WCG), high
          resolution (currently, up to 4K), high frame-rate content are
          important use cases, the codec should be able to encode such
          content efficiently.

nits: missing "and" in serial list, and the last comma is a comma splice.

Section 2.5

[Google didn't help me find reference [9].]

Section 2.6

The (long) list in Section 2.5 includes "cloud gaming"; how much overlap
does that have with this service?

Section 3.1, 3.2

What is the difference between "General Requirements" and "Basic
Requirements"?

ection 3.2.1

Is "Exemplary input source formats" supposed to just be an example, or
an indication of the pinnacle of possible values?

Section 4.1

  Initially, for the codec selected as a reference one (e.g., HEVC or
  VP9), a set of 10 QP (quantization parameter) values should be
  specified (in a separate document on Internet video codec testing)
  and corresponding quality values should be calculated. [...]

This seems to suggest ("Initially", "for the codec selected") that the
evaulation requirements are not yet complete.  Are they intended to be a
single set of requirements for the codec's development, or customized to
some per-application requirements?  (Is there a reference needed to
ongoing work to solidify these requirements?)

  QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) },
          i in R

I would suggest defining the argmin function.

It's surprising to see no reference to draft-ietf-netvc-testing from
this document.

Section 6

I don't see much need for a "Conclusions" section of this nature, in
this document.

Appendix B

Defining (e.g.) "high dynamic range" and "wide color gamut" with respect
to "normal" or "conventional" mechanisms does not really provide a
stable and archival reference for comparison.

[Ballot comment]
Thank you all for the work put into this document. Most of the content was way above my technical knowledge though; I would appreciate a reply to the two comments below.

== COMMENTS ==

-- Section 2.1 --

Please expand/explain 'PAM', 'RA', 'EDTV', ... before first use in table 1.

-- Section 4 --

This section is only on 'compression performance' and as I read it about compression ratio and nothing is said about packet-loss robustness which is identified in previous section as important. Am I missing something ?


-éric

[Ballot comment]
§4.2 mentions "Reference software provided to the NETVC WG for candidate codecs", but, if I understand correctly, the WG will not evaluate the candidates.

[Ballot discuss]
(1) I worry that the level of detail in the in the Compression Performance Evaluation (Section 4.1) is insufficient for implementation.  Specifically:

(a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated.”

-- How should a set of QPs be specified?

--How should the quality values be calculated?

-- What does the text “(in a separate document on Internet video codec testing)” mean?

(b) Per “A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document ", what document is that?  Is it draft-ietf-netvc-testing?

(2) Per the Security Considerations Section (Section 5)

-- What does “codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size” mean?

-- Please add additional language that codec should be written in a defensive style as they will be processing untrusted input.

[Ballot comment]
(1) There is something odd about the document formatting – the title and the first author last name in the footer is wrapped in “< … >”

(2) It would be helpful to forward reference that acronyms are explained in Appendix A. 

(3) This draft uses the words should and must to prescribe action.  Why wasn’t RFC2119 cited to explain these words?

(4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be helpful

(5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I initiate read that as a qualitative measure.  However, the text says “up to 10x and more” so that implies some quantitative measure.  What is that?

(6) Section 2.1.  Expand QP values on first use

(7) Section 2.x.  The language around content doesn’t appear to be consist.  For example:

-- Section 2.1, Internet Video Streaming says “movies, TV-series and shows, and animation.”

-- Section 2.2, IPTV says “television content”

-- Section 2.5, Screen casting says “business presentations …, animation (cartoons), gaming content, data visualization, …, virtual desktop infrastructure (VDI), screen/desktop sharing and collaboration, supervisory control and data acquisition (SCADA) display, automotive/navigation display, cloud gaming, factory automation display, wireless display, display wall, digital operating room (DiOR), etc. 

What the difference between Section 2.1’s animation and Section 2.5’s cartoons?

What’s the different between Section 2.1’s “movies, TV series …” and Section 2.2’s “television content”?

(8) Section 2.5.  The sentence “Currently, …” is very challenging to parse as it includes inline “i.e.,” and “etc”.

(9) Section 2.5.  Per “powerpoint, word documents”, these are specific Microsoft products.  I recommend using more generic names.

(10) Section 4.  I found it confusing that an evaluation methodology was in a requirements document.  I would have expected it in the draft-ietf-netvc-testing

(11) Section 4.1.  VP9 needs a reference.

(12) Section 6.  I don’t think this entire section is necessary.

(13) Editorial Nits:
-- Section 3.  Style nit.  s/chapter/section/

-- Section 4.1. Typo.  s/computged/computed

[Ballot comment]
I would expect that this document also mentions the interaction with congestion control, e.g. see RMCAT. All I can find is this, which seems very high level:
"3.1.7. Specifications providing integration with system and delivery
  layers should be developed."

Further the security consideration section is rather brief while it actually does mention security requirements. In a requirement document I would have expected that these requirements are clearly spelled out and explained, as well as risks/attacks are discussed.

And why is section 4 in this document and not in draft-ietf-netvc-testing?

Another minor comment:
Sec 2: "for instance, wired channels are considerably more error-
  free than wireless channels and therefore require different QoS
  approaches."
I don't think this is generally true as most mobile networks repair losses on the lower layers.

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has reviewed draft-ietf-netvc-requirements-09, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2019-06-04):

From: The IESG
To: IETF-Announce
CC: adam@nostrum.com, netvc-chairs@ietf.org, draft-ietf-netvc-requirements@ietf.org, mzanaty@cisco.com, Mo Zanaty , video-codec@ietf.org
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  () to Informational RFC


The IESG has received a request from the Internet Video Codec WG (netvc) to
consider the following document: - ''
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2019-06-04. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the beginning of
the Subject line to allow automated sorting.

Abstract


  This document provides requirements for a video codec designed
  mainly for use over the Internet. In addition, this document
  describes an evaluation methodology needed for measuring the
  compression efficiency to ensure whether the stated requirements are
  fulfilled or not.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested?

This document is Informational, as it specifies the requirements for an internet video codec.

(2) The IESG approval Document Announcement Write-Up:

Technical Summary

This document specifies the requirements for a video codec designed mainly for use over the internet, encompassing a wide range of interactive and non-interactive applications. Evaluation methodology is also specified for quantitative requirements such as compression efficiency.

Working Group Summary

There is broad working group consensus on the document contents, with no notable exceptions. The working group deliberated on whether to publish this as an RFC or keep as a living document during codec candidate development. The deciding factors were the requirements had become stable, and other consortiums (Alliance for Open Media) were also using this document for their codec development activities.

Document Quality

There have been several reviews of this document from several working group participants, as well as external review from other consortiums (Alliance for Open Media). Andrey Norkin, who also participates in AOM, became a co-author to maintain alignment between working group review and external (AOM) review. The first WGLC had extensive review comments, resulting in Andrey becoming a co-author. The last WGLC had no comments, as the document is mature and stable.

Personnel

Mo Zanaty is the Document Shepherd.
Adam Roach is the Responsible Area Director.

(3) Document Shepherd Review

The Document Shepherd has reviewed this version of the document and prior versions. Shepherd feedback on prior versions has been addressed sufficiently in this latest version.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No concerns about the level of reviews, which have been sufficient to ensure quality.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No particular or broader IETF review is needed. External review (AOM) was needed to ensure alignment with similar external activities (AV1 codec development).

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No specific concerns or issues with this document.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Each author confirmed to the Document Shepherd that they are not aware of any IPR requiring disclosure per BCP 78 and BCP 79.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No IPR disclosures have been filed that reference any version of this document or prior documents it replaces.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong broad consensus with no notable exceptions.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No discontent has been expressed.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

1 ID nit found, Appendix A. title extends to column 75 not 72.
This will be fixed with the update after AD/IESG review.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No formal reviews required.

(13) Have all references within this document been identified as
either normative or informative?

Although this document is Informational, it includes both normative and informative references.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

No normative references to any draft.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

No downward normative references.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No status change of existing RFCs.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

No IANA considerations.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

No new IANA registries.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

No formal language text.

(1) What type of RFC is being requested?

This document is Informational, as it specifies the requirements for an internet video codec.

(2) The IESG approval Document Announcement Write-Up:

Technical Summary

This document specifies the requirements for a video codec designed mainly for use over the internet, encompassing a wide range of interactive and non-interactive applications. Evaluation methodology is also specified for quantitative requirements such as compression efficiency.

Working Group Summary

There is broad working group consensus on the document contents, with no notable exceptions. The working group deliberated on whether to publish this as an RFC or keep as a living document during codec candidate development. The deciding factors were the requirements had become stable, and other consortiums (Alliance for Open Media) were also using this document for their codec development activities.

Document Quality

There have been several reviews of this document from several working group participants, as well as external review from other consortiums (Alliance for Open Media). Andrey Norkin, who also participates in AOM, became a co-author to maintain alignment between working group review and external (AOM) review. The first WGLC had extensive review comments, resulting in Andrey becoming a co-author. The last WGLC had no comments, as the document is mature and stable.

Personnel

Mo Zanaty is the Document Shepherd.
Adam Roach is the Responsible Area Director.

(3) Document Shepherd Review

The Document Shepherd has reviewed this version of the document and prior versions. Shepherd feedback on prior versions has been addressed sufficiently in this latest version.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No concerns about the level of reviews, which have been sufficient to ensure quality.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No particular or broader IETF review is needed. External review (AOM) was needed to ensure alignment with similar external activities (AV1 codec development).

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No specific concerns or issues with this document.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Each author confirmed to the Document Shepherd that they are not aware of any IPR requiring disclosure per BCP 78 and BCP 79.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No IPR disclosures have been filed that reference any version of this document or prior documents it replaces.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong broad consensus with no notable exceptions.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No discontent has been expressed.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

1 ID nit found, Appendix A. title extends to column 75 not 72.
This will be fixed with the update after AD/IESG review.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No formal reviews required.

(13) Have all references within this document been identified as
either normative or informative?

Although this document is Informational, it includes both normative and informative references.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

No normative references to any draft.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

No downward normative references.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No status change of existing RFCs.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

No IANA considerations.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

No new IANA registries.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

No formal language text.