RTP Payload for Society of Motion Picture and Television Engineers (SMPTE) ST 291-1 Ancillary Data
draft-ietf-payload-rtp-ancillary-14

[Ballot discuss]
The grouping semantics here are problematic. Section 4.1 talks about LS to associate ANC streams with other streams. LS is strictly a synchronization construct, not used to otherwise state relationships between streams. More importantly, there's nothing that prevents it being used to synchronize an arbitrary number of video streams with each other. The problem that arises is best illustrated by this example, where I'm going to transmit two video streams, each with their own ANC data, but need them to be synchronized (e.g., perhaps I'm going to have two border-less screens physically right next to each other for a double-wide image, and synchronization is important to avoid tearing):

        v=0
        o=Al 123456 11 IN IP4 host.example.com
        s=Professional Networked Media Test
        i=A test of synchronized video and ANC data
        t=0 0
        a=group:LS V1 V2 M1 M2
        m=video 50000 RTP/AVP 96
        c=IN IP4 233.252.0.1/255
        a=rtpmap:96 raw/90000
        a=fmtp:96 sampling=YCbCr-4:2:2; width=1280; height=720; depth=10
        a=mid:V1
        m=video 50002 RTP/AVP 96
        c=IN IP4 233.252.0.1/255
        a=rtpmap:96 raw/90000
        a=fmtp:96 sampling=YCbCr-4:2:2; width=1280; height=720; depth=10
        a=mid:V2
        m=video 50010 RTP/AVP 97
        c=IN IP4 233.252.0.2/255
        a=rtpmap:97 smpte291/90000
        a=fmtp:97 DID_SDID={0x61,0x02};DID_SDID={0x41,0x05}
        a=mid:M1
        m=video 50012 RTP/AVP 97
        c=IN IP4 233.252.0.2/255
        a=rtpmap:97 smpte291/90000
        a=fmtp:97 DID_SDID={0x61,0x02};DID_SDID={0x41,0x05}
        a=mid:M2

If the recipient of these mutually synchronized streams needs to reintegrate the ANC data into the video for offloading to an HDMI connection, how does it know which ANC stream to insert into which video stream?

There needs to be some additional association mechanism here. You can either add an attribute to the smpte291 media sections that clearly indicates which video stream they correspond to, or you can include the smpte291 packets in the same stream as the corresponding video, like so:

        v=0
        o=Al 123456 11 IN IP4 host.example.com
        s=Professional Networked Media Test
        i=A test of synchronized video and ANC data
        t=0 0
        m=video 50000 RTP/AVP 96 97
        c=IN IP4 233.252.0.1/255
        a=rtpmap:96 raw/90000
        a=rtpmap:97 smpte291/90000
        a=fmtp:96 sampling=YCbCr-4:2:2; width=1280; height=720; depth=10
        a=fmtp:97 DID_SDID={0x61,0x02};DID_SDID={0x41,0x05}
        m=video 50002 RTP/AVP 96 97
        c=IN IP4 233.252.0.1/255
        a=rtpmap:96 raw/90000
        a=rtpmap:97 smpte291/90000
        a=fmtp:96 sampling=YCbCr-4:2:2; width=1280; height=720; depth=10
        a=fmtp:97 DID_SDID={0x61,0x02};DID_SDID={0x41,0x05}

(See RFC4733 for an example of a payload that performs stream association in this fashion)

[Ballot comment]
It is common in examples of RTP payloads (such as Figure 1) to include values for those fields whose values are illustrative of the format itself. The "V=2" in Figure 1 is an example of this. I would suggest that the example would be slightly improved by including a similar notation for (e.g.) "Length = 32", "ANC_Count = 2", and the two Data_Count fields.

Section 2.1 defines "b01" as "invalid" for the F field. It does not, however, prescribe what recipients of such a value should do; reasonable options would include (a) treat it as 0b00; (b) ignore the ANC but process any others that might be present; or (c) ignore the entire packet.  I'll note that (b) allows you to come back and add semantics for "0b01" at a later date, if you find a reason to do so. In any case, to minimize variations among recipients, I would suggest defining concrete behavior here.

[Warning: I'm not terribly familiar with SMPTE video formats, and with the UHD-and-higher specs in particular, so it may be that the following comment can be answered with "that can never happen no matter how high-res streams get in the future."] As far as I know, SMPTE defines 4k as having 4 streams (each efectively a normal 1080 stream), 8k as having 16, and the future-looking 16k (presumably) having 64. Given that the Horizontal_Offset can communicate widths only up to 4093 samples (well within striking distance of the 2640 samples used by a single stream), and that the StreamNum field can communicate at most 128 streams, it seems that any sizes larger than 16k run the risk of exceeding the ability for these fields to place ancillary data packets at the correct horizontal position. I would recommend defining another reserved value for Horizontal_Offset (0xFFD), that can be used to mean "longer field follows" in a future specification; and a reserved value (0x7F) for StreamNum for the same purpose. Line_Number, with a maximum value of 2045, is similarly within striking distance of the 1125 lines used by a single stream, and would likely need the same accommodations.

It is implied, but not stated, that implementations should use different sessions (or at least different media sections) to convey smpte291 payloads than the video streams they correspond to. If this is the intention, please include a normative statement to that effect. If it is okay to include both payload types in the same media section, please add a statement explicitly saying so, and indicate that this can be used as an alternative to using LS groups.

Section 5.1 indicates that the answerer can respond with "all or a subset" of the DID_SDID parameters, or that it "MAY reject the offer."  Of course, an answerer is always able to reject an offer, but this text seems to imply that it might do so if it doesn't want to receive any of the DID_SDID parameters indicated in the offer. That seems like massive overkill. Since the model here appears to be using multiple sessions (or, at least, separate media sections) to convey smpte291 data, I would offer that this should instead say something like "MAY set the corresponding port number to 0 to decline the smpte291 stream". (If you opt to allow the smpte291 payloads to be in the same media section as their corresponding video streams, this would instead involve removing the corresponding PT).

Nit: The final line in the final example of section 4 is too long. Consider outdenting by two characters.

[Ballot comment]
I wasn’t sure whether this text was giving three examples of SDIs, or two. Am I the only one who’s confused?

  ANC is generally associated with the carriage of metadata within the
  bit stream of a Serial Digital Interface (SDI) such as SMPTE ST 259
  [ST259], the standard definition (SD) Serial Digital Interface (with
  ANC data inserted as per SMPTE ST 125 [ST125]), or SMPTE ST 292-1
  [ST292], the 1.5 Gb/s Serial Digital Interface for high definition
  (HD) television applications.

Perhaps using the descriptions first in each case, and ending with the name and reference in parentheses, would be clearer?

[Ballot comment]
One comment at the end of section 2:
"One millisecond is a reasonable upper bound for the
  amount of time between when an ANC data packet becomes available to a
  sender and the emission of an RTP payload containing that ANC data
  packet."
While it makes sense to send out the packet as soon as possible, I'm not sure what this sentence gives you. I don't think you can guanrantee 1ms as there might be additional delays on the NIC and as such I don't think there are any actions that could follow based on this value. To avoid that anybody is taking this as a hard requirement, I would maybe rather just remove this note.

[Ballot comment]
This is generally a well written document.

As the media type registration template in Section 3.1 is supposed to be self contained (it will be posted to IANA website as a web page), it should define (or point to) ABNF for different parameters and specify that some parameters (e.g. DID_SDID) can be repeated multiple times. I can find this information in Section 4, so please add references to section 4 in the template.

[Ballot comment]
I'd like to begin by noting that I know basically nothing about SMPTE or ST 291 or Ancillary Data. I opened the document thinking that this was going to be either impenetrable, a huge waste of my time, or both...

I was very pleasantly surprised by how clear, accessible and interesting the document is -- it clearly describes the use case / problem, provides just enough background to make understanding the discussion (relatively) easy, and documents the solution / protocol in a clear and concise manner.

I'm not really qualified to evaluate if this is the best way to implement / if it has hidden landmines (see first sentence!), but I'd like to thank the author and WG for producing a document which was a pleasure to read.

I'd also like to draw your attention to Nevil's OpsDir review, which contains some good questions, and some nits: https://datatracker.ietf.org/doc/review-ietf-payload-rtp-ancillary-06-opsdir-lc-brownlee-2016-11-03/

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Services Operator has completed its review of draft-ietf-payload-rtp-ancillary-10.txt. If any part of this review is inaccurate, please let us know.

The IANA Services Operator understands that, upon approval of this document, there is a single action which we must complete.

In the video registry on the Media Types registry page located at:

https://www.iana.org/assignments/media-types/

a single, new media type will be added as follows:

Name: smpte291
Template: [ TBD-at-registration ]
Reference: [ RFC-to-be ]

The IANA Services Operator understands that this is the only action required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is only to confirm what actions will be performed.

Thank you,

Sabrina Tanamal
IANA Services Specialist
PTI

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC: ben@nostrum.com, payload-chairs@ietf.org, draft-ietf-payload-rtp-ancillary@ietf.org, payload@ietf.org, acbegen@gmail.com
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (RTP Payload for SMPTE ST 291 Ancillary Data) to Proposed Standard


The IESG has received a request from the Audio/Video Transport Payloads WG
(payload) to consider the following document: - 'RTP Payload for SMPTE ST 291
Ancillary Data'
  as Proposed Standard

This is a repeated IETF Last Call due to material changes since the original
last call of version 6.

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2017-07-25. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the beginning of
the Subject line to allow automated sorting.

Abstract


  This memo describes a real-time transport protocol (RTP) payload
  format for the Society of Motion Picture and Television Engineers
  (SMPTE) Ancillary data, as defined by SMPTE ST 291-1.  SMPTE
  Ancillary data is generally used along with professional video
  formats to carry a range of ancillary data types, including time
  code, Closed Captioning, and the Active Format Description (AFD).




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-ancillary/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-ancillary/ballot/


No IPR declarations have been submitted directly on this I-D.

(Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs:

The IANA Services Operator has completed its review of draft-ietf-payload-rtp-ancillary-06.txt. If any part of this review is inaccurate, please let us know.

Upon approval of this document, we understand that there is a single registry action to complete.

In the video media types registry located at:

https://www.iana.org/assignments/media-types/

a single new media type is to be registered as follows:

Name: smpte291
Template: [ TBD-at-registration ]
Reference: [ RFC-to-be ]

We understand that these are the only actions required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is only to confirm what actions will be performed.

Thank you,

Sabrina Tanamal
IANA Services Specialist
PTI

The following Last Call announcement was sent out:

From: The IESG
To: "IETF-Announce"
CC: ben@nostrum.com, "Ali C. Begen" , payload-chairs@ietf.org, draft-ietf-payload-rtp-ancillary@ietf.org, payload@ietf.org, acbegen@gmail.com
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (RTP Payload for SMPTE ST 291 Ancillary Data) to Proposed Standard


The IESG has received a request from the Audio/Video Transport Payloads
WG (payload) to consider the following document:
- 'RTP Payload for SMPTE ST 291 Ancillary Data'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2016-11-03. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This memo describes an RTP Payload format for SMPTE Ancillary data,
  as defined by SMPTE ST 291-1.  SMPTE Ancillary data is generally used
  along with professional video formats to carry a range of ancillary
  data types, including time code, Closed Captioning, and the Active
  Format Description (AFD).




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-ancillary/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-ancillary/ballot/


No IPR declarations have been submitted directly on this I-D.

This is my AD evaluation of draft-ietf-payload-rtp-ancillary-05. I'd like to at least resolve
the substantive comments before going to IETF last call.

*** Substantive Comments *** :

- General:
There's a lot of use of 2119 keywords in situations where you talk more about definitions/
statements of fact than procedure. I mention some specific instances below, but may not
have caught all of them.

- Section 1:

A quick mention of the meaning of horizontal and vertical ancillary spaces would be
helpful.

The 6th paragraph says ancillary data flag (ADF) word is not carried in this payload.  But
section 2.1, under "horizontal offset" says "A value of 0 means that the Ancillary Data
Flag (ADF) of the ANC data packet begins immediately following SAV." These seem to
conflict--are they talking about something different?

- section 2:
-- timestamp:
"For progressive scan video, the timestamp SHALL denote the
sampling instant of the frame to which the ancillary data in
the RTP packet belongs."

That's a definition. I suggest "... timestamp denotes..." (occurs twice)

-- Marker Bit:

"The marker bit set to "1" SHALL indicate the last ANC RTP packet ..."

Consider "The marker bit set to "1" indicates..."

- section 2.1:
Am I correct to assume the various length or count fields are unsigned integers? Network
byte order?

-- ANC count:
"A single ANC RTP packet payload SHALL NOT carry more than 255 ANC data packets."
Isn't this a statement of fact, since the number is limited by the 8 bit ANC_Count field?
If so, consider s/SHALL NOT/cannot

On the other hand, the "may" in "additional RTP packets carrying ANC data may be sent
with the same RTP timestamp but with different sequence numbers." seems to be an
appropriate place for a 2119 "MAY"

"ANC_Count of 0 SHALL indicate that there are no ANC data packets"
Please consider s/SHALL indicate/indicates

-- Horizontal Offset:
All 3 instances of "SHALL" seem more like statements of fact.
 
- 3.1, Required Parameters:
"When an ANC RTP stream is to be associated with an RTP video stream, the RTP timestamp
rates SHOULD be the same to ensure that ANC data packets can be associated with the
appropriate frame or field."

Why not MUST? Do you anticipate that there might be situations where it's reasonable for
the rates not to match?

-- intended usage: Is "Common" the right choice? Do you expect this to be widely used, or
just in some very specific applications?

- 3.2, encoding of the subtype name:
It would be helpful to mention the "rate" parameter here as well.

-- mapping of "DID_SDID" parameters:
It would be helpful to mention the ability to carry multiple types earlier in the document
(perhaps where types are first mentioned, or where DID_SDID was first mentioned.)

- 5: The security considerations do not include the recommended text from
draft-ietf-payload-rtp-howto-14, section A.13. Should it?

- References: It seems like the references to [RFC3264] and [RFC5888] should be normative.

*** Editorial Comments ***:

- General: An early description of the meaning of ANC data packet types would be helpful.

- section 1: A quick mention of the meaning of horizontal and vertical ancillary spaces
would be helpful.

- section 1.1, last paragraph: By "data model", does this talk about the packet format, or
something else?

-3.1, optional parameters:
Is this link reasonably permanent? Also, consider moving this to the references section
(perhaps in a separate "URLs" section) and citing it normally.

It would be helpful to mention that DID_SDIS can occur multiple times (or be a list of
values, depending on your perspective.)

-- Change controller:
I suspect this should say “ The IETF PAYLOAD working group, or other party as designated
by the IESG.”"

- 3.2.1

What does "essence" mean in the context of "video essence stream"? Am I correct to assume
it refers to the actual video content?

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

Proposed standard (as other payload RFCs).

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

This memo describes an RTP Payload format for SMPTE Ancillary data,
  as defined by SMPTE ST 291-1.  SMPTE Ancillary data is generally used
  along with professional video formats to carry a range of ancillary
  data types, including time code, Closed Captioning, and the Active
  Format Description (AFD).

Working Group Summary

The document was presented once during a meeting and follow-up revisions have been made based on the WG consensus.

Document Quality

  Are there existing implementations of the protocol? Have a
  significant number of vendors indicated their plan to
  implement the specification?

There are two known implementations: BBC Research & Development and Snell Advanced Media. The following vendors have committed to implement the protocol: COVELOZ, Imagine Communications, Lawo, Grass Valley, Utah Scientific.

Personnel

  Who is the Document Shepherd? Who is the Responsible Area
  Director?

Ali Begen is the doc shepherd, Ben Campbell is the responsible AD.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

It is ready for the IESG processing.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No, multiple reviewers have performed a review.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

The author confirmed he was not aware of any IPR.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No IPR disclosure has been submitted.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

No nits.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

Media Type review was requested August 9, 2016. Minor comments have been received on August 11, and they were fixed in -05.

(13) Have all references within this document been identified as
either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

No.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

There are non-RFC normative references. BT1120, BT1700 and ST291, which are ITU and SMPTE documents.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No change to other RFCs.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

IANA considerations section has been reviewed, and no issues have been identified.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

No new registries.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

Nits and BNF rules have been checked. No issues.