Deterministic Networking (DetNet) Data Plane Framework
RFC 8938

Note: This ballot was opened for revision 04 and is now closed.

(Deborah Brungard) Yes

(Alissa Cooper) No Objection

Roman Danyliw No Objection

Martin Duke No Objection

Comment (2020-04-21 for -04)
No email
send info
My colleagues have covered the nits quite well.

Benjamin Kaduk No Objection

Comment (2020-04-23 for -04)
Other than some remarks on the security considerations, I basically just
have a bunch of editorial comments.  I've tried to deduplicate with what was
already reported by other ADs, but no doubt have kept a few things in that
have already been mentioned.

Section 1

   The DetNet Architecture models the DetNet related data plane
   functions decomposed into two sub-layers: a service sub-layer and a
   forwarding sub-layer.  The service sub-layer is used to provide

nit: "models" needs two objects to act on/compare, so maybe "as being
decomposed into".

   replicated in other forwarding technologies.  Most of DetNet benefits
   can be gained by running over a data link layer that has not been
   specifically enhanced to support all TSN capabilities but for certain
   networks and traffic mixes delay and jitter performance may vary due
   to the forwarding sub-layer intrinsic properties.

nit: I think the "certain networks and traffic mixes" are supposed to be
contingent on "a data link layer that [is not a TSN]" but the current text
does not quite match that.  Perhaps "certain such networks" is the smallest
fix, though it does conflate "data link layer" and "network" in an
unfortunate manner.

Section 3

   connectivity function of the forwarding sub-layer.  An example of
   this is Packet Replication, Elimination, and Ordering functions see
   Section 4.3.  The ordering (POF) uses sequence numbers added to

nit: the punctuation around the reference should be tweaked.

   The method of instantiating each of the layers is specific to the
   particular DetNet data plane method, and more than one approach may
   be applicable to a given bearer network type.

What is a "bearer network"?

Section 3.1.1

   applying existing standardized headers and/or encapsulations.  The
   Detnet forwarding sub-layer may provide capabilities leveraging that
   same header or encapsulation technology (e.g., DN IP or DN MPLS) or
   it may be achieved by other technologies (e.g., Figure 2).  DetNet is

nit: Figure 2 is not a technology; maybe "as shown in Figure 2".

Section 3.1.2

   number) in packets.  For example, in DetNet IP, zero encapsulation is
   used and no sequence number is available, and in DetNet MPLS, DetNet
   specific information may be added explicitly to the packets in the
   format of S-label and d-CW [I-D.ietf-detnet-mpls] .

nit: s/format of/form of/

Section 3.3

   Number (for PREOF) for each DetNet flow.  The DetNet Service sub-
   layer requires both; the DetNet forwarding sub-layer requires only
   Flow-ID.  Metadata can also be used for OAM indications and

nit(?): to say that "the service sublayer requires both" to me implies that
it always and unconditionally requires both, but IIUC the PREOF functions
are optional, and so for a given flow the service sublayer might not require
the sequence number.  Perhaps swapping it around to be structured more like
"the flow-ID is used by both the service and forwarding sub-layers, but the
sequence number is only used by the service layer" would help?

   Some MPLS examples of implicit metadata include the sequence number
   for use by the PREOF function, or even all the essential information
   being included into the DetNet over MPLS label stack (the DetNet
   Control Word and the DetNet Service label).

I would have thought that the detnet elements in the label stack would be
considered explicit metadata, but perhaps I'm just looking through the wrong
lens.

Section 3.4

   One method of operating an IP DetNet data plane without encapsulation
   is to use "6-tuple" based flow identification, where "6-tuple" refers
   to information carried in IP and higher layer protocol headers.
   General background on the use of IP headers, and "6-tuples", to
   identify flows and support Quality of Service (QoS) can be found in
   [RFC3670].  [RFC7657] provides useful background on differentiated

I did not see the explicit phrase "6-tuple" used in RFC 3670, so I don't
think this text is quite right as written.  (7657 does talk about it, but is
not currently being referenced for that purpose.)

Section 3.6.1.5

Is there a difference between "packet-by-packet distribution [of the same
flow]" and "packet-by-packet load sharing"?

Section 3.6.1.6

   as those in use by IP and MPLS networks.  At the Application layer a
   client of a DetNet service can use existing techniques to detect and
   monitor delay and loss.

[Is there a good reference for these "existing techniques"?  I recognize
that we shouldn't go into extensive detail here, and absent a single
consolidated reference, no-reference may be best.]

Section 3.6.2

   Service protection allow DetNet services to increase reliability and
   maintain a DetNet Service Assurance in the case of network congestion
   or network failure.  Detnet relies on the underlying technology

My understanding is that the whole point of DetNet was to provide
reliability in the face of network congestion (e.g., by resource
reservation).  So is it really only the "network failure" case that benefits
from service protection?

Section 3.6.3.1

   IP aggregation has both data plane and controller plane aspects.  For
   the data plane, flows may be aggregated for treatment based on shared
   characteristics such as 6-tuple.  Alternatively, an IP encapsulation

In what way would 6-tuple be a "shared characteristic" for *different*
flows?  I thought 6-tuple was typically used as a unique flow identifier...
(I see in Section 4.2.1 we refer to flows that "share 6-tuple attributes".)

Section 4.1

   However, from a practical and implementation standpoint, they are not
   equivalent at all.  Some approaches are more scalable than others in
   terms of signaling load on the network.  Some can take advantage of
   global tracking of resources in the DetNet domain for better overall
   network resource optimization.  Some are more resilient than others
   if link, node, or management equipment failures occur.  While a
   detailed analysis of the control plane alternatives is out of the
   scope of this document, the requirements from this document can be
   used as the basis of a later analysis of the alternatives.

side note: using "some" so much requires the reader to know which are which
... that may be reasonable, but perhaps we want the reader's brainpower
working on other things.

Section 4.2

nit: RestConf and YANG probably are best considered together as a single
"management mechanism" for these purposes.

Section 4.2.2

   DetNet application with the required service.  A requirement for the
   DetNet Controller Plane will be the ability to assign a particular
   identified DetNet IP flow to a path through the DetNet domain that
   has been assigned the required nodal resources.  This provides the

nit: I don't think that "nodal" is quite the right word, here, as it refers
to a property of the (network) graph more than the specific element
colocated with the node in the graph.  "per-node" seems like a better fit,
to me.

Section 4.3

   domain requires explicit support.  There may be capabilities that can
   be used, or extended, for example GMPLS end-to-end recovery [RFC4872]
   and GMPLS segment recovery [RFC4873].

nit: maybe this is "existing capabilities"?

Section 5

I guess there's some standard security considerations for when flow
aggregation is in play, namely that misbehavior from one component flow can
affect sibling flows as collateral damage.

   Security considerations for DetNet are described in detail in
   [I-D.ietf-detnet-security].  General security considerations are

The referenced document (now at -09) seems significantly improved from when
I previously made an in-passing review of the -03
(https://mailarchive.ietf.org/arch/msg/detnet/jZLodXBmQa7ZFDbBIvDO_xCDD0E/),
however, some of the issues I mentioned there remain, at least in part
(e.g., mentioning the use of HMAC without a colocated discussion of the need
for key distribution and having a taxonomy of threats titled "threat
model").  I would like to know what the current maturity state of the
detnet-security document is believed to be, so as to judge how appropriate
it is for the data-plane-framework to refer to it in this manner.

   described in [RFC8655].  This section considers general security

I think this is more like "Architecture-level DetNet" than "Generic", as the
current formulation sounds like it should be referring to BCP 72 :)

   Security aspects which are unique to DetNet are those whose aim is to
   provide the specific quality of service aspects of DetNet, which are
   primarily to deliver data flows with extremely low packet loss rates
   and bounded end-to-end delivery latency.

I can't find a way to read this other than "security aspects [...] whose aim
is to provide the specific quality of service aspects", and I'm puzzling at
why it's phrased in this way.  Specifically, I thought that *providing* the
QoS aspects is the core role of DetNet, and thus that the *security* aspects
would be protecting those aspects in the face of (some class of) attackers.
Would it be appropriate to replace this with something like the following?

% Part of what makes DetNet unique is its ability to provide specific and
% reliable quality of service (delivering data flows with extremely low
% packet loss rates and bounded end-to-end delivery latency), and the
% security-related aspects of protecting that quality of service are
% similarly unique.

This might also be a good place to note that correct operation of the PREOF
functions is pretty critical, and failures there could lead to pretty
catastrophic situations for the operational technology relying on them.  I
don't think there are ways for an attacker to try to attack them directly,
but Murphy's Law can be a pretty effective attacker, too.

   The primary considerations for the data plane is to maintain
   integrity of data and delivery of the associated DetNet service
   traversing the DetNet network.  Application flows can be protected

I'd suggest leading in to this with "As for all communications protocols,"
to be clear that this paragraph is not trying to cover the bits that are
"unique to DetNet" as much as remind the reader of general best practices.

   At the management and control level DetNet flows are identified on a
   per-flow basis, which may provide controller plane attackers with
   additional information about the data flows (when compared to
   controller planes that do not include per-flow identification).  This
   is an inherent property of DetNet which has security implications
   that should be considered when determining if DetNet is a suitable
   technology for any given use case.

I might also note that attackers with access to the controller plan already
have pretty significant attacks available to them.

   To provide uninterrupted availability of the DetNet service,
   provisions can be made against DOS attacks and delay attacks.  To
   protect against DOS attacks, excess traffic due to malicious or
   malfunctioning devices can be prevented or mitigated, for example
   through the use of existing mechanism such as policing and shaping
   applied at the input of a DetNet domain.  To prevent DetNet packets

In my head this would include things like disabling switch ports that are
receiving traffic floods that might overload the switch CPU, in order to
protect the DetNet traffic on other ports.  I don't see much value in saying
that in the document, though, since it's more of an operational practice
thing.

Section 9.1

RFC 3209 is cited only once, and it does not seem like a particularly
normative one.  Similarly, RFC 3473 is cited only as a "such as" example
(though several times).

Erik Kline No Objection

Murray Kucherawy No Objection

Comment (2020-04-12 for -04)
Editorial nits mostly.  The more interesting stuff first:

Throughout the document, various terms are sometimes capitalized and sometimes not.  A pass should be done to make this uniform throughout the document.  This becomes especially confusing in Section 3.6.2.1.  Examples: "App-flows". "Service sub-layer", "Forwarding sub-layer", "Packet".

Section 2.2:
* "CW" by itself doesn't appear anywhere in this document.  Neither do "LSR", "MPLS-TE", or "PW".

Section 3.1:
* Given the sentence in 3.1, I would expect the "technology" and "encapsulation" sections to be at the same level in the Table Of Contents tree, but they aren't.  That is, since "Data Plane Technology" is 3.1.1, I would expect "Encapsulation" to be 3.1.2.  Is some adjustment in order here?

Section 3.3:
* This paragraph needs a bit of editing:
-- OLD --
   Metadata can be included implicit or explicit.  Explicit means that a
   dedicated header field is used to include metadata in a DetNet
   packet.  In case of implicit method a part of an already existing
   header field is used to encode the metadata.
-- SUGGEST --
   Metadata inclusion can be implicit or explicit.  Explicit inclusions
   involve a dedicated header field that is used to include metadata
   in a DetNet packet.  In the implicit method, part of an already existing
   header field is used to encode the metadata.

Section 3.6.1.2:
* "Use of a specific path for a flow." -- this isn't a sentence; suggest: "A flow can be routed over a specific, pre-computed path."

Section 3.6.1.3:
* "Use of multiple packet streams using multiple paths, for example 1+1 or 1:1 linear protection." -- again not a sentence; maybe start with "Service protection involves use of ..."

Section 3.6.1.8:
* "... are currently out of scope for this document." -- remove "currently"; these issues won't be in scope for this document later

Section 3.6.2.1:
* "An OAM scheme that monitors the paths detects the loss of path or traffic is required to initiate the switch." -- I can't parse this sentence.  Maybe "detects" should be "to detect"?

Section 3.6.3:
* "the sum of the reservations should be the sum of all the individual reservations" -- there may be a term of art in play here, but that phrase seems self-redundant; what else could the latter be but the former?

Section 4.2.1:
-- OLD --
         The assignment of resources along the path depends on the
         technology and it includes assignment of specific links and
         coordination of the queuing and other traffic management
         capabilities such as policing and shaping.
-- NEW --
The assignment of resources along the path depends on the technology
and includes assignment of specific links, coordination of queueing, and
other traffic management capabilities such as policing and shaping.

And now the boring stuff, to save the RFC Editor some work:

Section 1:
* "Different application flows (e.g., Ethernet, IP, etc.), can be mapped on top of DetNet." -- remove the comma

Section 3:
* "Figure 1 reproduced from the [RFC8655],shows ..." -- remove "the" and replace the comma with a space
* "The ordering (POF) uses sequence numbers ..." -- should probably be "The ordering function (POF) uses sequence numbers ..."

Section 3.3:
* "... can provide or carry metadata:" -- add "the following" after "carry"

Section 3.4:
* "... for example GRE, IPSec etc." -- comma after "IPSec"

Section 3.5:
* "... the d-CW [I-D.ietf-detnet-mpls], [RFC4385],can be used." -- add a space before "can"
* "... no architectural constraint on its size of this structure ... -- s/its/the/

Section 3.6.1.2:
* "... for a certain characteristic e.g. highest ..." -- s/characteristic e.g. highest/characteristic, e.g., highest/

Section 3.6.1.3:
* "For DetNet this primarily ..." -- add a comma after "DetNet"

Section 3.6.1.4:
* "Network Coding, [nwcrg] not to ..." -- move the comma to after "[nwcrg]"
* "DetNet could utilized Network coding ..." -- s/utilized/utilize/ (or simply "use")

Section 3.6.2:
* "Service protection allow DetNet services ..." -- s/allow/allows/

Section 3.6.2.1:
* "Figure 3: Example Packet Flow in DetNet protected Network" -- capitalize "protected"
* "copy of packet 1.2 etc." -- add a comma after "1.2"
* "...entities shown in Figure 3 ." -- s/3 ./3./
* "... traffic at a time, could use the ..." -- remove comma

Section 3.6.2.3:
* "Many of the same concepts apply however rings ..." -- add a comma after "apply"

Section 3.6.3:
* "When aggregating DetNet flows the flows should ..." -- add a comma after the first "flows"
* "... should be compatible i.e. the same ..." -- s/compatible i.e./compatible, i.e.,/
* "When an encapsulation is used the choice ..." -- add comma after "used"

Section 3.6.5:
* "L2 represent a generic ..." s/represent/represents/

Section 4.1.:
* The second citation of RFC8655 in the first paragraph is probably not needed.

Section 4.2:
* "While management plane and control planes ..." -- maybe "While the management and control planes ..."
* Add a comma after "For example" in the last paragraph.

Section 4.2.1:
* "... provisioned or requested one or more ..." -- add comma after "requested"

Section 4.2.4:
-- OLD --
An example control plane solution for MPLS can be
   found in [RFC3473] , [RFC6387] and [RFC7551].
-- NEW --
Example control plane solutions for MPLS can be found in
[RFC3473], [RFC6387], and [RFC7551].

Warren Kumari No Objection

Comment (2020-04-22 for -04)
No email
send info
Doh! I had collected a whole pile of editorial nits and comments, but see that others have already covered them -  that'll larn me to procrastinate...

(Barry Leiba) No Objection

Comment (2020-04-14 for -04)
Thanks for this document.  Just lots of editorial comments here:

The RPC will need to do a lot of comma editing.  I’m not going to call them all out here.

— Section 1 —

   It describes the function
   and operation of the Packet Replication (PRF) Packet Elimination
   (PEF) and the Packet Ordering (POF) functions

You need commas between the items in the list.  I also find “the fuction of the functions” to be very odd, and suggest eliminating “functiin and” from the beginning of the sentence.

   Furthermore, it also describes the forwarding sub-layer.

“Furthermore” and “also” together is redundant, and I suggest eliminating one of them.

   Different application flows (e.g., Ethernet, IP, etc.) can be mapped

“e.g.” and “etc.” together is redundant, and I suggest “Different application flows, such as Ethernet or IP, can be mapped”.  If you keep the parentheses, the comma needs to disappear.

— Section 3 —

   The DetNet Architecture, [RFC8655],
   models the DetNet related data plane functions

You don’t need the commas to set off the citation, as the brackets already do that.  And “DetNet-related” needs a hyphen.

   Figure 1 reproduced from the [RFC8655],shows

Make this, “Figure 1, reproduced from [RFC8655], shows”

— Section 3.1.1 —

   currently defined for operation over packet switched (IP) networks or
   label switched (MPLS) networks.

Hyphenate “packet-switched” and “label-switched”.

— Section 3.1.2 —

   used and no sequence number is available, and in DetNet MPLS, DetNet
   specific information

Hyphenate “DetNet-specific”.

— Section 3.2 —

   An example of such metadata is
   the inclusion of a sequence number required by the PREOF function.

“PREOF function” is redundant, as the “F” already means “function”.  Also later in the document.

— Section 3.4 —

   may be deployed, for example GRE, IPSec etc.

“for example” and “etc.” together is redundant.

— Section 3.6.1.1 —

   Reservation of resources can allocate resources to specific DetNet
   flows.

This sounds a bit odd.  Maybe, “Resources might be reserved in order to make them available for allocation to specific DetNet flows.” ?

   Misbehaving DetNet flows
   must be able to be detected and ensure that they do not compromise
   QoS of other flows.

This sounds like it’s something that misbehaving DetNet flows have to do.  It would be better this way:

NEW
   It must be possible to detect misbehaving DetNet flows
   and to ensure that they do not compromise QoS of other flows.
END

— Section 3.6.1.2 —

   Explicit route
   computation can encompass a wide set of constraints and optimize the
   path for a certain characteristic e.g. highest bandwidth or lowest
   jitter.

This would sound better with “and can optimize”.  And “e.g.” needs commas bith before and after it, always.

— Section 3.6.1.4 —

   DetNet could utilized Network coding

Make it “utilize” (or, better, “use”).

— Section 3.6.2.1 —

   traffic over each segment of the end to end path.

Hyphenate “end-to-end” here.

   In this case there is no PRF
   function

“PRF function” is redundant.

— Section 3.6.2.2 —

   DetNet uses flow specific requirements (e.g., maximum
   number of out-of-order packets, maximum latency of the flow, etc.)
   for configuration of POF related buffers.

Hyphenate “flow-specific” and “POF-related”, and eliminate either “e.g.” or “etc.”

— Section 3.6.2.3 —

   Many of the same concepts apply however rings are

There needs to be a comma before “however”.

— Section 3.6.3 —

   How this is accomplished is data plane or control plane dependent.

NEW
   How this is accomplished is data-plane or control-plane dependent.
END

   When aggregating
   DetNet flows the flows should be compatible i.e. the same or very
   similar QoS and CoS characteristics.

NEW
   When aggregating
   DetNet flows, the flows should be compatible, i.e., have the same or
   very similar QoS and CoS characteristics.
END

— Section 3.6.5 —

   MPLS nodes may interconnected by different sub-network

“may be interconnected”

   Each of these
   sub-network technologies need to provide appropriate service

The subject is “each”, which is singular, so “needs to provide”.

— Section 4.2 —

   While management plane and control planes are traditionally
   considered separately, from the Data Plane perspective

Don’t capitalize “data plane” here.

— Section 4.2.1 —

   There are many techniques to achieve aggregation,
   for example in case of IP, it can be grouping of IP flows that share
   6-tuple attributes or flow identifiers at the DetNet sub-layer.

Very awkward sentence.  Split and rephrase:

NEW
   There are many techniques to achieve aggregation.
   For example, in the case of IP,  IP flows that share 6-tuple
   attributes or flow identifiers at the DetNet sub-layer can be
   grouped.
END

— Section 5 —

   The primary considerations for the data plane is to maintain
   integrity of data and delivery of the associated DetNet service

“consideration”, singular.

   through the use of existing mechanism such as policing and shaping

Make it “an existing mechanism” or “existing mechanisms”.

Alvaro Retana No Objection

Comment (2020-04-21 for -04)
(1) References:  I don't think that RFC3209, RFC3473 or RFC4385 need to be Normative as they seem to be used mostly as examples.


(2) [nits]

s/reproduced from the [RFC8655],shows/reproduced from [RFC8655] shows

s/[RFC4385],can be used/[RFC4385], can be used

s/constraint on its size of/constraint on the size of

s/RFC5575/draft-ietf-idr-rfc5575bis

Robert Wilton No Objection

Comment (2020-04-24 for -05)
Sorry, I ran out of time to fully review this document, but one surprise what that the title and abstract only mentions the dataplane, but then the document also has a reasonably sized section on control plane considerations, and I didn't know whether it would be helpful for the abstract to also mention this fact.