Deterministic Networking (DetNet) Data Plane: IP
draft-ietf-detnet-ip-07

I support some of the other DISCUSS items as well.


[[ recommendations ]]

[ whole document ]

* Given the use of 2-, 3-, 5-, and 6-tuple would it be a readability
  improvement if section 3 rolled them all up in to one more general term,
  like N-tuple?

  Quite possibly I have misunderstood, though.

* Instead of L2, or in the terminology definition of L2, clarify "sub-IP layer"
  so as to avoid concerns about things like "layer 2.5" and handling in other
  future link types?


[[ nits ]]

[ section 3 ]

* "IP DetNet End Systems can communicate over DetNet IP network with
   IP End System"

   The grammar toward the end of the sentence is off.

   "...over DetNet IP networks with other IP End Systems", perhaps?

* "from data plane perspective" -> "from the data plane perspective"

* "discribed" -> "described"

[ section 4.2 ]

* Suggest, maybe, "DN adjacent" instead of "DN attached".

  This use of "attached" seems to stretch to meaning of the word.

[ section 5.1.1.5 ]

* If you feel you want a reference, check out RFC 6437.

I support Alvaro’s DISCUSS points.  Otherwise, just a pile of nits here:

Section 1:

* "... defined to support IP flows, instead ..." -- "instead" should start a new sentence, or at least the comma should be a semi-colon

* "... data plane in Section 3, considerations that ..." -- add "and" before "considerations"

Section 3:

* The parentheses in the second paragraph are unbalanced.

* "Same can be valid for flow aggregates." -- To what does "Same" refer?

* "In such cases using smaller tuples are appropriate." -- s/are/is/

* "... forwarded unmodified, however ..." -- "however" should start a new sentence, or at least the comma should be a semi-colon

* "Note, that Figure 1 and ..." -- remove the comma

* "TSN over MPLS is discribed in ..." -- s/discribed/described/

Section 4.1:

* "... treatment on the connected sub-network, see ..." -- the comma should be a semi-colon, or the references should be in parentheses, etc.

Section 4.2:

* "... the underlying link / sub net specific ..." -- suggest "link/sub-network", if I'm reading the rest of this right (just to match other text, and tighten up the spaces)

* Some places in this section use "end-to-end", some use "end to end".  I'm not sure which is correct, but in any case the document should be consistent.

* "... link / sub-network ..." -- same point as before, twice more

* ""R" and "E" denotes replication ..." -- s/denotes/denote/

Section 4.3.2:

* "... subject of a completed reservation, can disrupt ..." -- remove the comma

* "Remarking packets ..." -- this should be "Re-marking" (two instances)

Section 4.3.3:

* "... when a single 5-tuple application flow experience reordering ..." -- s/experience/experiences/

Section 4.4:

* "... flows can be aggregated using any of the 6-tuple, ..." -- I can't quite parse this.  Do you mean "any subset of the elements of the 6-tuple"?

OLD:
   It is the responsibility of the DetNet controller plane to properly
   provision the use of these aggregation mechanisms.  This includes
   ensuring that aggregated flows have compatible e.g., the same or very
   similar QoS and/or CoS characteristics, see Section 4.3.2.  It also
   includes ensuring that per component-flow service requirements are
   satisfied by the aggregate, see Section 5.3.

NEW:
   It is the responsibility of the DetNet controller plane to properly
   provision the use of these aggregation mechanisms.  This includes
   ensuring that aggregated flows have compatible (e.g., the same or very
   similar) QoS and/or CoS characteristics, per Section 4.3.2.  It also
   includes ensuring that per component-flow service requirements are
   satisfied by the aggregate, per Section 5.3.

Section 5:

* "... Traffic classification, for example ..." -- that should be a semi-colon, or start a new sentence

Section 5.1:

* "Note, that additional ..." -- remove comma

Section 5.1.1.1:

* "... prefix matching for this field, see ..." -- parenthesize the "see" clause, or maybe change "see" to "per"

Section 5.1.1.2:

* Same issue with the tacked-on "see" and references.

* "Note: any IP address ..." -- capitalize "any"

Section 5.1.1.3:

* "Other, non-zero values, MUST be ..." -- remove the commas

Section 6:

* "When used, can be used ..." -- suggest "When used, the field can be used ..."

* "Exact and wildcard matching is required." (two instances) -- suggest "Support for both exact and wildcard matching is required."

Section 7:

* "... through whatever means is provided by the ..." -- s/is/are/

Thanks for responding to the SECDIR review (and thank you Tero for doing it).

Thanks for addressing my DISCUSS and COMMENT items.

I'm balloting NoObj in the "I read the protocol action, and I trust the sponsoring AD so have no problem." / "I have no cycles" sense. 
Due to other work, I was not able to review this document nearly as thoroughly as I would have liked, but I *do* support Alvaro's (and others) DISCUSS - there are many places where the document says that the network MUST do something, but without the something being particularly well defined;I'm sure that the authors / DetNet WG understand what these are, but it's not clear from the document itself -- I'm guessing that this is defined in other documents, but I wasn't able to easily find them.

I was mystified by " For some of the protocols 5-tuples and 6-tuples cannot be used because the port information is not available (e.g., ICMP, IPSec ESP).  Same can be valid for flow aggregates. " - is this that the 5/6 tuples cannot be used for flow aggregates? I went to try and find more info on flow aggregates, and looked in ietf-detnet-data-plane-framework (which, like many of the referenced draft-detnet- documents should be Normative references), but it didn't seem to have much detail (other than that "There are many techniques to achieve aggregation" -- where is how flow aggregation actually works documented? This document (S4.4) says that: flow aggregation "is an important technique for improving
   scaling by reducing the state per hop", but without understanding how flows are aggregated, and the state needed per flow, this is hard to evaluate. In addition "In either case, the management or control function that provisions the aggregate flows must ensure that adequate resources are allocated and configured to provide combined service requirements of the individual flows." -- if this "of the individual flows", or "of the aggregate flows"? Presumably there will be a difference.

I found Section "5.3.  DetNet IP Traffic Treatment Procedures" to be very terse -- I was expecting to read this and understand what *exactly* the IP Traffic Treatment Procedures are - instead it seemed to feel more like "Implementations must make sure that they provide the service that has been configured". The section says: " Typical mechanisms used to provide different treatment to different flows includes the allocation of system resources (such as queues and buffers) and provisioning or related parameters (such as shaping, and policing). " -- but this all feels fairly circular - where is the treatment of traffic "when operating in an IP packet switched network." actually defined? 
If I'm building a router/switch/similar, and what to provide support for DetNet over IP, what *exactly* do I do with traffic once it has been identified? How do I queue it differently? What happens if I cannot meet the requirement? Where is this documented? 

Apologies for not being able to review this in the depth it deserves,
W

Thank you for the work put into this document. 

I support Roman's first DISCUSS.

Please find below a couple on non-blocking COMMENTs; but, I would really appreciate a reply/answer/comment on all my COMMENTs.

I hope that this helps to improve the document,

Regards,

-éric

== COMMENTS ==

Please bear with my lack of DetNet expertise... but I have to ask the following question: how are IP fragments (i.e. lacking transport ports) processed? as they cannot match the 6-tuple. Related question: what about ICMP messages? They are often critical to a flow (PMTUd for instance, or traceroute) and should perhaps inherit the DetNet service?

-- Abstract --
Beside being the smallest abstract I have ever read on an IETF document, I also wonder about the wording "IP packet switched network". May be I am a purist, but, IP packet are forwarded and not switched. => recommend to add more meat in the abstract (notably differences with diffserv and intserv) *AND* remove the 'switched' word.

-- Section 4.5 --
"  While the DetNet IP data plane must support bidirectional DetNet
   flows, there are no special bidirectional features with respect to
   the data plane other than the need for the two directions of a co-
   routed bidirectional flow to take the same path."
   
This section does not use normative wording and I wonder whether the 'need' to take the same path will always be true as some links have different throughput up/down or their link loads could be different.

-- Section 5.1.2.1. --
"  The rules defined in this section only apply when the IPv4 Protocol
   or IPv6 Next Header Field contains the IANA defined value for UDP or
   TCP."
   
Is the intent to ignore those field when there is an IPv6 extension header between the IP and transport headers ? Same question for section 5.1.2.3.

Does it also mean that SCTP is not supported?

I support Alvaro's DISCUSS and I agree that the data plane draft should be a normative reference.

Thanks for addressing my DISCUSS.

[I saw the proposed changes on GitHub.  I trust the authors so I'm clearing.]

The other ADs caught a bunch of editorial stuff that I'll try to not
duplicate, though I do list a bunch of editorial stuff as well.

Section 1

   and routers that provide DetNet service to IP encapsulated data.  No
   DetNet-specific encapsulation is defined to support IP flows, instead
   the existing IP and higher layer protocol header information is used
   to support flow identification and DetNet service delivery.  Common

nit: comma splice

   This document provides an overview of the DetNet IP data plane in
   Section 3, considerations that apply to providing DetNet services via
   the DetNet IP data plane in Section 4.  Section 5 provides the

nit: this looks like it's trying to be a list but failing, so it's an
incomplete sentence.

Section 3

Do we want to have a note anywhere in here about how information about
which flows are DetNet flows and what behavior(s) they get (e.g., the
controller plane, or perhaps a forward reference to section 5), to
remind the reader about the architecture?

   Non-DetNet and DetNet IP packets are identical on the wire.

I suggest a different word than "identical", perhaps
"indistinguishable" (or maybe "the non-DetNet and DetNet IP packet
layouts")

   DetNet flow aggregation may be enabled via the use of wildcards,
   masks, lists, prefixes and ranges.  IP tunnels may also be used to

side note: I'd consider rewording so this doesn't sound like an
exhaustive list.

   Figure 1 illustrates a DetNet enabled IP network.  The DetNet enabled
   end systems originate IP encapsulated traffic that is identified
   within the DetNet domain as DetNet flows.  Relay nodes understand the

nit: I think the description of "identified" could be tightened up -- at
present, one could read this as either "the DetNet ingress classifies IP
traffic as DetNet or non DetNet, thus identifying the flows that are
DetNet flows", or "within the DetNet domain, these flows (originated
from the specific end systems) are identifiable as being DetNet flows".

   As non-DetNet and DetNet IP packets are identical on the wire, from
   data plane perspective, the only difference is that there is flow-

[whatever is done for "identical" above could be done here as well]
nit: missing article before "data-plane perspective"

Section 4.1

   IP flow identification means that DetNet must be aware of not only
   the format of the IP header, but also of the next protocol carried
   within an IP packet (see Section 5.1.1.3).

The way this is written it seems to bind as "(must be aware of) (not
only the format of) (the IP header) (but also of [the format of] the next
protocol)", but following the reference, it seems to only require
inspecting the value in the "next protocol" IP header field.
There would, of course, *also* be a requirement to look into TCP/UDP
headers to extract the ports needed for the 6-tuple, which is where I
originally thought this was going, but given the apparent intent, my
suggestion is to just s/next protocol/next protocol value/.

   End systems need to ensure that DetNet service requirements are met
   when processing packets associated to a DetNet flow.  When forwarding

We just said in the previous paragraph that end systems can be
DetNet-unaware; how would such DetNet-unaware systems ensure that DetNet
service requirements are met?

   packets, this means that packets are appropriately shaped on

Also, are end systems going to be *forwarding* packets?  I thought that
it was DetNet edge nodes (and transit/relay nodes) that would do so.

Section 4.2

   aware.  Such routers identify DetNet flows based on the IP 6-tuple,
   and ensure that the DetNet service required traffic treatment is
   provided both on the node and on any attached sub-network.

nit: I suggest hyphenating the compound adjective(s?) in "DetNet service
required traffic treatment" or otherwise rewording it, as I'm not
entirely clear on how to parse it.

   This solution provides DetNet functions end to end, but does so on a
   per link and sub-network basis.  Congestion protection and latency

We should probably note that there is a single point of failure at the
routers/etc. that remove and reapply these functions.

   Note that not mixing DetNet and non-DetNet traffic within a single
   5-tuple, as described above, enables simpler 5-tuple filters to be
   used (or re-used) at the edges of a DetNet network to prevent non-
   congestion-responsive DetNet traffic from escaping the DetNet domain.

I'd consider s/described/recommended/

Also, does the "simpler filters" also apply to determining what inbound
traffic should receive DetNet treatment?

Section 4.3.2

   future document.  From an encapsulation perspective, the combination
   of the 6-tuple i.e., the typical 5-tuple enhanced with the DSCP and
   previously mentioned optional field (i.e., flow label), uniquely
   identifies a DetNet IP flow.

I agree with the directorate reviewer that the language here needs to
get cleaned up.

   queue or shaper, is used by such flows.  There are multiple methods
   that MAY be used by an implementation to defend service delivery to
   reserved DetNet flows, including but not limited to:

nit: this doesn't seem like a normative MAY.

Section 4.3.3

   next hops.  As mentioned above, the DetNet IP data plane identifies
   flows using "6-tuple" header information as well as the additional
   optional header field.  DetNet generally allows for both flow-

Please just name the "optional header field" field directly; there's no
need to be oblique about it.

   Care should be taken when using different next hops for the same
   5-tuple.  As discussed in [RFC7657], unexpected behavior can occur
   when a single 5-tuple application flow experience reordering due to
   being split across multiple next hops.  Understanding of the

For some reason I thought the detnet architecture doc also had some
discussion of potential consequences of having a single flow split
across multiple paths, but all I can find is Section 3.2.2.2 that points
out there may be need to have additional buffering in such cases, in
order to equalize delays.  (Maybe I'm thinking of a different detnet
document, though I'm not sure which one that would be.)

   application and transport protocol impact of using different next
   hops for the same 6-tuple is required.  Again, this impacts path

I don't think I understand why we went from "5-tuple" to "6-tuple" in
the same paragraph.

Section 4.4

   flow identification.  DetNet IP flows can be aggregated using any of
   the 6-tuple, and an additional optional field (i.e., flow label).

Similarly to the previous comments, perhaps this should be "any of the
6-tuple and flow label"?  There's no need to add "optional" when already
subject to "using any of".

Section 5.1

   IP and higher layer protocol header information is used to identify
   DetNet flows.  All DetNet implementations that support this document
   MUST identify individual DetNet flows based on the set of information
   identified in this section.  Note, that additional flow

Does this requirement apply even to those implementations that have no
need to act at flow-level granularity, e.g., a router that only provide
DetNet service for aggregated flows, as recommended in Section 4.4?

   The configuration and control information used to identify an
   individual DetNet flow MUST be ordered by an implementation.
   Implementations MUST support a fixed order when identifying flows,
   and MUST identify a DetNet flow by the first set of matching flow
   information.

I'm a little surprised that this type of determinism is only an
implementation-level requirement.  If I'm understanding correctly, this
requirement would still allow for two different implementations to use
different rules for flow identification, and thus, potentially, to map a
single end-to-end flow to different classes of service.

Section 5.1.1.3

   An implementation MUST support flow identification based on the next
   protocol values defined in Section 5.1.2.  Other, non-zero values,
   MUST be used for flow identification.  Implementations SHOULD allow

I don't understand what distinction we're trying to draw between flow
identification based on the values defined in Section 5.1.2 and flow
identification based on other (non-zero) values.  Aren't they both
MUST-level requirements, as written?

   for these fields to be ignored for a specific DetNet flow.

side note: it seems like we're using "flow" to mean a couple different
(but similar) things in this paragraph (e.g., "considering IPv4
Protocol/IPv6 Next Header to be distinct or not").  Perhaps that's
unavioidable, though.

Section 5.1.1.4

   Traffic Class Field when processing IPv6 packets.  Implementations
   MUST support list based matching of DSCP values, where the list is

nit: hyphenate "list-based matching"

Section 6

   Information identifying a DetNet flow is ordered and implementations
   use the first match.  This can, for example, be used to provide a

[this bit might perhaps be affected if my previous comment about
implementation-specific results in any changes]

   DetNet service for a specific UDP flow, with unique Source and
   Destination Port field values, while providing a different service
   for the aggregate of all other flows with that same UDP Destination
   Port value.

[and this part actually seems to be setting at least an expectation that
the order is tied down more tightly than just "implementation specific".
Is this order perhaps something that the controller plane is expected to
provision?]

Section 7

We discuss elsewhere the need for end systems to (paraphrasing) "not
shoot themself in the foot", i.e., actually provide the resources
necessary to meet DetNet traffic guarantees.  For the case of DN
attached systems that are behind a sub-network (e.g., ES2 in Figure 3),
are there also considerations on the ability of that sub-network to
provide the necessary level of service?

Is there a need for synchronization of information about DetNet flows?
For example, would the controller plane need to ensure that information
about a new flow is fully propagated to the entire DetNet domain before
any traffic on that flow starts to flow?

If an attacker knows the 6-tuple used by a DetNet flow, isn't there a
risk that they could use IP-level spoofing to generate traffic that gets
classified as a DetNet flow, potentially causing that flow to exceed its
reservation and get erminated?  This may be a slightly different risk
than the one discussed in the last paragraph as requiring policing at
the input of the DetNet domain.  (This is related to the "source address
spoofing" risk that Bob Briscoe raised.)

   Security considerations for DetNet are described in detail in
   [I-D.ietf-detnet-security].  General security considerations are
   described in [RFC8655].  This section considers exclusively security

side note: the wording of these first two sentences feels a little
strange, as if the two documents should be somehow the same but are not.
Perhaps:

% Detailed security considerations for DetNet are catalogued in
% [I-D.ietf-detnet-security], and a more general (abbreviated) treatment
% is provided in [RFC8655].

Also, it looks like the comments I made on 2020-04-23 regarding
draft-ietf-detnet-security (as a side note in my ballot position on
draft-ietf-detnet-data-plane-framework-04) remain applicable:

% The referenced document (now at -09) seems significantly improved from
% when I previously made an in-passing review of the -03
% (https://mailarchive.ietf.org/arch/msg/detnet/jZLodXBmQa7ZFDbBIvDO_xCDD0E/),
% however, some of the issues I mentioned there remain, at least in part
% (e.g., mentioning the use of HMAC without a colocated discussion of
% the need for key distribution and having a taxonomy of threats titled
% "threat model").

   Security aspects which are unique to DetNet are those whose aim is to
   provide the specific quality of service aspects of DetNet, which are
   primarily to deliver data flows with extremely low packet loss rates
   and bounded end-to-end delivery latency.

I recognize that this is essentially verbatim from RFC 8655, but it
still reads somewhat strangely to me: specifically, it seems to say that
there are "security aspects [...] whose aim is to provide [QoS, low
packet loss, and bounded latency]", and I don't think it's the security
aspects specifically that aim to provide those.  Rather, I think it's
the overall goal of DetNet to provide those attributes, and there are
some security aspepcts specific to attempting to provide those
attributes.

   The primary considerations for the data plane is to maintain
   integrity of data and delivery of the associated DetNet service
   traversing the DetNet network.  Application flows can be protected
   through whatever means is provided by the underlying technology.  For
   example, encryption may be used, such as that provided by IPSec
   [RFC4301] for IP flows and/or by an underlying sub-net using MACSec
   [IEEE802.1AE-2018] for IP over Ethernet (Layer-2) flows.

It's good that you call this out, so thank you for that.
I'd suggest a slight addition, prefacing the sentence that currently
starts with "Application flows" with a lead-in that "Since no
DetNet-specific fields are available for DetNet IP, data integrity must
be provided by a lower (or, potentially, higher) layer; application
flows can [...]".  I'd also consider noting that providing guaranteed
delivery is impossible in the face of a sufficiently powerful attacker
(that can drop all traffic, cut fiber, etc.), so the attacker in the
DetNet threat model is necessarily slightly weaker than a typical BCP 72
threat model.

Section 11.1

The way in which RFC 3473 is used does not seem to require a normative
reference, to me.

(RFC 4301 is perhaps in a similar situation, though given that we need
normative references to 4302 and 4303, it doesn't stick out as much.)

Section 11.2

Was it considered to make draft-ietf-detnet-data-plane-framework a
normative reference?  The reference in Section 3 that says "common data
plane procedures and control information for all DetNet data planes" can
be found in it makes me wonder whether it contains information that I
need to know in order to use detnet-ip properly.

Thank you for addressing most of the extensive tsvart comments.

Sec 3 says DetNet might alter the DSCPs. Sec 7 says “ From a data plane perspective this document does not add or modify any header information.” These appear to be in contradiction.

Sec 4.1 “In order to maximize reuse of existing mechanisms, DetNet-aware
   applications and end systems SHOULD NOT mix DetNet and non-DetNet
   traffic within a single 5-tuple.“

I don’t understand this. What’s the point of having a 6-tuple if you can’t keep 5 constant and vary the DSCP?

Sec 5.2 discourages the use of “multiple paths.” But Figure 4 clearly shows packets taking multiple paths to the destination.

Nit:
Sec 5.1 s/end systems/end system

Hi,

Thanks for this document.  The concept seems pretty simple, I just have a few comments.

3.  DetNet IP Data Plane Overview

In the introduction, I wondered whether it would be useful to mention that the Detnet forwarding is based on the classification of the 6-tuple that is programmed via the management plane with the next hop.  E.g. introducing the key forwarding concept defined in 5.2.

4.2.  DetNet Domain-Specific Considerations

Are the sub networks shown in Figure 4 all detnet aware supporting the service function?  If so, it may be helpful to explicitly clarify this.

4.3.3.  Path Selection

Like the other reviewers, I found the terminology around using 5-tuple/6-tuples slightly confusing in places.  In particular, the text in 4.3.3 regarding 5 tuples seemed confusing.  I basically interpreted the text as saying that 5-tuple forwarding already has these issues that must be avoided, and 6 tuple forwarding is no different.

Should it also mention that EMCP should be avoided as part of this path selection section?

I note that there is an informative reference to ietf-detnet-yang.  I wasn't sure whether an informative reference to YANG itself might also be worth adding.

Regards,
Rob