Deterministic Networking (DetNet) Data Plane: IP
Note: This ballot was opened for revision 06 and is now closed.
(Deborah Brungard) Yes
(Alissa Cooper) No Objection
Comment (2020-06-24 for -06)
I support Alvaro's DISCUSS and I agree that the data plane draft should be a normative reference.
Roman Danyliw (was Discuss) No Objection
Thanks for responding to the SECDIR review (and thank you Tero for doing it). Thanks for addressing my DISCUSS and COMMENT items.
Martin Duke No Objection
Comment (2020-06-15 for -06)
Thank you for addressing most of the extensive tsvart comments. Sec 3 says DetNet might alter the DSCPs. Sec 7 says “ From a data plane perspective this document does not add or modify any header information.” These appear to be in contradiction. Sec 4.1 “In order to maximize reuse of existing mechanisms, DetNet-aware applications and end systems SHOULD NOT mix DetNet and non-DetNet traffic within a single 5-tuple.“ I don’t understand this. What’s the point of having a 6-tuple if you can’t keep 5 constant and vary the DSCP? Sec 5.2 discourages the use of “multiple paths.” But Figure 4 clearly shows packets taking multiple paths to the destination. Nit: Sec 5.1 s/end systems/end system
Benjamin Kaduk No Objection
Comment (2020-06-24 for -06)
The other ADs caught a bunch of editorial stuff that I'll try to not duplicate, though I do list a bunch of editorial stuff as well. Section 1 and routers that provide DetNet service to IP encapsulated data. No DetNet-specific encapsulation is defined to support IP flows, instead the existing IP and higher layer protocol header information is used to support flow identification and DetNet service delivery. Common nit: comma splice This document provides an overview of the DetNet IP data plane in Section 3, considerations that apply to providing DetNet services via the DetNet IP data plane in Section 4. Section 5 provides the nit: this looks like it's trying to be a list but failing, so it's an incomplete sentence. Section 3 Do we want to have a note anywhere in here about how information about which flows are DetNet flows and what behavior(s) they get (e.g., the controller plane, or perhaps a forward reference to section 5), to remind the reader about the architecture? Non-DetNet and DetNet IP packets are identical on the wire. I suggest a different word than "identical", perhaps "indistinguishable" (or maybe "the non-DetNet and DetNet IP packet layouts") DetNet flow aggregation may be enabled via the use of wildcards, masks, lists, prefixes and ranges. IP tunnels may also be used to side note: I'd consider rewording so this doesn't sound like an exhaustive list. Figure 1 illustrates a DetNet enabled IP network. The DetNet enabled end systems originate IP encapsulated traffic that is identified within the DetNet domain as DetNet flows. Relay nodes understand the nit: I think the description of "identified" could be tightened up -- at present, one could read this as either "the DetNet ingress classifies IP traffic as DetNet or non DetNet, thus identifying the flows that are DetNet flows", or "within the DetNet domain, these flows (originated from the specific end systems) are identifiable as being DetNet flows". As non-DetNet and DetNet IP packets are identical on the wire, from data plane perspective, the only difference is that there is flow- [whatever is done for "identical" above could be done here as well] nit: missing article before "data-plane perspective" Section 4.1 IP flow identification means that DetNet must be aware of not only the format of the IP header, but also of the next protocol carried within an IP packet (see Section 18.104.22.168). The way this is written it seems to bind as "(must be aware of) (not only the format of) (the IP header) (but also of [the format of] the next protocol)", but following the reference, it seems to only require inspecting the value in the "next protocol" IP header field. There would, of course, *also* be a requirement to look into TCP/UDP headers to extract the ports needed for the 6-tuple, which is where I originally thought this was going, but given the apparent intent, my suggestion is to just s/next protocol/next protocol value/. End systems need to ensure that DetNet service requirements are met when processing packets associated to a DetNet flow. When forwarding We just said in the previous paragraph that end systems can be DetNet-unaware; how would such DetNet-unaware systems ensure that DetNet service requirements are met? packets, this means that packets are appropriately shaped on Also, are end systems going to be *forwarding* packets? I thought that it was DetNet edge nodes (and transit/relay nodes) that would do so. Section 4.2 aware. Such routers identify DetNet flows based on the IP 6-tuple, and ensure that the DetNet service required traffic treatment is provided both on the node and on any attached sub-network. nit: I suggest hyphenating the compound adjective(s?) in "DetNet service required traffic treatment" or otherwise rewording it, as I'm not entirely clear on how to parse it. This solution provides DetNet functions end to end, but does so on a per link and sub-network basis. Congestion protection and latency We should probably note that there is a single point of failure at the routers/etc. that remove and reapply these functions. Note that not mixing DetNet and non-DetNet traffic within a single 5-tuple, as described above, enables simpler 5-tuple filters to be used (or re-used) at the edges of a DetNet network to prevent non- congestion-responsive DetNet traffic from escaping the DetNet domain. I'd consider s/described/recommended/ Also, does the "simpler filters" also apply to determining what inbound traffic should receive DetNet treatment? Section 4.3.2 future document. From an encapsulation perspective, the combination of the 6-tuple i.e., the typical 5-tuple enhanced with the DSCP and previously mentioned optional field (i.e., flow label), uniquely identifies a DetNet IP flow. I agree with the directorate reviewer that the language here needs to get cleaned up. queue or shaper, is used by such flows. There are multiple methods that MAY be used by an implementation to defend service delivery to reserved DetNet flows, including but not limited to: nit: this doesn't seem like a normative MAY. Section 4.3.3 next hops. As mentioned above, the DetNet IP data plane identifies flows using "6-tuple" header information as well as the additional optional header field. DetNet generally allows for both flow- Please just name the "optional header field" field directly; there's no need to be oblique about it. Care should be taken when using different next hops for the same 5-tuple. As discussed in [RFC7657], unexpected behavior can occur when a single 5-tuple application flow experience reordering due to being split across multiple next hops. Understanding of the For some reason I thought the detnet architecture doc also had some discussion of potential consequences of having a single flow split across multiple paths, but all I can find is Section 22.214.171.124 that points out there may be need to have additional buffering in such cases, in order to equalize delays. (Maybe I'm thinking of a different detnet document, though I'm not sure which one that would be.) application and transport protocol impact of using different next hops for the same 6-tuple is required. Again, this impacts path I don't think I understand why we went from "5-tuple" to "6-tuple" in the same paragraph. Section 4.4 flow identification. DetNet IP flows can be aggregated using any of the 6-tuple, and an additional optional field (i.e., flow label). Similarly to the previous comments, perhaps this should be "any of the 6-tuple and flow label"? There's no need to add "optional" when already subject to "using any of". Section 5.1 IP and higher layer protocol header information is used to identify DetNet flows. All DetNet implementations that support this document MUST identify individual DetNet flows based on the set of information identified in this section. Note, that additional flow Does this requirement apply even to those implementations that have no need to act at flow-level granularity, e.g., a router that only provide DetNet service for aggregated flows, as recommended in Section 4.4? The configuration and control information used to identify an individual DetNet flow MUST be ordered by an implementation. Implementations MUST support a fixed order when identifying flows, and MUST identify a DetNet flow by the first set of matching flow information. I'm a little surprised that this type of determinism is only an implementation-level requirement. If I'm understanding correctly, this requirement would still allow for two different implementations to use different rules for flow identification, and thus, potentially, to map a single end-to-end flow to different classes of service. Section 126.96.36.199 An implementation MUST support flow identification based on the next protocol values defined in Section 5.1.2. Other, non-zero values, MUST be used for flow identification. Implementations SHOULD allow I don't understand what distinction we're trying to draw between flow identification based on the values defined in Section 5.1.2 and flow identification based on other (non-zero) values. Aren't they both MUST-level requirements, as written? for these fields to be ignored for a specific DetNet flow. side note: it seems like we're using "flow" to mean a couple different (but similar) things in this paragraph (e.g., "considering IPv4 Protocol/IPv6 Next Header to be distinct or not"). Perhaps that's unavioidable, though. Section 188.8.131.52 Traffic Class Field when processing IPv6 packets. Implementations MUST support list based matching of DSCP values, where the list is nit: hyphenate "list-based matching" Section 6 Information identifying a DetNet flow is ordered and implementations use the first match. This can, for example, be used to provide a [this bit might perhaps be affected if my previous comment about implementation-specific results in any changes] DetNet service for a specific UDP flow, with unique Source and Destination Port field values, while providing a different service for the aggregate of all other flows with that same UDP Destination Port value. [and this part actually seems to be setting at least an expectation that the order is tied down more tightly than just "implementation specific". Is this order perhaps something that the controller plane is expected to provision?] Section 7 We discuss elsewhere the need for end systems to (paraphrasing) "not shoot themself in the foot", i.e., actually provide the resources necessary to meet DetNet traffic guarantees. For the case of DN attached systems that are behind a sub-network (e.g., ES2 in Figure 3), are there also considerations on the ability of that sub-network to provide the necessary level of service? Is there a need for synchronization of information about DetNet flows? For example, would the controller plane need to ensure that information about a new flow is fully propagated to the entire DetNet domain before any traffic on that flow starts to flow? If an attacker knows the 6-tuple used by a DetNet flow, isn't there a risk that they could use IP-level spoofing to generate traffic that gets classified as a DetNet flow, potentially causing that flow to exceed its reservation and get erminated? This may be a slightly different risk than the one discussed in the last paragraph as requiring policing at the input of the DetNet domain. (This is related to the "source address spoofing" risk that Bob Briscoe raised.) Security considerations for DetNet are described in detail in [I-D.ietf-detnet-security]. General security considerations are described in [RFC8655]. This section considers exclusively security side note: the wording of these first two sentences feels a little strange, as if the two documents should be somehow the same but are not. Perhaps: % Detailed security considerations for DetNet are catalogued in % [I-D.ietf-detnet-security], and a more general (abbreviated) treatment % is provided in [RFC8655]. Also, it looks like the comments I made on 2020-04-23 regarding draft-ietf-detnet-security (as a side note in my ballot position on draft-ietf-detnet-data-plane-framework-04) remain applicable: % The referenced document (now at -09) seems significantly improved from % when I previously made an in-passing review of the -03 % (https://mailarchive.ietf.org/arch/msg/detnet/jZLodXBmQa7ZFDbBIvDO_xCDD0E/), % however, some of the issues I mentioned there remain, at least in part % (e.g., mentioning the use of HMAC without a colocated discussion of % the need for key distribution and having a taxonomy of threats titled % "threat model"). Security aspects which are unique to DetNet are those whose aim is to provide the specific quality of service aspects of DetNet, which are primarily to deliver data flows with extremely low packet loss rates and bounded end-to-end delivery latency. I recognize that this is essentially verbatim from RFC 8655, but it still reads somewhat strangely to me: specifically, it seems to say that there are "security aspects [...] whose aim is to provide [QoS, low packet loss, and bounded latency]", and I don't think it's the security aspects specifically that aim to provide those. Rather, I think it's the overall goal of DetNet to provide those attributes, and there are some security aspepcts specific to attempting to provide those attributes. The primary considerations for the data plane is to maintain integrity of data and delivery of the associated DetNet service traversing the DetNet network. Application flows can be protected through whatever means is provided by the underlying technology. For example, encryption may be used, such as that provided by IPSec [RFC4301] for IP flows and/or by an underlying sub-net using MACSec [IEEE802.1AE-2018] for IP over Ethernet (Layer-2) flows. It's good that you call this out, so thank you for that. I'd suggest a slight addition, prefacing the sentence that currently starts with "Application flows" with a lead-in that "Since no DetNet-specific fields are available for DetNet IP, data integrity must be provided by a lower (or, potentially, higher) layer; application flows can [...]". I'd also consider noting that providing guaranteed delivery is impossible in the face of a sufficiently powerful attacker (that can drop all traffic, cut fiber, etc.), so the attacker in the DetNet threat model is necessarily slightly weaker than a typical BCP 72 threat model. Section 11.1 The way in which RFC 3473 is used does not seem to require a normative reference, to me. (RFC 4301 is perhaps in a similar situation, though given that we need normative references to 4302 and 4303, it doesn't stick out as much.) Section 11.2 Was it considered to make draft-ietf-detnet-data-plane-framework a normative reference? The reference in Section 3 that says "common data plane procedures and control information for all DetNet data planes" can be found in it makes me wonder whether it contains information that I need to know in order to use detnet-ip properly.
Erik Kline No Objection
Comment (2020-06-24 for -06)
I support some of the other DISCUSS items as well. [[ recommendations ]] [ whole document ] * Given the use of 2-, 3-, 5-, and 6-tuple would it be a readability improvement if section 3 rolled them all up in to one more general term, like N-tuple? Quite possibly I have misunderstood, though. * Instead of L2, or in the terminology definition of L2, clarify "sub-IP layer" so as to avoid concerns about things like "layer 2.5" and handling in other future link types? [[ nits ]] [ section 3 ] * "IP DetNet End Systems can communicate over DetNet IP network with IP End System" The grammar toward the end of the sentence is off. "...over DetNet IP networks with other IP End Systems", perhaps? * "from data plane perspective" -> "from the data plane perspective" * "discribed" -> "described" [ section 4.2 ] * Suggest, maybe, "DN adjacent" instead of "DN attached". This use of "attached" seems to stretch to meaning of the word. [ section 184.108.40.206 ] * If you feel you want a reference, check out RFC 6437.
Murray Kucherawy No Objection
Comment (2020-06-23 for -06)
I support Alvaro’s DISCUSS points. Otherwise, just a pile of nits here: Section 1: * "... defined to support IP flows, instead ..." -- "instead" should start a new sentence, or at least the comma should be a semi-colon * "... data plane in Section 3, considerations that ..." -- add "and" before "considerations" Section 3: * The parentheses in the second paragraph are unbalanced. * "Same can be valid for flow aggregates." -- To what does "Same" refer? * "In such cases using smaller tuples are appropriate." -- s/are/is/ * "... forwarded unmodified, however ..." -- "however" should start a new sentence, or at least the comma should be a semi-colon * "Note, that Figure 1 and ..." -- remove the comma * "TSN over MPLS is discribed in ..." -- s/discribed/described/ Section 4.1: * "... treatment on the connected sub-network, see ..." -- the comma should be a semi-colon, or the references should be in parentheses, etc. Section 4.2: * "... the underlying link / sub net specific ..." -- suggest "link/sub-network", if I'm reading the rest of this right (just to match other text, and tighten up the spaces) * Some places in this section use "end-to-end", some use "end to end". I'm not sure which is correct, but in any case the document should be consistent. * "... link / sub-network ..." -- same point as before, twice more * ""R" and "E" denotes replication ..." -- s/denotes/denote/ Section 4.3.2: * "... subject of a completed reservation, can disrupt ..." -- remove the comma * "Remarking packets ..." -- this should be "Re-marking" (two instances) Section 4.3.3: * "... when a single 5-tuple application flow experience reordering ..." -- s/experience/experiences/ Section 4.4: * "... flows can be aggregated using any of the 6-tuple, ..." -- I can't quite parse this. Do you mean "any subset of the elements of the 6-tuple"? OLD: It is the responsibility of the DetNet controller plane to properly provision the use of these aggregation mechanisms. This includes ensuring that aggregated flows have compatible e.g., the same or very similar QoS and/or CoS characteristics, see Section 4.3.2. It also includes ensuring that per component-flow service requirements are satisfied by the aggregate, see Section 5.3. NEW: It is the responsibility of the DetNet controller plane to properly provision the use of these aggregation mechanisms. This includes ensuring that aggregated flows have compatible (e.g., the same or very similar) QoS and/or CoS characteristics, per Section 4.3.2. It also includes ensuring that per component-flow service requirements are satisfied by the aggregate, per Section 5.3. Section 5: * "... Traffic classification, for example ..." -- that should be a semi-colon, or start a new sentence Section 5.1: * "Note, that additional ..." -- remove comma Section 220.127.116.11: * "... prefix matching for this field, see ..." -- parenthesize the "see" clause, or maybe change "see" to "per" Section 18.104.22.168: * Same issue with the tacked-on "see" and references. * "Note: any IP address ..." -- capitalize "any" Section 22.214.171.124: * "Other, non-zero values, MUST be ..." -- remove the commas Section 6: * "When used, can be used ..." -- suggest "When used, the field can be used ..." * "Exact and wildcard matching is required." (two instances) -- suggest "Support for both exact and wildcard matching is required." Section 7: * "... through whatever means is provided by the ..." -- s/is/are/
Warren Kumari No Objection
Comment (2020-06-24 for -06)
I'm balloting NoObj in the "I read the protocol action, and I trust the sponsoring AD so have no problem." / "I have no cycles" sense. Due to other work, I was not able to review this document nearly as thoroughly as I would have liked, but I *do* support Alvaro's (and others) DISCUSS - there are many places where the document says that the network MUST do something, but without the something being particularly well defined;I'm sure that the authors / DetNet WG understand what these are, but it's not clear from the document itself -- I'm guessing that this is defined in other documents, but I wasn't able to easily find them. I was mystified by " For some of the protocols 5-tuples and 6-tuples cannot be used because the port information is not available (e.g., ICMP, IPSec ESP). Same can be valid for flow aggregates. " - is this that the 5/6 tuples cannot be used for flow aggregates? I went to try and find more info on flow aggregates, and looked in ietf-detnet-data-plane-framework (which, like many of the referenced draft-detnet- documents should be Normative references), but it didn't seem to have much detail (other than that "There are many techniques to achieve aggregation" -- where is how flow aggregation actually works documented? This document (S4.4) says that: flow aggregation "is an important technique for improving scaling by reducing the state per hop", but without understanding how flows are aggregated, and the state needed per flow, this is hard to evaluate. In addition "In either case, the management or control function that provisions the aggregate flows must ensure that adequate resources are allocated and configured to provide combined service requirements of the individual flows." -- if this "of the individual flows", or "of the aggregate flows"? Presumably there will be a difference. I found Section "5.3. DetNet IP Traffic Treatment Procedures" to be very terse -- I was expecting to read this and understand what *exactly* the IP Traffic Treatment Procedures are - instead it seemed to feel more like "Implementations must make sure that they provide the service that has been configured". The section says: " Typical mechanisms used to provide different treatment to different flows includes the allocation of system resources (such as queues and buffers) and provisioning or related parameters (such as shaping, and policing). " -- but this all feels fairly circular - where is the treatment of traffic "when operating in an IP packet switched network." actually defined? If I'm building a router/switch/similar, and what to provide support for DetNet over IP, what *exactly* do I do with traffic once it has been identified? How do I queue it differently? What happens if I cannot meet the requirement? Where is this documented? Apologies for not being able to review this in the depth it deserves, W
(Barry Leiba) No Objection
Alvaro Retana (was Discuss) No Objection
Comment (2020-06-26 for -06)
Thanks for addressing my DISCUSS. [I saw the proposed changes on GitHub. I trust the authors so I'm clearing.]
Éric Vyncke No Objection
Comment (2020-06-25 for -06)
Thank you for the work put into this document. I support Roman's first DISCUSS. Please find below a couple on non-blocking COMMENTs; but, I would really appreciate a reply/answer/comment on all my COMMENTs. I hope that this helps to improve the document, Regards, -éric == COMMENTS == Please bear with my lack of DetNet expertise... but I have to ask the following question: how are IP fragments (i.e. lacking transport ports) processed? as they cannot match the 6-tuple. Related question: what about ICMP messages? They are often critical to a flow (PMTUd for instance, or traceroute) and should perhaps inherit the DetNet service? -- Abstract -- Beside being the smallest abstract I have ever read on an IETF document, I also wonder about the wording "IP packet switched network". May be I am a purist, but, IP packet are forwarded and not switched. => recommend to add more meat in the abstract (notably differences with diffserv and intserv) *AND* remove the 'switched' word. -- Section 4.5 -- " While the DetNet IP data plane must support bidirectional DetNet flows, there are no special bidirectional features with respect to the data plane other than the need for the two directions of a co- routed bidirectional flow to take the same path." This section does not use normative wording and I wonder whether the 'need' to take the same path will always be true as some links have different throughput up/down or their link loads could be different. -- Section 126.96.36.199. -- " The rules defined in this section only apply when the IPv4 Protocol or IPv6 Next Header Field contains the IANA defined value for UDP or TCP." Is the intent to ignore those field when there is an IPv6 extension header between the IP and transport headers ? Same question for section 188.8.131.52. Does it also mean that SCTP is not supported?
(Magnus Westerlund) No Objection
Robert Wilton No Objection
Comment (2020-06-24 for -06)
Hi, Thanks for this document. The concept seems pretty simple, I just have a few comments. 3. DetNet IP Data Plane Overview In the introduction, I wondered whether it would be useful to mention that the Detnet forwarding is based on the classification of the 6-tuple that is programmed via the management plane with the next hop. E.g. introducing the key forwarding concept defined in 5.2. 4.2. DetNet Domain-Specific Considerations Are the sub networks shown in Figure 4 all detnet aware supporting the service function? If so, it may be helpful to explicitly clarify this. 4.3.3. Path Selection Like the other reviewers, I found the terminology around using 5-tuple/6-tuples slightly confusing in places. In particular, the text in 4.3.3 regarding 5 tuples seemed confusing. I basically interpreted the text as saying that 5-tuple forwarding already has these issues that must be avoided, and 6 tuple forwarding is no different. Should it also mention that EMCP should be avoided as part of this path selection section? I note that there is an informative reference to ietf-detnet-yang. I wasn't sure whether an informative reference to YANG itself might also be worth adding. Regards, Rob