Deterministic Networking (DetNet) Data Plane: MPLS
draft-ietf-detnet-mpls-13

[[ questions ]]

[ section 4.1 ]

* "The 16 bit sequence number is needed to support some types of legacy
  clients."

  Should there be a SHOULD in here about defaulting to, say, use of 28 bit
  sequence numbers?


[[ nits ]]

[ section 4.2.1 ]

* s/were the sequence/where the sequence/

[ section 4.2.2 ]

* s/as outlines in/as outlined in/

[ section 4.2.3 ]

* Maybe "are supported the" -> "support the"?

[ section 4.2.3.2 ]

* s/a service parameters/a service parameter/?

* s/provisioning or related/provisioning of related/

* Maybe "requirement for MPLS LSRs to that CoS LSPs do not"
  -> "the requirement for MPLS LSRs that CoS LSPs do not"

[ section 4.4.2 ]

* s/to to/to/

[ section 4.6.2 ]

* s/that maybe used/that may be used/

* s/can provided/can be provided/

[ section 5.1 ]

* "each of the service's outgoing DetNet (member) flow" ->
  "each of the service's outgoing DetNet (member) flows"

[ section 5.1.1 ]

* Perhaps s/may be provided discussed/may be provided is discussed/

[ section 5.2 ]

* Perhaps s/flow specific resources/flow-specific resources/

Not much to add here.  I concur explicitly though with Barry's points about the document references.

Thank you for addressing my DISCUSS and COMMENT feedback.

Thanks to Shwetha for the OpsDir review, it was helpful to me for knowing where to focus my review. 

I think it would be nice if  the document had a bit more text around the (lack of) issues caused by: 
"It is important to note that this document differs from [RFC4448]
   where a sequence number of zero (0) is used to indicate that the
   sequence number check algorithm is not used."

I had to re-read this a few times and then spend some time trying to figure out what happens with implementations that follow RFC4448; as far as I can tell nothing bad happens, but having an extra sentence or two reassuring readers of this would be nice.

(0) I support Magnus' DISCUSS.  [I also have some comments about §4.2.2.2/§4.2.2.3 below.]


(1) §4.6.2: "DetNet provides zero congestion loss and bounded latency and jitter"

This phrase is a variation of the general statement in the Introduction: "DetNet provides these flows with extremely low packet loss rates and assured maximum end-to-end delivery latency."  Specific to this document, how does DetNet guarantee this behavior while operating on an MPLS network?

I see that §4.5/rfc8655 focuses a series of examples on TSN (which is not required by the architecture, right?), and this document mentions other technology as examples. Still, there is no explicit indication of how the "zero congestion loss and bounded latency and jitter" can be achieved.  Instead, the use of these mechanisms is left to the implementation/operator without any guidance.  I think that not offering any guidance is a significant gap in a Standards Track document.

Most of the mechanisms mentioned as examples in §4.6.2 are "old", for example, RSVP-TE.  Pretty much the same mechanisms are listed in §4.2.3.2 when talking about the option of using DetNEt unaware nodes to deliver the same "service parameters":

   When the DetNet service parameters of the DetNet flow or flows carried in
   an LSP represented by an F-Label can be met by an existing TE mechanism,
   the forwarding sub-layer processing node MAY be a DetNet unaware, i.e.,
   standard, MPLS LSR.  Such TE LSPs may provide LSP forwarding service as
   defined in, but not limited to, [RFC3209], [RFC3270], [RFC3272], [RFC3473],
   [RFC4875], [RFC5440], and [RFC8306].


How does DetNet guarantee "zero congestion loss and bounded latency and jitter" while operating on an MPLS network?  The more I read, the more it seems to me that the examples provided support the concept that the same results can be obtained in an MPLS network without DetNet.

[I am out of my depth and feel like I'm probably missing something, which is why I am not balloting DISCUSS on this point.  However, I believe that the issue of specific guidance remains.]



(2) Figure 4 shows the encapsulation of a DetNet App-Flow, which includes several labels + CW...  Figure 6 shows an even bigger stack (also with several F-Labels and the new A-Label, S-Label, CWs...).

What are the considerations related to the nodes' ability to support the label stack needed for DetNet operation and aggregation?  What is the relationship between the maximum number of labels supported in the network and the ability to push the required label stack onto the packet?  [Take a look at rfc8491 for a sample definition related to the Max SID Depth (MSD).]



(3) §4.2.1

   A 0 bit sequence number field length indicates that there is no
   DetNet sequence number used for the flow.  When the length is zero,
   the sequence number field MUST be set to zero (0) on all packets sent
   for the flow.

   ...and ignored by the receiver??

There are several other "MUST be set to zero" phrases in the text.


(4) §4.2.2

   An S-Label will normally be at the bottom of the label stack once the
   last F-Label is removed, immediately preceding the d-CW.  To support
   service sub-layer level OAM, an OAM Associated Channel Header (ACH)
   [RFC4385] together with a Generic Associated Channel Label (GAL)
   [RFC5586] MAY be used in place of a d-CW.

§4.2.1 says that the d-CW "MUST be present in all DetNet packets containing app-flow data".  What are the cases when the d-CW may be substituted by the ACH/GAL combination?  The PEF/POF functions are specified in terms of the d-CW -- is that functionality lost if the ACH/GAL are used instead?



(5) §4.2.2

   "...zero or more F-Labels and optionally, the incoming interface,
   proceeding the S-Label MUST be used together with the S-Label to
   uniquely identify the DetNet service associated with a received
   packet.  The incoming interface MAY also be used together with any
   present F-Label(s) and the S-Label to uniquely identify an incoming
   DetNet service..."


These two phrases seem to say the same thing, but with different normative expectations.



(6) §4.2.2.2:

The interoperable action is not "MUST check", or "MUST ensure", but "MUST discard".

OLD>
   After a DetNet service is identified for a received DetNet MPLS
   packet, as described above, an implementation MUST check if PEF is
   configured for that DetNet service.  When configured, the
   implementation MUST track the sequence number contained in received
   d-CWs and MUST ensure that duplicate (replicated) instances of a
   particular sequence number are discarded. 

NEW>
   After a DetNet service is identified for a received DetNet MPLS packet, 
   as described above, if PEF is configured for that DetNet service, 
   duplicate (replicated) instances of a particular sequence number MUST be 
   discarded.



(7) §4.2.2.2:

"MAY wish" is not an action that can be enforced

OLD>
   Note that an implementation MAY wish to constrain the maximum number
   of sequence numbers that are tracked, on platform-wide or per flow
   basis.  Some implementations MAY support the provisioning of the
   maximum number of sequence numbers that are tracked on either a
   platform-wide or per flow basis.


NEW>
   An implementation MAY constrain the maximum number of sequence numbers 
   that are tracked on either a platform-wide or per flow basis.



(8) §4.2.2.3:

The interoperable action is not "MUST check", or "MUST ensure", but "MUST process in order".

OLD>
   After a DetNet service is identified for a received DetNet MPLS
   packet, as described above, an implementation MUST check if POF is
   configured for that DetNet service.  When configured, the
   implementation MUST track the sequence number contained in received
   d-CWs and MUST ensure that packets are processed in the order
   indicated in the received d-CW sequence number field, which may not
   be in the order the packets are received.

NEW>
   After a DetNet service is identified for a received DetNet MPLS packet, 
   as described above, if POF is configured for that DetNet service, packets 
   MUST be processed in the order indicated in the received d-CW sequence 
   number field, which may not be in the order the packets are received.


(9) §4.2.2.3:

"MAY wish" is not an action that can be enforced

OLD>
   Note that an implementation MAY wish to constrain the maximum number
   of out of order packets that can be processed, on platform-wide or
   per flow basis.  Some implementations MAY support the provisioning of
   this number on either a platform-wide or per flow basis.  The number
   of out of order packets that can be processed also impacts the
   latency of a flow.

NEW>
   An implementation MAY constrain the maximum number of out of order 
   packets that can be processed, on either a platform-wide or per flow 
   basis.  The number of out of order packets that can be processed also 
   impacts the latency of a flow.



(10) [nits]

s/MAY required/MAY require

s/otherwise received/otherwise receive

s/a service parameters that dictates/service parameters that dictate

s/that maybe used/that may be used

Just some small, non-blocking, and easy-to-address comments here:

Abstract:
   This document builds on the DetNet
   Architecture and Data Plane Framework.

Section 2.1:
   This document uses the terminology established in the DetNet
   architecture [RFC8655] and the DetNet Data Plane Framework
   [I-D.ietf-detnet-data-plane-framework].  The reader is assumed to be
   familiar with these documents, any terminology defined therein

Then how is it possible that data-plane-framework is informative?  I think it has to be normative.

— Section 4.2.1 —

   and (2) to allow non-skip zero S/N what simplifies implementation.

I can’t parse that.  Can you fix the wording or explain what I’m missing?

— Section 6 —

   for the mitigation of Man-In-The-Middle attacks

This is entirely up to your judgment, but please consider using phrasing such as “for mitigating attackers-in-the-middle” here.  Thanks.

There's nothing particularly earth-shattering in these remarks; just
some things to double-check and potential minor improvements.  (The
combination of generic DetNet and MPLS security considerations cover
just about everything that's going on here.)  I also have a number of
nit-level notes that I'll send to the authors separately.

Section 2.2

I agree with the directorate reviewer that references for at least some
of these terms would be useful.

Section 3.1

I trust there's a reason for the figure to have "bottom of stack" at the
top of the figure (and vice versa).

   The DetNet MPLS data plane representation is illustrated in Figure 1.
   The service sub-layer includes a DetNet control word (d-CW) and an
   identifying service label (S-Label).  The DetNet control word (d-CW)
   conforms to the Generic PW MPLS Control Word (PWMCW) defined in
   [RFC4385].  An aggregation label (A-Label) is a special case of

Just to check my understanding: we conform to the generic format from
RFC 4385 but not the preferred one?

Section 4.2.1

Do we want to say anything about why the RFC 4385 preferred-format's
flags, fragmentation indicator, and length fields are not applicable to
the DetNet usage?  (I do not recall any restrictions that prevent DetNet
flows from traversing Ethernet segments, one of the justifications given
in RFC 4385 for the length field.)

I would also suggest avoiding the "S/N" abbreviation (as colliding with
"signal/noise"), and note that it is not listed at
https://www.rfc-editor.org/materials/abbrev.expansion.txt .

   The sequence number length MUST be provisioned on a per Detnet
   service basis via configuration, i.e., via the controller plane
   described in [I-D.ietf-detnet-data-plane-framework].

(side note) I didn't find a great definition for "DetNet service" as a
standalone term in RFC 8655, though it's clearly already in use there
for this meaning.

   When the sequence number field length is 16 or 28 bits for a flow,
   the sequence number MUST be incremented by one for each new app-flow
   packet sent.  When the field length is 16 bits, d-CW bits 4 to 15

Are there any considerations on the initial sequence number value?
Randomization of the initial sequence number may be a generic best
practice, as discussed in draft-gont-numeric-ids-sec-considerations
(which I am AD sponsoring).

Section 4.2.2

   S-Label values MUST be provisioned per DetNet service via
   configuration, e.g., via the controller plane described in
   [I-D.ietf-detnet-data-plane-framework].  Note that S-Labels provide
   [...]
   MAY be allocated from the platform label space [RFC3031].  Because
   S-Labels are local to each node rather than being a global identifier
   within a domain, they must be advertised to their upstream DetNet
   service-aware peer nodes (e.g., a DetNet MPLS End System or a DetNet
   Relay or Edge Node) and interpreted in the context of their received

I'm having a hard time reconciling "provisioned via configuration" with
"advertised to their upstream peer nodes" -- can you help explain what
is expected to happen here?

   An S-Label will normally be at the bottom of the label stack once the
   last F-Label is removed, immediately preceding the d-CW.  To support
   service sub-layer level OAM, an OAM Associated Channel Header (ACH)
   [RFC4385] together with a Generic Associated Channel Label (GAL)
   [RFC5586] MAY be used in place of a d-CW.

Does this preclude using an ACH in the absence of a GAL?

                        In the case where platform labels are not used,
   zero or more F-Labels and optionally, the incoming interface,
   proceeding the S-Label MUST be used together with the S-Label to
   uniquely identify the DetNet service associated with a received
   packet.  The incoming interface MAY also be used together with any
   present F-Label(s) and the S-Label to uniquely identify an incoming
   DetNet service, for example, in the case where PHP is used.  [...]

I'm not sure what the difference in meaning between these two sentences
is supposed to be.

Section 4.2.3.1

   When multiple sets of outgoing F-Labels or interfaces are provisioned
   for a particular DetNet service, a copy of the outgoing packet,

Would it be banal to note that this occurs as part of the PRF?

   S-label uniquely identifies the DetNet service.  In the case where
   platform labels are not used, incoming F-Label(s) and, optionally,
   the incoming interface MUST also be provisioned for a DetNet service.

This might be the third time we've mentioned this behavior; is it
perhaps getting redundant?

Section 4.3

   OAM follows the procedures set out in [RFC5085] with the restriction
   that only Virtual Circuit Connectivity Verification (VCCV) type 1 is
   supported.

Earlier we talked about OAM as just using the regular RFC 4385 ACH
method (which is what VCCV type 1 corresponds to); why is it necessary
to pull in the RFC 5085 procedures now (especially when we follow up two
paragraphs down with "the reader is referred to [RFC5058] for a more
detailed description")?

Section 4.4.1

   into and from an H-LSP.  Both carried LSPs and H-LSPs may or may not
   use the TC field, i.e., L-LSPs or E-LSPs.  Such nodes will need to

Perhaps a RFC 3270 reference is in order here, to pick up L-LSPs and
E-LSPs?  We seem to not really mention it in this context until Section
4.6.1 otherwise.

   Additional details of the traffic control capabilities needed at a
   DetNet-aware node may be covered in the new service definitions
   mentioned above or in separate future documents.  Controller plane

Er, mentioned where?  This seems to be the first instance of "service
definition" in this document.

Section 4.5

   latency the impact on the DetNet application would be severe.  To
   avoid the problem of a transient forwarding loop, changes to an LSP
   supporting DetNet MUST be loop-free.

Just to check my understanding: it's possible to get this loop-free
behavior with all the common control planes, e.g., RSVP-TE, MPLS-TP,
MPLS SR, etc.?

Section 5

   and hybrid combinations of the two.  The details of the controller
   plane solution required for the label distribution and the management
   of the label number space are out of scope of this document.  There

The details are out of scope, sure, but the requirements for what it
needs to provide seem to be in scope.  It looks like this is what is in
the following sub-sections, which is good, but perhaps the transition to
them could be written more explicitly.

(I did not try to cross-reference the lists given here with the in-line
requirements stated throughout the document, and trust the authors to
have done so.)

Section 5.1.1

Section 6

   plane.  The considerations raised related to MPLS networks in general
   in [RFC5920] are equally applicable to the the DetNet MPLS data
   plane.

In line of Roman's remarks, I'd suggest calling out a few key pieces
from the RFC 5920 security considerations, especially boundary filtering
to protect against label spoofing.

We might pull in the considerations from the relevant control word RFCs
as well, and from RFC 4206 for H-LSPs.

Some of the service label stuff is fairly analogous to the ongoing SFC
work, but it's probably a stretch to say that we should reference their
security considerations from here.

There's a couple chunks of text that are essentially copied from RFC
8655 but seem incoherent or incorrect, e.g.:

   Security aspects which are unique to DetNet are those whose aim is to
   provide the specific quality of service aspects of DetNet, which are

(It's DetNet's aim to provide those, not the security aspects' aim.)

   traffic.  To prevent DetNet packets from being delayed by an entity
   external to a DetNet domain, DetNet technology definition can allow
   for the mitigation of Man-In-The-Middle attacks, for example through
   use of authentication and authorization of devices within the DetNet
   domain.

(I don't think we've seen a clear portrayal of what these MITM
protection schemes would actually do, what is being
authenticated/authorized, etc., any of the times that it has come up.)

I hope we can improve them in this iteration.

Section 9

We're already over the 5-author limit; I, for one, would not mind having
7 authors (and skipping this section) instead of the current 6.

Section 10.1

Some of these may not strictly need to be in the normative references
section, e.g., 2211/2212, but it doesn't seem particularly harmful to
keep them here.

Section 10.2

I'm quite surprised that draft-ietf-detnet-data-plane-framework is not
listed as normative.

The reference to RFC 5586 is in "an OAM Associated Channel Header (ACH)
[RFC4385] together with a Generic Associated Channel Label (GAL)
[RFC5586] MAY be used in place of a d-CW", the normative keyword of
which suggests that it should properly be classified as normative.

Likewise for RFC 6790 ("Similarly, an Entropy Label Indicator/Entropy
Label (ELI/EL) [RFC6790] MAY be carried below the S-Label").

Thanks for the discussion and resolution of the issues I raised.

I found the position of the OAM indicator to be somewhat confusing.  4.2 says it happens in the d-cw layer but there is no mentioning of it in the corresponding section, 4.2.1, instead buried in the S-label text.

More importantly, it is correct that an OAM packet cannot test the PREOF functions because it does not have a d-cw, and therefore no sequence number? If so, that seems like a significant capability gap.

4.2.2. I would like to see some guidance for the POF and PEF as to how long it should store observed sequence numbers. This would appear to be one of the harder design decisions in the space, and some general principles would be helpful. For example, are there certain latency guarantees in the Detnet, so that after some time the sequence number can be safely discarded?

Hi,

thank you for this document. I have few COMMENTs.

I support Magnus' DISCUSS. I think implementing replication/duplicate elimination is non trivial at layer "2.5" and implementers would, I guess, benefit from extra clarifications. I'm not an expert in that field but I sense some challenges around the size of the tables needed to keep track of the received S/N, associated expiry timers, race conditions, ...

   A DetNet control word (d-CW) conforms to the Generic PW MPLS Control
   Word (PWMCW) defined in [RFC4385].
This kind of suggests that there is a *single* format for d-CW which is:
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0 0 0 0|                Sequence Number                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

but then
   As shown in Figure 3 of [RFC5085] when the first nibble of the d-CW
   is 0x0 the payload following the d-CW is normal user data.  However,
   when the first nibble of the d-CW is 0X1, the payload that follows
   the d-CW is an OAM payload with the OAM type indicated by the value
   in the d-CW Channel Type field.
suggests that, for OAM, d-CW format becomes
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0 0 0 1|Version|   Reserved    |         Channel Type          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Is that a d-CW with a different format or is that something else (an OAM ACH) as this text suggests:
   To support service sub-layer level OAM, an OAM Associated Channel
   Header (ACH) [RFC4385] together with a Generic Associated Channel
   Label (GAL) [RFC5586] MAY be used in place of a d-CW.


   The sequence number length MUST be provisioned on a per Detnet
   service basis via configuration, i.e., via the controller plane
   described in [I-D.ietf-detnet-data-plane-framework].
Should there be a requirement that the sequence number length be the same on both ends of the service?


   In some PREOF topologies, the node performing
   replication will be sending to multiple nodes performing PEF or POF,
   and may need to send different S-Label values for the different
   member flows for the same DetNet service.

   When replication is
   configured, the same app-flow data will be sent over multiple
   outgoing DetNet member flows using forwarding sub-layer LSPs.  An
   S-Label value MUST be configured per outgoing member flow.
It's not clear to me whether that last sentence means "a *different* S-Label value per ..." but I kind of sense a conflict between these two paragraphs. Am I wrong?


   zero or more F-Labels and optionally, the incoming interface,
   proceeding the S-Label MUST be used
Maybe I'm not reading this correctly, but if using the incoming interface is optional, and using zero F-labels is allowed, then it seems to me that this sentence allows for "nothing MUST be used"

Also, right after you say:
   The incoming interface MAY also be used ...
So I wonder if mentioning the incoming interface in the previous sentence is necessary.


   The edge and relay node internal procedures related to PREOF are
   implementation specific.  The order of a packet elimination or
   replication is out of scope in this specification.

   For example, a relay node that
   performs both PEF and PRF first performs PEF on incoming packets to
   create a compound flow.  It then performs PRF and copies the app-flow
   data and the d-CW into packets for each outgoing DetNet member flow.
Again, I kind of sense a conflict between these two paragraphs (first says order is OOS, second gives an order). Am I wrong?


   RFC7322 limits the number of authors listed on the front page of a
   draft to a maximum of 5.  The editor wishes to thank and acknowledge
   the follow author for contributing text to this draft.
I see 6 authors on front page. I don't have any specific opinion on that but there seems to be a mismatch between that text and the current number of authors on front page.


s/MAY required that PEF/MAY require that PEF/
s/F-Labels are supported the DetNet forwarding sub-layer./F-Labels are supported by the DetNet forwarding sub-layer./
s/Valid values included/Valid values include/
s/traffic paramters/traffic parameters/
s/the follow author/the following author/

Discuss resolved.  

Previous discuss comment:

Hi,

Thank you for this document.

Hopefully a trivial discuss to resolve ...

4.2.1.  DetNet Control Word and the DetNet Sequence Number

Does this section need to specify the initial value for the sequence number for a new flow?

Regards,
Rob