Label Switched Path (LSP) Ping and Traceroute Multipath Support for Link Aggregation Group (LAG) Interfaces
draft-ietf-mpls-lsp-ping-lag-multipath-08

[Ballot comment]
Very useful document and techniques; but, I am afraid that I have some issues with this document in its present form: nothing on the actual technique but rather on how the specification is written.

COMMENTS

1) Section 3, I wonder why the "LSR Capability Discovery" TLV is not part of another document: it seems so important to me that it would have deserved its own document (and avoiding the fate sharing with the LAG discovery). Probably too late to change anyway.

2) Section 3.2, while this section is about the generic discovery TLV, the text in 3.2 is only about "LAG Description Indicator" flag. This text should rather be in Section 4.

3) Traceroute with TTL expiring, will it require the 'expiring' LSR to check for capability discovery ? Or is it a 2-step procedure (discover the path, then ask for capabilities)?

4) Section 8, unclear to me where "Local Interface Index" is coming from... is it an opaque value for the initiator or does it have any semantic ? Same question applies to section 9 of course.

5) Section 10, in IPv6 any interface can have multiple IPv6 addresses, so, the text such as "or the interface address" is not applicable, suggestion "or any interface global address" (which can be ambiguous as well, perhaps the 'lowest' address ?)

NITS

N1) Section 3.1, "it can send an MPLS each request message" I cannot parse this part of the sentence

N2)Section 3.2, "When a responder LSR received" the use of past tense seems weird in this sentence, esp when present tense is use after.

[Ballot discuss]
Thanks for the updates in the -07; we seem to be in agreement on the main
path forward.  That said, in Section 4.2 we still see that:

  Based on the procedures described above, every LAG member link will
  have a Local Interface Index Sub-TLV and a Multipath Data Sub-TLV
  entries in the DDMAP TLV.  The order of the Sub-TLVs in the DDMAP TLV
  for a LAG member link MUST be Local Interface Index Sub-TLV
  immediately followed by Multipath Data Sub-TLV.  A LAG member link
  MAY also have a corresponding Remote Interface Index Sub-TLV.  When a
  [...]

I think we need "except as follows" or similar at the end of the second sentence, since
otherwise we go on to have a MUST that contradicts the "MUST be Local Interface Index [...]
immediately followed by Multipath Data".

(Also, we missed one instance of "optional" in Section 8: "Local Interface Index Sub-TLV is an optional TLV")

[Ballot comment]
A couple nits in the new text in the -07: "an new" (should be "a new") and
"mechanims" (should be "mechanism").

[the following COMMENT portion has not been revised since the ballot on the -06
and may contain stale information]

Am I reading this correctly that the "LSR Capability TLV" created here
is basically only applicable to properties of MPLS echo request/reply
exchanges?  If so, then perhaps the moniker "LSR Capability" is overly
generic.

Section 1.2

  o  Label switching over some member links of the LAG is successful,
      but will be failed over other member links of the LAG.

nit: there's some verb tense inconsistency here; maybe "but fails over
other member links" would help.

Section 2

  This document defines an optional TLV to discover the capabilities of
  a responder LSR and extensions for use with the MPLS LSP Ping and

nit: is it only the *responder* LSR we care about, or are there
potentially intermediaries in play?

  The solution consists of the MPLS echo request containing a DDMAP TLV
  and the optional LSR capability TLV to indicate that separate load

nit: Is this "optional capability TLV" the same "optional TLV" that we are
defining in this document?  If so, calling it "the new optional LSR
capability" would aid clarity.  (Also, we seem to use both the
"capability" and "Capability" capitzliations for describing this TLV.)

Section 3

  Capability TLV in the MPLS echo request message.  When the responder
  LSR receives an MPLS echo request message with the LSR Capability TLV
  included, if it knows the LSR Capability TLV, then it MUST include
  the LSR Capability TLV in the MPLS echo reply message with the LSR
  Capability TLV describing features and extensions supported by the
  local LSR.  Otherwise, an MPLS echo reply MUST be sent back to the
  initiator LSR with the return code set to "One or more of the TLVs
  was not understood".  [...]

This last MUST seems like something that this document cannot mandate
(as it attempts to apply to non-implementations of this document); it
could only work if it was an existing requirement from the core MPLS
spec, in which case lower-case "must" is appropriate (possibly with
section reference).

  o  If the responder LSR does not understand the "LAG Description
      Indicator flag":

      *  Clear both the "Downstream LAG Info Accommodation flag" and the
        "Upstream LAG Info Accommodation flag".

Similarly, if an LSR does not understand a flag, it cannot give special
treatment to packets containing that flag.  (Why an LSR would not
understand a flag defined in the same document that defines the
Capabilities TLV is beyond me, but you seem to want to allow for that
case.)

  If the responder does not know the LSR Capability TLV, an MPLS echo
  reply with the return code set to "One or more of the TLVs was not
  understood" MUST be sent back to the initiator LSR.

This duplicates the content I quoted at the top of this section's
comments.

Section 4.1

  Once the initiator LSR knows the capabilities that a responder
  supports, then it sends an MPLS echo request carrying a DDMAP with
  the "LAG Description Indicator flag" (G) set to the responder LSR.

nit: I think the key point is not that the initiator knows the
capabilities of the peer, but rather that the initiator knows that the
peer supports this specific mechanism.
Also, is this "a DDMAP TLV"?

Section 4.2

Reading this makes it sound like the "LAG Description Indicator flag" is
completely orthogonal to regular DDMAP functionality, so that if I set
the LAG description indicator flag but do not otherwise request detailed
descriptions, only the LAG interfaces are returned.  However, the flag
in question has to be inside a DDMAP TLV (and we should really say so,
perhaps as "[flag] set *in the DDMAP TLV*"!), so it seems like in practice
the LAG information can only be obtained in conjunction with full
detailed description information.  (The specific suggestion here is to
note clearly that the flag is se in the DDMAP TLV.)

      *  The responder LSR MUST include a DDMAP TLV when sending the
        MPLS echo reply.

I got confused on my first pass through this section; adding "There is a
single DDMAP TLV for the LAG interface, with member links described
using sub-TLVs" here would have kept me from veering onto the wrong path.
(I thought there was going to be one DDMAP TLV per member link, plus one
for the LAG aggregate, with lots of duplication.)

  Based on the procedures described above, every LAG member link will
  have a Local Interface Index Sub-TLV and a Multipath Data Sub-TLV
  entries in the DDMAP TLV.  The order of the Sub-TLVs in the DDMAP TLV
  for a LAG member link MUST be Local Interface Index Sub-TLV
  immediately followed by Multipath Data Sub-TLV.  A LAG member link
  MAY also have a corresponding Remote Interface Index Sub-TLV.  When a
  Local Interface Index Sub-TLV, a Remote Interface Index-Sub-TLV and a
  Multipath Data Sub-TLV are placed in the DDMAP TLV to describe a LAG
  member link, they MUST be placed in the order of Local Interface
  Index Sub-TLV, Remote Interface Index-Sub-TLV and Multipath Data Sub-
  TLV.

This last MUST contradicts the previous MUST.  I suggest some more text
to clarify under which conditions each requirement applies.

I'd also suggest not using the figure for conveying a (apparent?) mandatory
requirement, and instead adding some text at the end: "The block of local
interface index, (optional remote interface index) and multipath data
sub-TLVs for each member link MUST appear adjacent to each other in
order of increasing local interface index."

It's confusing to label the multiplath data sub-TLV as "MANDATORY" when
the body text says "MUST add an [sic] Multipath data Sub-TLV [...] if
the received DDMAP TLV requested multipath information", i.e., it is not
always present, based on the contents of the echo request.

  When none of the received multipath information maps to a particular
  LAG member link, then the responder LSR MUST still place the Local
  Interface Index Sub-TLV and the Multipath Data Sub-TLV for that LAG
  member link in the DDMAP TLV, the value of Multipath Length field of
  the Multipath Data Sub-TLV is set to zero.

nit: the last comma is a comma splice.

Section 5.1.2

It seems like this places a slightly stronger burden on the responder
than stock 8029 does, in that there is a MUST-level requirement to act
on the 'I' flag in combination with the 'G' flag, whereas for the 'I'
flag alone 8029 only has a SHOULD-level requirement to respond
accordingly.  We may want to call this out as a change in behavior.

Section 5.1.3

  Expectation is that there's a relationship between the interface
  index of the outgoing LAG member link at TTL=n and the interface
  index of the incoming LAG member link at TTL=n+1 for all discovered
  entropies.  In other words, set of entropies that load balances to

nit: "The expectation"
nit: "discovered entropies" is a strange wording given how the entropy
label works; is this more like "all entropies examined"?


  The initiator LSR sends two MPLS echo request messages to traverse
  the two LAG member links at TTL=n+1:

How does the initiator know (which entropy label values?) to get the echo
requests to traverse different member links?

  o  Error case:

      *  One MPLS echo request message reaches TTL=n+1 on an LAG member
        link 1.

      *  The other MPLS echo request message also reaches TTL=n+1 on an
        LAG member link 1.

      One or two MPLS echo request messages sent by the initiator LSR
      cannot reach the immediate downstream LSR, or the two MPLS echo
      request messages reach at the immediate downstream LSR from the
      same LAG member link.

The description paragraph doesn't seem to match up the scenario
described in the sub-bullets, which leaves me confused as to what
scenario(s) are intended to be considered as the "error case".

Section 6

  This document defines a new optional TLV which is referred to as the
  "LSR Capability TLV.  [...]

Is this optional to use or comprehension-optional?  (We elsewhere rely
on comprehension-mandatory behavior, so in the RFC 8029 terminology this
seems to be a "mandatory TLV" even if it is not mandatory to use.)

  included in the MPLS echo reply message.  Otherwise, if the responder
  does not know the LSR Capability TLV, an MPLS echo reply with the
  return code set to "One or more of the TLVs was not understood" MUST
  be sent back to the initiator LSR.

This duplicates a requirement stated elsewhere (that really ought to
just be using existing required behavior from 8029 anyway).

  LSR Capability TLV Type is TBD1.  Length is 4.  The value field of

In a hypothetical future where we need to allocate a 33rd capability
flag, would we rather create a new "extended capabilities" TLV (burning
another 32 bits for type+length) or leave flexibility now for 'length'
to increase in multiples of 4 with receivers ignoring bits past what
they know about?

      The "LSR Capability Flags" field is 4 octets in length, this
      document defines the following flags:

      0                  1                  2                  3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                  Must Be Zero (Reserved)                  |U|D|
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

I'm wary of describing these as "Must Be Zero", since they're just
unallocated but not required to be zero other than by the current state
of bit allocations.  (I guess this is conventional for MPLS, though?  So
maybe it's not a problem.)  Just "Reserved" would be fine.

      This document defines two flags.  The remaining flags MUST be set
      to zero when sending and ignored on receipt.  Both the U and the D
      flag MUST be cleared in the MPLS echo request message when
      sending, and ignored on receipt.  Neither, either or both the U
      and the D flag MAY be set in the MPLS echo reply message.

Similarly, I think we want to say "unallocated flags" rather than "the
remaining flags".

It also seems like the semantics being described here are "the TLV value
is ignored in the echo request and only has meaning in the response"
(i.e., this is a "tell me your capabilities" exchange, not a "here are
mine; what are yours?" exchange).  As written, that applies only to the
U and D bits, and is not a general property.  That may well be a fine
choice if we think we might want the two-way exchange for some future
allocated bit, but if we do think it's a general property of the
mechanism, I'd suggest rewording the text.

Section 7

(Per IANA, bits 4 and 5 are already assigned but are not reflected in the
diagram.)

Section 10

  The Detailed Interface and Label Stack TLV format is derived from the
  Interface and Label Stack TLV format (from [RFC8029]).  Two changes
  are introduced.  The first is that the label stack is converted into
  a sub-TLV.  The second is that a new sub-TLV is added to describe an
  interface index.  These fields of Detailed Interface and Label Stack
  TLV have the same use and meaning as in [RFC8029].  A summary of
  these fields is as below:

nit: Are "These fields" just "The other fields"?

        The Address Type indicates if the interface is numbered or
        unnumbered.  It also determines the length of the IP Address
        and Interface fields.  The resulting total length of the
        initial part of the TLV is listed as "K Octets".  The Address
        Type is set to one of the following values:

nit: is it better to list the currently allocated values or just refer
to the IANA registry?

I seee that this "index assigned to the interface" language for
unnumbered address types originates from RFC 8029, but both this
document and 8029 leave me confused as to the encoding used for this
index (for either/both IPv4 or IPv6 address types).

[My comment about "Must Be Zero" and allocatable bits also applies here]

Section 12

This document also adds a capability "negotiation" mechanism for MPLS
echo request/reply exchanges.  As MPLS does not offer integrity
protection for its messages, this negotiation is only as trustworthy as
the network from which messages are accepted as valid, and initiators
have no recourse but to accept faithfully the echo replies received.

  To prevent leakage of vital information to untrusted users, a
  responder LSR MUST only accept MPLS echo request messages from
  designated trusted sources via filtering source IP address field of
  received MPLS echo request messages.  [...]

I'd prefer to see a note added here about how "source IP address
filtering provides only a weak form of access control and is not, in
general, a reliable security mechanism.  Nonetheless, it is required
here in the absence of any more robust mechanism that might be used."

Section 13

Should we ask IANA to update the "Interface and Label Stack Address
Types" registry to also refer to this document, since we are sharing the
registry between the Interface and Label Stack TLV and the Detailed
version?

Section 13.2

The range 4-31743 spans two different allocation procedures (Standards
Action and Specification Required - Experimental RFC needed); I assume
that we want the 4-16383 range for Standards Action - mandatory TLVs.

Section 13.4

We don't say what range we want this to be allocated from -- I assume
4-16383 since this is a negotiated feature and sending it outside the
negotiated scenario should be an error.

Section 13.4.1

(I note that some existing sub-TLV registries have "Experimental RFC
required" for the experimental ranges, but that is arguably more
stringent than necessary; Specification Required seems more appropriate
for this case.)

Appendix A

I was kind of hoping for some more guidance on what an initiator could
do in these scenarios to try to get better visibility into the switch
behavior (and, hopefully, a workaround to get reliable echo responses
that cover all LAG member links).  AFAICT there's not a better option
than "try a bunch of entropy labels and see what responses you can get
back" and that's the same remedy in all the described topologies, but it
would be nice to have this stated explicitly for the reader.

Section A.1

                    In the worst case, MPLS echo request messages with
  specific entropies to exercise every LAG members from R1 to S1 can
  all reach R2 via same LAG member.  [...]

I'm confused by the phrase "specific entropies to exercise every LAG
members [sic] from R1 to S1" -- is there some deterministic algorithm by
which a specific entropy label value will force a specific LAG member
link?  My understanding was that the entropy was used as input to an
opaque hash function and that trial-and-error was the only way to
explore the mapping from entropy value to LAG member link taken.

[Ballot comment]
Due to lack of time (traveling) I'm balloting NoObj in the "I read the protocol action, and I trust the sponsoring AD so have no problem." sense. I would like to have been able to fully review it, as it sounds both fascinating and useful, but ...

[Ballot discuss]
Thanks for this document; it's clearly going to provide value, and it
gives a pretty well-readable description of how things are expected to
work.  That said, there's a number of details that need to be worked out
before publication...

"optional TLV" seems to be used in MPLS contexts as a technical term for
"comprehension-optional", not in the "optional to send" sense; it's the
counterpart of "mandatory TLV", and this terminology is even documented
in the MPLS TLVs registry.  It's my understanding that the LSP
Capabilities TLV and the Detailed Interface and Label Stack TLV are
intended to be comprehension-required (i.e., "mandatory TLVs"), and thus
that we must not use the phrase "optional TLV" in their description.
This would be made clear if we listed which range of values we intended
to allocate a TLV type from, in the IANA considerations, but that
remains unspecified at this time.

In a similar vein (the "comprehension-required" behavior), there are a
few places (Section 3 (twice), and Section 6; also Section 3 for the LAG
Description Indicator flag; see COMMENT) where we state new normative
language ("MUST") that is unenforceable, since it would need to apply to
MPLS implementations that do not implement this specification.
Fortunately, the comprehension-required TLV ranges provide this
functionality for us without the need to use normative language.

Section 4.2 has some conflicting "MUST"s about the ordering/presence of
sub-TLVs in the DDMAP TLV (see COMMENT, and also the "MANDATORY"
langauge in Figure 2).

If I'm reading Section 5.1.2 correctly, the described procedure is only
intended to apply when the "G" flag is present (as well as the "I" flag,
the requirement for which is explicitly stated), but this is not
explicitly stated.  In particular, the text as written says it applies
to all responders that "understand the LAG Description Indicator flag"
with no mention of runtime check for the presence of that flag.

I also worry that Sections 8, 9, and 10 are insufficiently clear about the
encoding of the interface index value -- is it an integer in NBO, an
opaque bitstring, or something else?

[Ballot discuss]
Thanks for this document; it's clearly going to provide value, and it
gives a pretty well-readable description of how things are expected to
work.  That said, there's a number of details that need to be worked out
before publication...

This document has six listed authors; per RFC 7322, documents listing
more than five authors are unusual, and six is greater than five.  Let's
talk about the author count.

"optional TLV" seems to be used in MPLS contexts as a technical term for
"comprehension-optional", not in the "optional to send" sense; it's the
counterpart of "mandatory TLV", and this terminology is even documented
in the MPLS TLVs registry.  It's my understanding that the LSP
Capabilities TLV and the Detailed Interface and Label Stack TLV are
intended to be comprehension-required (i.e., "mandatory TLVs"), and thus
that we must not use the phrase "optional TLV" in their description.
This would be made clear if we listed which range of values we intended
to allocate a TLV type from, in the IANA considerations, but that
remains unspecified at this time.

In a similar vein (the "comprehension-required" behavior), there are a
few places (Section 3 (twice), and Section 6; also Section 3 for the LAG
Description Indicator flag; see COMMENT) where we state new normative
language ("MUST") that is unenforceable, since it would need to apply to
MPLS implementations that do not implement this specification.
Fortunately, the comprehension-required TLV ranges provide this
functionality for us without the need to use normative language.

Section 4.2 has some conflicting "MUST"s about the ordering/presence of
sub-TLVs in the DDMAP TLV (see COMMENT, and also the "MANDATORY"
langauge in Figure 2).

If I'm reading Section 5.1.2 correctly, the described procedure is only
intended to apply when the "G" flag is present (as well as the "I" flag,
the requirement for which is explicitly stated), but this is not
explicitly stated.  In particular, the text as written says it applies
to all responders that "understand the LAG Description Indicator flag"
with no mention of runtime check for the presence of that flag.

I also worry that Sections 8, 9, and 10 are insufficiently clear about the
encoding of the interface index value -- is it an integer in NBO, an
opaque bitstring, or something else?

[Ballot comment]
Am I reading this correctly that the "LSR Capability TLV" created here
is basically only applicable to properties of MPLS echo request/reply
exchanges?  If so, then perhaps the moniker "LSR Capability" is overly
generic.

Section 1.2

  o  Label switching over some member links of the LAG is successful,
      but will be failed over other member links of the LAG.

nit: there's some verb tense inconsistency here; maybe "but fails over
other member links" would help.

Section 2

  This document defines an optional TLV to discover the capabilities of
  a responder LSR and extensions for use with the MPLS LSP Ping and

nit: is it only the *responder* LSR we care about, or are there
potentially intermediaries in play?

  The solution consists of the MPLS echo request containing a DDMAP TLV
  and the optional LSR capability TLV to indicate that separate load

nit: Is this "optional capability TLV" the same "optional TLV" that we are
defining in this document?  If so, calling it "the new optional LSR
capability" would aid clarity.  (Also, we seem to use both the
"capability" and "Capability" capitzliations for describing this TLV.)

Section 3

  Capability TLV in the MPLS echo request message.  When the responder
  LSR receives an MPLS echo request message with the LSR Capability TLV
  included, if it knows the LSR Capability TLV, then it MUST include
  the LSR Capability TLV in the MPLS echo reply message with the LSR
  Capability TLV describing features and extensions supported by the
  local LSR.  Otherwise, an MPLS echo reply MUST be sent back to the
  initiator LSR with the return code set to "One or more of the TLVs
  was not understood".  [...]

This last MUST seems like something that this document cannot mandate
(as it attempts to apply to non-implementations of this document); it
could only work if it was an existing requirement from the core MPLS
spec, in which case lower-case "must" is appropriate (possibly with
section reference).

  o  If the responder LSR does not understand the "LAG Description
      Indicator flag":

      *  Clear both the "Downstream LAG Info Accommodation flag" and the
        "Upstream LAG Info Accommodation flag".

Similarly, if an LSR does not understand a flag, it cannot give special
treatment to packets containing that flag.  (Why an LSR would not
understand a flag defined in the same document that defines the
Capabilities TLV is beyond me, but you seem to want to allow for that
case.)

  If the responder does not know the LSR Capability TLV, an MPLS echo
  reply with the return code set to "One or more of the TLVs was not
  understood" MUST be sent back to the initiator LSR.

This duplicates the content I quoted at the top of this section's
comments.

Section 4.1

  Once the initiator LSR knows the capabilities that a responder
  supports, then it sends an MPLS echo request carrying a DDMAP with
  the "LAG Description Indicator flag" (G) set to the responder LSR.

nit: I think the key point is not that the initiator knows the
capabilities of the peer, but rather that the initiator knows that the
peer supports this specific mechanism.
Also, is this "a DDMAP TLV"?

Section 4.2

Reading this makes it sound like the "LAG Description Indicator flag" is
completely orthogonal to regular DDMAP functionality, so that if I set
the LAG description indicator flag but do not otherwise request detailed
descriptions, only the LAG interfaces are returned.  However, the flag
in question has to be inside a DDMAP TLV (and we should really say so,
perhaps as "[flag] set *in the DDMAP TLV*"!), so it seems like in practice
the LAG information can only be obtained in conjunction with full
detailed description information.  (The specific suggestion here is to
note clearly that the flag is se in the DDMAP TLV.)

      *  The responder LSR MUST include a DDMAP TLV when sending the
        MPLS echo reply.

I got confused on my first pass through this section; adding "There is a
single DDMAP TLV for the LAG interface, with member links described
using sub-TLVs" here would have kept me from veering onto the wrong path.
(I thought there was going to be one DDMAP TLV per member link, plus one
for the LAG aggregate, with lots of duplication.)

  Based on the procedures described above, every LAG member link will
  have a Local Interface Index Sub-TLV and a Multipath Data Sub-TLV
  entries in the DDMAP TLV.  The order of the Sub-TLVs in the DDMAP TLV
  for a LAG member link MUST be Local Interface Index Sub-TLV
  immediately followed by Multipath Data Sub-TLV.  A LAG member link
  MAY also have a corresponding Remote Interface Index Sub-TLV.  When a
  Local Interface Index Sub-TLV, a Remote Interface Index-Sub-TLV and a
  Multipath Data Sub-TLV are placed in the DDMAP TLV to describe a LAG
  member link, they MUST be placed in the order of Local Interface
  Index Sub-TLV, Remote Interface Index-Sub-TLV and Multipath Data Sub-
  TLV.

This last MUST contradicts the previous MUST.  I suggest some more text
to clarify under which conditions each requirement applies.

I'd also suggest not using the figure for conveying a (apparent?) mandatory
requirement, and instead adding some text at the end: "The block of local
interface index, (optional remote interface index) and multipath data
sub-TLVs for each member link MUST appear adjacent to each other in
order of increasing local interface index."

It's confusing to label the multiplath data sub-TLV as "MANDATORY" when
the body text says "MUST add an [sic] Multipath data Sub-TLV [...] if
the received DDMAP TLV requested multipath information", i.e., it is not
always present, based on the contents of the echo request.

  When none of the received multipath information maps to a particular
  LAG member link, then the responder LSR MUST still place the Local
  Interface Index Sub-TLV and the Multipath Data Sub-TLV for that LAG
  member link in the DDMAP TLV, the value of Multipath Length field of
  the Multipath Data Sub-TLV is set to zero.

nit: the last comma is a comma splice.

Section 5.1.2

It seems like this places a slightly stronger burden on the responder
than stock 8029 does, in that there is a MUST-level requirement to act
on the 'I' flag in combination with the 'G' flag, whereas for the 'I'
flag alone 8029 only has a SHOULD-level requirement to respond
accordingly.  We may want to call this out as a change in behavior.

Section 5.1.3

  Expectation is that there's a relationship between the interface
  index of the outgoing LAG member link at TTL=n and the interface
  index of the incoming LAG member link at TTL=n+1 for all discovered
  entropies.  In other words, set of entropies that load balances to

nit: "The expectation"
nit: "discovered entropies" is a strange wording given how the entropy
label works; is this more like "all entropies examined"?


  The initiator LSR sends two MPLS echo request messages to traverse
  the two LAG member links at TTL=n+1:

How does the initiator know (which entropy label values?) to get the echo
requests to traverse different member links?

  o  Error case:

      *  One MPLS echo request message reaches TTL=n+1 on an LAG member
        link 1.

      *  The other MPLS echo request message also reaches TTL=n+1 on an
        LAG member link 1.

      One or two MPLS echo request messages sent by the initiator LSR
      cannot reach the immediate downstream LSR, or the two MPLS echo
      request messages reach at the immediate downstream LSR from the
      same LAG member link.

The description paragraph doesn't seem to match up the scenario
described in the sub-bullets, which leaves me confused as to what
scenario(s) are intended to be considered as the "error case".

Section 6

  This document defines a new optional TLV which is referred to as the
  "LSR Capability TLV.  [...]

Is this optional to use or comprehension-optional?  (We elsewhere rely
on comprehension-mandatory behavior, so in the RFC 8029 terminology this
seems to be a "mandatory TLV" even if it is not mandatory to use.)

  included in the MPLS echo reply message.  Otherwise, if the responder
  does not know the LSR Capability TLV, an MPLS echo reply with the
  return code set to "One or more of the TLVs was not understood" MUST
  be sent back to the initiator LSR.

This duplicates a requirement stated elsewhere (that really ought to
just be using existing required behavior from 8029 anyway).

  LSR Capability TLV Type is TBD1.  Length is 4.  The value field of

In a hypothetical future where we need to allocate a 33rd capability
flag, would we rather create a new "extended capabilities" TLV (burning
another 32 bits for type+length) or leave flexibility now for 'length'
to increase in multiples of 4 with receivers ignoring bits past what
they know about?

      The "LSR Capability Flags" field is 4 octets in length, this
      document defines the following flags:

      0                  1                  2                  3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                  Must Be Zero (Reserved)                  |U|D|
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

I'm wary of describing these as "Must Be Zero", since they're just
unallocated but not required to be zero other than by the current state
of bit allocations.  (I guess this is conventional for MPLS, though?  So
maybe it's not a problem.)  Just "Reserved" would be fine.

      This document defines two flags.  The remaining flags MUST be set
      to zero when sending and ignored on receipt.  Both the U and the D
      flag MUST be cleared in the MPLS echo request message when
      sending, and ignored on receipt.  Neither, either or both the U
      and the D flag MAY be set in the MPLS echo reply message.

Similarly, I think we want to say "unallocated flags" rather than "the
remaining flags".

It also seems like the semantics being described here are "the TLV value
is ignored in the echo request and only has meaning in the response"
(i.e., this is a "tell me your capabilities" exchange, not a "here are
mine; what are yours?" exchange).  As written, that applies only to the
U and D bits, and is not a general property.  That may well be a fine
choice if we think we might want the two-way exchange for some future
allocated bit, but if we do think it's a general property of the
mechanism, I'd suggest rewording the text.

Section 7

(Per IANA, bits 4 and 5 are already assigned but are not reflected in the
diagram.)

Section 10

  The Detailed Interface and Label Stack TLV format is derived from the
  Interface and Label Stack TLV format (from [RFC8029]).  Two changes
  are introduced.  The first is that the label stack is converted into
  a sub-TLV.  The second is that a new sub-TLV is added to describe an
  interface index.  These fields of Detailed Interface and Label Stack
  TLV have the same use and meaning as in [RFC8029].  A summary of
  these fields is as below:

nit: Are "These fields" just "The other fields"?

        The Address Type indicates if the interface is numbered or
        unnumbered.  It also determines the length of the IP Address
        and Interface fields.  The resulting total length of the
        initial part of the TLV is listed as "K Octets".  The Address
        Type is set to one of the following values:

nit: is it better to list the currently allocated values or just refer
to the IANA registry?

I seee that this "index assigned to the interface" language for
unnumbered address types originates from RFC 8029, but both this
document and 8029 leave me confused as to the encoding used for this
index (for either/both IPv4 or IPv6 address types).

[My comment about "Must Be Zero" and allocatable bits also applies here]

Section 12

This document also adds a capability "negotiation" mechanism for MPLS
echo request/reply exchanges.  As MPLS does not offer integrity
protection for its messages, this negotiation is only as trustworthy as
the network from which messages are accepted as valid, and initiators
have no recourse but to accept faithfully the echo replies received.

  To prevent leakage of vital information to untrusted users, a
  responder LSR MUST only accept MPLS echo request messages from
  designated trusted sources via filtering source IP address field of
  received MPLS echo request messages.  [...]

I'd prefer to see a note added here about how "source IP address
filtering provides only a weak form of access control and is not, in
general, a reliable security mechanism.  Nonetheless, it is required
here in the absence of any more robust mechanism that might be used."

Section 13

Should we ask IANA to update the "Interface and Label Stack Address
Types" registry to also refer to this document, since we are sharing the
registry between the Interface and Label Stack TLV and the Detailed
version?

Section 13.2

The range 4-31743 spans two different allocation procedures (Standards
Action and Specification Required - Experimental RFC needed); I assume
that we want the 4-16383 range for Standards Action - mandatory TLVs.

Section 13.4

We don't say what range we want this to be allocated from -- I assume
4-16383 since this is a negotiated feature and sending it outside the
negotiated scenario should be an error.

Section 13.4.1

(I note that some existing sub-TLV registries have "Experimental RFC
required" for the experimental ranges, but that is arguably more
stringent than necessary; Specification Required seems more appropriate
for this case.)

Appendix A

I was kind of hoping for some more guidance on what an initiator could
do in these scenarios to try to get better visibility into the switch
behavior (and, hopefully, a workaround to get reliable echo responses
that cover all LAG member links).  AFAICT there's not a better option
than "try a bunch of entropy labels and see what responses you can get
back" and that's the same remedy in all the described topologies, but it
would be nice to have this stated explicitly for the reader.

Section A.1

                    In the worst case, MPLS echo request messages with
  specific entropies to exercise every LAG members from R1 to S1 can
  all reach R2 via same LAG member.  [...]

I'm confused by the phrase "specific entropies to exercise every LAG
members [sic] from R1 to S1" -- is there some deterministic algorithm by
which a specific entropy label value will force a specific LAG member
link?  My understanding was that the entropy was used as input to an
opaque hash function and that trial-and-error was the only way to
explore the mapping from entropy value to LAG member link taken.

[Ballot comment]
Thank you for the responses to Jorg's TSV-ART review.

I did see one point in his review, that I'm not seeing a response or document change for.

He said,

1. With the potentially substantial stacking of TLVs, I am wondering how large
  packets can get, especially if numerous links might constitute a LAG and all
  of those are extensively described.  It may be useful to provide the reader
  with some intuition.  There are many useful examples in the document, but
  they all refer to individual fields.  A complete packet could be helpful.

My question is actually a follow-on - in a world where we even tunnel tunnels, are there going to be MTU size issues that might be mentioned?

[Ballot comment]
I wanted to comment on the same sentence/normative requirement as Alvaro did in his point (2). Given Alvaro's additional information that there is actually even a technical conflict with this requirement, I think this should be address before publication and might even be discuss-worthy. However, I'm really not an expert on MPLS and therefore leave the decision to state a discuss ballot position to potentially other, more knowledgable ADs.

Thanks for addressing the TSV-ART review comments (and thanks Jörg for the review)! I support adding another sentence with a pointer to rate-limit requirements in other docs. Thanks for proposing this change. Looking forward this see this in the doc!

[Ballot comment]
(1) The Update to rfc8029 is not clearly explained.  The new functionality introduced in this document is clear, but it seems to me that it is optional from the point of view that it is only needed when LAGs exist, and even then, only the Initiator and the Responder need to implement the enhancements.  IOW, the Update that this document presents is not one that is needed by all rfc8029 implementations.  I'm looking for text that explains that.


(2) §6: "Otherwise, if the responder does not know the LSR Capability TLV, an MPLS echo reply with the return code set to "One or more of the TLVs was not understood" MUST be sent back to the initiator LSR."  Given that this is the case where the new TLV is not known, then we can't Normatively direct those nodes to do anything (since they probably don't implement anything in this document).  s/MUST/must + add a reference to rfc8029 (where the behavior is specified) [The same text appears again in §3 and §3.2.  Writing the specification is one place is enough.]

BTW, according to rfc8029, the return code will be sent back only if the TLV is mandatory, but §6 defines it as optional.  Please be clear and specific about the definition and the instructions to IANA.


(3) This document doesn't talk about what should be done if a response is not received, at any point in the process.  I searched in rfc8029, but didn't find anything related to timeouts...only §4.8 (Non-compliant Routers), which includes a process on what to do if a reply is not received.  That process doesn't seem to apply to this document -- what should an initiator do if a reply is not received?  I am specially interested in the discovery phases.


(4) [nit] In §7, the DS Flags should look like this (see rfc8012):

      0 1 2 3 4 5 6 7
      +-+-+-+-+-+-+-+-+
      | MBZ |G|E|L|I|N|
      +-+-+-+-+-+-+-+-+


(5) RFC8126 should be a Normative reference.

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-mpls-lsp-ping-lag-multipath-05. If any part of this review is inaccurate, please let us know.

The IANA Functions Operator understands that, upon approval of this document, there are eight actions which we must complete.

First, in the TLVs registry on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

a single, new TLV is to be registered as follows:

Type: [ TBD-at-Registration ]
TLV Name: LSR Capability TLV
Reference: [ RFC-to-be ]

Second, a new registry is to be created called the LSR Capability Flags registry. The new registry will be on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

The registration policy for the new registry will be Standards Action as defined by RFC 8126. There are initial registrations in the new registry as follows:

Bit number Name Reference
---------- ---------------------------------------- ---------
31 D: Downstream LAG Info Accommodation [ RFC-to-be ]
30 U: Upstream LAG Info Accommodation [ RFC-to-be ]
0-29 Unassigned

Third, in the Sub-TLVs for TLV Type 20 subregistry of the TLVs registry also on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

a single, new registration is to be made as follows:

Sub-Type: [ TBD-at-Registration ]
Sub-TLV Name: Local Interface Index Sub-TLV
Reference: [ RFC-to-be ]
Comment:

Fourth, a new registry is to be created called the Interface Index Flags registry. The new registry will be on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

The registration policy for the new registry will be Standards Action as defined by RFC 8126. There are initial registrations in the new registry as follows:

Bit number Name Reference
---------- ---------------------------------------- ---------
15 M: LAG Member Link Indicator [ RFC-to-be ]
0-14 Unassigned

Fifth, in the Sub-TLVs for TLV Type 20 subregistry of the TLVs registry also on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

a single, new registration is to be made as follows:

Sub-Type: [ TBD-at-Registration ]
Sub-TLV Name: Remote Interface Index Sub-TLV
Reference: [ RFC-to-be ]
Comment:

Sixth, also in the TLVs registry on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

a single, new TLV is to be registered as follows:

Type: [ TBD-at-Registration ]
TLV Name: Detailed Interface and Label Stack TLV
Reference: [ RFC-to-be ]

Seventh, a new subregistry of the TLVs registry on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

The new registry will be called the:

Sub-TLVs for TLV Type [ TBD-at-Registration ]

where the value of [ TBD-at-Registration ] will match the TLV value registered in the sixth IANA action above.

Assignments of Sub-Types in the mandatory and optional spaces are via Standards Action as defined in RFC 8126. Assignments of Sub-Types in the experimental space is via Specification Required as defined in RFC 8126. There are initial registrations in the new subregistry as follows:

Sub-Type Name Reference
----------- -------------------------------------- ---------
1 Incoming Label Stack [ RFC-to-be ]
2 Incoming Interface Index [ RFC-to-be ]
3-16383 Unassigned (mandatory TLVs)
16384-31743 Experimental
32768-49161 Unassigned (optional TLVs)
49162-64511 Experimental

Eighth, in the DS Flags registry on the Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters registry page located at:

https://www.iana.org/assignments/mpls-lsp-ping-parameters/

a single, new DS Flag is to be registered as follows:

Bit number: [ TBD-at-Registration ]
Name: G: LAG Description Indicator
Reference: [ RFC-to-be ]

The IANA Functions Operator understands that these are the only actions required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2018-12-11):

From: The IESG
To: IETF-Announce
CC: mpls@ietf.org, draft-ietf-mpls-lsp-ping-lag-multipath@ietf.org, mpls-chairs@ietf.org, db3546@att.com, loa@pi.nu
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Label Switched Path (LSP) Ping/Trace Multipath Support for Link Aggregation Group (LAG) Interfaces) to Proposed Standard


The IESG has received a request from the Multiprotocol Label Switching WG
(mpls) to consider the following document: - 'Label Switched Path (LSP)
Ping/Trace Multipath Support for Link
  Aggregation Group (LAG) Interfaces'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2018-12-11. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the beginning of
the Subject line to allow automated sorting.

Abstract


  This document defines extensions to the MPLS Label Switched Path
  (LSP) Ping and Traceroute mechanisms as specified in RFC 8029.  The
  extensions allow the MPLS LSP Ping and Traceroute mechanisms to
  discover and exercise specific paths of Layer 2 (L2) Equal-Cost
  Multipath (ECMP) over Link Aggregation Group (LAG) interfaces.
  Additionally, a mechanism is defined to enable determination of the
  capabilities of an LSR supported.

  This document updates RFC8029.





The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-mpls-lsp-ping-lag-multipath/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-mpls-lsp-ping-lag-multipath/ballot/

The following IPR Declarations may be related to this I-D:

  https://datatracker.ietf.org/ipr/2496/
  https://datatracker.ietf.org/ipr/3192/

The MPLS wg requests that

  draft-ietf-mpls-lsp-ping-lag-multipath-04

is published as an RFC on the Standards Track

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

  We request that the document is published as a Proposed Standard.
  Proposed Standard is the correct type of RFC since the document
  specifies new protocol and protocol element, and creates IANA
  registries that allocate code-points through standards action,
  the document also allocates code-points from IANA registries that
  requires Standards Action allocation.
 
  The document header says: Standard Tracks.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  This document defines an extension to the MPLS Label Switched Path
  (LSP) Ping and Traceroute as specified in RFC 8029.  The extension
  allows the MPLS LSP Ping and Traceroute to discover and exercise
  specific paths of Layer 2 (L2) Equal-Cost Multipath (ECMP) over
  Link Aggregation Group (LAG) interfaces.

  MPLS LSP Ping and Traceroute tools were not designed to discover
  and exercise specific paths of L2 ECMP.  The result is certain
  limitations when LSP Ping is used with LAG.

  MPLS LSP Ping and Traceroute are not able to detect the presence of and
  localise certain LSP failures on all member links of the LAG. 

  Creation of this document was motivated by issues encountered
  in live networks.

Working Group Summary

  The only thing is that as the document was going through a first
  working group last call, most of the original authors had changed
  affiliation, and the document progress became slow. At one point
  in time the wg chairs thought the document would be abandonned, but
  when we solicitited the interest for progressing the document we
  found quite a bit of interest.
  The working group chairs then appointed an editor and the progress
  has been smooth after that. However, as a result the document has now
  six authors. The authors, wg chairs and the working group are
  comfortable with 6 authors.

  There were two working group last calls. The second one was issued
  because there was a long time between the first and the point in
  time when all comments were addressed.

  There is a solid support for this document within the working group.

Document Quality

  Yes we know of implementations of this protocol.


Personnel

  Loa ANdersson is the Document Shepherd.
  Deborah Brungard is the Responsible Area Director.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

  The document shepherd reviewed the document when it was first
  posted, as part of the working group adoption process and when
  preparing the two wglcs.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

  No such concerns.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

  No such reviews necessary.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

  No such concerns.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.


  All the authors has stated on the MPLS WG mailing list, that they
  are unaware on aany other IPRs that the disclosed.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

  The Datatracker indicates two IPR disclosures against this
  document, in reality it is the same disclosure that was renewed
  when the draft became a working group document.

  There were very little discussion of the IPR, the wg routinely
  accept IPR disclosures that says "free to implement".

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it?

  This document was initiated because operational problems were
  found in deployed networks. The working groups agreed that this
  problem had to be resolved and is fully behind the document.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

  No such threats.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

  The document passes the nits-tool clean, manual checks have not
  revealed any other nits.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

  No formal review necessary.

(13) Have all references within this document been identified as
either normative or informative?

  Yes all the references are correctly split in normative and informative.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

  No, all normative references are to existing RFCs.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

  No downward references.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

  This document will not change any existing RFCs.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

  The Shepherd has reviewed that IANA section and IANA allocation
  requests several times. The section is clear and well written.

  The IANA is requested to create and maintain a registry entitled "LSR
  Capability Flags", the allocation policies are clearly defined. The¨
  initial allocation is documented.

  The IANA is requested to create and maintain a registry entitled
  "Interface Index Flags",  the allocation policies are clearly
  defined. The initial allocation is documented.

  This document defines "Sub-TLVs for TLV Type TBD4", IANA is requested
  to create a new sub-registry for this TLV, the allocation policies
  are clearly defined. The initial allocations are documented.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

  No such registries.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

  No such automated reviews necessary.