Solutions for BGP Persistent Route Oscillation
draft-ietf-idr-route-oscillation-stop-04

[Ballot discuss]
Thank you for writing this document on this important issue.

I want to recommend its approval as an RFC and will do so
shortly, but a Gen-ART review from Paul Kyzivat
inspired me to read the document, and I have
some comments. I'm not necessarily looking for
document changes, but I am looking for some responses
to Paul's or my comments, to make sure we all agree
that we are doing this right.

Specifically,

> A detailed description of how these oscillations can occur
> can be found in [RFC3345]; the description of how a node would
> locally detect such condition is outside the scope of this document.

I am not an expert on this topic, but I'd like to understand the nature
of such detection algorithms. How hard problem is that? What is required?
I'm not looking for specifying or even referring to the algorithm here, but
it would be beneficial at least for this reader to understand the nature
of the class of solutions referred to here. Are solutions algorithmically
computable? Heuristic? Dependent on specific configurations?
Vendor solutions exist but aren't documented.

> When the ADD-PATH Capability is also received from
> the IBGP peer with the "Send/Receive" field set to 1 (receive
> multiple paths) or 3 (send/receive multiple paths) for the same  SAFI>, then the following procedures shall be followed:
>
> If the peer is a route reflection client, the route reflector SHOULD
> advertise to the peer the Group Best Paths (or the Available Paths)
> received from its non-client IBGP peers.  Depending on the
> configuration, the route reflector MAY also advertise to the peer the
> Group Best Paths (or the Available Paths) received from its clients.

I am picking this text excerpt as an example.

Like Paul, some of this language makes me slightly uncomfortable,
and it would be good to understand what the aim is. If the intent
is to describe a procedure that can be followed and fully specified,
the expectations on the text are different from an informative
description of the kinds of things an implementation could do.

In the above text, there's quite a lot of imprecision. That may
be fine, but I wanted to check that this is the case.

For instance, why is a "shall" (and in lowercase) followed by
a set of "SHOULD"s and "MAY"s? Is there any guidance on
when the "SHOULD"s should be followed? There seems to
be thinking behind, could that be documented? What kind
of configuration issues affect the decisions?

[Ballot comment]
The abstract and introduction should explain terms that are not commonly used (such as MED). BGP is a fine term to use without expansion.

[Ballot comment]

Jon mitchell performed the opsdir review against 02

I have reviewed draft-ietf-idr-route-oscilliation-02 as part of the
Operational directorate's ongoing effort to review all IETF documents
being processed by the IESG.  These comments were written with the
intent of improving the operational aspects of the IETF drafts. Comments
that are not addressed in last call may be included in AD reviews during
the IESG review.  Document editors and WG chairs should treat these
comments just like any other last call comments.

I found the document is clear and Ready w/Issues (primarily last
paragraph of this review).

This is a Informational draft from the IDR WG which describes the
necessary set of paths that have to be advertised in order to stop
BGP persistent route oscillation due to the MED attribute.

Nits/suggestions:

Abstract - I don't think it's required to put caveats in the abstract,
this is well covered in the introduction and the text, consider removing
the last sentence parenthesized content.

Introduction - paragraph 2, strike the word clearly, s/"right"/necessary,
consider finishing last sentence with "to prevent the condition."

paragraph 3, first sentence - s/present/describe (easier to understand
due to only one definition)

paragraph 4, last sentence - s/is/are

Section 3 - consider renaming section header to "Advertise All the
Available Paths", even though all is somewhat redundant, it aligns with
previous text, internal text, and addpath-guidelines description.

Section 4 - paragraph 2 - I let this slip by the first time around but
tried to get them to soften the language about intra-cluster metrics
versus inter-cluster metrics in RFC 4456, but maybe it's worth softening
the language here about it.  An obvious alternative topology of using
larger intra-cluster metrics than inter-cluster metrics also is
sufficient to prevent route oscillation in the case where all BGP
NEXT_HOP's are coming from clients in the cluster (so that a large
intra-cluster link metric is included in the calculation in either
case).  This is sometimes deployed by providers that want to ensure
intra-cluster links are depreferenced versus WAN links for transit
traffic.  As simple of a sentence as, or any technique that prefers
intra-cluster to inter-cluster clients from the perspective of the same
clusters route reflector, would be sufficient to describe what is
necessary to prevent oscillation.  A reference to RFC 4451 may be useful
as well which also discusses MED deployment considerations.

Section 5 - add-path guidelines draft seems to be a useful reference
here that describes the various implementation approaches of ADD-PATH
including the two required to stop MED-induced oscillation?

Section 6, last sentence - as an operator, I disagree that what appears
to be a recommendation (am I misreading this sentence?) for greater
utilizing MED in the decision making process is being done here.  MED as
is described by this draft often causes more issues than it solves as
seems evident by the necessity of this draft.  In fact, if the
recommendation instead was to flatten MED to the same value everywhere
or compare MED's between AS's (an often deployed knob which is not IETF
documented AFAIK), you can rule out the case of MED-induced oscillation
w/o the necessary enhancements proposed by this document.  I believe
this is clearly spelled out in RFC3345 as well (see section 2.3 and
3.2).  It's not clear to me that this sentence is even factually
correct, and would cause a reduction of the number of group best paths
that need to be advertised, maybe this is making some wider assumption
of how MED's are assigned by the peer networks?

Cheers,

Jon

_______________________________________________
OPS-DIR mailing list
OPS-DIR@ietf.org
https://www.ietf.org/mailman/listinfo/ops-dir

[Ballot comment]

- abstract: use of so many acronyms here is a pity, be good
to expand or not use some of 'em

- intro: "MED attribute" is not mentioned in RFC4271, at
least not obviously enough to find with a grep;-) I had to
search for "multi" to figure out that you meant the
MULTI_EXIT_DISC thing, so maybe point at 5.1.4 in 4271 to
be nice?

- intro: It'd also be nice to provide a reference for the
"almost all of the known" claim for readers who aren't
familiar with such examples. (And I do mean "nice" and not
"necessary":-)

(Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-idr-route-oscillation-stop-02.txt, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any IANA actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, IANA does not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
IANA Specialist
ICANN

The following Last Call announcement was sent out:

From: The IESG
To: "IETF-Announce"
CC: draft-ietf-idr-route-oscillation-stop@ietf.org, idr@ietf.org, shares@ndzh.com., idr-chairs@ietf.org, akatlas@gmail.com
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (BGP Persistent Route Oscillation Solutions) to Proposed Standard


The IESG has received a request from the Inter-Domain Routing WG (idr) to
consider the following document:
- 'BGP Persistent Route Oscillation Solutions'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2016-04-29. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document presents two sets of paths for an address prefix that
  can be advertised by a BGP route reflector or confederation ASBR to
  eliminate the MED-induced route oscillations in a network.  The first
  set involves all the available paths, and would achieve the same
  routing consistency as the full IBGP mesh.  The second set, which is
  a subset of the first one, involves the neighbor-AS based Group Best
  Paths, and would be sufficient to eliminate the MED-induced route
  oscillations (subject to certain commonly adopted topological
  constrains).




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-idr-route-oscillation-stop/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-idr-route-oscillation-stop/ballot/


No IPR declarations have been submitted directly on this I-D.

Version of Shepherd write-up: 2/24/2016  (RFC4858)
Type: Standards Track
Next step: AD and IESG review 
Last update: 4/11/2016


(1) What type of RFC: Standard 

Why: Because it specifies the solution to a day-one problem with BGP and hierarchical RRs/Confederations.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  This  document presents two sets of paths for an address prefix
  that can be advertised by a BGP route reflector or confederation ASBR
  to eliminate the MED-induced route oscillations in a network.  The
  first set involves all the available paths, and would achieve the
  same routing consistency as the full IBGP mesh.  The second set,
  which is a subset of the first one, involves the neighbor-AS based
  Group Best Paths, and would be sufficient to eliminate the MED-
  induced route oscillations (subject to certain commonly adopted
  topological constrains).

Working Group Summary

Strong consensus.  Good discussion on the list.
https://www.ietf.org/mail-archive/web/idr/current/msg14201.html


Document Quality

Implemented in Juniper, Cisco

Personnel
Document Shepherd: Susan Hares
AD: Alia Atlas (Alvaro Retana is the author)
RTG-DIR review: Tony Przygienda
http://www.ietf.org/mail-archive/web/idr/current/msg14431.html


(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

Reviewed text, ask questions on list on expansion to Data Center.
https://www.ietf.org/mail-archive/web/idr/current/msg14201.html

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No, this was discussed for over 5 years, and deployed.
We are just slow in getting it standardized.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No, normal reviews are fine.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No concerns.  Document has received review, and deployment.
It fixes an operational problem and has been tested for 3-4 years. 

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Alvaro Retana
https://www.ietf.org/mail-archive/web/idr/current/msg14249.html

Enke Chen:
https://www.ietf.org/mail-archive/web/idr/current/msg15463.html

Daniel Walton
https://www.ietf.org/mail-archive/web/idr/current/msg14262.html

John Scudder
https://www.ietf.org/mail-archive/web/idr/current/msg15462.html

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.
no IPR statements

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong agreement.  5 years of discussion, and operational deployment.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

Nit is an error in the NIT detection.  NIT program needs to be fixed.
All Shepherd's comments addressed.


(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No additional formal review.

(13) Have all references within this document been identified as
either normative or informative?

yes

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

draft-idr-add-paths - sent to IESG prior to this document, and
draft-idr-add-paths - is expected to go as a bundle with this draft.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

draft-idr-add-paths - going as a bundle for approval with this draft.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

Fixes the route-oscillation-stop problem by using add-paths.
However, add-paths solution is not required to RFC4271.


(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

Version of Shepherd write-up: 2/24/2016  (RFC4858)
Type: Standards Track
Next step: AD and IESG review 
Last update: 3/26/2016


(1) What type of RFC: Standard 

Why: Because it specifies the solution to a day-one problem with BGP and hierarchical RRs/Confederations.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  This  document presents two sets of paths for an address prefix
  that can be advertised by a BGP route reflector or confederation ASBR
  to eliminate the MED-induced route oscillations in a network.  The
  first set involves all the available paths, and would achieve the
  same routing consistency as the full IBGP mesh.  The second set,
  which is a subset of the first one, involves the neighbor-AS based
  Group Best Paths, and would be sufficient to eliminate the MED-
  induced route oscillations (subject to certain commonly adopted
  topological constrains).

Working Group Summary

Strong consensus.  Good discussion on the list.
https://www.ietf.org/mail-archive/web/idr/current/msg14201.html


Document Quality

Implemented in Juniper, Cisco

Personnel
Document Shepherd: Susan Hares
AD: Alia Atlas (Alvaro Retana is the author)
RTG-DIR review: Tony Przygienda
http://www.ietf.org/mail-archive/web/idr/current/msg14431.html


(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

Reviewed text, ask questions on list on expansion to Data Center.
https://www.ietf.org/mail-archive/web/idr/current/msg14201.html

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No, this was discussed for over 5 years, and deployed.
We are just slow in getting it standardized.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No, normal reviews are fine.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No concerns.  Document has received review, and deployment.
It fixes an operational problem and has been tested for 3-4 years. 

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Alvaro Retana
https://www.ietf.org/mail-archive/web/idr/current/msg14249.html

Enke Chen:
https://www.ietf.org/mail-archive/web/idr/current/msg15463.html

Daniel Walton
https://www.ietf.org/mail-archive/web/idr/current/msg14262.html

John Scudder
https://www.ietf.org/mail-archive/web/idr/current/msg15462.html

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.
no IPR statements

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong agreement.  5 years of discussion, and operational deployment.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

Nit is an error in the NIT detection.  NIT program needs to be fixed.


(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No additional formal review.

(13) Have all references within this document been identified as
either normative or informative?

yes

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

draft-idr-add-paths - sent to IESG prior to this document, and
draft-idr-add-paths - is expected to go as a bundle with this draft.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

draft-idr-add-paths - going as a bundle for approval with this draft.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

Fixes the route-oscillation-stop problem by using add-paths.
However, add-paths solution is not required to RFC4271.


(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.