Requirements for the Graceful Shutdown of BGP Sessions
draft-ietf-grow-bgp-graceful-shutdown-requirements-07

[Ballot discuss]
This is a fine document, and I will move to No Objection on the call.  However, there is one issue I would like to
discuss.  The document states:

      In case of partial deployment the proposed
  solution SHOULD incrementally improve the maintenance process. The
  solution SHOULD bring improvements even when one of the two ASes does
  not support graceful shutdown.

Is the second SHOULD really attainable?  Specifically, if a link exists between two ASBRs and only one supports
graceful shutdown, how can it result in improvements?

If this requirement is achievable, that is terrific.  If not, we should strike that requirement and set expectations that
can be met.

[Ballot discuss]
Discuss issues handled in revision -05.

I think this is a useful document and it is encouraging to see requirements like this developed in the Ops Area for use within the Routing Area. Thank you.

[Ballot comment]
It would be useful if the document made clear up front that the issue being addressed is not just shutdown, but includes startup. This is always tacked on as a sort of afterthought. It would probably be useful to the reader in terms of identifying the utility of the document if it were case in terms of graceful planned maintenance rather than simply shutdown.

In Section 3.1 it would be good to start by saying that the example was a very simple case rather than allowing the reader to conclude this and then be told.

"(e.g. ASBR1 is chosen based on a higher local pref, hierarchical route reflectors are used...)." This looks like authors gave up rather than indicating giving a proper introduction to the full scope of the problem.

Section 5 needs to make clear up front that good news is being considered as well as bad news.

====
This may be a style thing, but the incomplete lists hint that the author is not prepared to consider the issue in depth with the reader.

"Depending on the session type (EBGP, IBGP...), there may be some  variations in the proposed solution in order to fulfill the  requirements. "

and

(e.g. static routes, RIPv2, OSPF, IS-IS...)

====

Not a comment on the document per se, but it looks like the authors rev'd the draft during this review cycle, which is not helpful, since different ADs may be working to different versions.

[Ballot comment]
It would be useful if the document made clear up front that the issue being addressed is not just shutdown, but includes startup. This is always tacked on as a sort of afterthought. It would probably be useful to the reader in terms of identifying the utility of the document if it were case in terms of graceful planned maintenance rather than simply shutdown.

In Section 3.1 it would be good to start by saying that the example was a very simple case rather than allowing the reader to conclude this and then be told.

"(e.g. ASBR1 is chosen based on a higher local pref, hierarchical route reflectors are used...)." This looks like authors gave up rather than indicating giving a proper introduction to the full scope of the problem.

Section 5 needs to make clear up front that good news is being considered as well as bad news.

====
This may be a style thing, but the incomplete lists hint that the author is not prepared to consider the issue in depth with the reader.

"Depending on the session type (EBGP, IBGP...), there may be some  variations in the proposed solution in order to fulfill the  requirements. "

and

(e.g. static routes, RIPv2, OSPF, IS-IS...)

====

[Ballot discuss]
The IETF work on routing behavior under failure and in particular convergence control has been taking place in RTGWG. I see no record of a review request by that WG and I believe that the document would benefit from a review by the members of that WG.

The document talks about shut-down and bring-up, but cost changes may also introduce microloops.

The true complexity in a real case is significantly greater than the authors bring out, and it would be useful if this was described. One additional example is the case where ASBR2 is internally routing via ASBR1 to AS1.

"Transient routing inconsistencies happen during IBGP convergence  because all routers are not updating their RIB at the same time. " The FIBs also need to be updated which can add a significant time to the convergence process.

"Transient forwarding loops can be avoided by performing only one IP  lookup on BGP routes in each AS and by using tunnels (e.g. MPLS LSP) to send packets between ASBRs. As such, BGP/MPLS VPNs should be  immune to such micro forwarding loops."

This is a problem statement not a solutions specification, and so the document should not be hinting at solutions. However there is a significant ambiguity here because the authors are unclear whether  they are discussing BGP free core ( a significant constraint on the class of network they are considering) or tunnel based loop avoidance. In any case the document would benefit from references to RFC5714 and RFC5715 (and maybe even RFC5286) since many of the solutions described there would translate across to this space.

"The goal of BGP graceful shutdown is to initiate the BGP convergence to find the alternate paths before the nominal paths  are removed."

The primary goal is surely to accomplish the shutdown (or startup) with zero packet loss. PL is introduced later in the para but is the goal is "fewer packets (possibly none) are lost" and I do not see why the goal in the planned case (as opposed to the FRR case) is not zero, since this should be achievable. This issue continues in the commentary on req (a). In any case this requirement hides a lot of subtly that the reader should be alerted to. For example what is implicit advertisement? Also I think that co-ordination would be a better concept here rather than advertisement which hints at certain classes of solution.

"The mechanism SHOULD allow BGP routers to minimize packet loss when a path is removed or advertised. In particular, it SHOULD be ensured    that the old path is not removed from the routing tables of the affected routers before the new path is known. "

There is a class of solution to this problem in which an intermediate topology is used, but this requirement seems to be disallowing that approach.

Req(a) is actually a set of requirements and should perhaps be expanded.

The following sub-requirement of Req(a) "The solution mechanism MUST reduce packet loss but MAY provide only a reduction rather than full minimization, in order to trade off with  simplicity of implementation and operation as shown in some of the  following requirements. " Could use expanding to give more guidance.

Surely Req(d) should be a MUST?

In Requirement (e) - Incremental deployment on a per AS basis should be a MUST otherwise the technology would need an upgrade to the Internet.

I am not sure what Requirement (g) is saying. I think that it would be better if this was part of the solution figure of merit, rather than permission to include this well known concept in the design.

There needs to be rather more commentary concerning the solution selection criteria explaining why the ordering was chosen. It's not clear that this a complete list, for example congestion introduced and the topological spead of the event may be factors to consider.

In Section 6, it's not clear why these particular reference topologies were chosen, or indeed whether they should be part of the main requirement or an appendix. However, I do not understand what the authors mean when that say that the section 5 requirements "SHOULD" apply to each case.

"The requirements of section 5 do not translate as easily as in  the two previous topologies because we do not require propagating  the maintenance advertisement outside of the two ASes involved in an eBGP session."

I am not sure what the authors mean by "do not require". Is a solution allowed to propagate this information if it is advantageous?

One of the more difficult problems to deal with is multiple simultaneous toplogy changes (planned or accidental) and the document needs to discuss this.

[Ballot comment]
I don't object to this document moving forward if the working group and sponsoring AD believe it will be useful in informing future protocol work, but I worry about its utility. The document has 16 SHOULDs and 2 SHOULD NOTs, and only 2 MUSTs (one of which really doesn't add anything):

  In the end, once the planned maintenance is finished the nominal BGP routing **MUST** be reestablished.
  Security Considerations Security considerations **MUST** be addressed by the proposed solutions.

Are there really no other hard requirements the group could come to consensus on?

The document talks about only covering "a subset of the cases". Does it mean "these requirements" when it says "the cases"? If not, what cases is it referring to?

[Ballot comment]
Section 5., paragraph 2:
>    As a result, provided an alternate path is available in the AS, the
>    packets are rerouted before the BGP session termination and fewer
>    packets (possibly none) are lost during the BGP convergence process
>    since at any time, all routers have a valid path.

  There is obviously the unstated assumption here that the available
  capacity on the new path can sustain the current load on the old path.
  Otherwise, there will definitely be packet loss. This isn't something
  that BGP can necessarily guarantee.


Section 5., paragraph 5:
>    a)  A mechanism to advertise the maintenance action to all affected
>    routers is REQUIRED. Such mechanism may be either implicit or
>    explicit. Note that affected routers can be located both in the local
>    AS and in neighboring ASes. Note also that the maintenance action can
>    either be the shutdown of a BGP session or the establishment of a BGP
>    session.
>    The mechanism SHOULD minimize packet loss when a path is removed or
>    advertised. In particular, it SHOULD be ensured that the old path is
>    not removed from the routing tables of the affected routers before
>    the new path is known.

  A "mechanism to advertise" can't really "minimize packet loss". It's
  the *reaction* to that advertisement that you need to make this
  requirement for.

[Ballot comment]
Section 2

  The Border Gateway Protocol(BGP) is heavily used in Service Provider

Add citation of RFC 4271 [BGP-4]

  networks both for Internet and BGP/MPLS VPN services. For resiliency

Add citation of RFC 4364 [VPN]

---

Section 6

OLD
  The solution drafts
  should state its applicability for each of these possible
  topologies.
NEW
  The solution documents
  should state the applicability of the solutions for each of
  these possible topologies.

[Ballot discuss]
Discuss updated to include comments from my review

I think this is a useful document and it is encouraging to see requirements like this developed in the Ops Area for use within the Routing Area. Thank you.

---

Section 4

Please add "g-shut" and "g-noshut" to the terminology section

---

Section 5 Requirement a)

  The mechanism SHOULD minimize packet loss when a path is removed or
  advertised. In particular, it SHOULD be ensured that the old path is
  not removed from the routing tables of the affected routers before
  the new path is known.

Now, I may be a bit pednatic here, but I have two questions:

1. Is it the mechanisms that minimises packet loss, or does the
  mechanism enable an implementation to minimise packet loss?

2. Why is this "SHOULD" not "MUST"? What use would this mechanism be if
  it does not minimise packet loss? You might as well not have it!
  OTOH, if you believe that "SHOULD" is correct, you will need to
  give at least an example (using "MAY") of how the non-minimisation
  of packet loss is acceptable.

---

Section 5 Requirement c)

  This is out of scope of this
  document, but comprehensive graceful shutdown procedures should take
  this into account.

If it is out of scope, how can you say that the GS should do? I know you
are trying to say that a wider scope GS procedure should take this into
account, but that is clearly out of scope. I suggest you change to:

  This is out of scope of this
  document.

---

Section 7

I think you should require that solutions should specifically address
the issue of bogus g-shut messages and the impact they would have on
the n/w. Also the impact of hiding a g-shut message so that g-shut is
not performed.

=====

Enke Chen performed a Routing Directorate review on October 1st. I would like to see discussion or action on the following points he raised.

---

I suggest that "BGP graceful bringup" related text be removed from the document for the following reasons:

    - To reduce packet loss during BGP bringup, several techniques have been
      developed and are well known, including setting the "overload bit" in
      IS-IS, and using large metrics in OSPF; and also also postponing the
      bringup of external links until the IGPs and IBGPs have converged.

    - When a primary path is added (such as in the case of BGP bringup),
      routing needs to reconverge.  IMO there is not much more that can be
      done or needs to be done than what is already known.

    - It is not clear from the document if there is a new requirement.

---

Section 2, Introduction

The draft says:

    Currently, BGP [BGP-4] and MP-BGP [MP-BGP] do not include any
    operation to gracefully withdraw a prefix while traffic toward that
    prefix could still be correctly forwarded. When a BGP session is
    taken down, BGP behaves as if it was a sudden link or router failure
    and withdraws the prefixes learnt over that session, which may
    trigger traffic loss. There is no mechanism to advertise to its BGP
    peers that the prefix will soon be unreachable, while still being
    reachable. When applicable, such mechanism would reduce or prevent
    traffic loss.


This paragraph seems to suggest a particular enhancement ("graceful
prefix withdraw") in the protocol might be needed.  This is a bit
misleading, and may not be the intention.

First, such an enhancement is not really required as there are a number
of parameters (in particular LOCAL-PREF) in BGP that can be used to
influence the degree of preference in route selection, and thus
influence the traffic flow.

Second, this is a requirement document, and is not about a specific
solution to satisfy the requirement.

So I suggest that the text be removed, or expanded to include the
possible re-use of existing protocol machinery.

[Ballot discuss]
Enke Chen performed a Routing Directorate review on October 1st. I would like to see discussion or action on the following points he raised.

---

I suggest that "BGP graceful bringup" related text be removed from the document for the following reasons:

    - To reduce packet loss during BGP bringup, several techniques have been
      developed and are well known, including setting the "overload bit" in
      IS-IS, and using large metrics in OSPF; and also also postponing the
      bringup of external links until the IGPs and IBGPs have converged.

    - When a primary path is added (such as in the case of BGP bringup),
      routing needs to reconverge.  IMO there is not much more that can be
      done or needs to be done than what is already known.

    - It is not clear from the document if there is a new requirement.

---

Section 2, Introduction

The draft says:

    Currently, BGP [BGP-4] and MP-BGP [MP-BGP] do not include any
    operation to gracefully withdraw a prefix while traffic toward that
    prefix could still be correctly forwarded. When a BGP session is
    taken down, BGP behaves as if it was a sudden link or router failure
    and withdraws the prefixes learnt over that session, which may
    trigger traffic loss. There is no mechanism to advertise to its BGP
    peers that the prefix will soon be unreachable, while still being
    reachable. When applicable, such mechanism would reduce or prevent
    traffic loss.


This paragraph seems to suggest a particular enhancement ("graceful
prefix withdraw") in the protocol might be needed.  This is a bit
misleading, and may not be the intention.

First, such an enhancement is not really required as there are a number
of parameters (in particular LOCAL-PREF) in BGP that can be used to
influence the degree of preference in route selection, and thus
influence the traffic flow.

Second, this is a requirement document, and is not about a specific
solution to satisfy the requirement.

So I suggest that the text be removed, or expanded to include the
possible re-use of existing protocol machinery.

  (1.a)  Who is the Document Shepherd for this document?  Has the
          Document Shepherd personally reviewed this version of the
          document and, in particular, does he or she believe this
          version is ready for forwarding to the IESG for publication?

Peter Schoenmaker, co-chair of the GROW Working Group. 
I have reviewed this document, and i believe the document is ready for
publication.

  (1.b)  Has the document had adequate review both from key WG members
          and from key non-WG members?  Does the Document Shepherd have
          any concerns about the depth or breadth of the reviews that
          have been performed?

The document authors have actively participated in presenting the
content of this document in the work group at ietf meetings along with
discussion on the Grow mailing list.  They have recieved feedback that
they have incorporated into the document.

  (1.c)  Does the Document Shepherd have concerns that the document
          needs more review from a particular or broader perspective,
          e.g., security, operational complexity, someone familiar with
          AAA, internationalization, or XML?

I believe this document has recieved sufficient review and input from
network operators who are in a position to help define the requirements,
and understand the problem discussed.

  (1.d)  Does the Document Shepherd have any specific concerns or
          issues with this document that the Responsible Area Director
          and/or the IESG should be aware of?  For example, perhaps he
          or she is uncomfortable with certain parts of the document, or
          has concerns whether there really is a need for it.  In any
          event, if the WG has discussed those issues and has indicated
          that it still wishes to advance the document, detail those
          concerns here.  Has an IPR disclosure related to this document
          been filed?  If so, please include a reference to the
          disclosure and summarize the WG discussion and conclusion on
          this issue.

I have no concerns related to this document.  It has been well accepted
and appropriately addresses the needs of bgp networks.

  (1.e)  How solid is the WG consensus behind this document?  Does it
          represent the strong concurrence of a few individuals, with
          others being silent, or does the WG as a whole understand and
          agree with it?

The document recieve full consensus from the working group.

  (1.f)  Has anyone threatened an appeal or otherwise indicated extreme
          discontent?  If so, please summarize the areas of conflict in
          separate email messages to the Responsible Area Director.  (It
          should be in a separate email because this questionnaire is
          entered into the ID Tracker.)

No extreme discontent has been expressed on any portion of this document.

  (1.g)  Has the Document Shepherd personally verified that the
          document satisfies all ID nits?  (See
          http://www.ietf.org/ID-Checklist.html and
          http://tools.ietf.org/tools/idnits/.)  Boilerplate checks are
          not enough; this check needs to be thorough.  Has the document
          met all formal review criteria it needs to, such as the MIB
          Doctor, media type, and URI type reviews?  If the document
          does not already indicate its intended status at the top of
          the first page, please indicate the intended status here.

I have revied this document, and there are no nits.

  (1.h)  Has the document split its references into normative and
          informative?  Are there normative references to documents that
          are not ready for advancement or are otherwise in an unclear
          state?  If such normative references exist, what is the
          strategy for their completion?  Are there normative references
          that are downward references, as described in [RFC3967]?  If
          so, list these downward references to support the Area
          Director in the Last Call procedure for them [RFC3967].

The references are properly split into normative, and informative references.
There are no pending documents referenced in either section.

  (1.i)  Has the Document Shepherd verified that the document's IANA
          Considerations section exists and is consistent with the body
          of the document?  If the document specifies protocol
          extensions, are reservations requested in appropriate IANA
          registries?  Are the IANA registries clearly identified?  If
          the document creates a new registry, does it define the
          proposed initial contents of the registry and an allocation
          procedure for future registrations?  Does it suggest a
          reasonable name for the new registry?  See [RFC2434].  If the
          document describes an Expert Review process, has the Document
          Shepherd conferred with the Responsible Area Director so that
          the IESG can appoint the needed Expert during IESG Evaluation?

There are no IANA considerations for this document.

  (1.j)  Has the Document Shepherd verified that sections of the
          document that are written in a formal language, such as XML
          code, BNF rules, MIB definitions, etc., validate correctly in
          an automated checker?

There are no sections in this document written in a formal language.

  (1.k)  The IESG approval announcement includes a Document
          Announcement Write-Up.  Please provide such a Document
          Announcement Write-Up.  Recent examples can be found in the
          "Action" announcements for approved documents.  The approval
          announcement contains the following sections:

          Technical Summary
    This document specifies the situations where there maybe
            service impact on bgp routed networks as a results of
            maintanance related to BGP peers.  The document describes
            the service impact from maintainance along with the cause
            of said impact.  A series of goals, and network topologies
            for which solutions should encompass are laid out as a
    guideline for future work.

          Working Group Summary
            The document was actively worked on in the GROW working
            group, with members, including network operators, providing
            feedback on the document.  The document recieved working
            group concensus.

          Document Quality
            This document covers real world causes of the BGP protocol
            as deployed in a variety of networks.  Direct experience
            with the issues described have been incorporated in the
            document.

          Personnel
            The document has been reviewed by Peter Schoenmaker (WG
            co-chair.)  Ron Bonica and Dan Romascanu are the ops area
            directors.