Internet Exchange BGP Route Server Operations
draft-ietf-grow-ix-bgp-route-server-operations-05

[Ballot discuss]
I welcome this document and think it is a useful addition to the canon.
However, John Scudder did a Routing Directorate review during the IETF
last call period and emailed his comments to the authors and to the
GROW mailing list. I have seen no response to this directly or on the
GROW list.

Therefore, from a process point of view, I adopt all of John's comments
as a Discuss even though many of the points are small and would normally
be just Comments.

- Throughout the document, various terms are used to describe what RFC
  4271 calls a "route". The definition given in RFC 4271 is:

  Route
      A unit of information that pairs a set of destinations with the
      attributes of a path to those destinations.  The set of
      destinations are systems whose IP addresses are contained in one
      IP address prefix carried in the Network Layer Reachability
      Information (NLRI) field of an UPDATE message.  The path is the
      information reported in the path attributes field of the same
      UPDATE message.

  That is, one NLRI plus its path attributes, as carried in an UPDATE,
  is a "route". I would suggest adopting this term, or "BGP route" if
  you prefer, instead of terms such as "NLRI UPDATE message", "NLRI
  message", "prefix UPDATE message", and even just plain "NLRI" and
  "message". Also some, but not all, of the uses of "prefix". I think
  doing so will make the document clearer, more readable, and more
  technically accurate. A simple search for the terms I've called out
  should show most of them so I won't enumerate them here unless you
  ask me to (feel free, if you want).

- Reference [RS-ARCH] is a dead link. I found a live copy at
  http://www.cs.usc.edu/assets/003/83191.pdf. It might be worth
  checking with the authors of RS-ARCH to ask what a good archival
  reference is.

- S. 4.2 talks about scaling. I'm trying to make sense of the analysis:

  Regardless of any Loc-RIB optimization technique is implemented, the
  route server's control plane bandwidth requirements will scale
  according to O(P * N), where P is the total number of unique paths
  received by the route server and N is the total number of route
  server clients. 

  So far so good. (Except nit: there seems to be a word missing, such
  as "whether" as in "Regardless of whether any Loc-RIB...")

  In the case where P_avg (the arithmetic mean number
  of unique paths received per route server client) remains roughly
  constant even as the number of connected clients increases, this
  relationship can be rewritten as O((P_avg * N) * N) or O(N^2). 

  I don't see where the second factor of N comes from. You're basically
  expanding the P in the first expression as P_avg * N -- but why? I
  think this would only apply if add-path all-paths was chosen as the
  path hiding mitigation strategy -- but this is not touched on in
  route-server-operations, only in ix-bgp-route-server, and besides that
  the beginning of the paragraph implies you're analyzing the multiple
  Loc-RIB strategy, so I don't guess all-path is what you were thinking
  of. If you're not doing all-path, the O(N^2) analysis is wrong AFAICT.
  To see this, consider that the inbound routes require O(P_avg * N)
  which is just O(N), but the number of routes you're going to advertise
  is bounded by the size of the Internet routing table, which is a
  constant for purposes of this analysis, so also O(N). In and out are
  summed, not multiplied, so the whole thing works out to be O(N), not
  O(N^2).

  So I think this needs to either be corrected, or the assumptions need
  to be better explained. Moving on:

  This
  quadratic upper bound on the network traffic requirements indicates
  that the route server model will not scale to arbitrarily large
  sizes.

  If you continue to think this sentence is warranted, I think it should
  be better quantified. Of course nothing can scale to *arbitrarily*
  large sizes, but that still leaves a lot to the imagination. I would
  think it would be beneficial for an IX operator reading this document
  to be able to have some idea of how practical the limitation is. Since
  the analysis in question is looking at control traffic bandwidth
  consumption, it wouldn't be too onerous to throw some simple
  assumptions up against it -- for example, "if we suppose a RS receives
  on average 100,000 routes from each client with a rate of change of 10
  routes/second, sends on average 1,000,000 routes to each client with a
  rate of change of 100 routes/second, and that each route consumes on
  average 50 bytes in a BGP UPDATE message, simple arithmetic shows that
  a GigE connection to that RS will be fully saturated by the time the
  number of clients reaches 25,000." (Which does not seem like a very
  practical limitation, the RS will hit a CPU or memory bottleneck
  first.)

  Anyway, maybe you will decide on reconsideration of the big-O analysis
  that this bit is not needed at all, which would be OK with me.

- S 4.2.2.1,

  If the route server
  operator has prior knowledge of interconnection relationships between
  route server clients, then the operator may configure separate Loc-
  RIBs only for route server clients with unique outbound routing
  policies.

  It wasn't obvious to me what "outbound" applies to -- the client? The
  RS? -- and for that matter why an inbound policy (on the RS) might not
  apply. Possibly this could be remedied by simply dropping the
  adjective "outbound".

- S. 4.2.1.2,

  destination splitting would require significant co-ordination
  between the route server operator and each route server client

  It's not clear to me why it would "require significant co-ordination",
  depending on what resource you're trying to conserve. Two examples of
  how you could avoid coordination while still getting benefit: You
  could have clients send all their routes to all the RSes, but have
  RSes filter out the prefixes they don't care about. This gives the RS
  most of the CPU benefit it would have gotten had the client done the
  filtering (prefix filtering is cheap), almost all the memory benefit
  (the filtered routes need not be retained in the Adj-RIB-In), and
  around half the control traffic bandwidth benefit. The client incurs
  cost to send duplicate routes that are going to be discarded by the
  RS, but the client is presumably not the bottleneck resource. Or
  better still, the RS could use ORF towards the clients to control
  what routes the clients will send.

- S. 4.6.1,

  OLD:
  Prefixes sent to the route server are tagged with specific [RFC1997]
  or [RFC4360] BGP community attributes

  I don't think the naked references scan well as adjectives in this
  context. I suggest

  NEW:
  Prefixes sent to the route server are tagged with specific standard
  [RFC1997] or extended [RFC4360] BGP community attributes

- Also in S. 4.6.1,

  OLD:
  As both standard and extended BGP communities values are restricted
  to 6 octets

  Actually standard communities are restricted to less than that.
  Perhaps reword as

  NEW:
  As both standard and extended BGP communities values are restricted
  to 6 octets or fewer

- Also in S. 4.6.1,

  route server operator should take care to ensure
  that the predefined BGP community values mechanism used on their
  route server is compatible with [RFC4893] 4-octet autonomous system
  numbers.
 
  I suspect an RS operator reading this might be left scratching his or
  her head and asking "what does it mean for me to be compatible with
  RFC4893 in this context"? It would be kind to offer them some
  guidance, since after all this is a guidance document.

- S. 4.7: Where you say "non-commutative" I think you mean "non-
  transitive".

- S. 4.7:

  Problems of this form can be dealt with using [RFC5881] bidirectional
  forwarding detection.

  It's not clear to me how certain non-transitive forwarding failures
  can be dealt with using BFD. To take an example, suppose clients A, B
  and C peer with RS. The IX fabric has a failure such that A and B can
  both reach RS, but not each other. C has connectivity to everyone.
  Prefix X is advertised to RS by both B and C. For whatever reason, RS
  selects X via B to advertise to A. Even if A runs BFD towards B, at
  best A can determine that the route from RS can't be used. A isn't
  able to fail over to C's route as it would in the full-mesh case,
  since it's not aware of it. Depending on A's other connectivity, this
  may result in sub-optimal routing towards X, or complete loss of
  connectivity to X.

  It's beyond the scope of the draft to solve this problem, but the text
  could be made more accurate. A minimal fix would be

  Problems of this form can be partially mitigated using [RFC5881]
  bidirectional forwarding detection.

  although you might want to go on a bit longer to explain what problems
  can't be mitigated.

- S. 4.8:

  This problem is not specific to route servers and it can also be
  implemented using bilateral peering sessions.  However, the potential
  damage is amplified by route servers because a single BGP session can
  be used to affect many networks simultaneously.

  This is true, but there is a more severe way RSes aggravate the
  problem: In a full mesh, a router can (and usually does) directly
  enforce a "no third-party next hops" policy against its peers. An RS
  peer by definition cannot enforce this policy against the RS, so the
  RS is the only place it can be enforced.

- S. 4.8:

  Route server operators SHOULD check that the BGP NEXT_HOP attribute
  for NLRIs received from a route server client matches the interface
  address of the client.  If the route server receives an NLRI where
  these addresses are different

  so far so good (modulo my first comment about the use of "NLRI", of
  course), but:

  and where the announcing route server
  client is in a different autonomous system to the route server client
  which uses the next hop address,

  Is the RS sincerely expected to enforce the above? I suppose it could
  be implemented automatically although imperfectly, by noticing that
  multiple clients are in the same neighbor AS and noticing when they
  use each other as third-party next hops, but AFAIK people generally
  don't try to figure this out, they just do what you've said in the
  preceding sentence -- make sure the NH matches the interface address.
  If you really do propose that the RS should allow third-party next
  hops but only from clients in a common AS, I think you should talk
  about it specifically and in more detail. If you didn't really mean
  that, then I suggest you drop the clause.

- S. 5:

  On route server installations which do not employ path hiding
  mitigation techniques, the path hiding problem outlined in section
  Section 4.1 can be used in certain circumstances to proactively block
  third party prefix announcements from other route server clients.

  I don't understand what this means. Specifically, I don't know what it
  means to "proactively block third party prefix announcements" or for
  that matter, even what you mean by "third party prefix announcements"
  in this context. (As a term of art, I normally understand "third party
  announcement" in a BGP context to mean announcing a third-party next
  hop as you discuss in S. 4.8). I also don't know what the "certain
  circumstances" are, quite likely these should be given at least a
  little color if not entirely spelled out.

  Also, a nit -- the xref expansion has put "section section" into your
  text.

- S. 7:

  BIRD, OpenBGPD and Quagga, whose open source BGP implementations
  include route server capabilities

  Great, cool, but:

  which are compliant with this
  document.

  I'm not sure what it actually means to be "compliant" with a document
  that "describes operational considerations". Perhaps just drop the
  phrase?

[Ballot comment]
Nits also taken from John's review:

- In S. 2,

  OLD:
BGP sessions between each participant router
  NEW:
BGP sessions between each pair of participant routers

- In S. 4.2.1.1,

  OLD:
  In
  this situation, the multiple Loc-RIB views required by each client
  are merged into a single view.

  As written, this implies that each client requires multiple Loc-RIB
  views, which I don't think is what was intended. I suggest:

  NEW:
  In
  this situation, multiple Loc-RIB views
  are merged into a single view.

- I personally am strongly put off by the neologism "granular" to mean
  "fine-grained" and suggest the latter instead. I realize it's not an
  unusual usage so by all means disregard if you feel strongly about
  it.

- S. 4.6.2:

  OLD:
  server operators to implement construct per-client routing policies.
  NEW:
  server operators to construct per-client routing policies.

[Ballot comment]
With all of the RECOMMENDations and things you SHOULD do in this document in order to prevent "bad things" from happening, I'd have thought this to be BCPish. But I am made to understand that RSs are (somewhat) controversial beasts, and therefore making it a BCP would have been controversial as well. You might want to mention that fact in the document, but it's up to you.

[Ballot comment]
Thank you for responding to the SecDir review.  I see some wording suggestions were made in response to this review,  I do think they would be helpful and would like to see the updates made to the draft:
https://www.ietf.org/mail-archive/web/secdir/current/msg05065.html

From my review:
This is a non-blocking comment for consideration only.  In figure 2, I see the point of the connections from each router into the route server, demonstrating the point made in the section where you just need n connections to the route server instead of n*(n-1)/2.  What do the dots on the outside connecting each router represent?  I'm asking because the draft later describes use of a shared media like Ethernet, but this diagram looks like the routers are directly connected and it appears to require passing through other routers to route traffic.  If that's correct, you could probably just remove the dots on the outside edge connecting the routers since the point of this section is the connections to the route server.  Otherwise, if they have some sort of meaning, it might help to explain what the dots represent, I'm guessing it's not for the actual exchanges of data.

Section 4.2, The first sentence of paragraph 2 is missing a word making it a bit difficult to read.

IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-grow-ix-bgp-route-server-operations-03, which is currently in Last Call, and has the following comments:

We understand that, upon approval of this document, there are no IANA Actions that need completion.

While it is helpful for the IANA Considerations section of the document to remain in place upon publication, if the authors prefer to remove it, IANA doesn't object.

If this assessment is not accurate, please respond as soon as possible.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Internet Exchange Route Server Operations) to Informational RFC


The IESG has received a request from the Global Routing Operations WG
(grow) to consider the following document:
- 'Internet Exchange Route Server Operations'
  as
Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2014-09-22. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  The popularity of Internet exchange points (IXPs) brings new
  challenges to interconnecting networks.  While bilateral eBGP
  sessions between exchange participants were historically the most
  common means of exchanging reachability information over an IXP, the
  overhead associated with this interconnection method causes serious
  operational and administrative scaling problems for IXP participants.

  Multilateral interconnection using Internet route servers can
  dramatically reduce the administrative and operational overhead of
  IXP participation and these systems used by many IXP participants as
  a preferred means of exchanging routing information.

  This document describes operational considerations for multilateral
  interconnections at IXPs.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-grow-ix-bgp-route-server-operations/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-grow-ix-bgp-route-server-operations/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational.  This document discusses the operations of the route server.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:


Technical Summary:

The document discusses the operations of a BGP route server at Internet Exchagne Points (IXPs), including how they are used and what functions they require.

Working Group Summary:

This document was created to upto date documents on how route servers are used today at IXPs.  The original document was split between IDR and GROW, with IDR covering the protocol specific points, and GROW covering the requirements and operations.  The IDR document is draft-ietf-idr-ix-bgp-route-server-05.

Document Quality:

The document covers the existing implementations and usecases of a route server at an IXP.  It has been reviewed sufficiently by those that operate IXPs, and the working group.

Personnel:

Peter Schoenmaker is the document Shepherd.
Joel Jaeggli is the AD.


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

The shepherd has followed the document since its creation.  Followed and reviewed the comments and revisions on the list.  Along with individually reviewing the document.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No concerns.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No additional specific reviewed required.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

No additional concerns.  The document is a straight forward review of the usage of IXPs in exchange points.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

yes.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Yes. This document was smooth through the working group with no major concerns.  At the same time it had active contribution.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

It had a couple outdated references, but shoudl be resolved after review.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

No formal reviews required.

(13) Have all references within this document been identified as either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

The accompanying document ietf-idr-ix-bgp-route-server is progressing through IDR.  Last call has not yet been issued.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

draft-ietf-idr-ix-bgp-route-server-05 is still progressing, and a normative reference.

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

no.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

No IANA considerations.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

None required.