Address Resolution Problems in Large Data Center Networks
draft-ietf-armd-problem-statement-04

I am moving my Discuss to a Comment.

The substance is addressing the issues raised during IETF Last Call period by Manav Bhatia resulting from his Routing Directorate Review (see http://www.ietf.org/mail-archive/web/rtg-dir/current/msg01731.html).

The concern that led to this being a Discuss was that the issues were raised during IETF last call and should have been addressed at that time.

I understand that the authors are working on a new revision, and since the issues themselves would only have merited being entered as Comments, I am down-grading to a Comment.

The terminology distinction between "application" and "server" is really messy in this document. When you say "application", you are *always* talking about a piece of *server* application software (as distinct from *client* application software). When you say, e.g., "web server", sometimes you're referring to the software (the server application), and sometimes you're referring to the machine (hardware or virtual) on which it runs. For me sitting here in APP land, the use of the word "application" to only mean "server application" is weird, and the use of the word server in two ways makes this document all the more confusing to read. If you all in OPS land understand these sundry uses in this document, I'm not going to pitch a fit. But I would love to see "application" replaced by "server application" or "server software" or something like it, and I'd like to see "server", when used to talk about the machine, to be replaced by "host".

1. In section 7.1, does a high volume of ARP traffic have more impact
on routers than on hosts or VMs?  If so, why?

2. In section 7.1, does the total volume of ARP traffic ever become
great enough to have a measurable impact on available traffic
capacity?

3. Does this sentence from section 7.2 imply that IPv6 stacks that
exhibit the described behavior are compliant with RFC 4861?

   Consequently, some
   implementations will send out "probe" ND queries to validate in-use
   ND entries as frequently as every 35 seconds [RFC4861].

4. I suggest dropping the sentence about the impact of VMs in section
7.3.  Any growth in the datacenter that increases the number of
addresses used in an L2 domain, whether it be the physical span of the
L2 domain or the use of VMs, will have the impact described in section
7.3.  The impact of growth will also have an impact on the scenarios
in section 7.1 and 7.2.  The specific impact of VMs is also mentioned
earlier in the document.

5. Are the three problems described in sections 7.1-3 really the only
address resolution problems in large datacenters?  How do the three
problems interact with each other (as mentioned at the end of section
7.3), when the ARP and ND problems seem to be related to CPU usage and
the MAC table issue seems to be a memory problem.

6. It was a little surprising to me that section 5 describes multicast
ND for address resolution, but section 7.2 only cites the unicast use
of ND for NUD as a problem.

abstract: r/Our/The

abstract/s1: Is it massive scaling or scaling of massive data centers (i.e.,t those with a massive # of hosts ;)

s1: r/we/this document

s1: r/aims to/lists (I sure hope it will ;)

s2: Are these internet hosts or some other kind of host?  Should it be pointing to RFC 1123?

s2: ToR an unfortunate reuse of "The Onion Router" and more commonly used in reference to the ToR Browser (https://www.torproject.org/).  Any chance of calling it SoR _ single rack switch (SRS)?

s11: Agree Stephen.

This is a near-discuss. But I'm not a fan of putting use-cases or
problem statements on critical paths, so its not.

I think the security considerations ought to have made mention of
isolation of traffic, e.g. via separate VLANS or the moral
equivalent. In other words, I think section 11 ought be about the
security considerations of section 6 (mainly) and not a statement
about ARP which is (presumably) not the answer for armd. Since such
traffic isolation is a real requirement and a pain point for any armd
solution, and if this document does motivate armd design, then I
disagree that this document has no security implications.

Note - I'm not suggesting you try solve the pain here, but only that
you properly recognise it.

There are a couple of emotive terms - massive and "a lot" which should be replaced with a clear definition of the size the authors are considering.

----

and/or OpenFlow [OpenFlow] infused directory assistance approaches.  

There should be a reference to IDA approaches

----

   Current implementations
   today can support ARP processing in the low thousands of ARP packets
   per second, which is several orders of magnitude lower than the rate
   at which packets can be forwarded by ASICs.

I think that this is a bit misleading, since the ARP rate was never expected
to be close to the data rate. Are the authors stating that in these systems
the rates are similar, if so I think that should be made clearer, in either case
I think the para needs rewording.

----

" Some routers can be configured to broadcast periodic gratuitous ARPs.  

Needs a reference to gratuitous ARPs

-----

7.2.  IPv6 Neighbor Discovery

It would be useful to highlight in this section that ND is orders
of magnitude more aggressive in cache expiry than IPv4. I
understand 4hrs (for IPv4) vs 35s (for IPv6) and I further
understand that this scaling issue in much smaller systems that those
considered in this text. 


====

Nits

"Hypervisor:  Software running on a host that allows multiple VMs to run on the same host."

hypervisor surely more likely runs on a server

Just some comments that you are free to ignore ... I think there are some places where the terminology and background have been glossed over.

For instance, I don't know what "massive" means.  That's very subjective.  The number 100,000 physical machines is given at the end of section 3, but is that where massive starts or is that somewhere in the middle of the range?

It also seems to be assumed that the switches are multilayer switches, as there's text mentioning that they can be the gateway for a subnet (in section 3).  However, in the IETF, it seems to me that we've usually been pretty careful to distinguish between devices acting as L2 switches and devices acting as L3 routers, so it seems that the concept of MLS should be introduced.  The presence of things like load balancers that use higher-layer information for switching is another way that people have gotten scalability without encountering these ARP/ND issues, or overly complicating the L2 and L3 switching configuration.

Further, the VM systems that I know of require a SAN in order to failover and migrate from one physical machine to another, they aren't moving huge distances across the datacenter because of that.  I'm not sure what the assumptions about VM mobility across physical machines are that this document makes.  It seems to assume that any VM might run on any physical machine, which is only realistic for certain types of datacenter, and certainly not all.  But maybe this is also part of the definition of "massive" in that they're doing more of a VPS type of hosting?