COIN                                                            H. Singh
Internet-Draft                                   MNK Labs and Consulting
Intended status: Informational                            M-J. Montpetit
Expires: August 22, 2021                            Concordia Univeristy
                                                       February 18, 2021

 Requirements for P4 Program Splitting for Heterogeneous Network Nodes


   For distributed computing, the P4 research community has published a
   paper to show how to split a P4 program into sub-programs which run
   on heterogeneous network nodes in a network.  Examples of nodes are a
   network switch, a smartNIC, or a host machine.  The paper has
   developed artifacts to split program based on latency, data rate,
   cost, etc.  However, the paper does not mention any requirements.  To
   provide guidance, this document covers requirements for splitting P4
   programs for heterogeneous network nodes.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on August 22, 2021.

Copyright Notice

   Copyright (c) 2021 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect

Singh & Montpetit        Expires August 22, 2021                [Page 1]

Internet-Draft                  P4 Split                   February 2021

   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Requirements Language . . . . . . . . . . . . . . . . . . . .   2
   2.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   3.  Terminology and Abbreviations . . . . . . . . . . . . . . . .   3
   4.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .   4
   5.  Changes to P4 Compiler to Block Split . . . . . . . . . . . .   5
   6.  Discussion  . . . . . . . . . . . . . . . . . . . . . . . . .   5
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .   5
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   5
   9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   5
   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .   6
     10.1.  Normative References . . . . . . . . . . . . . . . . . .   6
     10.2.  Informative References . . . . . . . . . . . . . . . . .   6
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   7

1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.  Introduction

   The research paper [FLY] covers splitting a P4 program into sub-
   programs to run the sub-programs on heterogeneous network nodes.
   There are certain issues to discuss first because some P4 code cannot
   be split to run elsewhere.  There are other issues as well.  For
   brevity, this document uses the terms smartNIC and NIC

   In a data center, host machines are connected to a switch.  In an
   Enterprise network, P4 data plane replicates ARP [RFC0826] and IPv6
   ND [RFC4861] messages for layer-2 address resolution.  If a program
   split moves ARP and IPv6 code to smartNIC, the hosts should also move
   to smartNIC.  If hosts do not move, the switch resolves layer-2
   destinations and messages the NIC with ARP or IPv6 ND table update.
   But the switch is forwarding traffic at 12 Tbps and for any layer-2
   lookup, the switch has to message the NIC which slows down switch
   forwarding.  If hosts also move with ARP and IPv6 ND to the NIC,
   there are still issues.  A NIC with two 100G ports will not be able
   to support all 25G hosts on a switch with 32 ports.  So multiple NICs
   are used.  If a switch is used in bridged mode, there is a single

Singh & Montpetit        Expires August 22, 2021                [Page 2]

Internet-Draft                  P4 Split                   February 2021

   link-local domain for ARP and IPv6 ND.  If the switch is used as a
   layer-3 switch, then one interface with layer-3 addresses can operate
   the switch.  With multiple NICs, each NIC has its own link-local
   domain and if configured, a layer-3 interface.  So hosts on one NIC
   go through an additional router to communicate with hosts on another
   NIC.  On the switch, running in bridged mode, the router is not

   In a public cloud, Azure resolves layer-2 destination with a central
   controller and thus the switch does not use any data plane broadcast
   or IPv6 ND multicast addresses.  However, this network faces the same
   issue mentioned above when multiple NICs are used.  Google resolves
   layer-2 via a proprietary Neighbor Discover protocol [GOOG].  How
   does Flightplan [FLY] deal with three such disparate networks?

   Regarding BGP, if a CLOS network runs BGP, BGP operates between LEAF
   and SPINE switches.  If BGP data plane table splits to a smartNIC,
   you have to assign an IP address for BGP peer on host CPU.  Now the
   host CPU runs BGP control plane and NIC stores BGP data plane tables.
   But both Azure and AWS (Amazon Web Services) do not run any SDN or
   BGP control plane on host because such network activity steals key
   cycles from host CPU.  There is another major problem.  Hosts
   routinely move in the data center to load balance.  With a host move,
   the BGP peer may move to a totally different subnet and break the BGP

   The punt or divert path of a data plane processes ARP, IPv6 ND, and
   any routing control messages.  Production quality switches (or
   routers) also run a punt rate-limiter in the data plane so that the
   switch/router CPU is not inundated.  In a heterogeneous network, it
   is not just how close one punts packets to CPU, but also what else
   moves with punt path?  Certainly the data plane punt rate-limiter
   also moves.

3.  Terminology and Abbreviations

      CPU - Central Processing Unit.

      DPDK - Data Plane Development Kit from Intel.

      ClOS - leaf and spine switched network redundant topology.

      FPGA - Field Programmable Gate Array.

      NIC - Network Interface Card.

      npu - Network Processing Unit.

Singh & Montpetit        Expires August 22, 2021                [Page 3]

Internet-Draft                  P4 Split                   February 2021

      smartNIC - a NIC with processor/FPGA.

      TCAM - Ternary Content-Addressable Memory.

      VPP - Vector Packet Processing from Cisco.

4.  Requirements

   The requirements are:

   1.  If the heterogeneous network includes a switch, the ARP and IPv6
       ND data plane P4 code should not be split to run outside the

   2.  Likewise ARP or IPv6 ND Proxy data plane code should not be split
       to run outside the switch.

   3.  BGP table should not be split and move outside the switch.
       Distributed BGP is a research topic.

   4.  A switch likely includes TCAM and thus the P4 program may use P4
       ternary table match kind.  If such a table is moved to another
       node due to program split, the node the code moves to is
       important.  A FPGA (field-programmable gate array) does not use
       TCAM and a host machine may not either.  The FPGA and host use
       hash-based table lookup.  Depending on the table key size, an
       appropriate hash is required.  Either the splitting tool prompts
       the user for what hash to use or deduces what hash - user input
       is desirable.  For example, for a 6-tuple IPv4 key, a 128 bit key
       is used and for the same 6-tuple, the IPv6 key uses 320 bits.
       Appropriate hashes are required for such keys.

   5.  Splitting algorithms should not develop High Availability.
       Network deployments already use dual switches, or CLOS topology
       for redundancy.  BFD [RFC5880] is recommended for use with
       liveliness detection.

   6.  Any automated tool that splits a P4 program to run on
       heterogeneous nodes, should provide a manual override.  For
       example, a P4 program is compiled for a switching asic.  The
       compiler raises an error saying code fits in N+2 pipeline stages
       but the asic has only N stages.  In this case, an automated tool
       will just split the program.  However, a manual override allows
       the programmer to tweak the code manually to fit.  With manual
       tweaking I have been able to fit code in N-1 stages after getting
       an initial error from compiler for code using N+2 stages.  Manual
       override could kick in if the number of stages used is (N + 16% x

Singh & Montpetit        Expires August 22, 2021                [Page 4]

Internet-Draft                  P4 Split                   February 2021

   7.  The splitting tool should define clearly what is the punt path
       for P4 code running on a host.  The reason is because the host
       CPU is the data plane, so where is the punted packet to CPU sent?
       For DPDK, I expect Linux user space to receive punted packets.
       For VPP, it supports a punt node.

5.  Changes to P4 Compiler to Block Split

   Using P4 Annotations to pass information to p4c (P4 compiler) backend
   [P4C] to not split certain code is not desirable.  This document
   proposes to change p4c.  A new table implementation property called
   atomic is added to p4c.  If this atomic table implementation property
   is configured for a table in the P4 program, then the table and its
   actions and any table invocation code block are not split.

6.  Discussion

   The two largest public cloud operators are Amazon AWS and Microsoft
   Azure [NIC].  Both operators run Software Defined Networking (SDN) in
   the smartNIC.  The reason is running SDN stack in software on the
   host requires additional CPU cycles.  Burning CPUs for SDN services
   takes away from the processing power available to customer VMs, and
   increases the overall cost of providing cloud services.  Azure uses a
   FPGA on smartNIC and programs the FPGA in Verilog, not P4.  Amazon
   uses multi-core npu (Graviton uses 64 cores) on smartNIC and does not
   program Graviton in P4.  Both these operators do not use host CPU or
   network switch for SDN operations.  In future, even if both operators
   program smartNIC in P4, the operators do not have heterogeneous nodes
   running SDN.  Likewise, in future, the switch runs a new SDN feature,
   e.g.  switch caching of popular lookup, then there are heterogeneous
   nodes to apply Flightplan to.

7.  Security Considerations

   Use IPSec [RFC4301] to secure any control plane communications.

8.  IANA Considerations


9.  Acknowledgements

   Thanks (in alphabetical order by first name) to Nik Sultana for
   reviewing this document.

Singh & Montpetit        Expires August 22, 2021                [Page 5]

Internet-Draft                  P4 Split                   February 2021

10.  References

10.1.  Normative References

   [RFC0826]  Plummer, D., "An Ethernet Address Resolution Protocol: Or
              Converting Network Protocol Addresses to 48.bit Ethernet
              Address for Transmission on Ethernet Hardware", STD 37,
              RFC 826, DOI 10.17487/RFC0826, November 1982,

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,

   [RFC4301]  Kent, S. and K. Seo, "Security Architecture for the
              Internet Protocol", RFC 4301, DOI 10.17487/RFC4301,
              December 2005, <>.

   [RFC4861]  Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
              "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
              DOI 10.17487/RFC4861, September 2007,

   [RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
              (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,

10.2.  Informative References

   [FLY]      Sultana, N., Sonchack, J., Giesen, H., Pedisich, I., Han,
              Z., Shyamkumar, N., Burad, S., DeHon, A., and B. T. Loo,
              "Flightplan: Dataplane Disaggregation and Placement for P4
              Programs", November 2020,

   [GOOG]     Singh, A., "Jupiter Rising: A Decade of Clos Topologies
              and Centralized Control in Google Datacenter Network",
              September 2016,

   [NIC]      Firestone, D., "Azure Accelerated Networking: SmartNICs in
              the Public Cloud", April 2018, <

Singh & Montpetit        Expires August 22, 2021                [Page 6]

Internet-Draft                  P4 Split                   February 2021

   [P4C]      Community, P., "P4_16 Reference Compiler - Github", May
              2018, <>.

Authors' Addresses

   Hemant Singh
   MNK Labs and Consulting
   7 Caldwell Drive
   Westford, MA  01886


   Marie-Jose Montpetit
   Concordia Univeristy
   1455 Boulevard de Maisonneuve O
   Montreal, Quebec  01886


Singh & Montpetit        Expires August 22, 2021                [Page 7]