Network Virtualization Overlays (NVO3) Encapsulation Considerations

Abstract

The IETF Network Virtualization Overlays (NVO3) Working Group developed considerations for a common encapsulation that addresses various network virtualization overlay technical concerns. This document provides a record, for the benefit of the IETF community, of the considerations arrived at starting from the output of an NVO3 encapsulation design team. These considerations may be helpful with future deliberations by working groups over the choice of encapsulation formats.¶

There are implications of having different encapsulations in real environments consisting of both software and hardware implementations and within and spanning multiple data centers. For example, OAM functions such as path MTU discovery become challenging with multiple encapsulations along the data path.¶

Based on these considerations, the Working Group determined that Geneve with a few modifications as the common encapsulation. This document provides more details, particularly in Section 7.¶

1. Introduction

The NVO3 Working Group is chartered to gather requirements and develop solutions for network virtualization data planes based on encapsulation of virtual network traffic over an IP-based underlay data plane. Requirements include due consideration for OAM and security. Based on these requirements the WG was to select, extend, and/or develop one or more data plane encapsulation format(s).¶

This led to WG drafts and an RFC describing three encapsulations as follows:¶

• [RFC8926] Geneve: Generic Network Virtualization Encapsulation¶
• [ietf_intarea_gue] Generic UDP Encapsulation¶
• [nvo3_vxlan_gpe] Generic Protocol Extension for VXLAN (VXLAN-GPE)¶

Discussion on the list and in face-to-face meetings identified a number of technical problems with each of these encapsulations. Furthermore, there was clear consensus at the 96th IETF meeting in Berlin that, to maximize interoperability, the working group should progress only one data plane encapsulation. In order to overcome a deadlock on the encapsulation decision, the WG consensus was to form a Design Team [RFC2418] to resolve this issue and provide initial considerations.¶

2. Design Team and Working Group Process

The Design Team was to select one of the proposed encapsulations and enhance it to address the technical concerns. The simple evolution of deployed networks as well as applicability to all locations in the NVO3 architecture were goals. The Design Team was to specifically avoid selecting a design that is burdensome on hardware implementations but should allow future extensibility. The selected design also needed to operate well with ICMP and in Equal Cost Multi-Path (ECMP) environments. If further extensibility is required, then it should be done in such a manner that it does not require the consent of an entity outside of the IETF.¶

The output of the Design Team was then prcoessed through the working group resulting in working group consensus for this document.¶

3. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

4. Abbreviations and Acronyms

The following abbreviations and acronyms are used in this document:¶

ACL

- Access Control List¶

DT

- NVO3 encapsulation Design Team¶

ECMP

- Equal Cost Multi-Path¶

EVPN

- Ethernet VPN [RFC8365]¶

Geneve

- Generic Network Virtualization Encapsulation [RFC8926]¶

GPE

- Generic Protocol Extension¶

GUE

- Generic UDP Encapsulation [ietf_intarea_gue]¶

HMAC

- Hash based keyed Message Authentication Code [RFC2104]¶

IEEE

- Institute for Electrical and Electronic Engineers (www.ieee.org)¶

NIC

- Network Interface Card (refers to network interface hardware which is not necessarily a discrete "card")¶

NSH

- Network Service Header [RFC8300]¶

NVA

- Network Virtualization Authority¶

NVE

- Network Virtual Edge (device)¶

NVO3

- Network Virtualization Overlays over Layer 3¶

OAM

- Operations, Administration, and Maintenance [RFC6291]¶

PWE3

- Pseudowire Emulation Edge to Edge¶

TCAM

- Ternary Content-Addressable Memory¶

TLV

- Type, Length, and Value¶

Transit device

- Underlay network devices between NVE(s).¶

UUID

- Universally Unique Identifier¶

VNI

- Virtual Network Identifier¶

VXLAN

- Virtual eXtensible LAN [RFC7348]¶

5. Encapsulation Issues and Background

The following subsections describe issues with current encapsulations as discussed by the NVO3 WG. Numerous extensions and options have been designed for GUE and Geneve which may help resolve some of these issues but have not yet been validated by the WG.¶

Also included are diagrams and information on the candidate encapsulations. These are mostly copied from other documents. Since each protocol is assumed to be sent over UDP, an initial UDP Header is shown which would be preceded by an IPv4 or IPv6 Header.¶

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

Outer UDP Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |    Dest Port = 6081 Geneve    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          UDP Length           |        UDP Checksum           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Geneve Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Virtual Network Identifier (VNI)       |    Reserved   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                    Variable-Length Options                    ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

UDP Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Source port            |     Dest port = 6080 GUE      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        UDP Length             |          Checksum             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

GUE Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | 0 |C|   Hlen  |  Proto/ctype  |             Flags             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                  Extensions Fields (optional)                 ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

Outer UDP Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Source Port         |     Dest Port = 4790 GPE      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           UDP Length          |        UDP Checksum           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

VXLAN-GPE Header
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |R|R|Ver|I|P|B|O|       Reserved                | Next Protocol |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                VXLAN Network Identifier (VNI) |   Reserved    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6. Common Encapsulation Considerations

  IP|UDP|NVO3 Encap|DTLS/IPsec-ESP Extension|payload.

7. Recommendations

The Design Team (DT) reported that Geneve was most suitable as a starting point for a proposed standard for network virtualization, for the following reasons given below. This conclusion was supported by the NVO3 Working Group.¶

1. On whether VNI should be in the base header or in an extension header and whether it should be a 24-bit or 32-bit field (see Section 6.9), it was agreed that VNI is critical information for network virtualization and MUST be present in all packets. It was also agreed that a 24-bit VNI, which is supported by Geneve, matches the existing widely used encapsulation formats, i.e., VXLAN [RFC7348] and NVGRE [RFC7637], and hence is more suitable to use going forward.¶
2. The Geneve header has the total options length which allows skipping over the options for NIC offload operations and will allow transit devices to view flow information in the inner payload.¶
3. The option of using NSH [RFC8300] with VXLAN-GPE was considered but given that NSH is targeted at service chaining and contains service chaining information, it is less suitable for the network virtualization use case. The other downside for VXLAN-GPE was lack of a header length in VXLAN-GPE, which makes skipping over the headers to process inner payload more difficult. A Total Option Length is present in Geneve. It is not possible to skip any options in the middle with VXLAN-GPE. In principle a split between a base header and a header with options is interesting (whether that options header is NSH or some new header without ties to a service path). Whether it would make sense to either use NSH for this, or define a new NVO3 options header was explored. However, this makes it slightly harder to find the inner payload since the length field is not in the NVO3 header itself. Thus, one more field would have to be extracted to compute the start of the inner payload. Also, if the experience with IPv6 extension headers is a guide, there would be a risk that key pieces of hardware might not implement the options header, resulting in future calls to deprecate its use. Making the options part of the base NVO3 header has less of those issues. Even though the implementation of any particular option can not be predicted ahead of time, the option mechanism and ability to skip the options is likely to be broadly implemented.¶
4. The TLV style and bit field style of extension were compared. It was deemed that parsing either TLVs or bit fields is expensive and, while bit fields may be simpler to parse, it is also more restrictive and requires guessing which extensions will be widely implemented so they can get early bit assignments. Given that half the bits are already assigned in GUE, a widely deployed extension may appear in a flag extension, and this will require extra processing, to dig the flag from the flag extension and then look for the extension itself. Also bit fields are not flexible enough to address the requirements from OAM, Telemetry, and security extensions, for variable length option and different subtypes of the same option. While TLVs are more flexible, a control plane can restrict the number of option TLVs as well as the order and size of the TLVs to limit this flexibility and make the TLVs simpler for a dataplane implementation to handle.¶
5. The multi-vendor NVE case was briefly discussed, as was the need to allow vendors to put their own extensions in the NVE header. This is possible with TLVs.¶
6. It was agreed that the C (Critical) bit in Geneve is helpful. This bit indicates that the header includes options which must be parsed or the packet discarded. It allows a receiver NVE to easily decide whether to process options or not, for example a UUID based packet trace, and how an optional extension such as that can be ignored by a receiver NVE and thus make it easy for NVE to skip over the options. Thus, the C bit should remain as defined in Geneve.¶
7. There are already some extensions that are being discussed (see section 6.2) of varying sizes. By using Geneve options it is possible to get in-band parameters like switch id, ingress port, egress port, internal delay, and queue size using TLV extensions for telemetry purpose from switches. It is also possible to add security extension TLVs like HMAC [RFC2104] and DTLS/IPsec [RFC9147]/[RFC6071] to authenticate the Geneve packet header and secure the Geneve packet payload by software or hardware tunnel endpoints. A Group Based Policy extension TLV can be carried as well.¶
8. There are already implementations of Geneve options deployed in production networks. There is as well new hardware supporting Geneve TLV parsing. In addition, an In-band Telemetry [INT] specification is being developed by P4.org that illustrates the option of INT meta data carried over Geneve. OVN/OVS [OVN] have also defined some option TLV(s) for Geneve.¶
9. Usage requirements (see Section 6) have been addressed while considering the requirements and implementations in general including software and hardware.¶

There seems to be interest in standardizing some well-known secure option TLVs to secure the header and payload to guarantee encapsulation header integrity and tenant data privacy. The working group should consider standardizing such option(s).¶

The following enhancements to Geneve are recommended to make it more suitable to hardware and yet provide flexibility for software:¶

• The following sort of text is recommended: while TLVs are more flexible, a control plane can restrict the number of option TLVs as well the order and size of the TLVs to make it simpler for a data plane implementation in software or hardware to handle. For example, there may be some critical information such as a secure hash that must be processed in a certain order at lowest latency.¶
• A control plane can negotiate a subset of option TLVs and certain TLV ordering, as well as limiting the total number of option TLVs present in the packet, for example, to allow for hardware capable of processing fewer options. Hence, the control plane needs to have the ability to describe the supported TLVs subset and their order.¶
• The Geneve documents should specify that the subset and order of option TLVs SHOULD be configurable for each remote NVE in the absence of a protocol control plane.¶
• Geneve should follow fragmentation recommendations in overlay services like PWE3 and the L2/L3 VPN recommendations to guarantee larger MTU for the tunnel overhead ([RFC3985] Section 5.3).¶
• Geneve should provide a recommendation for critical bit processing - text could specify how critical bits can be used with control plane specifying the critical options.¶
• Given that there is a telemetry option use case for a length of 256 bytes, it is recommended that Geneve increase the Single TLV option length to 256.¶
• Geneve address requirements for OAM considerations for alternate marking and for performance measurements that need a 2 bit field in the header should be considered and the need for the current OAM bit in the Geneve Header clarified.¶
• The WG should work on security options for Geneve.¶

9. Security Considerations

This document does not introduce any additional security constraints; however, Section 6.2.2 discusess security/integrity extensions and this document suggests, in Section 7, that the the nvo3 WG work on security options for Geneve.¶

11. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.

12. Informative References

[ietf_gue_extensions]

Herbert, T., Yong, L., and F. Templin, "Extensions for Generic UDP Encapsulation", work in progress, 8 March 2019, <https://datatracker.ietf.org/doc/draft-ietf-intarea-gue-extensions/>.

[ietf_intarea_gue]

Herbert, T., Yong, L., and O. Zia, "Generic UDP Encapsulation", work in progress, 26 October 2019.

[IEEE802.1Q]

802.1 WG, IEEE., "Bridges and Bridged Networks", IEEE Std 802.1Q-2014, 3 November 2014.

[INT]

P4.org, "In-band Network Telemetry (INT) Dataplane Specification", November 2020, <https://p4.org/p4-spec/docs/INT_v2_1.pdf>.

[nvo3_vxlan_gpe]

Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol Extension for VXLAN (VXLAN-GPE)", work in progress, 4 November 2023, <https://datatracker.ietf.org/doc/draft-ietf-nvo3-vxlan-gpe/>.

[OVN]

Network, O. V., "", <https://www.openvswitch.org/>.

[RFC2104]

Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, DOI 10.17487/RFC2104, February 1997, <https://www.rfc-editor.org/info/rfc2104>.

[RFC2418]

Bradner, S., "IETF Working Group Guidelines and Procedures", BCP 25, RFC 2418, DOI 10.17487/RFC2418, September 1998, <https://www.rfc-editor.org/info/rfc2418>.

[RFC3985]

Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI 10.17487/RFC3985, March 2005, <https://www.rfc-editor.org/info/rfc3985>.

[RFC6071]

Frankel, S. and S. Krishnan, "IP Security (IPsec) and Internet Key Exchange (IKE) Document Roadmap", RFC 6071, DOI 10.17487/RFC6071, February 2011, <https://www.rfc-editor.org/info/rfc6071>.

[RFC6291]

Andersson, L., van Helvoort, H., Bonica, R., Romascanu, D., and S. Mansfield, "Guidelines for the Use of the "OAM" Acronym in the IETF", BCP 161, RFC 6291, DOI 10.17487/RFC6291, June 2011, <https://www.rfc-editor.org/info/rfc6291>.

[RFC7042]

Eastlake 3rd, D. and J. Abley, "IANA Considerations and IETF Protocol and Documentation Usage for IEEE 802 Parameters", BCP 141, RFC 7042, DOI 10.17487/RFC7042, October 2013, <https://www.rfc-editor.org/info/rfc7042>.

[RFC7348]

Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, <https://www.rfc-editor.org/info/rfc7348>.

[RFC7637]

Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network Virtualization Using Generic Routing Encapsulation", RFC 7637, DOI 10.17487/RFC7637, September 2015, <https://www.rfc-editor.org/info/rfc7637>.

[RFC8300]

Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., "Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300, January 2018, <https://www.rfc-editor.org/info/rfc8300>.

[RFC8365]

Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., Uttaro, J., and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, March 2018, <https://www.rfc-editor.org/info/rfc8365>.

[RFC8394]

Li, Y., Eastlake 3rd, D., Kreeger, L., Narten, T., and D. Black, "Split Network Virtualization Edge (Split-NVE) Control-Plane Requirements", RFC 8394, DOI 10.17487/RFC8394, May 2018, <https://www.rfc-editor.org/info/rfc8394>.

[RFC8926]

Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., "Geneve: Generic Network Virtualization Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November 2020, <https://www.rfc-editor.org/info/rfc8926>.

[RFC9147]

Rescorla, E., Tschofenig, H., and N. Modadugu, "The Datagram Transport Layer Security (DTLS) Protocol Version 1.3", RFC 9147, DOI 10.17487/RFC9147, April 2022, <https://www.rfc-editor.org/info/rfc9147>.

[RFC9197]

Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi, Ed., "Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022, <https://www.rfc-editor.org/info/rfc9197>.

[VXLANgroup]

Smith, M. and L. Kreeger, "VXLAN Group Policy Option", work in progress, 22 October 2018, <https://datatracker.ietf.org/doc/html/draft-smith-vxlan-group-policy-05>.

Appendix A. Encapsulation Comparison

+----------------+----------------+----------------+----------------+
|                |     Geneve     |      GUE       |   VXLAN-GPE    |
+----------------+----------------+----------------+----------------+
| Outer transport|     UDP/IP     |     UDP/IP     |     UDP/IP     |
| UDP Port Number|     6081       |     6080       |     4790       |
+----------------+----------------+----------------+----------------+
| Base header    |    8 octets    |    4 octets    |    8 octets    |
| length         |                |                |  (16 octets    |
|                |                |                |   using NSH)   |
+----------------+----------------+----------------+----------------+
| Extensibility  |Variable length |Extension fields| No native ext- |
|                |    options     |                | ensibility.    |
|                |                |                | Might use NSH. |
+----------------+----------------+----------------+----------------+
| Extension      |   TLV-based    |   Flag-based   |   TLV-based    |
| parsing method |                |                |(using NSH with |
|                |                |                |   MD Type 2)   |
+----------------+----------------+----------------+----------------+
| Extension      |    Variable    |     Fixed      |    Variable    |
| order          |                |                |  (using NSH)   |
+----------------+----------------+----------------+----------------+
| Length field   |       +        |       +        |       -        |
+----------------+----------------+----------------+----------------+
| Max Header     |   260 octets   |   128 octets   |    8 octets    |
| Length         |                |                |(264 using NSH) |
+----------------+----------------+----------------+----------------+
| Critical exte- |       +        |       -        |       -        |
| nsion bit      |                |                |                |
+----------------+----------------+----------------+----------------+
| VNI field size |    24 bits     |    32 bits     |    24 bits     |
|                |                |  (extension)   |                |
+----------------+----------------+----------------+----------------+
| Next protocol  |    16 bits     |     8 bits     |     8 bits     |
| field          |   Ethertype    | Internet prot- |  New registry  |
|                |   registry     | ocol registry  |                |
+----------------+----------------+----------------+----------------+
| Next protocol  |       -        |       -        |       +        |
| indicator      |                |                |                |
+----------------+----------------+----------------+----------------+
| OAM / control  |    OAM bit     |  Control bit   |    OAM bit     |
| field          |                |                |                |
+----------------+----------------+----------------+----------------+
| Version field  |    2 bits      |    2 bits      |    2 bits      |
+----------------+----------------+----------------+----------------+
| Reserved bits  |   14 bits      |     none       |   27 bits      |
+----------------+----------------+----------------+----------------+