ippm                                                          M. Spiegel
Internet-Draft                                                     Intel
Intended status: Informational                              F. Brockners
Expires: September 10, 2020                                  S. Bhandari
                                                          R. Sivakolundu
                                                                   Cisco
                                                           March 9, 2020


                 In-situ OAM raw data export with IPFIX
                  draft-spiegel-ippm-ioam-rawexport-03

Abstract

   In-situ Operations, Administration, and Maintenance (IOAM) records
   operational and telemetry information in the packet while the packet
   traverses a path between two points in the network.  This document
   discusses how In-situ Operations, Administration, and Maintenance
   (IOAM) information can be exported in raw, i.e. uninterpreted, format
   from network devices to systems, such as monitoring or analytics
   systems using IPFIX.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on September 10, 2020.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents



Spiegel, et al.        Expires September 10, 2020               [Page 1]


Internet-Draft            IOAM raw data export                March 2020


   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements  . . . . . . . . . . . . . . . . . . . . . .   4
     1.2.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . .   5
   2.  Conventions . . . . . . . . . . . . . . . . . . . . . . . . .   5
   3.  IPFIX for IOAM raw data export  . . . . . . . . . . . . . . .   6
     3.1.  Key IPFIX information elements leveraged for IOAM raw
           data export . . . . . . . . . . . . . . . . . . . . . . .   6
     3.2.  New IPFIX information elements leveraged for IOAM raw
           data export . . . . . . . . . . . . . . . . . . . . . . .   7
       3.2.1.  ioamReportFlags . . . . . . . . . . . . . . . . . . .   7
       3.2.2.  ioamEncapsulationType . . . . . . . . . . . . . . . .   7
       3.2.3.  ioamPreallocatedTraceData . . . . . . . . . . . . . .   8
       3.2.4.  ioamIncrementalTraceData  . . . . . . . . . . . . . .   9
       3.2.5.  ioamE2EData . . . . . . . . . . . . . . . . . . . . .   9
       3.2.6.  ioamPOTData . . . . . . . . . . . . . . . . . . . . .  10
       3.2.7.  ipHeaderPacketSectionWithPadding  . . . . . . . . . .  10
       3.2.8.  ethernetFrameSection  . . . . . . . . . . . . . . . .  11
   4.  Examples  . . . . . . . . . . . . . . . . . . . . . . . . . .  12
     4.1.  Fixed Length IP Packet  . . . . . . . . . . . . . . . . .  12
     4.2.  Variable Length IP Packet (length < 255)  . . . . . . . .  13
     4.3.  Variable Length IP Packet (length > 255)  . . . . . . . .  14
     4.4.  Variable Length ETHERNET Packet (length < 255)  . . . . .  15
     4.5.  Variable Length IP Packet with Fixed Length IOAM
           Incremental Trace Data  . . . . . . . . . . . . . . . . .  16
     4.6.  Variable Length IP Packet with Variable Length IOAM
           Incremental Trace Data  . . . . . . . . . . . . . . . . .  17
   5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  18
   6.  Manageability Considerations  . . . . . . . . . . . . . . . .  19
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  19
   8.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  19
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  19
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  19
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  20
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  21

1.  Introduction

   In-situ Operations, Administration, and Maintenance (IOAM) records
   operational and telemetry information in the packet while the packet
   traverses a path between two points in the network.  IOAM data fields



Spiegel, et al.        Expires September 10, 2020               [Page 2]


Internet-Draft            IOAM raw data export                March 2020


   are defined in [I-D.ietf-ippm-ioam-data].  This document discusses
   how In-situ Operations, Administration, and Maintenance (IOAM)
   information can be exported in raw format, i.e. uninterpreted format,
   from network devices to systems, such as monitoring or analytics
   systems using IPFIX [RFC7011].

   "Raw export of IOAM data" refers to a mode of operation where a node
   exports the IOAM data as it is received in the packet.  The exporting
   node neither interprets, aggregates nor reformats the IOAM data
   before it is exported.  Raw export of IOAM data is to support an
   operational model where the processing and interpretation of IOAM
   data is decoupled from the operation of encapsulating/updating/
   decapsulating IOAM data, which is also referred to as IOAM data-plane
   operation.  The figure below shows the separation of concerns for
   IOAM export: Exporting IOAM data is performed by the "IOAM node"
   which performs IOAM data-plane operation, whereas the interpretation
   of IOAM data is performed by the IOAM data processing system.  The
   separation of concerns is to off-load interpretation, aggregation and
   formatting of IOAM data from the node which performs data-plane
   operations.  In other words, a node which is focused on data-plane
   operations, i.e. forwarding of packets and handling IOAM data will
   not be tasked to also interpret the IOAM data, but can leave this
   task to another system.  Note that for scalability reasons, a single
   IOAM node could choose to export IOAM data to several IOAM data
   processing systems.


























Spiegel, et al.        Expires September 10, 2020               [Page 3]


Internet-Draft            IOAM raw data export                March 2020


                       +-------------+
                       | Monitoring/ |
                       | Analytics   |
                       | system      |
                       +-------------+
                              ^
                              |  Processed/interpreted/aggregated
                              |  IOAM data
                              |
                       +-------------+
                       | IOAM data   |
                       | processing  |
                       | system      |
                       +-------------+
                              ^
                              |  Raw export of
                              |  IOAM data
                              |
                       +------------+
                       |            |
                   ..--| IOAM node  |--..
                       |            |
                       +------------+


    IOAM node: IOAM encapsulating, IOAM decapsulating or
               IOAM transit node.

    IOAM data processing system: System that receives raw IOAM data
               and provides for formatting, aggregation and
               interpretation of the IOAM data.

    Monitoring/Analytics system: System that receives telemetry and
               other operational information from a variety of
               sources and provides for correlation and
               interpretation of the data received.


   Raw export of IOAM data is typically generated by network devices at
   the edges of the network.  Deployment and use-case dependent, such as
   in cases where the operator is interested in dropped packets, raw
   export of IOAM data may be generated by IOAM transit nodes.

1.1.  Requirements

   Requirements for raw export of IOAM data:

   o  Export all IOAM information contained in a packet.



Spiegel, et al.        Expires September 10, 2020               [Page 4]


Internet-Draft            IOAM raw data export                March 2020


   o  Export a specific IOAM data type - Incremental Trace type,
      Preallocated Trace type, Proof of Transit type, Edge to Edge type.

   o  Support coalescing of the IOAM data from multiple packets into a
      single raw export packet.

   o  Support export of additional parts of the packet, other than the
      IOAM data as part of the raw export.  This could be parts of the
      packet header and/or parts of the packet payload.  This additional
      information provides context to the IOAM data (e.g. to be used for
      flow identification) and is to enable the IOAM data processing
      system to perform further analysis on the received data.

   o  Report the reason why IOAM data was exported.  The "reason for
      export" is to complement the IOAM data retrieved from the packet.
      For example, if a packet was dropped by a node due to congestion,
      it could be helpful to export the IOAM data of this dropped packet
      along with an indication that the packet that the IOAM data
      belongs to was dropped due to congestion.

1.2.  Scope

   This document discusses raw export of IOAM data using IPFIX.

   The following is considered out of scope for this document:

   o  Protocols other than IPFIX for raw export of IOAM data.

   o  Interpretation or aggregation of IOAM data prior to exporting.

   o  Configuration of network devices so that they can determine when
      to generate IOAM reports, and what information to include in those
      reports.

   o  Events that trigger generation of IOAM reports.

   o  Selection of particular destinations within distributed telemetry
      monitoring systems, to which IOAM reports will be sent.

   o  Export format for flow statistics or processed/interpreted/
      aggregated IOAM data.

2.  Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].




Spiegel, et al.        Expires September 10, 2020               [Page 5]


Internet-Draft            IOAM raw data export                March 2020


   Abbreviations used in this document:

   E2E:       Edge to Edge

   IOAM:      In-situ Operations, Administration, and Maintenance

   MTU:       Maximum Transmit Unit

   OAM:       Operations, Administration, and Maintenance

   POT:       Proof of Transit

3.  IPFIX for IOAM raw data export

   IPFIX, being a generic export protocol, can export any Information
   Elements as long as they are described in the information model.  The
   IPFIX protocol is well suited for and is defined as the protocol for
   exporting packet samples in [RFC5476].

   IPFIX/PSAMP [RFC7011], [RFC5476] already define many of the
   information elements needed for exporting sections of packets needed
   for deriving context and raw IOAM data export.  This document
   specifies extensions of the IPFIX information model for meeting the
   requirements in Section 1.1.

3.1.  Key IPFIX information elements leveraged for IOAM raw data export

   The existing IPFIX Information Elements that are required for IOAM
   raw data export are listed here.  Their details are available in
   IANA's IPFIX registry [IANA-IPFIX].

   The existing IPFIX Information Elements used to carry the sections of
   the packets including IOAM data within it are as follows:

   313  - ipHeaderPacketSection

   315  - dataLinkFrameSection

   The following Information Elements will be used to provide context to
   the ipHeaderPacketSection and dataLinkFrameSection as described in
   [IANA-IPFIX]:

   408  - dataLinkFrameType

   409  - sectionOffset

   410  - sectionExportedOctets




Spiegel, et al.        Expires September 10, 2020               [Page 6]


Internet-Draft            IOAM raw data export                March 2020


   The following Information Element will be used to provide forwarding
   status of the flow and any attached reasons.

   89 - forwardingStatus

3.2.  New IPFIX information elements leveraged for IOAM raw data export

   IOAM data raw export using IPFIX requires a set of new information
   elements which are described in this section.

3.2.1.  ioamReportFlags

   Description:

   This Information Element describes properties associated with an IOAM
   report.

   The ioamReportFlags data type is an 8-bit field.  The following bits
   are defined here:

   Bit 0  Dropped Association - Dropped packet of interest.

   Bit 1  Congested Queue Association - Indicates the presence of
      congestion on a monitored queue.

   Bit 2  Tracked Flow Association - Matched a flow of interest.

   Bit 3-7  Reserved

   IANA is requested to create a new subregistry for IOAM Report Flags
   and fill it with the initial list from the description.  New
   assignments for IOAM Encapsulation Types are administered by IANA
   through Expert Review [RFC5226] i.e., review by one of a group of
   experts designated by an IETF Area Director.

   Abstract Data Type: unsigned8

   Data Type Semantics: flags

   ElementId: TBD1

   Status: current

3.2.2.  ioamEncapsulationType

   Description:





Spiegel, et al.        Expires September 10, 2020               [Page 7]


Internet-Draft            IOAM raw data export                March 2020


   This Information Element specifies the type of encapsulation to
   interpret ioamPreallocatedTraceHeader, ioamIncrementalTraceHeader,
   ioamE2EHeader, ioamPOTHeader.

   The following ioamEncapsulationType values are defined here:

   0  None : IOAM data follows format defined in
      [I-D.ietf-ippm-ioam-data]

   1  GRE : IOAM data follows format defined in [I-D.weis-ippm-ioam-eth]

   2  IPv6 : IOAM data follows format defined in
      [I-D.ietf-ippm-ioam-ipv6-options]

   3  VXLAN-GPE : IOAM data follows format defined in
      [I-D.brockners-ippm-ioam-vxlan-gpe]

   4  GENEVE Option: IOAM data follows format defined in
      [I-D.brockners-ippm-ioam-geneve]

   5  GENEVE Next Protocol: IOAM data follows format defined in
      [I-D.weis-ippm-ioam-eth]

   6  NSH : IOAM data follows format defined in [I-D.ietf-sfc-ioam-nsh]

   IANA is requested to create a new subregistry for IOAM Encapsulation
   Types and fill it with the initial list from the description.  New
   assignments for IOAM Encapsulation Types are administered by IANA
   through Expert Review [RFC5226] i.e., review by one of a group of
   experts designated by an IETF Area Director.

   Abstract Data Type: unsigned8

   Data Type Semantics: identifier

   ElementId: TBD2

   Status: current

3.2.3.  ioamPreallocatedTraceData

   Description:

   This Information Element carries n octets of IOAM Preallocated Trace
   data defined in [I-D.ietf-ippm-ioam-data].

   The format of the data is determined by the ioamEncapsulationType
   information element, if present.  When the ioamEncapsulationType



Spiegel, et al.        Expires September 10, 2020               [Page 8]


Internet-Draft            IOAM raw data export                March 2020


   information element is present and has a value other than "None", and
   with sufficient length, this element may also report octets from
   subsequent headers and payload.  If no ioamEncapsulationType
   information element is present, then the encapsulation type shall be
   assumed to be "None" and this information element only contains
   octets from the IOAM Preallocated Trace Option.

   Abstract Data Type: octetArray

   ElementId: TBD3

   Status: current

3.2.4.  ioamIncrementalTraceData

   Description:

   This Information Element carries n octets of IOAM Incremental Trace
   data defined in [I-D.ietf-ippm-ioam-data].

   The format of the data is determined by the ioamEncapsulationType
   information element, if present.  When the ioamEncapsulationType
   information element is present and has a value other than "None", and
   with sufficient length, this element may also report octets from
   subsequent headers and payload.  If no ioamEncapsulationType
   information element is present, then the encapsulation type shall be
   assumed to be "None" and this information element only contains
   octets from the IOAM Incremental Trace Option.

   Abstract Data Type: octetArray

   ElementId: TBD4

   Status: current

3.2.5.  ioamE2EData

   Description:

   This Information Element carries n octets of IOAM E2E data defined in
   [I-D.ietf-ippm-ioam-data].

   The format of the data is determined by the ioamEncapsulationType
   information element, if present.  When the ioamEncapsulationType
   information element is present and has a value other than "None", and
   with sufficient length, this element may also report octets from
   subsequent headers and payload.  If no ioamEncapsulationType
   information element is present, then the encapsulation type shall be



Spiegel, et al.        Expires September 10, 2020               [Page 9]


Internet-Draft            IOAM raw data export                March 2020


   assumed to be "None" and this information element only contains
   octets from the IOAM Edge-to-Edge Option.

   Abstract Data Type: octetArray

   ElementId: TBD5

   Status: current

3.2.6.  ioamPOTData

   Description:

   This Information Element carries n octets of IOAM POT data defined in
   [I-D.ietf-ippm-ioam-data].

   The format of the data is determined by the ioamEncapsulationType
   information element, if present.  When the ioamEncapsulationType
   information element is present and has a value other than "None", and
   with sufficient length, this element may also report octets from
   subsequent headers and payload.  If no ioamEncapsulationType
   information element is present, then the encapsulation type shall be
   assumed to be "None" and this information element only contains
   octets from the IOAM Proof of Transit Option.

   Abstract Data Type: octetArray

   ElementId: TBD6

   Status: current

3.2.7.  ipHeaderPacketSectionWithPadding

   Description:

   This Information Element carries a series of n octets from the IP
   header of a sampled packet, starting sectionOffset octets into the IP
   header.

   However, if no sectionOffset field corresponding to this Information
   Element is present, then a sectionOffset of zero applies, and the
   octets MUST be from the start of the IP header.

   With sufficient length, this element also reports octets from the IP
   payload.  However, full packet capture of arbitrary packet streams is
   explicitly out of scope per the Security Considerations sections of
   [RFC5477] and [RFC2804].




Spiegel, et al.        Expires September 10, 2020              [Page 10]


Internet-Draft            IOAM raw data export                March 2020


   When this Information Element has a fixed length, this MAY include
   padding octets that are used to fill out that fixed length.

   When this information element has a variable length, the variable
   length MAY include up to 3 octets of padding, used to preserve
   4-octet alignment of subequent Information Elements or subsequent
   records within the same set.

   In either case of fixed or variable length, the amount of populated
   octets MAY be specified in the sectionExportedOctets field
   corresponding to this Information Element, in which case the
   remainder (if any) MUST be padding.  If there is no
   sectionExportedOctets field corresponding to this Information
   Element, then all octets MUST be populated unless the total length of
   the IP packet is less than the fixed length of this Information
   Element, in which case the remainder MUST be padding.

   Abstract Data Type: octetArray

   ElementId: TBD7

   Status: current

3.2.8.  ethernetFrameSection

   Description:

   This Information Element carries a series of n octets from the IEEE
   802.3 Ethernet frame of a sampled packet, starting after the preamble
   and start frame delimiter (SFD), plus sectionOffset octets into the
   frame if there is a sectionOffset field corresponding to this
   Information Element.

   With sufficient length, this element also reports octets from the
   Ethernet payload.  However, full packet capture of arbitrary packet
   streams is explicitly out of scope per the Security Considerations
   sections of [RFC5477] and [RFC2804].

   When this Information Element has a fixed length, this MAY include
   padding octets that are used to fill out that fixed length.

   When this information element has a variable length, the variable
   length MAY include up to 3 octets of padding, used to preserve
   4-octet alignment of subequent Information Elements or subsequent
   records within the same set.

   In either case of fixed or variable length, the amount of populated
   octets MAY be specified in the sectionExportedOctets field



Spiegel, et al.        Expires September 10, 2020              [Page 11]


Internet-Draft            IOAM raw data export                March 2020


   corresponding to this Information Element, in which case the
   remainder (if any) MUST be padding.  If there is no
   sectionExportedOctets field corresponding to this Information
   Element, then all octets MUST be populated unless the total length of
   the Ethernet frame is less than the fixed length of this Information
   Element, in which case the remainder MUST be padding.

   Abstract Data Type: octetArray

   ElementId: TBD8

   Status: current

4.  Examples

   This section shows a set of examples of how IOAM information along
   with other parts of the packet can be carried using IPFIX.

4.1.  Fixed Length IP Packet

   This example shows a fixed length IP packet.  IOAM data is part of
   the ipHeaderPacketSection.





























Spiegel, et al.        Expires September 10, 2020              [Page 12]


Internet-Draft            IOAM raw data export                March 2020


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+---
|        Version Number         |             Length            |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Export Time (seconds)                     |IPFIX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message
|                        Sequence Number                        |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Observation Domain ID                     |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|    Set ID ( = Template ID)    |             Length            |SetHdr
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|ioamReportFlags| fwdingStatus  |     sectionExportedOctets     |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                 ipHeaderPacketSection (start)                 |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1
|                              ...                              |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                 ipHeaderPacketSection (end)                   |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|ioamReportFlags| fwdingStatus  |     sectionExportedOctets     |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                 ipHeaderPacketSection (start)                 |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2
|                              ...                              |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                 ipHeaderPacketSection (end)                   |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--


4.2.  Variable Length IP Packet (length < 255)

   This examples shows a variable length IP packet, with length < 255
   bytes.  IOAM data is part of the ipHeaderPacketSectionWithPadding.
















Spiegel, et al.        Expires September 10, 2020              [Page 13]


Internet-Draft            IOAM raw data export                March 2020


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|        Version Number         |             Length            |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Export Time (seconds)                     |IPFIX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message
|                        Sequence Number                        |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Observation Domain ID                     |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|    Set ID ( = Template ID)    |             Length            |SetHdr
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|ioamReportFlags| fwdingStatus  | paddingOctets | Length (< 255)|   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|            ipHeaderPacketSectionWithPadding (start)           |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1
|                              ...                              |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|            ipHeaderPacketSectionWithPadding (end)             |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|ioamReportFlags| fwdingStatus  | paddingOctets | Length (< 255)|   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|            ipHeaderPacketSectionWithPadding (start)           |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2
|                              ...                              |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|            ipHeaderPacketSectionWithPadding (end)             |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--


4.3.  Variable Length IP Packet (length > 255)

   This examples shows a variable length IP packet, with length > 255
   bytes.  IOAM data is part of the ipHeaderPacketSectionWithPadding.
















Spiegel, et al.        Expires September 10, 2020              [Page 14]


Internet-Draft            IOAM raw data export                March 2020


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|        Version Number         |             Length            |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Export Time (seconds)                     |IPFIX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message
|                        Sequence Number                        |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Observation Domain ID                     |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|    Set ID ( = Template ID)    |             Length            |SetHdr
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|ioamReportFlags| fwdingStatus  |         paddingOctets         |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
| paddingOctets |      255      |      Length (0 to 65535)      |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|            ipHeaderPacketSectionWithPadding (start)           |Record1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                              ...                              |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|            ipHeaderPacketSectionWithPadding (end)             |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--


4.4.  Variable Length ETHERNET Packet (length < 255)

   This examples shows a variable length Ethernet packet, with length <
   255 bytes.  IOAM data is part of the ethernetFrameSection.






















Spiegel, et al.        Expires September 10, 2020              [Page 15]


Internet-Draft            IOAM raw data export                March 2020


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|        Version Number         |             Length            |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Export Time (seconds)                     |IPFIX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message
|                        Sequence Number                        |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                     Observation Domain ID                     |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|    Set ID ( = Template ID)    |             Length            |SetHdr
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--
|ioamReportFlags| fwdingStatus  | paddingOctets | Length (< 255)|   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                  ethernetFrameSection (start)                 |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1
|                              ...                              |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   |
|                  ethernetFrameSection (end)                   |   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--


4.5.  Variable Length IP Packet with Fixed Length IOAM Incremental Trace
      Data

   This examples shows a variable length IP packet with length < 255
   bytes and fixed length ioamIncrementalTraceData carried separately.























Spiegel, et al.        Expires September 10, 2020              [Page 16]


Internet-Draft            IOAM raw data export                March 2020


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---
|        Version Number         |             Length            |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                     Export Time (seconds)                     |IPFIX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message
|                        Sequence Number                        |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                     Observation Domain ID                     |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---
|    Set ID ( = Template ID)    |             Length            |SetHdr
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---
|               ioamIncrementalTraceData (start)                |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                              ...                              |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|               ioamIncrementalTraceData (end)                  |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1
|ioamReportFlags| fwdingStatus  |ioamEncapType  | Length (< 255)|  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|            ipHeaderPacketSectionWithPadding (start)           |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                              ...                              |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|            ipHeaderPacketSectionWithPadding (end)             |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---


4.6.  Variable Length IP Packet with Variable Length IOAM Incremental
      Trace Data

   This examples shows a variable length IP packet with length < 255
   bytes and variable length ioamIncrementalTraceData with length < 255
   bytes carried separately.
















Spiegel, et al.        Expires September 10, 2020              [Page 17]


Internet-Draft            IOAM raw data export                March 2020


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---
|        Version Number         |             Length            |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                     Export Time (seconds)                     |IPFIX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message
|                        Sequence Number                        |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                     Observation Domain ID                     |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---
|    Set ID ( = Template ID)    |             Length            |SetHdr
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---
|ioamReportFlags| fwdingStatus  |ioamEncapType  | Length (< 255)|  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|               ioamIncrementalTraceData (start)                |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                              ...                              |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|               ioamIncrementalTraceData (end)                  |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1
|                 paddingOctets                 | Length (< 255)|  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|            ipHeaderPacketSectionWithPadding (start)           |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|                              ...                              |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
|            ipHeaderPacketSectionWithPadding (end)             |  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+---


5.  IANA Considerations

   IANA is requested to allocate code points for the following
   Information Elements in [IANA-IPFIX]:

   TBD1  ioamReportFlags

   TBD2  ioamEncapsulationType

   TBD3  ioamPreallocatedTraceData

   TBD4  ioamIncrementalTraceData

   TBD5  ioamE2EData

   TBD6  ioamPOTData




Spiegel, et al.        Expires September 10, 2020              [Page 18]


Internet-Draft            IOAM raw data export                March 2020


   TBD7  ipHeaderPacketSectionWithPadding

   TBD8  ethernetFrameSection

   See Section 3.2 for further details.

   IANA is requested to create subregistries for ioamReportFlags defined
   in Section 3.2.1 and ioamEncapsulationType defined in Section 3.2.2.

6.  Manageability Considerations

   Manageability considerations will be addressed in a later version of
   this document..

7.  Security Considerations

   Security considerations will be addressed in a later version of this
   document.

8.  Acknowledgements

   The authors would like to thank Barak Gafni, Tal Mizrahi, John Lemon,
   and Aviv Kfir for their thoughts and comments on raw IOAM data
   export.

9.  References

9.1.  Normative References

   [I-D.ietf-ippm-ioam-data]
              Brockners, F., Bhandari, S., Pignataro, C., Gredler, H.,
              Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov,
              P., Chang, R., Bernier, D., and J. Lemon, "Data Fields for
              In-situ OAM", draft-ietf-ippm-ioam-data-09 (work in
              progress), March 2020.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC5476]  Claise, B., Ed., Johnson, A., and J. Quittek, "Packet
              Sampling (PSAMP) Protocol Specifications", RFC 5476,
              DOI 10.17487/RFC5476, March 2009,
              <https://www.rfc-editor.org/info/rfc5476>.






Spiegel, et al.        Expires September 10, 2020              [Page 19]


Internet-Draft            IOAM raw data export                March 2020


   [RFC7011]  Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
              "Specification of the IP Flow Information Export (IPFIX)
              Protocol for the Exchange of Flow Information", STD 77,
              RFC 7011, DOI 10.17487/RFC7011, September 2013,
              <https://www.rfc-editor.org/info/rfc7011>.

9.2.  Informative References

   [I-D.brockners-ippm-ioam-geneve]
              Brockners, F., Bhandari, S., Govindan, V., Pignataro, C.,
              Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Lapukhov,
              P., Gafni, B., Kfir, A., and M. Spiegel, "Geneve
              encapsulation for In-situ OAM Data", draft-brockners-ippm-
              ioam-geneve-03 (work in progress), September 2019.

   [I-D.brockners-ippm-ioam-vxlan-gpe]
              Brockners, F., Bhandari, S., Govindan, V., Pignataro, C.,
              Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A.,
              Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE
              Encapsulation for In-situ OAM Data", draft-brockners-ippm-
              ioam-vxlan-gpe-03 (work in progress), November 2019.

   [I-D.ietf-ippm-ioam-ipv6-options]
              Bhandari, S., Brockners, F., Pignataro, C., Gredler, H.,
              Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B.,
              Lapukhov, P., Spiegel, M., Krishnan, S., and R. Asati,
              "In-situ OAM IPv6 Options", draft-ietf-ippm-ioam-
              ipv6-options-01 (work in progress), March 2020.

   [I-D.ietf-sfc-ioam-nsh]
              Brockners, F. and S. Bhandari, "Network Service Header
              (NSH) Encapsulation for In-situ OAM (IOAM) Data", draft-
              ietf-sfc-ioam-nsh-02 (work in progress), September 2019.

   [I-D.weis-ippm-ioam-eth]
              Weis, B., Brockners, F., Hill, C., Bhandari, S., Govindan,
              V., Pignataro, C., Gredler, H., Leddy, J., Youell, S.,
              Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M.
              Spiegel, "EtherType Protocol Identification of In-situ OAM
              Data", draft-weis-ippm-ioam-eth-02 (work in progress),
              September 2019.

   [IANA-IPFIX]
              "IP Flow Information Export (IPFIX) Entities",
              <https://www.iana.org/assignments/ipfix/ipfix.xhtml>.






Spiegel, et al.        Expires September 10, 2020              [Page 20]


Internet-Draft            IOAM raw data export                March 2020


   [RFC2804]  IAB and IESG, "IETF Policy on Wiretapping", RFC 2804,
              DOI 10.17487/RFC2804, May 2000,
              <https://www.rfc-editor.org/info/rfc2804>.

   [RFC5226]  Narten, T. and H. Alvestrand, "Guidelines for Writing an
              IANA Considerations Section in RFCs", RFC 5226,
              DOI 10.17487/RFC5226, May 2008,
              <https://www.rfc-editor.org/info/rfc5226>.

   [RFC5477]  Dietz, T., Claise, B., Aitken, P., Dressler, F., and G.
              Carle, "Information Model for Packet Sampling Exports",
              RFC 5477, DOI 10.17487/RFC5477, March 2009,
              <https://www.rfc-editor.org/info/rfc5477>.

Authors' Addresses

   Mickey Spiegel
   Barefoot Networks, an Intel company
   4750 Patrick Henry Drive
   Santa Clara, CA  95054
   US

   Email: mickey.spiegel@intel.com


   Frank Brockners
   Cisco Systems, Inc.
   Hansaallee 249, 3rd Floor
   DUESSELDORF, NORDRHEIN-WESTFALEN  40549
   Germany

   Email: fbrockne@cisco.com


   Shwetha Bhandari
   Cisco Systems, Inc.
   Cessna Business Park, Sarjapura Marathalli Outer Ring Road
   Bangalore, KARNATAKA 560 087
   India

   Email: shwethab@cisco.com










Spiegel, et al.        Expires September 10, 2020              [Page 21]


Internet-Draft            IOAM raw data export                March 2020


   Ramesh Sivakolundu
   Cisco Systems, Inc.
   170 West Tasman Dr.
   SAN JOSE, CA 95134
   U.S.A.

   Email: sramesh@cisco.com












































Spiegel, et al.        Expires September 10, 2020              [Page 22]