Internet-Draft E. Boschi draft-ietf-ipfix-reducing-redundancy-01.txt Hitachi Europe Expires: April 26, 2007 L. Mark Fraunhofer FOKUS B. Claise Cisco Systems October 23, 2006 Reducing redundancy in IPFIX and PSAMP reports draft-ietf-ipfix-reducing-redundancy-01 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 26, 2007. Copyright Notice Copyright (C) The Internet Society (2006). Boschi, Mark, Claise Expires April 2007 [Page 1]
Reducing redundancy in IPFIX and PSAMP reports Abstract This document describes a bandwidth saving method for exporting flow or packet information using the IP Flow Information Export (IPFIX) protocol. As the PSAMP protocol is based on IPFIX, these considerations are valid for PSAMP exports as well. This method works by separating information common to several flow records from information specific to an individual flow record. Common flow information is exported only once in a data record defined by an option template, while the rest of the specific flow information is associated with the common information via a unique identifier. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Table of Contents Copyright Notice...............................................1 Abstract.......................................................2 1. Introduction..............................................3 1.1 IPFIX Documents Overview...............................4 1.2 PSAMP Documents Overview...............................4 2. Terminology...............................................4 2.1 Terminology Summary Table.............................10 2.2 IPFIX Flows versus PSAMP Packets......................10 3. Problem Statement and High Level Solution................10 3.1 Per Flow Data Reduction...............................11 3.1.1 Unique Data Reduction..................................11 3.1.2 Multiple Data Reduction................................12 3.2 Per Packet Data Reduction.............................13 4. Specifications for bandwidth saving information export...15 4.1 Per Flow Data Reduction...............................15 4.1.1 Unique Data Reduction..................................15 4.1.2 Multiple Data Reduction................................17 4.2 Per-Packet Data Reduction.............................17 5. Transport Protocol Choice................................18 5.1 PR-SCTP...............................................18 5.2 UDP...................................................18 5.3 TCP...................................................18 6. commonPropertiesID Management............................18 7. The Collecting Process Side..............................19 7.1 UDP...................................................20 7.2 TCP...................................................21 8. Export and Evaluation Considerations.....................21 8.1 Transport Protocol Choice.............................22 Boschi, Mark, Claise Expires April 2007 [Page 2]
Reducing redundancy in IPFIX and PSAMP reports 8.2 Reduced Size Encoding.................................22 8.3 commonPropertiesID vs. TemplateID scope...............22 8.4 Efficiency Gain.......................................22 9. IANA Considerations......................................22 10. Security Considerations..................................23 11. Appendix A: Examples.....................................23 11.1 Per Flow Data Reduction...............................23 11.1.1 Unique Data Reduction.................................23 11.1.2 Multiple Data Reduction...............................26 11.2 Per-Packet Information Export.........................30 11.3 commonPropertiesID Template Withdrawal Message........32 12. References...............................................33 12.1 Normative References..................................33 12.2 Informative References................................33 13. Acknowledgements.........................................34 14. Author's Addresses.......................................34 15. Intellectual Property Statement..........................35 16. Copyright Statement......................................35 17. Disclaimer...............................................35 1. Introduction The IPFIX working group has specified a protocol to export IP Flow information [IPFIX-PROTO]. This protocol is designed to export information about IP traffic flows and related measurement data, where a flow is defined by a set of key attributes (e.g. source and destination IP address, source and destination port, etc.). However, thanks to its template mechanism, the IPFIX protocol can export any type of information, as long as the information element is specified in [IPFIX-INFO] or registered with IANA. Regardless of the fields content, flow records with common properties export the same fields in every single flow record. These common properties may represent values common to a collection of flows or packets, or values that are invariant over time. The reduction of redundant data from the export stream can result in a significant reduction of the transferred data. This draft specifies a way to export these invariant or common properties only once, while the rest of the flow specific properties are exported in regular data records. Unique common properties identifiers are used to link data records and the common attributes. The proposed method is applicable to IPFIX flow and to PSAMP per packet information, without any changes to both the IPFIX and PSAMP protocol specifications. Boschi, Mark, Claise Expires April 2007 [Page 3]
Reducing redundancy in IPFIX and PSAMP reports 1.1 IPFIX Documents Overview The IPFIX protocol [IPFIX-PROTO] provides network administrators with access to IP flow information. The architecture for the export of measured IP flow information out of an IPFIX exporting process to a collecting process is defined in [IPFIX-ARCH], per the requirements defined in [RFC3917]. [IPFIX-ARCH] specifies how IPFIX data record and templates are carried via a congestion-aware transport protocol from IPFIX exporting processes to IPFIX collecting process. IPFIX has a formal description of IPFIX information elements, their name, type and additional semantic information, as specified in [IPFIX-INFO]. Finally [IPFIX-AS] describes what type of applications can use the IPFIX protocol and how they can use the information provided. It furthermore shows how the IPFIX framework relates to other architectures and frameworks. 1.2 PSAMP Documents Overview The document "A Framework for Packet Selection and Reporting" [PSAMP- FMWK], describes the PSAMP framework for network elements to select subsets of packets by statistical and other methods, and to export a stream of reports on the selected packets to a collector. The set of packet selection techniques (sampling, filtering, and hashing) supported by PSAMP are described in "Sampling and Filtering Techniques for IP Packet Selection" [PSAMP-TECH]. The PSAMP protocol [PSAMP-PROTO] specifies the export of packet information from a PSAMP exporting process to a PSAMP collecting process. Like IPFIX, PSAMP has a formal description of its information elements, their name, type and additional semantic information. The PSAMP information model is defined in [PSAMP-INFO]. Finally [PSAMP-MIB] describes the PSAMP Management Information Base. 2. Terminology The terms in this section are in line with the IPFIX terminology section [IPFIX-PROTO], and [PSAMP-PROTO]. Note that this document selected the IPFIX definition of the term Exporting Process [IPFIX- PROTO], as this definition is more generic than the PSAMP definition [PSAMP-PROTO]. Observation Point An Observation Point is a location in the network where IP packets can be observed. Examples include: a line to which a probe is attached, a shared medium, such as an Ethernet-based LAN, a single port of a router, or a set of interfaces (physical or logical) of a router. Boschi, Mark, Claise Expires April 2007 [Page 4]
Reducing redundancy in IPFIX and PSAMP reports Note that every Observation Point is associated with an Observation Domain (defined below), and that one Observation Point may be a superset of several other Observation Points. For example one Observation Point can be an entire line card. That would be the superset of the individual Observation Points at the line card's interfaces. Observation Domain An Observation Domain is the largest set of Observation Points for which Flow information can be aggregated by a Metering Process. For example, a router line card may be an Observation Domain if it is composed of several interfaces, each of which is an Observation Point. In the IPFIX Message it generates, the Observation Domain includes its Observation Domain ID, which is unique per Exporting Process. That way, the Collecting Process can identify the specific Observation Domain from the Exporter that sends the IPFIX Messages. Every Observation Point is associated with an Observation Domain. It is RECOMMENDED that Observation Domain IDs are also unique per IPFIX Device. IP Traffic Flow or Flow There are several definitions of the term 'flow' being used by the Internet community. Within the context of IPFIX we use the following definition: A Flow is defined as a set of IP packets passing an Observation Point in the network during a certain time interval. All packets belonging to a particular Flow have a set of common properties. Each property is defined as the result of applying a function to the values of: 1. one or more packet header field (e.g. destination IP address), transport header field (e.g. destination port number), or application header field (e.g. RTP header fields [RFC1889]) 2. one or more characteristics of the packet itself (e.g. number of MPLS labels, etc...) 3. one or more of fields derived from packet treatment (e.g. next hop IP address, the output interface, etc...) A packet is defined to belong to a Flow if it completely satisfies all the defined properties of the Flow. This definition covers the range from a Flow containing all packets observed at a network interface to a Flow consisting of just a single packet between two applications. It includes packets selected by a sampling mechanism. Boschi, Mark, Claise Expires April 2007 [Page 5]
Reducing redundancy in IPFIX and PSAMP reports Flow Record A Flow Record contains information about a specific Flow that was observed at an Observation Point. A Flow Record contains measured properties of the Flow (e.g. the total number of bytes for all the Flow's packets) and usually characteristic properties of the Flow (e.g. source IP address). Metering Process The Metering Process generates Flow Records. Inputs to the process are packet headers and characteristics observed at an Observation Point, and packet treatment at the Observation Point (for example the selected output interface). The Metering Process consists of a set of functions that includes packet header capturing, timestamping, sampling, classifying, and maintaining Flow Records. The maintenance of Flow Records may include creating new records, updating existing ones, computing Flow statistics, deriving further Flow properties, detecting Flow expiration, passing Flow Records to the Exporting Process, and deleting Flow Records. Exporting Process The Exporting Process sends Flow Records to one or more Collecting Processes. The Flow Records are generated by one or more Metering Processes. Exporter A device which hosts one or more Exporting Processes is termed an Exporter. IPFIX Device An IPFIX Device hosts at least one Exporting Process. It may host further Exporting processes and arbitrary numbers of Observation Points and Metering Process. Collecting Process A Collecting Process receives Flow Records from one or more Exporting Processes. The Collecting Process might process or store received Flow Records, but such actions are out of scope for this document. Boschi, Mark, Claise Expires April 2007 [Page 6]
Reducing redundancy in IPFIX and PSAMP reports Template Template is an ordered sequence of <type, length> pairs, used to completely specify the structure and semantics of a particular set of information that needs to be communicated from an IPFIX Device to a Collector. Each Template is uniquely identifiable by means of a Template ID. Template Record A Template Record defines the structure and interpretation of fields in a Data Record. Data Record A Data Record is a record that contains values of the parameters corresponding to a Template Record. Options Template Record An Options Template Record is a Template Record that defines the structure and interpretation of fields in a Data Record, including defining how to scope the applicability of the Data Record. Set Set is a generic term for a collection of records that have a similar structure. In an IPFIX Message, one or more Sets follow the Message Header. There are three different types of Sets: Template Set, Options Template Set, and Data Set. Template Set A Template Set is a collection of one or more Template Records that have been grouped together in an IPFIX Message. Options Template Set An Options Template Set is a collection of one or more Options Template Records that have been grouped together in an IPFIX Message. Data Set A Data Set is one or more Data Records, of the same type, that are grouped together in an IPFIX Message. Each Data Record is Boschi, Mark, Claise Expires April 2007 [Page 7]
Reducing redundancy in IPFIX and PSAMP reports previously defined by a Template Record or an Options Template Record. Information Element An Information Element is a protocol and encoding independent description of an attribute which may appear in an IPFIX Record. The IPFIX information model [IPFIX-INFO] defines the base set of Information Elements for IPFIX. The type associated with an Information Element indicates constraints on what it may contain and also determines the valid encoding mechanisms for use in IPFIX. Observed Packet Stream The Observed Packet Stream is the set of all packets observed at the Observation Point. Packet Content The packet content denotes the union of the packet header (which includes link layer, network layer and other encapsulation headers) and the packet payload. Selection Process A Selection Process takes the Observed Packet Stream as its input and selects a subset of that stream as its output. Selector A Selector defines the action of a Selection Process on a single packet of its input. If selected, the packet becomes an element of the output Packet Stream. The Selector can make use of the following information in determining whether a packet is selected: (i) the Packet Content; (ii) information derived from the packet's treatment at the Observation Point; (iii) any selection state that may be maintained by the Selection Process. PSAMP Device A PSAMP Device is a device hosting at least an Observation Point, a Selection Process and an Exporting Process. Typically, corresponding Observation Point(s), Selection Boschi, Mark, Claise Expires April 2007 [Page 8]
Reducing redundancy in IPFIX and PSAMP reports Process(es) and Exporting Process(es) are co-located at this device, for example at a router. Filtering A filter is a Selector that selects a packet deterministically based on the Packet Content, or its treatment, or functions of these occurring in the Selection State. Examples include field match Filtering, and Hash-based Selection. commonPropertiesID An identifier of a set of common properties that is locally unique to an Exporting Process and to Observation Domain. This ID can be used to link to information reported in separate records. See [IPFIX-INFO] for the Information Element definition. Common Properties Common Properties are a collection of one or more attributes shared by a set of different Flow Records. Each set of Common Properties is uniquely identifiable by means of a commonPropertiesID. Specific Properties Specific Properties are a collection of one or more attributes reported in a Flow Record that are not included in the Common Properties defined for that Flow Record. Transport Session In SCTP, the transport session is known as the SCTP association, which is uniquely identified by the SCTP endpoints [RFC2960]; in TCP, the transport session is known as the TCP connection, which is uniquely identified by the combination of IP addresses and TCP ports used; In UDP, the transport session is known as the UDP session, which is uniquely identified by the combination of IP addresses and UDP ports used. Boschi, Mark, Claise Expires April 2007 [Page 9]
Reducing redundancy in IPFIX and PSAMP reports 2.1 Terminology Summary Table. +------------------+---------------------------------------------+ | | Contents | | +--------------------+------------------------+ | Set | Template | Record | +------------------+--------------------+------------------------+ | Data Set | / | Data Record(s) | +------------------+--------------------+------------------------+ | Template Set | Template Record(s) | / | +------------------+--------------------+------------------------+ | Options Template | Options Template | / | | Set | Record(s) | | +------------------+--------------------+------------------------+ Figure 1: Terminology Summary Table A Data Set is composed of Data Record(s). No Template Record is included. A Template Record or an Options Template Record defines the Data Record. A Template Set contains only Template Record(s). An Options Template Set contains only Options Template Record(s). 2.2 IPFIX Flows versus PSAMP Packets As described in [PSAMP-PROTO], the major difference between IPFIX and PSAMP is that the IPFIX protocol exports Flow Records while the PSAMP protocol exports Packet Records. From a pure export point of view, IPFIX will not distinguish a Flow Record composed of several packets aggregated together from a Flow Record composed of a single packet. So the PSAMP export can be seen as special IPFIX Flow Record containing information about a single packet. For this document clarity, the term Flow Record represents a generic term expressing an IPFIX Flow Record or a PSAMP packet record, as foreseen by its definition. However, when appropriate, a clear distinction between Flow Record or packet Record will be made. 3. Problem Statement and High Level Solution Several Flow Records often share a set of common properties. Repeating the information about these common properties for every Flow Record introduces a huge amount of redundancy. This document proposes a method to reduce this redundancy. The section 3.1.1 describes the generic concept. Section 3.1.2 identifies that the proposed solution can be applied multiple times. Section 3.2 utilizes the concept to export per-packet information. Boschi, Mark, Claise Expires April 2007 [Page 10]
Reducing redundancy in IPFIX and PSAMP reports 3.1 Per Flow Data Reduction 3.1.1 Unique Data Reduction Consider a set of properties "A", e.g. common sourceAddressA and sourcePortA, equivalent for each Flow Records exported. Figure 2 shows how this information is repeated with classical IPFIX Flow Records, expressing the waste of bandwidth to export redundant information. +----------------+-------------+---------------------------+ | sourceAddressA | sourcePortA | <flow1 information> | +----------------+-------------+---------------------------+ | sourceAddressA | sourcePortA | <flow2 information> | +----------------+-------------+---------------------------+ | sourceAddressA | sourcePortA | <flow3 information> | +----------------+-------------+---------------------------+ | sourceAddressA | sourcePortA | <flow4 information> | +----------------+-------------+---------------------------+ | ... | ... | ... | +----------------+-------------+---------------------------+ Figure 2: Common and Specific Properties exported in the same record Figure 3 shows how this information is exported when applying the specifications of this document. The Common Properties are separated from the Specific Properties for each Flow Record. The Common Properties would be exported only once in a specific Data Record (defined by an Option Template), while each Flow Record contains a pointer to the Common Properties A, along with its Flow specific information. In order to maintain the relationship between these sets of properties, we introduce indices (in this case: index for properties A) for the Common Properties that are unique for all Common Properties entries within an Observation Domain. The purpose of the indices is to serve as a "key" identifying "rows" of the Common Properties table. The rows are then referenced by the Specific Properties by using the appropriate value for the Common Properties identifier. +------------------------+-----------------+-------------+ | index for properties A | sourceAddressA | sourcePortA | +------------------------+-----------------+-------------+ | ... | ... | ... | +------------------------+-----------------+-------------+ Boschi, Mark, Claise Expires April 2007 [Page 11]
Reducing redundancy in IPFIX and PSAMP reports +------------------------+---------------------------+ | index for properties A | <flow1 information> | +------------------------+---------------------------+ | index for properties A | <flow2 information> | +------------------------+---------------------------+ | index for properties A | <flow3 information> | +------------------------+---------------------------+ | index for properties A | <flow4 information> | +------------------------+---------------------------+ Figure 3: Common and Specific Properties exported in different records This unique export of the Common Properties results in a decrease of the bandwidth requirements for the path between the Exporter and the Collector. 3.1.2 Multiple Data Reduction A Flow Record can refer to one or more Common Properties sets; the use of multiple Common Properties can lead to more efficient exports. Note that in the case of multiple Common Properties, the different sets of Common Properties MUST be disjoint (i.e. MUST not have information elements in common), to avoid potential collisions. Consider a set of properties "A", e.g. common sourceAddressA and sourcePortA and another set of properties "B", e.g. destinationAddressB and destinationPortB. Figure 4 shows how this information is repeated with classical IPFIX export in several Flow Records. +--------+--------+---------+---------+---------------------+ |srcAddrA|srcPortA|destAddrB|destPortB| <flow1 information> | +--------+--------+---------+---------+---------------------+ |srcAddrA|srcPortA|destAddrC|destPortC| <flow2 information> | +--------+--------+---------+---------+---------------------+ |srcAddrD|srcPortD|destAddrB|destPortB| <flow3 information> | +--------+--------+---------+---------+---------------------+ |srcAddrE|srcPortE|destAddrF|destPortF| <flow4 information> | +--------+--------+---------+---------+---------------------+ | ... | ... | ... | ... | ... | +--------+--------+---------+---------+---------------------+ Figure 4: Common and Specific Properties exported in the same record We can separate the Common Properties into the properties A composed of sourceAddressA and sourcePortA, and into the properties B composed of destinationAddressB and destinationPortB. The Flow Record that only contain the property A will only contain the index for property A, the Flow Record that only contain the property B will contain the index for property B, while the Flow Record that Boschi, Mark, Claise Expires April 2007 [Page 12]
Reducing redundancy in IPFIX and PSAMP reports contain both the properties A and B contains both indexes (see Figure 5). +-------------------+-----------------+-------------+ | index for prop. A | sourceAddressA | sourcePortA | +-------------------+-----------------+-------------+ +-------------------+---------------------+------------------+ | index for prop. B | destinationAddressB | destinationPortB | +-------------------+---------------------+------------------+ +-----------------+-----------------+-----------------------+ |index for prop. A|index for prop. B| <flow1 information> | +-----------------+-----------------+-----------------------+ |index for prop. A| <flow2 information> | +-----------------+-----------------+-----------------------+ |index for prop. B| <flow3 information> | +-----------------+-----------------+-----------------------+ | <flow4 information> | +-----------------+-----------------+-----------------------+ Figure 5: Multiple Common (above) and Specific Properties (below) exported in different records The advantage of the multiple Common Properties is that the objective of reducing the bandwidth is met while the number of index is kept to a minimum. Indeed, an alternative solution would have been to have an extra index for the property C, composed of sourceAddressA, sourcePortA, destinationAddressB, destinationPortB. 3.2 Per Packet Data Reduction The PSAMP specifications are used for the export of per-packet information, exporting the specific observed packet in an IPFIX Flow Record. This can be considered as a special Flow Record case, composed of a single packet. If filtering is applied to select a series of packets, using the PSAMP specifications to export per-packet information might be relatively inefficient if the filtered fields (the common attributes) are exported in every single record. For example, if filtering restricts the observation of packets to the packets having the source IP address A, exporting the common properties (the source IP address A) in every record is not efficient. Figure 6, which displays the high level solution for the per-packet reduction, depicts three packets belonging to different flows, as the destination IP address is different. Note that all packets share the same source IP address A. The export of the source IP Boschi, Mark, Claise Expires April 2007 [Page 13]
Reducing redundancy in IPFIX and PSAMP reports address A introduces a huge amount of redundancy, as they are repeated for every packet in every Data Record. +----------+-----------+--------------------------+ | srcAddrA | destAddrB | <packet1 information> | +----------+-----------+--------------------------+ | srcAddrA | destAddrC | <packet2 information> | +----------+-----------+--------------------------+ | srcAddrA | destAddrD | <packet3 information> | +----------+-----------+--------------------------+ Figure 6: Common and Specific Properties represented in one record In Figure 7 we separate Common Properties from Specific Properties, i.e. Common Properties from specific packet information. In order to maintain the relation between Specific (Packet) Properties and Common Properties we introduce indices (index for properties A and index B for properties B), as previously explained. +----------+------------------------+ | srcAddrA | index for properties A | +----------+------------------------+ +------------------------+-----------+-----------------------+ | index for properties A | destAddrB | <packet1 information> | +------------------------+-----------+-----------------------+ | index for properties A | destAddrC | <packet2 information> | +------------------------+-----------+-----------------------+ | index for properties A | destAddrD | <packet3 information> | +------------------------+-----------+-----------------------+ Figure 7: Common and Specific (packet) Properties exported separately An example of the per packet data reduction is the measurement of One-Way Delay (OWD), where the exact same specific packet must be observed at the source and destination of the path to be measured. By subtracting the time of observation of the same packet at the two end points with synchronized clocks, the OWD is computed. As the OWD is measured for a specific application on which a Service Level Agreement (SLA) is bound, this translates into the observation of multiple packets with specific properties, results of filtering. In order to match the identical packet at both Observation Points, a series of packets with a set of properties (For example, all the packets of a specific source and destination IP addresses, of a specific DSCP value, and of a specific destination transport port) must be observed at both ends of the measurements. This implies Boschi, Mark, Claise Expires April 2007 [Page 14]
Reducing redundancy in IPFIX and PSAMP reports that the source and destination must export of a series of Flow Records composed of two types of information: some common information for all packets, and some unique information about packets in order to generate a unique identifier for each packet passing this Observation Point (for example, a hash value on the invariant fields of the packet). So, the source and destination composing the measurements end points can individually and independently apply the redundancy technique described in this draft in order to save some bandwidth for their respective Flow Records export. 4. Specifications for bandwidth saving information export The IPFIX protocol [IPFIX-PROTO] is Template based. Templates define how data should be exported, describing data fields together with their type and meaning. IPFIX specifies two types of Templates: the Template Record and the Options Template Record. The difference between the two is that the Options Template Record includes the notion of scope, defining how to scope the applicability of the Data Record. The scope, which is only available in the Options Template Record, gives the context of the reported Information Elements in the Data Records. The Template Records and Options Template Records are necessary to decode the Data Records. Indeed, by only looking at the Data Records themselves, this is impossible to distinguish a Data Record defined by Template Record from a Data Record defined by an Option Template Record. To export information more efficiently, this specification proposes to group Flow Records by their common properties. We define Common Properties as a collection of attributes shared by a set of different Flow Records. An implementation using the proposed specification MUST follow the IPFIX transport protocol specifications defined in [IPFIX-PROTO]. 4.1 Per Flow Data Reduction 4.1.1 Unique Data Reduction As explained in Figure 8, the information is split into two parts, using two different Data Records. Common Properties MUST be exported via Data Records defined by an Option Template Record and MUST be sent only once with the reliable stream of PR-SCTP association or within the TCP connection. These properties represent values common to several Flow Records (e.g. IP source and destination address). The Common Properties Data Records MUST be sent prior to the corresponding Specific Properties Data Records. The Data Records reporting Specific Properties MUST be associated with the Data Records reporting the Common Properties using a unique identifier for the Common Properties, the commonPropertiesID Information Element [IPFIX-INFO]. The commonPropertiesID MUST be Boschi, Mark, Claise Expires April 2007 [Page 15]
Reducing redundancy in IPFIX and PSAMP reports exported as the scope in the Options Template Record, and also exported in the associated Template Record. +---------------------------+ +---------------------+ | Common Properties | | Specific Properties | Template | Option Template Record | | Template Record | Definition | | | | | scope: commonPropertiesID | | commonPropertiesID | | Common Properties | | Specific Properties | +------------+--------------+ +---------+-----------+ .............|...............................|....................... | | +------------v-------------+ +----------v----------+ | Common Properties | | Specific Properties |+ Exported | Data Record |------> Data Records || Data +--------------------------+ +---------------------+| Records +---------------------+ Figure 8: Template Record and Data Record dependencies The assignment of Flow Records to common attributes could be alternatively provided by the templateID Information Element (instead of the commonPropertiesID Information Element). In this case, the scope in the Common Properties Option Template Record must contain the Template ID used in the Specific Properties Template Record, as displayed in Figure 9. The Common Properties are valid for all data records of the specified Template. In this case the use of commonPropertiesID is not required. +---------------------------+ +---------------------+ | Common Properties | | Specific Properties | Template | Option Template Record | | Template Record | Definition | | | | | scope: Template ID | | Specific Properties | | Common Sroperties | | | +------------+--------------+ +---------+-----------+ .............|...............................|....................... | | +------------v-------------+ +----------v----------+ | Common Properties | | Specific Properties |+ Exported | Data Record |------> Data Records || Data +--------------------------+ +---------------------+| Records +---------------------+ Figure 9: Template Records and Data Records linked with TemplateID Boschi, Mark, Claise Expires April 2007 [Page 16]
Reducing redundancy in IPFIX and PSAMP reports 4.1.2 Multiple Data Reduction If a set of Flow Records share multiple sets of Common Properties, multiple commonPropertiesID instances MAY be used to increase export efficiency even further, as displayed in the Figure 10. +----------------------------+ +---------------------+ | Common Properties | | Specific Properties | Template | Option Template Record | | Template Record | Definition | | | | | Scope: commonPropertiesID1 | | commonPropertiesID1 | | Scope: commonPropertiesID2 | | commonPropertiesID2 | | Common Properties | | Specific Properties | +------------+---------------+ +---------+-----------+ .............|...............................|....................... | | +------------v-------------+ +----------v----------+ | Common Properties | | Specific Properties |+ Exported | Data Record |------> Data Records || Data +--------------------------+ +---------------------+| Records +---------------------+ Figure 10: Multiple data reduction 4.2 Per-Packet Data Reduction From the IPFIX protocol, there are no differences between the Flow Record or per packet record data reduction, except maybe the terminology where the Specific Properties could be called packet specific properties in the following Figure 11. +---------------------------+ +---------------------+ | Common Properties | | Specific Properties | Template | Option Template Record | | Template Record | Definition | | | | | scope: commonPropertiesID | | commonPropertiesID | | Common Properties | | Specific Properties | +------------+--------------+ +---------+-----------+ .............|...............................|....................... | | +------------v-------------+ +----------v----------+ | Common Properties | | Specific Properties |+ Exported | Data Record |------> Data Records || Data +--------------------------+ +---------------------+| Records +---------------------+ Figure 11: Per-packet data reduction Boschi, Mark, Claise Expires April 2007 [Page 17]
Reducing redundancy in IPFIX and PSAMP reports 5. Transport Protocol Choice This document follows the IPFIX transport protocol specifications defined in [IPFIX-PROTO]. However, depending on the transport protocol choice, this document imposes some more constraints. If PR-SCTP is selected as the IPFIX protocol, the SCTP sub-section specifications MUST be respected. If UDP is selected as the IPFIX protocol, the UDP sub-section specifications MUST be respected. If TCP is selected as the IPFIX protocol, the TCP sub-section specifications MUST be respected. 5.1 PR-SCTP The active Common Properties MUST be sent after the SCTP association establishment before the corresponding Specific Properties Data Records. In case of SCTP association re-establishment, all active Common Properties MUST be re-sent before the corresponding Specific Properties Data Records. The Common Properties Flow Records MUST be sent on a reliable SCTP stream. 5.2 UDP Common Properties Data Records MUST be re-sent at regular intervals, whose frequency MUST be configurable. The default value for the frequency of Common Properties transmission is 10 minutes. If a commonPropertiesID is not used anymore the Exporting Process stops re-sending the related Common Properties Data Record. The old commonPropertiesID MUST NOT be used until its lifetime (see 7.1) has expired. 5.3 TCP Common Properties MUST be sent after the TCP connection establishment before the corresponding Specific Properties Data Records. In case of TCP connection re-establishment, all active Common Properties MUST be re-sent before the corresponding Specific Properties Data Records. 6. commonPropertiesID Management The commonPropertiesId is an identifier of a set of common properties that is locally unique per Observation Domain and Transport Session. The Exporting Process MUST manage the commonPropertiesIDs allocations for its Observation Domains and Transport Session. Different Observation Domains from the same Boschi, Mark, Claise Expires April 2007 [Page 18]
Reducing redundancy in IPFIX and PSAMP reports Exporter MAY use the same commonPropertiesID value to refer to different sets of Common Properties. The commonPropertiesID values MAY be assigned sequentially, but its NOT REQUIRED. Particular commonPropertiesID ranges or values MAY have explicit meanings for the IPFIX Device. For example, commonPropertiesID values may be assigned based on the result of a hash function, etc... Using a 64-bit commonPropertiesID Information Element allows the export of 2**64 -1 active sets of Common Properties, per Observation Domain and per Transport Session: as consequence, per Exporting Process. commonPropertiesIDs that are not used anymore SHOULD be withdrawn. The Common Properties ID withdrawal message is an Option Data Record consisting of only one scope field namely the commonPropertiesID (with a type of 137 [IPFIX-INFO]) and no non-scope fields. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 14 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID N | Field Count = 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = 137 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 12: commonPropertiesID Withdrawal Message If UDP is selected as the transport protocol, the commonPropertiesID Template Withdraw Messages MUST not be used, as this method is inefficient due to the unreliable nature of UDP. 7. The Collecting Process Side This section describes the Collecting Process when using SCTP and PR- SCTP as the transport protocol. Any necessary changes to the Collecting Process specifically related to TCP or UDP transport protocols are specified in the subsections. The Collecting Process MUST store the commonPropertiesId information for the duration of the association so that it can interpret the corresponding Data Records that are received in subsequent Data Sets. The Collecting Process can either store the Flow Records as they arrive, without reconstructing the initial Flow Record, or Boschi, Mark, Claise Expires April 2007 [Page 19]
Reducing redundancy in IPFIX and PSAMP reports reconstruct the initial Flow Record. In the former case, there might be less storage capacity required at the Collector side. In the latter case, the collector job is more complex and time- consuming due to the higher resource demand for record processing in real time. If the Collecting Process has received the Specific Properties Data Record before the associated Common Properties Data Record, the Collecting Process MAY store the Specific Properties Data Record and await the retransmission or out-of-order arrival of the Common Properties Data Record. Like TemplateIDs the commonPropertiesIDs are generated dynamically by the Exporting Process. Hence a restart of the Exporting Process may imply a renumbering of commonProperiesIDs. Common Properties IDs are unique per SCTP association and per Observation Domain. If the Collecting Process receives a common Properties ID which has already been received but which has not previously been withdrawn (i.e. a commonPropertiesID from the same Exporter Observation Domain with the same Template ID received on the SCTP association), then the Collecting Process MUST shutdown the association. When an SCTP association is closed, the Collecting Process MUST discard all Common Properties IDs received over that association and stop decoding IPFIX Messages that use those Common Properties IDs. If a Collecting Process receives a Common Properties Withdrawal message, the Collecting Process MUST delete the corresponding Common Properties associated with the specific SCTP association and specific Observation Domain, and stop decoding IPFIX Messages that use the withdrawn Templates. If the Collecting Process receives a Common Properties Withdrawal message for a Common Properties that it has not received before on this SCTP assocation, it MUST reset the SCTP association, discard the IPFIX Message, and SHOULD log the error as it does for malformed IPFIX Messages. 7.1 UDP The Collecting Process MUST associate a lifetime with each Common Property received via UDP. Common Properties not refreshed by the Exporting Process within the lifetime are expired at the Collecting Process. If the Common Properties are not refreshed before that lifetime has expired, the Collecting Process MUST discard the corresponding Boschi, Mark, Claise Expires April 2007 [Page 20]
Reducing redundancy in IPFIX and PSAMP reports definition of the commonPropertiesID and any current and future associated Data Records. In which case, an alarm MUST be logged. The Collecting Process MUST NOT decode any further Data Records which are associated with the expired Common Properties. If a Common Property is refreshed with a Template Record that differs from the previous received Template Record, the Collecting Process SHOULD log a warning and replace the previous received Common Property with the new one. The Template lifetime at the Collecting Process MUST be at least 3 times higher than the Template refresh timeout configured on the Exporting Process. The Collecting Process SHOULD accept Data Records without the associated Common Property(ies) required to decode the Data Record. If the Common Property(ies) have not been received at the time Data Records are received, the Collecting Process SHOULD store the Data Records for a short period of time and decode them after the Template Records (or other definitions) are received. The short period of time MUST be lower than the lifetime of definitions associated with identifiers considered unique within the UDP session. 7.2 TCP When the TCP connection is reset, either gracefully or abnormally, the Collecting Processes MUST delete all commonPropertiesID values corresponding to that connection. If a Collection Process receives a commonPropertiesID Withdraw message, the Collection Process MUST expire the related Common Properties data. 8. Export and Evaluation Considerations The main advantage of the method specified in this document is the reduction in the amount of measurement data that has to be transferred from the Exporter to the Collector. In addition there might be less storage capacity required at the Collector side if the Collector decides to store the Flow Records as they arrive, without reconstructing the initial Flow Record. On the other hand, these methods require additional resources on both the Exporter and the Collector. The Exporter has to manage Common Properties information and to assign commonPropertiesId values to Flow Records. The Collector has to process records described by two templates instead of just one. Additional effort is also required when post processing the measurement data, in order to correlate Flow Records with Common Properties information. Boschi, Mark, Claise Expires April 2007 [Page 21]
Reducing redundancy in IPFIX and PSAMP reports 8.1 Transport Protocol Choice The proposed method is most effective using a reliable transport protocol for the transfer of the Common Properties. Therefore the use of SCTP or TCP is recommended. However, if the path from the Exporting Process to the Collecting Process is not fully reliable, the SCTP or TCP retransmission might reduce the benefits of this specification. If the path from the Exporting Process to the Collecting Process is full reliable, the use of UDP is less effective because the common properties have to be re-sent regularly. 8.2 Reduced Size Encoding The transfer of the commonPropertiesIDs originates some overhead. Note that IPFIX allows reduced-size encoding of Information Elements. In cases where the range of the commonPropertiesID can be restricted, reduced-size encoding can be applied also to the commonPropertiesID, and would result in a further bandwidth efficiency gain. 8.3 commonPropertiesID vs. TemplateID scope The assignment of Flow Records to common attributes could be done via the commonPropertiesID and alternatively via the templateID Information Element. In the second case the commonPropertiesID is not required: this reduces the overhead but the Exporting Process must use one templateID per set of Common Properties. In the general case, this method is not scalable, but it can be suitable for certain applications. 8.4 Efficiency Gain The example in section 11.2 below uses IPFIX to export measurement data for each received packet. In that case, for a flow of 1000 packets the amount of data can be decreased more than 33 percent. While the goal of this specification is to reduce the bandwidth, the efficiency might be limited. Indeed, the efficiency gain is based on the numerous redundant information in flows. While the Exporting Process can evaluate the direct gain for the Flow Records to be exported, it cant predict whether future Flow Records would contain the information specified by active commonPropertiesID values. This implies that the efficiency factor of this specification is higher for specific applications where filtering is involved, such as one- way delay or trajectory sampling. 9. IANA Considerations This document has no actions for IANA. Boschi, Mark, Claise Expires April 2007 [Page 22]
Reducing redundancy in IPFIX and PSAMP reports 10. Security Considerations For the proposed use of the IPFIX protocol for bandwidth-saving export the security considerations as for the IPFIX protocol apply. 11. Appendix A: Examples 11.1 Per Flow Data Reduction 11.1.1 Unique Data Reduction In this section we show how flow information can be exported efficiently using the method described in this draft. Let's suppose we have to periodically export data about two IPv6 Flows. In this example we report the following information: Flow| dstIPv6Address | dst- |nPkts|nBytes | | Port | | ---------------------------------------------------------------- A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 30 | 6000 | | | | A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 50 | 9500 | | | | B |5F05:2000:80AD:5800:0058:00AA:00B7:AF2B| 1932 | 60 | 8000 | | | | A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 40 | 6500 | | | | A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 60 | 9500 | | | | B |5F05:2000:80AD:5800:0058:00AA:00B7:AF2B| 1932 | 54 | 7600 The Common Properties in this case are the destination IPv6 address and the destination port. We first define an Option Template that contains the following Information Elements: - Scope: the commonPropertiesID, with a type of 137 [IPFIX-INFO] and a length of 8 octets. - The destination IPv6 address, destinationIPv6Address [IPFIX- INFO], with a type of 28 and a length of 16 octets - The destination port, destinationTransportPort [IPFIX-INFO] with a type of 11, and a length of 2 octets Figure 13 shows the Option template defining the Common Properties with commonPropertiesID as scope: Boschi, Mark, Claise Expires April 2007 [Page 23]
Reducing redundancy in IPFIX and PSAMP reports 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = 137 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 8 |0| destinationIPv6Address = 28| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 16 |0|destinationTransportPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 13: Common Properties Option Template The Specific Properties Template consists of the information not contained in the Option Templates, i.e. flow specific information, in this case the number of packets and the number of bytes to be reported. Additionally, this Template contains the commonPropertiesID. In Data Records, the value of this field will contain one of the unique indices of the Option Records exported before. It contains the following Information Elements (see also Figure 14): - commonPropertiesID with a length of 8 octets - The number of packets of the Flow: inPacketDeltaCount in [IPFIX-INFO], with a length of 4 octets - The number of octets of the Flow: inOctetDeltaCount in [IPFIX- INFO], with a length of 4 octets 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 20 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 258 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = 137 | Field Length = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inPacketDeltaCount = 2 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inOctetDeltaCount = 1 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 14: Specific Properties Template Boschi, Mark, Claise Expires April 2007 [Page 24]
Reducing redundancy in IPFIX and PSAMP reports Considering the data shown at the beginning of this example, the following two Data Records will be exported: Common- | dstAddress | dst- PropertiesID | | Port -------------+-----------------------------------------+------- 101 | 5F05:2000:80AD:5800:0058:0800:2023:1D71 | 80 | | 102 | 5F05:2000:80AD:5800:0058:00AA:00B7:AF2B | 1932 The Data Records reporting the Common Properties will look like: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 257 | Length = 60 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 101 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5F05:2000: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 80AD:5800: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 0058:0800: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 2023:1D71 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 80 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 102 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5F05:2000: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 80AD:5800: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 0058:00AA: ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... 00B7:AF2B | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1932 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 15: Data Records reporting Common Properties Boschi, Mark, Claise Expires April 2007 [Page 25]
Reducing redundancy in IPFIX and PSAMP reports The Data Records will in turn be: commonPropertiesID | inPacketDeltaCount | inOctetDeltaCount --------------------------------------------------------------- 101 | 30 | 6000 101 | 50 | 9500 102 | 60 | 8000 101 | 40 | 6500 101 | 60 | 9500 102 | 54 | 7600 Figure 16 shows the first Data Record listed in the table: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 258 | Length = 16 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 101 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 30 | 6000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 16: Data Record reporting Specific Properties 11.1.2 Multiple Data Reduction In this example we export the following flow information: Flow | srcAddr | srcPort | dstAddr | dstPort | nPackets | nBytes ---------------------------------------------------------------- A |10.0.0.1 | 1932 |10.0.1.2 | 80 | 30 | 6000 B |10.0.0.3 | 2032 |10.0.1.2 | 80 | 50 | 9500 Figure 17 shows the Option Templates, containing the Common Properties together with the commonPropertiesID as Scope. In the first Common Properties Option Template we export the following Information Elements: - Scope 1: the Common Properties ID, commonPropertiesId with a type of 137 [IPFIX-INFO]. Note that the commonProperties IE has a length of 8 octets, but if smaller size is sufficient to Boschi, Mark, Claise Expires April 2007 [Page 26]
Reducing redundancy in IPFIX and PSAMP reports carry any value the Exporter may need to deliver, reduced size encoding can be used. In this example we use reduced sizing, of 4 octets. - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type of 8 and a length of 4 octets - the source Port, sourceTransportPort [IPFIX-INFO], with a type of 7 and a length of 2 octets The second Option Template contains the following Information Elements: - Scope 2: the commonPropertiesID, with a type of 137 [IPFIX- INFO] and a length of 4 octets (reduced sizing). - the destination IPv4 Address, destinationIPv4Address [IPFIX- INFO], with a type of 12 and a length of 4 octets - the destination port, destinationTransportPort [IPFIX-INFO] with a type of 11, and a length of 2 octets The commonPropertiesId Information Element is used in both cases as the Scope Field. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = 137 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| transportSourcePort = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Boschi, Mark, Claise Expires April 2007 [Page 27]
Reducing redundancy in IPFIX and PSAMP reports 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = 137 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| destinationIPv4Address = 12| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0|transportDestinationPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 17: Example Common Properties Template Considering the values given at the beginning of this section we will export the Common Properties using the following Data Records: commonPropertiesID | sourceAddress | sourcePort --------------------+-----------------+------------- 101 | 10.0.0.1 | 1932 102 | 10.0.0.3 | 2032 and commonPropertiesID | dstAddress | dstPort --------------------+---------------+----------- 103 | 10.0.1.2 | 80 The Specific Properties Template consists of the information not contained in the Option Templates, i.e. flow specific information. Additionally, this Template contains the two commonPropertiesID. In Data Records, the values of each of these fields will contain one of the unique indices specified in the Option Records exported previously. Figure 18 displays the Template including the commonPropertiesID plus the Specific Properties. In this example we export the following Information Elements: - commonPropertiesID for the source fields with a length of 4 octets (reduced size encoding) Boschi, Mark, Claise Expires April 2007 [Page 28]
Reducing redundancy in IPFIX and PSAMP reports - commonPropertiesID for the destination fields with a length of 4 octets (reduced size encoding) - the number of packets of the Flow: inPacketDeltaCount in [IPFIX-INFO], with a length of 4 octets - the number of octets of the Flow: inOctetDeltaCount in [IPFIX- INFO], with a length of 4 octets 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 259 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = 137 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = 137 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inPacketDeltaCount = 2 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inOctetDeltaCount = 1 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 18: Example Specific Properties Template Considering the values given at the beginning of this section, the Data Records of the two flows will look like: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 256 | Length = 28 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 101 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 103 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 30 | 6000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 102 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 103 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 50 | 9500 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 19: Specific Properties Boschi, Mark, Claise Expires April 2007 [Page 29]
Reducing redundancy in IPFIX and PSAMP reports 11.2 Per-Packet Information Export This section demonstrates per-packet information export to support passive One-Way Delay (OWD) measurement. The Templates required for exporting measurement data of this kind are illustrated in the figures below. Figure 20 shows the Option Template containing the information concerning Flows using the commonPropertiesID as scope. In the Common Properties Template we export the following Information Elements: - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type of 8 and a length of 4 octets - the destination IPv4 Address, destinationIPv4Address [IPFIX- INFO], with a type of 12 and a length of 4 octets - the Class of Service field, ClassOfServiceIPv4 [IPFIX-INFO], with a type of 5 and a length of 1 octet - the Protocol Identifier, protocolIdentifier [IPFIX-INFO], with a type of 4 and a length of 1 octet - source port, sourceTransportPort [IPFIX-INFO], with a type of 7 and and a length of 2 octets - destination port, destinationTransportPort [IPFIX-INFO], with a type of 11 and a length of 2 octets The commonPropertiesID Information Element, is used as the Scope Field. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 40 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| destinationIPv4Address = 12 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| classOfServiceIPv4 = 5 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 1 |0| protocolIdentifier = 4 | Boschi, Mark, Claise Expires April 2007 [Page 30]
Reducing redundancy in IPFIX and PSAMP reports +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 1 |0| transportSourcePort = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 |0|transportDestinationPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 20: Example Flow Properties Template For passive One-Way-Delay measurement, the Packet Properties Template, or Specific Properties Template, consists of at least Timestamp and Packet ID. Additionally, this template contains a commonPropertiesId field to associate the packet with a Flow. Figure 21 displays the template with the packet properties. In this example we export the following Information Elements: - commonPropertiesID. In this case reduced size encoding is used, and the Information Element is declared with a length of 4 octets instead of 8. - packetTimestamp, packetID, and packetLength. Since packetTimestamp, packetID, and packetLength are not (yet) IETF- defined information elements, we export them as enterprise- specific IEs. The three IEs have respectively a type of 220, 221, and 222 and a length of 8, 4, and 4 octets. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 36 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| commonPropertiesID = 137 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetTimestamp = 220 | Field Length = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetID = 221 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetLength = 222 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 21: Example Packet Properties Template Boschi, Mark, Claise Expires April 2007 [Page 31]
Reducing redundancy in IPFIX and PSAMP reports At the collection point, packet records from the two measurement points are gathered and correlated by means of the packet ID. The resulting delay data records are exported in a similar manner as the packet data. One-way delay data is associated with flow information by the commonPropertiesId field. The OWD properties contain the Packet Pair ID (which is the packet ID of the two contributing packet records), the timestamp of the packet passing the reference monitor point in order to reconstruct a time series, the calculated delay value, and the commonPropertiesID. In this example using IPFIX to export the measurement data for each received packet 30 bytes have to be transferred (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2, destionationTransportPort=2, packetTimestamp=8, packetID=4, packetLength=4). Without considering the IPFIX protocol overhead a flow of 1000 packets produces 30000 bytes of measurement data. Using the proposed optimization each packet produces an export of only 20 bytes (packetTimestamp=8, packetID=4, packetLength=4, commonPropertiesID=4). The export of the flow information produces 18 bytes (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2, destionationTransportPort=2, commonPropertiesID =4). For a flow of 1000 packets this sums up to 20018 bytes. This is a decrease of more than 33 percent. 11.3 commonPropertiesID Template Withdrawal Message This section shows an example commonPropertiesIDs Withdrawal message. Figure 22 depicts the Option Template Record with the commonPropertiesID as unique scope field, and no non-scope fields. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 14 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID 259 | Field Count = 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| commonPropertiesID 137 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 22: example commonPropertiesID withdrawal template Figure 23 shows the Option Data Record withdrawing commonPropertiesID N: Boschi, Mark, Claise Expires April 2007 [Page 32]
Reducing redundancy in IPFIX and PSAMP reports 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 259 | Length = 12 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | N | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 23: commonPropertiesID withdrawal record, withdrawing commonPropertiesID N 12. References 12.1 Normative References [RFC2119] S. Bradner, Key words for use in RFCs to Indicate Requirement Levels, BCP 14, RFC 2119, March 1997 [IPFIX-PROTO] B. Claise et Al, IPFIX Protocol Specification, <draft-ietf-ipfix-protocol-23.txt>, Internet-Draft work in progress, October 2006 [IPFIX-INFO] J. Quittek, S.Bryant, B.Claise, J. Meyer, Information Model for IP Flow Information Export, <draft-ietf-ipfix-info-13.txt>, Internet-draft work in progress, September 2006 [PSAMP-PROTO] B. Claise, J. Quittek, A. Johnson, PSAMP Protocol Specification, <draft-ietf-psamp-protocol-06.txt>, Internet-Draft work in progress, June 2006 12.2 Informative References [IPFIX-ARCH] G. Sadasivan, N. Brownlee, B. Claise, J. Quittek, Architecture Model for IP Flow Information Export, <draft-ietf-ipfix-arch-12.txt>, Internet-Draft work in progress, September 2006 [IPFIX-AS] T. Zseby, E. Boschi, N. Brownlee, B. Claise, IPFIX Applicability, <draft-ietf-ipfix-as-10.txt>, Internet- Draft work in progress, August 2006 [PSAMP-TECH] T. Zseby, M. Molina, N. Duffield, S. Niccolini, F. Raspall, Sampling and Filtering Techniques for IP Packet Selection, <draft-ietf-psamp-sample-tech- 07.txt>, Internet-Draft work in progress, January 2006 Boschi, Mark, Claise Expires April 2007 [Page 33]
Reducing redundancy in IPFIX and PSAMP reports [PSAMP-INFO] T. Dietz, F. Dressler, G. Carle, B. Claise, P. Aitken, Information Model for Packet Sampling Exports, <draft- ietf-psamp-info-04.txt>, Internet-Draft work in progress, March 2006 [PSAMP-MIB] T. Dietz, B. Claise, Definitions of Managed Objects for Packet Sampling, <draft-ietf-psamp-mib-06.txt>, Internet-Draft work in progress, June 2006 [PSAMP-FMWK] N. Duffield, D. Chiou, B. Claise, A. Greenberg, M. Grossglauser, P. Marimuthu, J. Rexford, G. Sadasivan, A Framework for Passive Packet Measurement, <draft-ietf- psamp-framework-10.txt>, Internet-Draft work in progress, January 2005 [RFC3917] J. Quittek, T. Zseby, B. Claise, S. Zander, Requirements for IP Flow Information Export, RFC 3917, October 2004 13. Acknowledgements The authors would like to thank Guido Pohl for initiating this work and for his contribution to early versions of this document. 14. Author's Addresses Elisa Boschi Hitachi Europe SAS Immeuble Le Theleme 1503 Route des Dolines 06560 Valbonne, France Phone: +33 4 89874180 Email: elisa.boschi@hitachi-eu.com Lutz Mark Fraunhofer Institute for Open Communication Systems Kaiserin-Augusta-Allee 31 10589 Berlin Germany Phone: +49-30-34 63 7306 Fax: +49-30-34 53 8306 Email: mark@fokus.fraunhofer.de Benoit Claise Cisco Systems De Kleetlaan 6a b1 Diegem 1813 Belgium Boschi, Mark, Claise Expires April 2007 [Page 34]
Reducing redundancy in IPFIX and PSAMP reports Phone: +32 2 704 5622 Email: bclaise@cisco.com 15. Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. 16. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. 17. Disclaimer This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Boschi, Mark, Claise Expires April 2007 [Page 35]
