Reducing redundancy in IPFIX and PSAMP reports
Network Working Group E. Boschi
Internet-Draft Hitachi Europe
Expires: August 30, 2006 L. Mark
Fraunhofer FOKUS
February 2006
Reducing redundancy in IPFIX and PSAMP reports
draft-boschi-ipfix-reducing-redundancy-00.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that
any applicable patent or other IPR claims of which he or she is
aware have been or will be disclosed, and any of which he or she
becomes aware will be disclosed, in accordance with Section 6 of
BCP 79.
Internet-Drafts are working documents of the Internet
Engineering Task Force (IETF), its areas, and its working
groups. Note that other groups may also distribute working
documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other than
as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 30, 2006.
Copyright Notice
Copyright (C) The Internet Society (2006).
Boschi, Mark Expires August 2006 [Page 1]
Reducing redundancy in IPFIX and PSAMP reports
Abstract
This document describes a bandwidth saving methodology for
exporting flow or packet information using the IP Flow
Information Export (IPFIX) protocol. Being the PSAMP protocol
based on IPFIX, these considerations are valid for PSAMP exports
as well.
The main idea is to separate the export of information common to
several flows (or packets) and the specific flow (or packet)
information, using two different records. The association
between the records is kept using unique Identifiers.
Table of Contents
1. Introduction············································2
2. Terminology·············································3
3. OPEN POINTS·············································3
4. Techniques for bandwidth saving information export······3
4.1 Single and multiple associations······················4
4.2 Export of Per-Packet Information······················5
4.3 Using Scopes··········································6
4.4 CommonPropertiesID Management·························7
5. Examples················································7
5.1 Multiple CommonPropertiesIDs··························7
5.2 Per-Packet Information Export························10
6. Export and evaluation considerations···················12
7. IANA Consideration·····································13
8. Security Considerations································13
9. Normative References···································13
10. Author's Addresses·····································13
11. Intellectual Property Statement························13
12. Copyright Statement····································14
13. Disclaimer·············································14
1. Introduction
In [IPFIX-PROTO] the IPFIX working group has defined a protocol
to export IP flow information. The main purpose of the protocol
is to exchange information about IP traffic flows in combination
with related measurement data. The export of that information is
done via flow records. In this scope a flow is defined by a set
of key attributes (e.g. source and destination IP address,
source and destination port, etc.).
When exporting flow records sharing a set of common attributes,
these attributes have to be repeated for each data record even
though the values remain the same. The elimination of redundant
data from the export stream can result in a significant
reduction of the transferred data.
This draft proposes to use option data records to export common
properties, whose value remains constant, and to send these
properties only once. Measured properties vary over time and are
sent multiple times in regular data records. The linkage of the
flow records with the common properties exported via option data
records is done via a CommonPropertiesID.
The proposed method can be applied also to export per packet
information (e.g. for Passive One-Way Delay measurements or
PSAMP exports). In this case a flow is a collection of packets
that share a set of common attributes.
Boschi, Mark Expires August 2006 [Page 2]
Reducing redundancy in IPFIX and PSAMP reports
The proposed method does not need any changes to the IPFIX
protocol.
2. Terminology
Collecting Process
The collecting process receives records of flow or packet
information. The data is stored for later processing (by
the calculation process)
Exporting Process
The exporting processes send flow and packet records to the
collecting processes. The records are generated by the
measurement process.
Filtering
Filtering selects a subset of packets by applying
deterministic functions on parts of the packet content like
header fields or parts of the payload. A filtering process
needs to process the packet (look at packet header and/or
payload) in order to make the selection decision.
Measurement Device
A measurement device has access to at least one observation
point. It is hosting at least one measurement process and
one export process.
Metering/Measurement Process
The measurement process generates records of packet and
flow information. Packets passing the observation point are
captured, time stamped, filtered and classified. The
measurement process calculates the packet Ids.
Passive One-Way-Delay Measurement
Abbreviated: POWD Measurement
CommonPropertiesID
An identifier of a set of common properties that is unique
within an Observation Domain. This ID can be used to link
to information reported in separate records.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT","SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as
described in RFC 2119.
3. OPEN POINTS
[1] The current definition (and name) of FlowID needs to be
generalised as well. It is currently more an identifier for a
flow than a generic index. This draft proposes a new IE,
CommonPropertiesID; its definition should be included in [IPFIX-
INFO]. Throughout this draft flowID has been replaced with
CommonPropertiesID.
4. Techniques for bandwidth saving information export
The IPFIX protocol is template based. Templates define the
structure of data to be exported, describing data fields
together with their type and meaning. IPFIX specifies two types
of records to export data: data records and option data records.
Boschi, Mark Expires August 2006 [Page 3]
Reducing redundancy in IPFIX and PSAMP reports
These records are defined via template records and option
template records.
To export information more efficiently we group different flow
records depending on their common properties. We define Common
Properties as a collection of attributes shared by a set of
different flow records. Specific Properties are the remaining
flow or packet attributes, not shared with the same set of
flows. Since a packet can be considered as a special kind of
flow, these considerations remain valid for packets too.
We separate the information export using two different records.
Common Properties are exported via option data records and are
sent only once. These properties represent values common to
several flows. The associated Specific Properties are
subsequently sent using data records. The association is kept
using a particular index that uniquely identifies the Common
Properties, the CommonPropertiesID.
Figure 1 shows the relation between template and data sets for
common and specific properties. The Common Properties option
template defines the common attributes of a flow or a group of
flows (e.g. IP source and destination address) and the
CommonPropertiesID. The CommonPropertiesID is a unique
identifier for a set of attributes; this field allows the
linkage between those attributes and flow data records.
Subsequent option data records of this template export the
Common Properties values. The assignment of flow records to
common attributes could be alternatively provided by the
TemplateID, as explained in Section 4.3. In this case the
CommonPropertiesID can be omitted.
The format for the information related to single flows or
packets is defined in the Specific Properties template. This
information is flow/packet specific and normally not shared
between many flows/packets.
+---------------------+ +---------------------+
| Option Template Set | | Template Set |
| | | | Description of
| Common Properties | | Specific Properties | exported data
+----------+----------+ +----------+----------+
...........|............................|.........................
| |
+----------v----------+ +----------v----------+ Exported data
| Option Data Set | | Data Set | with references
| <------+ | by means of flow
| Common Properties | | Specific Properties | or template
+---------------------+ +---------------------+ identifiers
Figure 1: Template Set and Data Set dependencies
The Common Properties option data records SHOULD be sent prior
to the corresponding Specific Properties data records.
4.1 Single and multiple associations
Several flow records often share a set of common properties.
Repeating the information about these common properties for
every record introduces a huge amount of redundancy. Let's
consider a set of properties "A", e.g. common sourceAddressA and
sourcePortA and another set of properties "B", e.g.
destinationAddressB and destinationPortB. Figure 2 shows how
Boschi, Mark Expires August 2006 [Page 4]
Reducing redundancy in IPFIX and PSAMP reports
this information is repeated with classical IPFIX export in
several records.
In Figure 3 we separate Common Properties from Specific flow (or
packet) Properties. In order to maintain the relation between
these sets of properties we introduce indices (idxA and idxB)
for the Common Properties that are unique for all Common
Property entries. The purpose of the indices is to serve as
"primary key" that identifies rows of the Common Properties.
More details about these indices will be given in section 4.3.
The rows are then referenced by the Specific Properties by using
the appropriate value for the Common Properties identifier.
A flow record can refer to one or more Common Properties sets,
achieving thus, in the latter case, a better export efficiency.
+--------------------+---------------------------------+
| properties A | <flow info> |
+--------------------+---------------------------------+
| properties A | <flow info> |
+--------------------+---------------------------------+
| properties B | <flow info> |
+--------------------+-------------------+-------------+
| properties A | properties B | <flow info> |
+--------------------+-------------------+-------------+
Figure 2: Common and Specific Properties exported in the same
record
Specific Properties
+------+----------------------+
Common Properties > idxA | <flow info> |
+--------------+-----+ +------+----------------------+
| properties A |idxA < > idxA | <flow info> |
+--------------+-----+ +------+----------------------+
| properties B |idxB <--------> idxB | <flow info> |
+--------------+-----+ +------+------+---------------+
> idxA | idxB | <flow info> |
+------+------+---------------+
Figure 3: Common and Specific Properties exported separately
4.2 Export of Per-Packet Information
A particular case is the export of per packet information (e.g.
for Passive One-Way Delay measurements or PSAMP exports). In
this case a single packet could be considered a special case of
a flow and consequently per-packet information could be exported
using flow records. Doing this though would introduce additional
overhead, since packets belonging to the same flow share common
attributes (e.g. source address, destination address, etc).
Exporting these Common Properties on a per-packet basis, would
be redundant.
Figure 4 depicts three packets belonging to flow A and one
packet belonging to flow B, respectively. It shows export
records containing packet information plus flow information
(source and destination address). Undoubtedly, flow information
(or Common Properties, as defined in this draft) introduces a
Boschi, Mark Expires August 2006 [Page 5]
Reducing redundancy in IPFIX and PSAMP reports
huge amount of redundancy, as it is repeated for every packet in
every record.
In Figure 5 we separate Common Properties from Specific
Properties, i.e. flow from packet information. In order to
maintain the relation between Specific (Packet) Properties and
Common (Flow) Properties we introduce indices (idxA and idxB),
much in the same way as explained in the previous sections. The
linkage of one packet and flow B (flow properties B, idxB) is
explicitly drawn.
One-packet flows
+-------------------+----------------------+
| flow properties A | <packet info> |
+-------------------+----------------------+
| flow properties A | <packet info> |
+-------------------+----------------------+
| flow properties B | <packet info> |
+-------------------+----------------------+
| flow properties A | <packet info> |
+-------------------+----------------------+
Figure 4: Flow and packet information represented in one-packet
flows
Specific (Packet) Properties
+------+--------------------+
Common (Flow) Properties > idxA | <packet info> |
+------------------+-----+ +------+--------------------+
|flow properties A |idxA < > idxA | <packet info> |
+------------------+-----+ +------+--------------------+
|flow properties B |idxB <-------> idxB | <packet info> |
+------------------+-----+ +------+--------------------+
> idxB | <packet info> |
+------+--------------------+
Figure 5: Flow information and packet information
4.3 Using Scopes
Common Properties are sent via IPFIX option records. IPFIX
option records contain one or more scope fields. The Flow
Properties record can contain the CommonPropertiesID and/or the
TemplateID as scope fields. There are three options:
1) Use CommonPropertiesID as scope
The common properties are valid for all data records
containing that CommonPropertiesID. This solution limits the
number of different flows that can be exported at the same
time in the same observation domain to 2**32 (using 32 bits
CommonPropertiesIDs)
2) Use CommonPropertiesID and TemplateID as scope
The common properties are valid only for data records
referring to the template specified by the TemplateID and
containing that CommonPropertiesID. This allows the export up
to 2**32 flows per template. The solution is to be chosen
when the number of flows to be exported is expected to be
very high (and beyond the limit posed by solution 1)
Boschi, Mark Expires August 2006 [Page 6]
Reducing redundancy in IPFIX and PSAMP reports
3) Use TemplateID as scope
The common properties are valid for all data records of the
specified template. In this case CommonPropertiesIDs are not
needed but the exporting process requires a templateID per
flow. In the general case this solution is not scalable but
can be suitable for certain applications (e.g. flow
aggregation).
4.4 CommonPropertiesID Management
The management of CommonPropertiesIDs is very similar to the
management of TemplateIDs described in [IPFIX-PROTO]. The
Exporting Process maintains the CommonPropertiesIDs for the
exporter's Observation Domains. Like templateIDs, a
CommonPropertiesID MUST be unique per Observation Domain (source
identifier in the IPFIX header). Different Observation Domains
from the same exporter may use the same CommonPropertiesID value
to refer to different flows.
There are no constraints regarding the order of the
CommonPropertiesID allocation. When limiting the scope to
special templates, the CommonPropertiesIDs have to be unique per
Observation Domain and template.
Using 32 bit CommonPropertiesIDs allows the export of 2**32
active flows in parallel. CommonPropertiesIDs have a certain
lifetime inside which they cannot be reused. After that time a
CommonPropertiesID can be assigned to another flow.
CommonPropertiesID whose lifetime has expired from longer SHOULD
be preferred. The lifetime MUST be configurable.
The collecting process associates a lifetime with each
CommonPropertiesID. The lifetime MUST be configurable. The
mapping of data records to flow properties uses the most recent
flow definition of the specified CommonPropertiesID. If there is
no flow definition of that CommonPropertiesID or the lifetime of
the flow definition has been expired, no mapping is possible. In
this case the collecting process SHOULD log an error.
When IPFIX uses an unreliable transport protocol to export the
option data records containing the flow properties and the
CommonPropertiesIDs these records MUST be re-sent at regular
intervals, whose frequency MUST be configurable.
When using a connection oriented transport protocol the flow
properties have to be re-sent after a connection re-
establishment in prior to the corresponding Packet Properties
data records.
5. Examples
5.1 Multiple CommonPropertiesIDs
In this section we show how flow information can be exported
efficiently using the method described in this draft. Let's
suppose we have to export the following flow information:
Flow | srcAddr | srcPort | dstAddr | dstPort | nPackets | nBytes
----------------------------------------------------------------
A |10.0.0.1 | 1932 |10.0.1.2 | 80 | 30 | 6000
B | -- | -- |10.0.1.2 | 80 | 50 | 9500
C |10.0.0.1 | 1932 | -- | -- | 60 | 8000
Boschi, Mark Expires August 2006 [Page 7]
Reducing redundancy in IPFIX and PSAMP reports
Figure 6 shows the option templates, containing the Common
Properties together with the CommonPropertiesID, which is the
scope.
In the first Common Properties template we export the following
Information Elements:
- the source IPv4 Address, sourceIPv4Address [IPFIX-INFO],
with a type of 8 and a length of 4 octets
- the source Port, transportSourcePort [IPFIX-INFO], with a
type of 7 and a length of 2 octets
- the Common Properties ID, with a type [TO BE ASSIGNED, in
this Draft referenced to as XX] and a length of 4 octets.
The second option template contains the following Information
Elements:
- the destination IPv4 Address, destinationIPv4Address
[IPFIX-INFO], with a type of 12 and a length of 4 octets
- the destination port, transportDestinationPort [IPFIX-INFO]
with a type of 11, and a length of 2 octets
- the Common Properties ID, with a type [TO BE ASSIGNED, in
this Draft referenced to as XX] and a length of 4 octets.
The flow identifier, which is represented by the
CommonPropertiesId Information Element [NOTE: to be included in
IPFIX-INFO], is used in both cases as the Scope Field.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| CommonPropertiesID = XX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| transportSourcePort = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| CommonPropertiesID = XX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| destinationIPv4Address = 12|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0|transportDestinationPort = 11|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6: Example Common Properties Templates
Boschi, Mark Expires August 2006 [Page 8]
Reducing redundancy in IPFIX and PSAMP reports
Considering the values given at the beginning of this section we
will export the following option data records:
CommonPropertiesID | sourceAddress | sourcePort
----------------------------------------------------
1 | 10.0.0.1 | 1932
and
CommonPropertiesID | dstAddress | dstPort
------------------------------------------------
2 | 10.0.1.2 | 80
The Specific Properties template consists of the information not
contained in the Option Templates, i.e. flow specific
information. Additionally, this template contains the
CommonPropertiesID. In data records, the value of this field
will contain one of the unique indices of the option records
exported before.
Figure 7 displays the template with the Specific Properties for
Flow A (above) and Flow B (below). Note that for flow A two
Information Elements of the same type are exported
(CommonPropertiesID). In this example we export the following
Information Elements:
- CommonPropertiesID with a length of 4 octets
- The number of packets of the Flow: inPacketDeltaCount in
[IPFIX-INFO], with a length of 4 octets
- The number of octets of the Flow: inOctetDeltaCount in
[IPFIX-INFO], with a length of 4 octets
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 259 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| CommonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| CommonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inPacketDeltaCount = 2 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inOctetDeltaCount = 1 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 20 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 258 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| CommonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inPacketDeltaCount = 2 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inOctetDeltaCount = 1 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Boschi, Mark Expires August 2006 [Page 9]
Reducing redundancy in IPFIX and PSAMP reports
Figure 7: Example Specific Properties Template
Considering the values given at the beginning of this section,
the data records will look like in the following table. The
first row represent the Specific Properties of Flow B
(CommonPropertiesID = 2), while the second row those of flow C
(CommonPropertiesID = 1).
CommonPropertiesID | inPacketDeltaCount | inOctetDelataCount
---------------------------------------------------------------
2 | 50 | 9500
| |
1 | 60 | 8000
The Specific Properties for Flow A will report both
CommonPropertiesID values. The data record for Flow A will look
like:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 30 | 6000 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8: Specific Properties for Flow A
5.2 Per-Packet Information Export
[NOTE: this example uses the FlowID IE to associate Common
Properties to Specific Properties. In case of per-packet
information export, this is an alternative solution. FlowID
management is the same as the CommonPropertiesID management
described in Section 4.4]
To demonstrate how to use IPFIX efficiently to export per-packet
information, this section proposes how to use the IPFIX protocol
for exporting flow information (Common Properties) and
per-packet information (Specific Properties, in this case
related to a long-lived flow) for OWD computation.
In order to acquire a One-Way path delay information, two
measurement points with synchronized clocks must exist, one at
each end of the path under examination. Both measurement points
will capture packets, assign them timestamps and generate an
identifier for a packet passing that point. Each measurement
point will export its measurement data to a collecting process
where the data are correlated based on the packet identifiers
and timestamps and then the delay is calculated as a difference
of two timestamps of a packet pair.
The templates that would be needed for exporting measurement
data of this kind are illustrated in the figures below.
Figure 9 shows the option template containing the information
concerning flows using the FlowID as scope.
Boschi, Mark Expires August 2006 [Page 10]
Reducing redundancy in IPFIX and PSAMP reports
In the Flow Properties template we export the following
Information Elements:
- the source IPv4 Address, sourceIPv4Address [IPFIX-INFO],
with a type of 8 and a length of 4 octets
- the destination IPv4 Address, destinationIPv4Address
[IPFIX-INFO], with a type of 12 and a length of 4 octets
- the Class of Service field, ClassOfServiceIPv4 [IPFIX-
INFO], with a type of 5 and a length of 1 octet
- source and destination ports, transportSourcePort and
transportDestinationPort [IPFIX-INFO]with a type of 7 and
11 respectively, and a length of 2 octets each
The flow identifier, which is represented by the FlowId
Information Element [IPFIX-INFO], is used as the Scope Field.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 40 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| FlowID = 148 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| destinationIPv4Address = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| classOfServiceIPv4 = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| protocolIdentifier = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| transportSourcePort = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |0|transportDestinationPort = 11|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9: Example Flow Properties Template
For passive One-Way-Delay measurement, the Packet Properties
template consists of at least Timestamp and Packet ID.
Additionally, this template contains a flow identifier field. In
packet records, the value of this field will contain one of the
unique indices of the flow records exported before.
Figure 10 displays the template with the packet properties. In
this example we export the following Information Elements:
- FlowID [IPFIX-INFO] with a length of 4 octets
- packetTimestamp, packetID, and packetLength. Since
packetID, packetLength and flowID are not (yet) IETF-
defined information elements, we export them as enterprise-
specific IEs. The three IEs have respectively a type of
220, 221, and 222 and a length of 8, 4, and 4 octets.
Boschi, Mark Expires August 2006 [Page 11]
Reducing redundancy in IPFIX and PSAMP reports
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 36 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| FlowID = 148 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| packetTimestamp = 220 | Field Length = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| packetID = 221 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| packetLength = 222 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 10: Example Packet Properties Template
The delay is derived by a calculation step: at the collection
point packet records of two measurement points are gathered and
correlated by means of the packet ID. The resulting delay data
records are exported in a similar manner as the packet data have
been. Especially, the linkage between delay data and flow
information is represented with the discussed flow identifier
fields. The OWD properties contain the Packet Pair ID (which is
the packet ID matching that of the two contributing packet
records), a timestamp (which is the timestamp of the packet
passing the reference monitor point) in order to reconstruct a
time series, the calculated delay value, and finally a flow
identifier.
6. Export and evaluation considerations
The main advantage of the methods proposed in this draft is the
reduced amount of measurement data that has to be transferred
from the exporter to the collector. In addition there is less
storage capacity needed at the collector side.
On the other hand there is some extra processing power needed on
the exporter side to manage Common Properties information and to
assign them to flow records. The collector has to process
records of two templates instead of just one but has to read and
write less data. Additional effort is needed when post
processing the measurement data, because now the correlation of
flow records with Common Properties information is needed.
In the example in section 5.2 using IPFIX to export the
measurement data for each received packet 30 bytes have to be
transferred (sourceAddressV4=4, destinationAddressV4=4,
classOfServiceV4=1, protocolIdentifier=1, transportSourcePort=2,
transportDestionationPort=2, packetTimestamp=8, packetID=4,
packetLength=4). Disregarding the IPFIX protocol overhead a flow
of 1000 packets produces 28000 bytes of measurement data. Using
the proposed optimization each packet produces an export of only
20 bytes (packetTimestamp=8, packetID=4, packetLength=4,
flowID=4). The export of the flow information produces 16 bytes
(sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1,
protocolIdentifier=1, transportSourcePort=2,
transportDestionationPort=2, flowID =4). For a flow of 1000
Boschi, Mark Expires August 2006 [Page 12]
Reducing redundancy in IPFIX and PSAMP reports
packet this sums up to 16016 bytes. This is a decrease of more
than 40 percent.
Note that IPFIX allows the reduction of the size of exported
Information Elements. This can be applied also to the
CommonPropertiesID and would result in a further reduction of
bytes.
7. IANA Consideration
This document does not imply any IANA action.
8. Security Considerations
For the proposed use of the IPFIX protocol for bandwidth-saving
export the security considerations as for the IPFIX protocol
apply.
9. Normative References
[IPFIX-PROTO] Benoit Claise et Al.: IPFIX Protocol
Specification, IETF draft work in progress
<draft-ietf-ipfix-protocol-19.txt>, September 2005
[IPFIX-INFO] J. Quittek, S.Bryant, B.Claise, J. Meyer:
Information Model for IP Flow Information Export
Internet-draft work in progress <draft-ietf-ipfix-
info-11.txt>, September 2005
[PSAMP-PROTO] Benoit Claise: PSAMP Protocol Specification,
Internet Draft <draft-ietf-psamp-protocol-03.txt>,
October 2005
10. Author's Addresses
Elisa Boschi
Hitachi Europe SAS
Immeuble Le Theleme
1503 Route des Dolines
06560 Valbonne, France
Phone: +33 4 89874180
Email: elisa.boschi@hitachi-eu.com
Lutz Mark
Fraunhofer Institute for Open Communication Systems
Kaiserin-Augusta-Allee 31
10589 Berlin
Germany
Phone: +49-30-34 63 7306
Fax: +49-30-34 53 8306
Email: mark@fokus.fraunhofer.de
11. Intellectual Property Statement
The IETF takes no position regarding the validity or scope of
any Intellectual Property Rights or other rights that might be
claimed to pertain to the implementation or use of the
technology described in this document or the extent to which any
license under such rights might or might not be available; nor
does it represent that it has made any independent effort to
identify any such rights. Information on the procedures with
respect to rights in RFC documents can be found in BCP 78 and
BCP 79.
Boschi, Mark Expires August 2006 [Page 13]
Reducing redundancy in IPFIX and PSAMP reports
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the
use of such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR
repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention
any copyrights, patents or patent applications, or other
proprietary rights that may cover technology that may be
required to implement this standard. Please address the
information to the IETF at ietf-ipr@ietf.org.
12. Copyright Statement
Copyright (C) The Internet Society (2006). This document is
subject to the rights, licenses and restrictions contained in
BCP 78, and except as set forth therein, the authors retain all
their rights.
13. Disclaimer
This document and the information contained herein are provided
on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
Boschi, Mark Expires August 2006 [Page 14]