Reducing redundancy in IPFIX and PSAMP reports
Internet-Draft E. Boschi
draft-boschi-ipfix-reducing-redundancy-01.txt Hitachi Europe
Expires: September 7, 2006 L. Mark
Fraunhofer FOKUS
March 2006
Reducing redundancy in IPFIX and PSAMP reports
draft-boschi-ipfix-reducing-redundancy-01.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that
any applicable patent or other IPR claims of which he or she is
aware have been or will be disclosed, and any of which he or she
becomes aware will be disclosed, in accordance with Section 6 of
BCP 79.
Internet-Drafts are working documents of the Internet
Engineering Task Force (IETF), its areas, and its working
groups. Note that other groups may also distribute working
documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other than
as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on September 7, 2006.
Copyright Notice
Copyright (C) The Internet Society (2006).
Boschi, Mark Expires September 2006 [Page 1]
Reducing redundancy in IPFIX and PSAMP reports
Abstract
This document describes a bandwidth saving methodology for
exporting flow or packet information using the IP Flow
Information Export (IPFIX) protocol. Being the PSAMP protocol
based on IPFIX, these considerations are valid for PSAMP exports
as well.
The main idea is to separate the export of information common to
several flows (or packets) and the specific flow (or packet)
information, using two different records. The association
between the records is kept using unique Identifiers.
Table of Contents
1. Introduction·········································2
2. Terminology··········································3
3. Open issues··········································3
4. Techniques for bandwidth saving information export···3
4.1 Single and multiple associations···················4
4.2 Export of Per-Packet Information···················5
4.3 Using Scopes·······································6
4.4 CommonPropertiesID Management······················7
5. Examples·············································7
5.1 Single commonPropertiesIDs·························7
5.2 Multiple commonPropertiesIDs······················10
5.3 Per-Packet Information Export·····················12
6. Export and evaluation considerations················14
7. IANA Considerations·································15
8. Security Considerations·····························15
9. Normative References································15
10. Author's Addresses··································16
11. Intellectual Property Statement·····················16
12. Copyright Statement·································16
13. Disclaimer··········································16
1. Introduction
In [IPFIX-PROTO] the IPFIX working group has defined a protocol
to export IP Flow information. The main purpose of the protocol
is to exchange information about IP traffic Flows in combination
with related measurement data. The export of that information is
done via Flow Records. In this scope a Flow is defined by a set
of key attributes (e.g. source and destination IP address,
source and destination port, etc.).
When exporting Flow Records sharing a set of common attributes,
these attributes have to be repeated for each Data Record even
though the values remain the same. The elimination of redundant
data from the export stream can result in a significant
reduction of the transferred data.
This draft proposes to use Option Data Records to export common
properties, whose value remains constant, and to send these
properties only once. Measured properties vary over time and are
sent multiple times in regular Data Records. The linkage of the
Flow Records with the common properties exported via option data
records is done via a Common Properties Identifier.
The proposed method can be applied also to export per packet
information (e.g. for Passive One-Way Delay measurements or
Boschi, Mark Expires September 2006 [Page 2]
Reducing redundancy in IPFIX and PSAMP reports
PSAMP exports). In this case a Flow is a collection of packets
that share a set of common attributes.
The proposed method does not need any changes to the IPFIX
protocol.
2. Terminology
Collecting Process
The collecting process receives records of flow or packet
information. The data is stored for later processing (by
the calculation process)
Exporting Process
The exporting processes send flow and packet records to the
collecting processes. The records are generated by the
measurement process.
Filtering
Filtering selects a subset of packets by applying
deterministic functions on parts of the packet content like
header fields or parts of the payload. A filtering process
needs to process the packet (look at packet header and/or
payload) in order to make the selection decision.
Measurement Device
A measurement device has access to at least one observation
point. It is hosting at least one measurement process and
one export process.
Metering/Measurement Process
The measurement process generates records of packet and
flow information. Packets passing the observation point are
captured, time stamped, filtered and classified. The
measurement process calculates the packet Ids.
Passive One-Way-Delay Measurement
Abbreviated: POWD Measurement
CommonPropertiesID
An identifier of a set of common properties that is unique
within an Observation Domain. This ID can be used to link
to information reported in separate records.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT","SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as
described in RFC 2119.
3. Open issues
[1] This draft proposes a new IE, commonPropertiesID; its
definition should be included in [IPFIX-INFO] and is currently
specified in the "IANA considerations" of this draft.
4. Techniques for bandwidth saving information export
The IPFIX protocol is template based. Templates define how data
should be exported, describing data fields together with their
type and meaning. IPFIX specifies two types of records to export
data: Data Records and Option Data Records. The layout of these
records is specified by Template Records and Option Template
Records.
Boschi, Mark Expires September 2006 [Page 3]
Reducing redundancy in IPFIX and PSAMP reports
To export information more efficiently we group different flow
records depending on their common properties. We define Common
Properties as a collection of attributes shared by a set of
different Flow Records. Specific Properties are the remaining
flow or packet attributes, not shared with the same set of
flows. Since a packet can be considered as a special kind of
flow, these considerations remain valid for packets too.
We split the information export into two parts, using two
different records. Common Properties are exported via Option
Data Records and are sent only once. These properties represent
values common to several flows. The associated Specific
Properties are subsequently sent using Data Records. The
association is kept using a particular index that uniquely
identifies the Common Properties, the commonPropertiesID.
Figure 1 shows the relation between Template and Data Sets for
Common and Specific Properties. The Common Properties Option
Template defines the common flow attributes (e.g. IP source and
destination address) and the commonPropertiesID. The
commonPropertiesID is a unique identifier for a set of
attributes; this field allows the linkage between those
attributes and Flow Data Records. Subsequent Option Data Records
of this template export the Common Properties values. The
assignment of Flow Records to common attributes could be
alternatively provided by the templateID, as explained in
Section 4.3. In this case the commonPropertiesID can be omitted.
The format for the information related to single flows or
packets is defined in the Specific Properties template. This
information is flow/packet specific and normally not shared
between many flows/packets.
+---------------------+ +---------------------+
| Option Template Set | | Template Set |
| | | | Description of
| Common Properties | | Specific Properties | exported data
+----------+----------+ +----------+----------+
...........|............................|.........................
| |
+----------v----------+ +----------v----------+ Exported data
| Option Data Set | | Data Set | with references
| <------+ | through common
| Common Properties | | Specific Properties | properties or
+---------------------+ +---------------------+ template IDs
Figure 1: Template Set and Data Set dependencies
The Common Properties option data records SHOULD be sent prior
to the corresponding Specific Properties data records.
4.1 Single and multiple associations
Several Flow Records often share a set of common properties.
Repeating the information about these common properties for
every record introduces a huge amount of redundancy. Let's
consider a set of properties "A", e.g. common sourceAddressA and
sourcePortA and another set of properties "B", e.g.
destinationAddressB and destinationPortB. Figure 2 shows how
this information is repeated with classical IPFIX export in
several records.
Boschi, Mark Expires September 2006 [Page 4]
Reducing redundancy in IPFIX and PSAMP reports
+--------------------+---------------------------------+
| properties A | <flow info> |
+--------------------+---------------------------------+
+--------------------+-----------------+---------------+
| properties A | properties B | <flow info> |
+--------------------+-----------------+---------------+
Figure 2: Common and Specific Properties exported in the same
record
In Figure 3 we separate Common Properties from Specific flow (or
packet) properties. In order to maintain the relation between
these sets of properties we introduce indices (idxA and idxB)
for the Common Properties that are unique for all Common
Property entries. The purpose of the indices is to serve as
"primary key" that identifies rows of the Common Properties.
More details about these indices will be given in section 4.3.
The rows are then referenced by the Specific Properties by using
the appropriate value for the Common Properties identifier.
A Flow Record can refer to one or more Common Properties sets,
achieving thus, in the latter case, a better export efficiency.
Note that in this case potential collisions between different
properties must be avoided. The Exporting Process must make sure
that if two or more sets of Common Properties are referenced by
the same Specific Properties, there are no ambiguities or
conflicts on the values of the common attributes.
Common Properties Specific Properties
+--------------+-----+ +------+----------------------+
| properties A |idxA <--------> idxA | <flow info> |
+--------------+-----+ +------+----------------------+
| properties B |idxB < > idxA | idxB | <flow info> |
+--------------+-----+ +------+------+---------------+
Figure 3: Common and Specific Properties exported separately
4.2 Export of Per-Packet Information
A particular case is the export of per packet information (e.g.
for Passive One-Way Delay measurements or PSAMP exports). In
this case a single packet could be considered a special case of
a flow and consequently per-packet information could be exported
using Flow Records. Doing this though would introduce additional
overhead, since packets belonging to the same flow share common
attributes (e.g. source address, destination address, etc).
Exporting these Common Properties on a per-packet basis, would
be redundant.
Figure 4 depicts three packets belonging to flow A and one
packet belonging to flow B, respectively. It shows export
records containing packet information plus flow information
(source and destination address). Undoubtedly, flow information
(or Common Properties, as defined in this draft) introduces a
huge amount of redundancy, as it is repeated for every packet in
every record.
Boschi, Mark Expires September 2006 [Page 5]
Reducing redundancy in IPFIX and PSAMP reports
One-packet flows
+-------------------+----------------------+
| flow properties A | <packet info> |
+-------------------+----------------------+
| flow properties A | <packet info> |
+-------------------+----------------------+
| flow properties B | <packet info> |
+-------------------+----------------------+
| flow properties A | <packet info> |
+-------------------+----------------------+
Figure 4: Flow and packet information represented in one-packet
flows
In Figure 5 we separate Common Properties from Specific
Properties, i.e. flow from packet information. In order to
maintain the relation between Specific (Packet) Properties and
Common (Flow) Properties we introduce indices (idxA and idxB),
much in the same way as explained in the previous sections. The
linkage of one packet and flow B (flow properties B, idxB) is
explicitly drawn.
Specific (Packet) Properties
+------+--------------------+
Common (Flow) Properties > idxA | <packet info> |
+------------------+-----+ +------+--------------------+
|flow properties A |idxA < > idxA | <packet info> |
+------------------+-----+ +------+--------------------+
|flow properties B |idxB <-------> idxB | <packet info> |
+------------------+-----+ +------+--------------------+
> idxB | <packet info> |
+------+--------------------+
Figure 5: Flow information and packet information
4.3 Using Scopes
Common Properties are sent via IPFIX option records. IPFIX
option records contain one or more scope fields. The Flow
Properties record can contain the commonPropertiesID and/or the
TemplateID as scope fields. There are three options:
1) Use commonPropertiesID as scope
The Common Properties are valid for all data records
containing that commonPropertiesID. This solution limits the
number of different flows that can be exported at the same
time in the same observation domain to 2**32 (using 32 bits
commonPropertiesIDs)
2) Use commonPropertiesID and templateID as scope
The Common Properties are valid only for data records
referring to the template specified by the templateID and
containing that commonPropertiesID. This allows the export up
to 2**32 flows per template. The solution is to be chosen
when the number of flows to be exported is expected to be
very high (and beyond the limit posed by solution 1)
3) Use templateID as scope
Boschi, Mark Expires September 2006 [Page 6]
Reducing redundancy in IPFIX and PSAMP reports
The common properties are valid for all data records of the
specified template. In this case commonPropertiesIDs are not
needed but the Exporting Process requires a templateID per
flow. In the general case this solution is not scalable but
can be suitable for certain applications (e.g. flow
aggregation).
4.4 CommonPropertiesID Management
The management of commonPropertiesIDs is very similar to the
management of templateIDs described in [IPFIX-PROTO]. The
Exporting Process maintains the commonPropertiesIDs for the
exporter's Observation Domains. Like templateIDs, a
commonPropertiesID MUST be unique per Observation Domain (source
identifier in the IPFIX header). Different Observation Domains
from the same exporter may use the same commonPropertiesID value
to refer to different sets of Common Properties.
There are no constraints regarding the order of the
commonPropertiesID allocation. When limiting the scope to
special templates, the commonPropertiesIDs have to be unique per
Observation Domain and Template.
Using 32 bit commonPropertiesIDs allows the export of 2**32
active flows in parallel. CommonPropertiesIDs have a certain
lifetime inside which they cannot be reused. After that time a
commonPropertiesID can be assigned to another set of Common
Properties. CommonPropertiesID whose lifetime has longer expired
SHOULD be preferred. The lifetime MUST be configurable.
The collecting process associates a lifetime with each
commonPropertiesID. The lifetime MUST be configurable. The
mapping of Data Records to Common Properties uses the most
recent Common Properties definition associated to the specified
commonPropertiesID. If there is no flow definition associated
with that commonPropertiesID or the lifetime of the flow
definition has expired, no mapping is possible. In this case the
collecting process SHOULD log an error.
When IPFIX uses an unreliable transport protocol to export the
Option Data Records containing the Common Properties and the
commonPropertiesIDs these records MUST be re-sent at regular
intervals, whose frequency MUST be configurable.
When using a connection oriented transport protocol the Common
Properties have to be re-sent after a connection re-
establishment before the corresponding Specific Properties Data
Records.
5. Examples
5.1 Single commonPropertiesIDs
In this section we show how flow information can be exported
efficiently using the method described in this draft. Let's
suppose we have to periodically export data about two IPv6
Flows.
In this example we report the following information:
Boschi, Mark Expires September 2006 [Page 7]
Reducing redundancy in IPFIX and PSAMP reports
Flow| dstIPv6Address | dst- |nPkts|nBytes
| | Port | |
----------------------------------------------------------------
A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 30 | 6000
| | | |
A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 50 | 9500
| | | |
B |5F05:2000:80AD:5800:0058:00AA:00B7:AF2B| 1932 | 60 | 8000
| | | |
A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 40 | 6500
| | | |
A |5F05:2000:80AD:5800:0058:0800:2023:1D71| 80 | 60 | 9500
| | | |
B |5F05:2000:80AD:5800:0058:00AA:00B7:AF2B| 1932 | 54 | 7600
The Common Properties in this case are the destination IPv6
address and the destination port. We first define an Option
Template that contains the following Information Elements:
- the destination IPv6 address, destinationIPv6Address
[IPFIX-INFO], with a type of 28 and a length of 16 octets
- the destination port, destinationTransportPort [IPFIX-INFO]
with a type of 11, and a length of 2 octets
- the Common Properties ID, with a type [TO BE ASSIGNED, in
this Draft referenced to as XX] and a length of 4 octets.
Figure 6 shows the Option template defining the Common
Properties with commonPropertiesID as scope:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| commonPropertiesID = XX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| destinationIPv6Address = 28|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 16 |0|destinationTransportPort = 11|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6: Common Properties Option Template
The Specific Properties Template consists of the information not
contained in the Option Templates, i.e. flow specific
information, in this case the number of packets and the number
of bytes to be reported. Additionally, this Template contains
the commonPropertiesID. In Data Records, the value of this field
will contain one of the unique indices of the Option Records
exported before. It contains the following Information Elements
(see also Figure 7):
- commonPropertiesID with a length of 4 octets
- the number of packets of the Flow: inPacketDeltaCount in
[IPFIX-INFO], with a length of 4 octets
Boschi, Mark Expires September 2006 [Page 8]
Reducing redundancy in IPFIX and PSAMP reports
- the number of octets of the Flow: inOctetDeltaCount in
[IPFIX-INFO], with a length of 4 octets
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 20 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 258 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| commonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inPacketDeltaCount = 2 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inOctetDeltaCount = 1 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7: Specific Properties Template
Considering the data shown at the beginning of this example, the
following two Option Data Records will be exported:
Common- | dstAddress | dst-
PropertiesID | | Port
-------------+-----------------------------------------+-------
1 | 5F05:2000:80AD:5800:0058:0800:2023:1D71 | 80
| |
2 | 5F05:2000:80AD:5800:0058:00AA:00B7:AF2B | 1932
The Option Data Records reporting the Common Properties will
look like:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 257 | Length = 52 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 5F05:2000: ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 80AD:5800: ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0058:0800: ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 2023:1D71 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 80 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 5F05:2000: ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 80AD:5800: ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0058:00AA: ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 00B7:AF2B |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 1932 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8: Option Data Records reporting Common Properties
Boschi, Mark Expires September 2006 [Page 9]
Reducing redundancy in IPFIX and PSAMP reports
The Data Records will in turn be:
commonPropertiesID | inPacketDeltaCount | inOctetDeltaCount
---------------------------------------------------------------
1 | 30 | 6000
1 | 50 | 9500
2 | 60 | 8000
1 | 40 | 6500
1 | 60 | 9500
2 | 54 | 7600
The Figure below shows the first Data Record listed in the
table:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 258 | Length = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 30 | 6000 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9: Data Record reporting Specific Properties
5.2 Multiple commonPropertiesIDs
In case a flow shares two set of Common Properties, an
additional overhead reduction can be obtained. Let's suppose we
have to export the following flow information:
Flow | srcAddr | srcPort | dstAddr | dstPort | nPackets | nBytes
----------------------------------------------------------------
A |10.0.0.1 | 1932 |10.0.1.2 | 80 | 30 | 6000
B |10.0.0.3 | 2032 |10.0.1.2 | 80 | 50 | 9500
Fehler! Verweisquelle konnte nicht gefunden werden. shows the
Option Templates, containing the Common Properties together with
the commonPropertiesID, which is the scope.
In the first Common Properties Option Template we export the
following Information Elements:
- the source IPv4 Address, sourceIPv4Address [IPFIX-INFO],
with a type of 8 and a length of 4 octets
- the source Port, sourceTransportPort [IPFIX-INFO], with a
type of 7 and a length of 2 octets
- the Common Properties ID, commonPropertiesId with a type
[TO BE ASSIGNED, in this Draft referenced to as XX] and a
length of 4 octets.
The second Option Template contains the following Information
Elements:
- the destination IPv4 Address, destinationIPv4Address
[IPFIX-INFO], with a type of 12 and a length of 4 octets
Boschi, Mark Expires September 2006 [Page 10]
Reducing redundancy in IPFIX and PSAMP reports
- the destination port, destinationTransportPort [IPFIX-INFO]
with a type of 11, and a length of 2 octets
- the Common Properties ID, with a type [TO BE ASSIGNED, in
this Draft referenced to as XX] and a length of 4 octets.
The Flow identifier, which is represented by the
commonPropertiesId Information Element [NOTE: to be included in
IPFIX-INFO], is used in both cases as the Scope Field.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| commonPropertiesID = XX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| transportSourcePort = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| commonPropertiesID = XX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| destinationIPv4Address = 12|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0|transportDestinationPort = 11|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 10: Example Common Properties Template
Considering the values given at the beginning of this section we
will export the following Option Data Records:
commonPropertiesID | sourceAddress | sourcePort
--------------------+-----------------+-------------
1 | 10.0.0.1 | 1932
2 | 10.0.0.3 | 2032
and
commonPropertiesID | dstAddress | dstPort
--------------------+---------------+-----------
3 | 10.0.1.2 | 80
The Specific Properties Template consists of the information not
contained in the Option Templates, i.e. flow specific
information. Additionally, this Template contains the two
commonPropertiesIDs. In Data Records, the values of these fields
Boschi, Mark Expires September 2006 [Page 11]
Reducing redundancy in IPFIX and PSAMP reports
will contain one of the unique indices specified in the Option
Records exported before.
Figure 11 displays the Template including the
commonPropertiesIDs plus the Specific Properties. In this
example we export the following Information Elements:
- commonPropertiesID for the source fields with a length of 4
octets
- commonPropertiesID for the destination fields with a length
of 4 octets
- the number of packets of the Flow: inPacketDeltaCount in
[IPFIX-INFO], with a length of 4 octets
- the number of octets of the Flow: inOctetDeltaCount in
[IPFIX-INFO], with a length of 4 octets
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 259 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| commonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| commonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inPacketDeltaCount = 2 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| inOctetDeltaCount = 1 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 11: Example Specific Properties Template
Considering the values given at the beginning of this section,
the Data Records of the two flows will look like:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Length = 28 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 30 | 6000 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 50 | 9500 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 12: Specific Properties
5.3 Per-Packet Information Export
Boschi, Mark Expires September 2006 [Page 12]
Reducing redundancy in IPFIX and PSAMP reports
To demonstrate how to use IPFIX efficiently to export per-packet
information, this section proposes how to use the IPFIX protocol
for exporting Flow information (Common Properties) and
per-packet information (Specific Properties, in this case
related to a long-lived flow) for OWD computation.
In order to acquire a One-Way path delay information, two
measurement points with synchronized clocks must exist, one at
each end of the path under examination. Both measurement points
will capture packets, assign them timestamps and generate an
identifier for a packet passing that point. Each measurement
point will export its measurement data to a Collecting Process
where the data are correlated based on the packet identifiers
and timestamps and then the delay is calculated as a difference
of two timestamps of a packet pair.
The Templates that would be needed for exporting measurement
data of this kind are illustrated in the figures below.
Figure 13 shows the Option Template containing the information
concerning Flows using the commonPropertiesID as scope. In the
Flow Properties Template (the Common Porperties Template) we
export the following Information Elements:
- the source IPv4 Address, sourceIPv4Address [IPFIX-INFO],
with a type of 8 and a length of 4 octets
- the destination IPv4 Address, destinationIPv4Address
[IPFIX-INFO], with a type of 12 and a length of 4 octets
- the Class of Service field, ClassOfServiceIPv4 [IPFIX-
INFO], with a type of 5 and a length of 1 octet
- source and destination ports, sourceTransportPort and
destinationTransportPort [IPFIX-INFO], with a type of 7 and
11 respectively, and a length of 2 octets each
The Flow identifier, which is represented by the
commonPropertiesID Information Element, is used as the Scope
Field.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 40 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field count = 1 |0| commonPropertiesID = XX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| destinationIPv4Address = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| classOfServiceIPv4 = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| protocolIdentifier = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| transportSourcePort = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |0|transportDestinationPort = 11|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | (Padding) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 13: Example Flow Properties Template
Boschi, Mark Expires September 2006 [Page 13]
Reducing redundancy in IPFIX and PSAMP reports
For passive One-Way-Delay measurement, the Packet Properties
Template, or Specific Properties Template, consists of at least
Timestamp and Packet ID. Additionally, this template contains a
flow identifier field. In packet Records, the value of this
field will contain one of the unique indices of the Flow Records
exported before.
Figure 14 displays the template with the packet properties. In
this example we export the following Information Elements:
- commonPropertiesID with a length of 4 octets
- packetTimestamp, packetID, and packetLength. Since
packetTimestamp, packetID, and packetLength are not (yet)
IETF-defined information elements, we export them as
enterprise-specific IEs. The three IEs have respectively a
type of 220, 221, and 222 and a length of 8, 4, and 4
octets.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 36 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| commonPropertiesID = XX | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| packetTimestamp = 220 | Field Length = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| packetID = 221 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| packetLength = 222 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Enterprise number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 14: Example Packet Properties Template
The delay is derived by a calculation step: at the collection
point packet records of two measurement points are gathered and
correlated by means of the packet ID. The resulting delay data
records are exported in a similar manner as the packet data have
been. Especially, the linkage between delay data and flow
information is represented with the Flow identifier
commonPropertiesID fields. The OWD properties contain the Packet
Pair ID (which is the packet ID matching that of the two
contributing packet records), a timestamp (which is the
timestamp of the packet passing the reference monitor point) in
order to reconstruct a time series, the calculated delay value,
and finally the flow identifier (commonPropertiesID).
6. Export and evaluation considerations
The main advantage of the methods proposed in this draft is the
reduced amount of measurement data that has to be transferred
from the Exporter to the Collector. In addition there is less
storage capacity needed at the Collector side.
Boschi, Mark Expires September 2006 [Page 14]
Reducing redundancy in IPFIX and PSAMP reports
On the other hand there is some extra processing power needed on
the Exporter side to manage Common Properties information and to
assign them to Flow Records. The Collector has to process
records of two templates instead of just one but has to read and
write less data. Additional effort is needed when post
processing the measurement data, because now the correlation of
Flow Records with Common Properties information is needed.
In the example in section 5.3 using IPFIX to export the
measurement data for each received packet 30 bytes have to be
transferred (sourceAddressV4=4, destinationAddressV4=4,
classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2,
destionationTransportPort=2, packetTimestamp=8, packetID=4,
packetLength=4). Without considering the IPFIX protocol overhead
a flow of 1000 packets produces 28000 bytes of measurement data.
Using the proposed optimization each packet produces an export
of only 20 bytes (packetTimestamp=8, packetID=4, packetLength=4,
commonPropertiesID=4). The export of the flow information
produces 16 bytes (sourceAddressV4=4, destinationAddressV4=4,
classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2,
destionationTransportPort=2, commonPropertiesID =4). For a flow
of 1000 packet this sums up to 16016 bytes. This is a decrease
of more than 40 percent.
Note that IPFIX allows the reduction of the size of exported
Information Elements. This can be applied also to the
commonPropertiesID and would result in a further reduction of
bytes.
7. IANA Considerations
IANA should provide an ElementId for the commonPropertiesID
Information Element.
commonPropertiesId
Description
An identifier of a set of common properties that is unique
within an Observation Domain. This ID can be used to link
to information reported in separate records
Abstract Data Type: unsigned32
Data Type Semantics: identifier
ElementId: TBA
Status: proposed
8. Security Considerations
For the proposed use of the IPFIX protocol for bandwidth-saving
export the security considerations as for the IPFIX protocol
apply.
9. Normative References
[IPFIX-PROTO] Benoit Claise et Al.: IPFIX Protocol
Specification, IETF draft work in progress
<draft-ietf-ipfix-protocol-19.txt>, September 2005
[IPFIX-INFO] J. Quittek, S.Bryant, B.Claise, J. Meyer:
Information Model for IP Flow Information Export
Internet-draft work in progress <draft-ietf-ipfix-
info-11.txt>, September 2005
Boschi, Mark Expires September 2006 [Page 15]
Reducing redundancy in IPFIX and PSAMP reports
[PSAMP-PROTO] Benoit Claise: PSAMP Protocol Specification,
Internet Draft <draft-ietf-psamp-protocol-03.txt>,
October 2005
10. Author's Addresses
Elisa Boschi
Hitachi Europe SAS
Immeuble Le Theleme
1503 Route des Dolines
06560 Valbonne, France
Phone: +33 4 89874180
Email: elisa.boschi@hitachi-eu.com
Lutz Mark
Fraunhofer Institute for Open Communication Systems
Kaiserin-Augusta-Allee 31
10589 Berlin
Germany
Phone: +49-30-34 63 7306
Fax: +49-30-34 53 8306
Email: mark@fokus.fraunhofer.de
11. Intellectual Property Statement
The IETF takes no position regarding the validity or scope of
any Intellectual Property Rights or other rights that might be
claimed to pertain to the implementation or use of the
technology described in this document or the extent to which any
license under such rights might or might not be available; nor
does it represent that it has made any independent effort to
identify any such rights. Information on the procedures with
respect to rights in RFC documents can be found in BCP 78 and
BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the
use of such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR
repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention
any copyrights, patents or patent applications, or other
proprietary rights that may cover technology that may be
required to implement this standard. Please address the
information to the IETF at ietf-ipr@ietf.org.
12. Copyright Statement
Copyright (C) The Internet Society (2006). This document is
subject to the rights, licenses and restrictions contained in
BCP 78, and except as set forth therein, the authors retain all
their rights.
13. Disclaimer
This document and the information contained herein are provided
on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
Boschi, Mark Expires September 2006 [Page 16]