Internet Engineering Task Force L. Peluso
Internet-Draft University of Napoli
Intended status: Standards Track T. Zseby
Expires: June 4, 2011 Fraunhofer Institute FOKUS
S. D'Antonio
CINI Consortium/University of
Napoli "Parthenope"
C. Henke
Fraunhofer Institute FOKUS
December 01, 2010
Flow Selection Techniques
draft-ietf-ipfix-flow-selection-tech-03.txt
Abstract
Flow selection is the process of selecting a subset of flows from all
flows observed at an observation point. The objective of flow
selection is to reduce the effort of post-processing flow data and
transferring flow records. The flow selection process can be enabled
at different stages of the measurement process. It can be applied
either directly after classification or at recording/exporting time
by limiting the number of flows to be stored and/or exported to the
collecting process. This document describes motivations for flow
selection and presents flow selection techniques. It furthermore
provides an information model for configuring flow selection
techniques and discusses what information about a flow selection
process is worth exporting through a suitable information model.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
Peluso, et al. Expires June 4, 2011 [Page 1]
Internet-Draft Flow Selection Techniques December 2010
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 4, 2011.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
Peluso, et al. Expires June 4, 2011 [Page 2]
Internet-Draft Flow Selection Techniques December 2010
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6
4. Flow selection as a function of the IPFIX Exporter . . . . . . 7
4.1. Flow selection in the metering process . . . . . . . . . . 9
4.2. Flow selection in the flow recording process . . . . . . . 9
4.3. Flow selection during the exporting process . . . . . . . 10
5. Flow selection as a function of the IPFIX Mediator . . . . . . 11
6. Flow selection techniques . . . . . . . . . . . . . . . . . . 13
6.1. Flow selection based on flow record content . . . . . . . 13
6.2. Flow selection based on flow record arrival time or
sequence . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.3. Flow selection on external events . . . . . . . . . . . . 13
7. Information model for flow selection information exporting . . 14
7.1. Meter process related (TBD1-TBD2) . . . . . . . . . . . . 15
7.1.1. FsMeter_UnmeasPacketCount . . . . . . . . . . . . . . 16
7.1.2. FsMeter_UnmeasBytesCount . . . . . . . . . . . . . . . 16
7.2. Flow recording process related (TBD3-TBD8) . . . . . . . . 16
7.2.1. FsFrec_PacketInDroppedRecsCount . . . . . . . . . . . 17
7.2.2. FsFrec_ByteInDroppedRecsCount . . . . . . . . . . . . 17
7.2.3. FsFrec_FrecDroppedCount . . . . . . . . . . . . . . . 17
7.2.4. FsFrec_UnexportedFrecCount . . . . . . . . . . . . . . 18
7.2.5. FsFrec_UnexportedPacketInFrecCount . . . . . . . . . . 18
7.2.6. FsFrec_UnexportedBytesInFrecCount . . . . . . . . . . 18
7.3. Flow exporting process related (TBD9-TBD14) . . . . . . . 19
7.3.1. FsExp_PacketInDroppedRecsCount . . . . . . . . . . . . 19
7.3.2. FsExp_ByteInDroppedRecsCount . . . . . . . . . . . . . 19
7.3.3. FsExp_FrecDroppedCount . . . . . . . . . . . . . . . . 20
7.3.4. FsExp_UnexportedCount . . . . . . . . . . . . . . . . 20
7.3.5. FsExp_UnexportedPacketCount . . . . . . . . . . . . . 20
7.3.6. FsExp_UnexportedByteInExpCount . . . . . . . . . . . . 21
8. Implementation requirements . . . . . . . . . . . . . . . . . 21
9. Information Model for Configuration of Flow Selection
Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 22
9.1. selectorMethod . . . . . . . . . . . . . . . . . . . . . . 22
9.2. flowMaxAdmitFlowRecords . . . . . . . . . . . . . . . . . 23
9.3. flowRecordBytesSize . . . . . . . . . . . . . . . . . . . 24
9.4. flowRecordPacketsSize . . . . . . . . . . . . . . . . . . 24
9.5. flowInactivityTime . . . . . . . . . . . . . . . . . . . . 24
9.6. flowIPAddressVersion . . . . . . . . . . . . . . . . . . . 25
9.7. flowIPv4SourceAddress . . . . . . . . . . . . . . . . . . 25
9.8. flowIPv4DestinationAddress . . . . . . . . . . . . . . . . 26
9.9. flowIPv6SourceAddress . . . . . . . . . . . . . . . . . . 26
9.10. flowIPv6DestinationAddress . . . . . . . . . . . . . . . . 27
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Peluso, et al. Expires June 4, 2011 [Page 3]
Internet-Draft Flow Selection Techniques December 2010
11.1. Normative References . . . . . . . . . . . . . . . . . . . 27
11.2. Informative References . . . . . . . . . . . . . . . . . . 27
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 29
Peluso, et al. Expires June 4, 2011 [Page 4]
Internet-Draft Flow Selection Techniques December 2010
1. Introduction
This document describes flow selection techniques for traffic
measurements. As stated in [PSAMP-TECH], packet selection is the
process of selecting a subset of packets among those collected at an
observation point. The element on which this selection mechanism is
performed is a single packet and the selection decision is random,
deterministic or based on packet properties. In contrast to this,
flow selection techniques consider flows as the basic elements on
which a selection process is performed. A flow is defined as a set
of packets with common properties [IPFIX-REQ]. Flow selection
reduces the resource demands for capturing, storing, exporting and
post-processing flow-based measurement results. Maintaining and
exporting all flow records to the collecting process can exhaust
available resources with the result that measurement data is
discarded which reduces the utility of the measurement data and makes
accurate estimations impossible. Flow Selection can be applied to
only select flows that are of interest for a certain application or
to select a representative subset of the flows. Selecting
representative subsets of flows allows for flow characteristic
estimation while reducing the measurement data. Example applications
in which flow selections can be applied are accounting, attack and
intrusion detection, traffic engineering, traffic classification,
etc.. In many networks few large flows contribute to the majority of
the overall traffic volume [DuLT01a], [DuLT01b], which is also
referred to as "Quasi-Zipf-Law" [KuXW04]. This "elephant and mice"
phenomenon plays an important role in flow based measurements as it
poses challenges on the flow selection process depending on the
application and the available resources. For example in accounting
purposes it is useful to concentrate on the so-called "heavy hitter"
flows to cope with a limited flow cache size or limited transmission
capacity in times when resources are scarce.
2. Scope
This document describes flow selection techniques and their
parameters. It addresses the configuration of flow selection
techniques and defines which information should be reported by
devices that perform flow selection. It only describes processes
directly acting on traffic flows during the metering phase and/or the
exporting phase. Therefore it is assumed that flow selection is
performed after packets are classified into flows. This document
does not address the flow selection effects that might arise from
sampling or filtering packets in the metering process before the
classification process is performed. Such packet selection
techniques are described in [PSAMP-TECH] and, therefore, are out of
the scope of this document.
Peluso, et al. Expires June 4, 2011 [Page 5]
Internet-Draft Flow Selection Techniques December 2010
3. Terminology
This document is consistent with the terminology introduced in
[IPFIX-ARCH], [PSAMP-TECH] and [IPFIX-REQ]. Further some additional
terms are presented which extend the terminology.
* Flow
A Flow is defined as a set of packets with common properties
passing an Observation Point in the network during a certain time
interval.
* Flow Record
A Flow Record contains information about a specific flow that was
metered at an observation point. A Flow Record contains measured
properties of the flow (e.g., the total number of bytes of all
packets of the flow) and usually characteristic properties of the
flow (e.g., source IP address).
* Classification
Is a filtering process in which packets are mapped to specific
flow records based on the packet properties. These properties
make up the flow key (e.g. header information, packet content, AS
number). In case a flow record for this specific flow key already
exist the flow record is updated, otherwise a new flow record is
created.
* Flow Selection Process
A Flow Selection Process takes a set of Flow Records as its input
and selects a subset of that set as its output.
* Flow Selection State
A Flow Selection Process may maintain state information for use by
the Flow Selection Process. At a given time, the Flow Selection
State may depend on flows observed at and before that time, as
well as other variables. Examples include:
(i) number of accounted flow records;
(ii) memory space available for flow recording;
Peluso, et al. Expires June 4, 2011 [Page 6]
Internet-Draft Flow Selection Techniques December 2010
(iii) state of the pseudorandom number generators;
(iv) hash values calculated during the selection.
(v) timestamps of flow observation (e.g. first or last packet
of flow) and flow duration
* Flow Selector
A Flow Selector defines the action of a Flow Selection Process on
a single flow of its input. The Flow Selector can make use of the
following information in order to establish whether or not a flow
has to be selected or not:
(i) the content of the flow record;
(ii) any information state related to the flow recording
process;
(iii) any flow selection state that may be maintained by the
Flow Selection Process.
* IPFIX Exporter
The IPFIX exporter is a software that receives measurement data
from e.g. the Metering Process and exports this data to a
collector.
* IPFIX Mediator
The IPFIX Mediator receives flow records from IPFIX Exporters and,
after some post-processing (e.g. more fine-grained selection),
sends them to one or multiple Collectors. This selection can be
driven based on the information requested by the individual
Collector.
4. Flow selection as a function of the IPFIX Exporter
Figure 1 shows the IPFIX reference model as defined in [IPFIX-ARCH],
and extends it by introducing the functional components where flow
selection can take place.
Peluso, et al. Expires June 4, 2011 [Page 7]
Internet-Draft Flow Selection Techniques December 2010
Packet(s) coming in to Observation Point(s)
| |
v v
+----------------+---------------------------+ +-----+-------+
| Metering Process on an | | |
| Observation Point | | |
| | | |
| packet header capturing | | |
| | |...| Metering |
| timestamping | | Process N |
| | | | |
| packet selection | | |
| | | | |
| classification | | |
| | | | |
| flow state dependent packet sampling (*) | | |
| | | | |
| aggregation | | |
| | | | |
| flow recording (*) | | |
| | | | |
| | Timing out Flows | | |
| | Handle resource overloads | | |
+--------|-----------------------------------+ +-----|-------+
| |
Flow Records (selected by Observation Domain) Flow Records
| |
+----------------------+----------------------+
|
+----------------------|---------------+
| Exporting Process v |
| +---------------+-----------+ |
| | flow export (*) | |
| +---------------+-----------+ |
| | |
+----------------------+---------------+
|
v
IPFIX export packet to Collector
(*) indicates where flow selection can take place.
Figure 1: Flow selection as a function of the IPFIX Exporter
In contrast to packet selection, flow selection is always applied
after the packets are classified into flows. Flows can be selected
at different stages of the measurement chain:
Peluso, et al. Expires June 4, 2011 [Page 8]
Internet-Draft Flow Selection Techniques December 2010
1. during metering [PSAMP-TECH];
2. during flow recording;
3. during flow exporting.
4.1. Flow selection in the metering process
The main reason for applying flow state dependent sampling during the
metering process is that the flow recording process may not have, at
a certain point in time, enough memory positions to record all
observable flows. Another reason may be that there might not be
enough processing resources to create and manage a new flow record.
To overcome these limitations, a number of possible policies can be
applied, the simplest one being to discard new packets which cannot
be assigned to existing flow records (i.e. that would require the
creation of a new flow record). More complex policies are
applicable, mainly aimed at detecting the so called elephant flows,
so as to prioritize flows carrying a higher traffic volume in the
flow recording process . For instance, [EsVa01] proposes criteria to
define a packet as eligible to create a new flow record (sample and
hold, multistage filters). Regardless of specific algorithms, we are
concerned about identifying what information on the flow state
dependent packet sampling is worth keeping and making available to
applications (by exporting it out of an IPFIX device). An option
could be to keep a cumulative counter of the total number of packets
and bytes that were not taken into account for measurement purposes
because of flow state dependent sampling. Furthermore, it is
possible to keep a timestamp for the first and last of these
discarded packets. In practice, this implies aggregating all these
packets in a single macro flow, and keeping track of its volume and
duration. Storing more detailed information about packets which have
not been measured because of flow state dependent sampling would
contradict the fact that the sampling is done because of lack of
memory and/or processing resources.
4.2. Flow selection in the flow recording process
As described in the previous section, because of lack of memory
positions in the flow recording process some incoming packets might
be discarded if they lead to the creation of a new flow record.
However, under certain circumstances, it may be advantageous to
discard an existing flow record during the flow recording process in
order to make room for a new one which has been created upon arrival
of a new packet. For example, an algorithm to make the decision
whether to discard a new incoming packet or an existing flow record
is described in [Moli03]. In this section we focus on the selection
of the information to be stored regarding the record removal rather
Peluso, et al. Expires June 4, 2011 [Page 9]
Internet-Draft Flow Selection Techniques December 2010
than on the details of the decision making algorithm. For the
reasons we mentioned above, it does not make sense to store separate
information for each discarded flow record, as it would contradict
the motivation why discarding is done (i.e. lack of memory
resources). The information that can be kept with a limited overhead
is the cumulative counter of the total number of not yet exported
packets and bytes belonging to flow records that were removed during
the flow recording process. Ideally, we would like to keep also a
timestamp for the first (T_fd) and last (T_ld) not yet exported
packets belonging to every discarded flow record. This would mean
aggregating all these packets in a macro flow, and keeping track of
its volume and duration. To do so, we would need to maintain a
timestamp of the first and last non-exported packets in each flow
record, check the values of such timestamps whenever a record is
discarded in order to verify whether they are smaller or larger than
T_fd and T_ld, respectively, and if so update them. Another
information that can be easily maintained is the number of discarding
actions, along with the timestamps of the first and last action.
This information should not be used by applications to re-normalize
the received per flow statistics (because a flow may be discarded and
created multiple times), but rather to monitor and control the
performance of the implemented policy. Note that we take into
account a discarding event only when the discarded flow record
contains data about traffic which has not been exported. Otherwise,
the removal of a record whose traffic was exported (either after a
timeout or after the arrival of specific packets, e.g. TCP FIN or
RST) is part of the normal operation of an IPFIX flow metering
system. Note also that we consider only the case when the
elimination of a flow record during the flow recording process leads
to the complete loss of all the information contained in the flow
record itself. The implementation of a different policy, such as
immediate exporting of the flow record before elimination, or
freezing of the flow record and moving it to another area of memory
for later exporting, is not considered as an elimination and
therefore it is out of the scope of this document. Along with the
information about the number of discarded flow records and associated
packets and bytes, it is useful to keep cumulative information about
the number of flow records containing not yet exported traffic and
being currently handled by the flow recording process, as well as the
cumulative number of not exported packets and bytes contained in
them.
4.3. Flow selection during the exporting process
The exporting process may implement policies for exporting only a
subset of the flow records which have been stored in the system
memory. The decision to export only a subset of the flow records can
be motivated by the existence of an explicit policy which filters out
Peluso, et al. Expires June 4, 2011 [Page 10]
Internet-Draft Flow Selection Techniques December 2010
the flow records to be exported. An example of such a policy could
be to export only the flow records associated to flows whose
accounted traffic is below a certain threshold, or to implement a
more complex mechanism such as the one described in [DuLT01a] or
[DuLT01b]. Another motivation which might bring to the exporting of
a subset of stored flow records is resource limitation. For example,
the exporting process has been assigned a limited time slot to
operate or it exports only a predefined number of packets. Hybrid
cases can happen where the exporting of a subset of the flow records
is motivated by the co-existence of resource limitations and ad-hoc
policies which are applied in order to optimize the exporting process
(e.g. given that the exporting process applies to a subset of the
flow records, such subset is selected so that the overall number of
exported packets and bytes belonging to the subset is maximized).
Selecting flow records during the exporting process raises the issue
of identifying the information which is worth keeping about the flow
selection process. Two different scenarios cab be envisaged. If a
flow record is not exported and then it does not feed the flow
recording process, the scenario is the same as when the deletion of
the flow record is caused by the need to make room to another record.
The metrics to be kept are cumulative packets and bytes associated
with non-exported flow records, timestamps of the first and last
packets belonging to non-exported flow records, counter of dropping
events and timestamps of first and last dropping event. If a record
eligible for exporting is not exported and it enters the flow
recording process it has a chance of being exported in the future.
It would be beneficial for an application to get information, in
terms of number of generated packets and bytes, about the flow
records which are not being exported due to the existence of
exporting policies and/or resource limitations. This is intended to
make it possible to detect potential pathologic conditions, like the
missed exporting of a large number of flow records and/or associated
traffic, or the growing number of flow records being involved in the
flow recording process. The selection of the flow records to be
exported implies performing a complete scanning of the memory area
where flow information is stored, thus jeopardizing the efficiency of
the overall exporting process. For this reason, flow exporting
protocol specification does not include flow selection during the
recording process as a mandatory function even if the information
model has been designed to enable such function.
5. Flow selection as a function of the IPFIX Mediator
As shown in Figure 2, flow selection can be performed as an
intermediate process within an IPFIX Mediator. This process selects
the flow records from a sequence which meets pre-defined criteria and
exports them to an IPFIX Collector. This selection function can be
Peluso, et al. Expires June 4, 2011 [Page 11]
Internet-Draft Flow Selection Techniques December 2010
seen as a more fine-grained process with respect to the selection
performed by an IPFIX Exporter. The criteria used to drive the
selection process at Mediator's level might be applied to the set of
flow records coming from the IPFIX Exporter, thus triggering a
further flow selection process.
Packet(s) coming in to Observation Point(s)
|
IPFIX Original |
Exporter v
+------------------+-------------------+
| |
| Metering Process on an |
| Observation Point |
| | |
| Flow metering and selection |
| | |
| Flow recording and selection |
| | |
| Flow exporting and selection |
| | |
+------------------+-------------------+
|
v
Flow Records (selected by Observation Domain)
|
IPFIX Mediator v
+------------------+-------------------+
| |
| Collecting process |
| | |
| Flow selection (*) |
| | |
| Exporting process |
| | |
+------------------+-------------------+
|
v
Flow Records
Figure 2: Flow selection as a function of the IPFIX Mediator
As an example, if an IPFIX Mediator interacts with a set of IPFIX
Collectors, flow records arriving at the IPFIX Mediator might be
selected based on the IPFIX Collector requesting flow information.
As described in previous sections, flow selection can take place
during metering, recording, and exporting processes of an IPFIX
Peluso, et al. Expires June 4, 2011 [Page 12]
Internet-Draft Flow Selection Techniques December 2010
exporter depending on the policies which are implemented to meet
application requirements. In case flow selection is performed at
Mediator's level, we envisage the use of flow selection techniques as
a step of the exporting process aimed to identify the flow records to
be exported among those stored in the system's memory. This is
because the lighter is the intermediate selection process, the better
is the performance of the mediation framework.
6. Flow selection techniques
We can distinguish the following selection techniques:
1. based on flow record content (i.e. all reported flow
characteristics);
2. based on flow record arrival time;
3. based on external events like the exhaustion of local resources.
6.1. Flow selection based on flow record content
Flow selection can be done based on fields in an IPFIX flow record.
This can be done similarly to field match filtering for packet
selection described in [PSAMP-TECH]. The difference is that instead
of packet fields flow record fields are here used to drive the
selection decision. An example would be to select flow records based
on parameters like the flow size in bytes or the number of packets.
Another application would be to select flow records based on either
flow start time or flow keys (IP addresses, ports) of the stored flow
record.
6.2. Flow selection based on flow record arrival time or sequence
Flow records can be selected based on their arrival time at the
exporting process. An example would be to select a number of flow
records during certain time intervals. Another option is to select
flow records based on the order they arrive at the exporting process.
In such case, one can either periodically select the k-th records or
select randomly a set of flow records.
6.3. Flow selection on external events
The selection of flow records can be also triggered by external
events. An example would be to activate flow selection based on the
value of a router state parameter like the number of entries in the
flow cache.
Peluso, et al. Expires June 4, 2011 [Page 13]
Internet-Draft Flow Selection Techniques December 2010
7. Information model for flow selection information exporting
In this section we define the elements devoted to containing the
information described in the previous sections. Some elements have
been associated with a pair of timestamps, which are referred to as
Tfirst and Tlast (instead of element_nameTfirst, element_nameTlast).
Note that all the following information elements are aimed at
describing macro flows parameters (e.g. the total number of packets
and bytes contained in all dropped or not created flow records).
Some of these macro flows parameters are additive, in the sense that
it is only possible to add contributions to them, but never subtract
some amount. For example, the macro flow of the packets contained in
flow records that are discarded from the flow reporting process
receives an addition when a flow record is discarded, and this
addition can never be subtracted. On the other side, some macro flow
parameters can dynamically receive and loose additions. For example,
the macro flows of packets not yet exported receives an addition when
a new packet arrives, and looses addition as an exporting event takes
place. Associating a timestamp to the oldest and most recent
additions in case of additive flows is an easy task, while it is
complicate for the others(it would require to maintain full state
information) and that is why we did not define timestamps for these
information elements.
The information elements herein introduced are defined in accordance
with the IPFIX information model [RFC5102]. Furthermore, the data
types used to represent the Flow Selection-related information
elements are those defined in section 3.1 of the IPFIX information
model [RFC 2051].
List of additional Flow Selection information elements:
Peluso, et al. Expires June 4, 2011 [Page 14]
Internet-Draft Flow Selection Techniques December 2010
+-------+------------------------------------+
| ID | Name |
+-------+------------------------------------+
| TBD1 | FsMeter_UnmeasPacketCount |
+-------+------------------------------------+
| TBD2 | FsMeter_UnmeasBytesCount |
+-------+------------------------------------+
| TBD3 | FsFrec_PacketInDroppedRecsCount |
+-------+------------------------------------+
| TBD4 | FsFrec_ByteInDroppedRecsCount |
+-------+------------------------------------+
| TBD5 | FsFrec_FrecDroppedCount |
+-------+------------------------------------+
| TBD6 | FsFrec_UnexportedFrecCount |
+-------+------------------------------------+
| TBD7 | FsFrec_UnexportedPacketInFrecCount |
+-------+------------------------------------+
| TBD8 | FsRec_UnexportedBytesInFrecCount |
+-------+------------------------------------+
| TBD9 | FsExp_PacketInDroppedRecsCount |
+-------+------------------------------------+
| TBD10 | FsExp_BytesInDroppedRecsCount |
+-------+------------------------------------+
| TBD11 | FsExp_FrecDroppedCount |
+-------+------------------------------------+
| TBD12 | FsExp_UnexportedCount |
+-------+------------------------------------+
| TBD13 | FsExp_UnexportedPacketCount |
+-------+------------------------------------+
| TBD14 | FsExp_UnexportedByteInExpCount |
+-------+------------------------------------+
7.1. Meter process related (TBD1-TBD2)
Information Elements presented in this section are related to Flow
Selection performed during the Metering Process.
+------+---------------------------+
| ID | Name |
+------+---------------------------+
| TBD1 | FsMeter_UnmeasPacketCount |
+------+---------------------------+
| TBD2 | FsMeter_UnmeasBytesCount |
+------+---------------------------+
Peluso, et al. Expires June 4, 2011 [Page 15]
Internet-Draft Flow Selection Techniques December 2010
7.1.1. FsMeter_UnmeasPacketCount
Specifies the count of packets that were not measured because of the
use of the flow state dependent sampling, in terms of:
TsFirst: timestamp of the first packet not measured because of the
use of theflow state dependent sampling (Type: dateTime)
TsLast: timestamp of the last packet not measured because of the use
of the flow state dependent sampling (Type: dataTime)
7.1.2. FsMeter_UnmeasBytesCount
Description:
This Information Element specifies the count of bytes that were
not measured because of the use of the flow state dependent
sampling
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD2
Status: Proposed
Units: bytes
7.2. Flow recording process related (TBD3-TBD8)
Information Elements presented in this section are related to Flow
Selection performed during the Flow Recording Process.
+------+------------------------------------+
| ID | Name |
+------+------------------------------------+
| TBD3 | FsFrec_PacketInDroppedRecsCount |
+------+------------------------------------+
| TBD4 | FsFrec_ByteInDroppedRecsCount |
+------+------------------------------------+
| TBD5 | FsFrec_FrecDroppedCount |
+------+------------------------------------+
| TBD6 | FsFrec_UnexportedFrecCount |
+------+------------------------------------+
| TBD7 | FsFrec_UnexportedPacketInFrecCount |
+------+------------------------------------+
Peluso, et al. Expires June 4, 2011 [Page 16]
Internet-Draft Flow Selection Techniques December 2010
+------+------------------------------------+
| TBD8 | FsFrec_UnexportedBytesInFrecCount |
+------+------------------------------------+
7.2.1. FsFrec_PacketInDroppedRecsCount
Specifies the count of the non-exported packets that were contained
in the flow records eliminated from the flow recording process
because of resource limitations/policies in the flow recording
process. It is defined in terms of:
TsFirst: timestamp of the first non-exported packet belonging to an
eliminated flow record (Type: dateTime)
TsLast: timestamp of the last non-exported packet belonging to an
eliminated flow record (Type: dateTime)
7.2.2. FsFrec_ByteInDroppedRecsCount
Description:
This Information Element specifies the count of the non-exported
bytes that were contained in the flow records eliminated from the
flow recording process because of resource limitations/policies in
the flow recording process.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD4
Status: Proposed
Units: bytes
7.2.3. FsFrec_FrecDroppedCount
Specifies the count of flow records containing non-exported packets
eliminated from the flow recording process because of resources
limitations/policies in the flow recording process. It is defined in
terms of:
TsFirst: timestamp of the first flow record elimination event
occurring during the flow recording process (Type: dateTime)
TsLast: timestamp of the last flow record elimination event occurring
during the flow recording process (Type: dateTime)
Peluso, et al. Expires June 4, 2011 [Page 17]
Internet-Draft Flow Selection Techniques December 2010
7.2.4. FsFrec_UnexportedFrecCount
Description:
This Information Element specifies the count of the flow records
currently existing in the flow recording process containing at
least one non-exported packet.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD6
Status: Proposed
Units: flow records
7.2.5. FsFrec_UnexportedPacketInFrecCount
Description:
This Information Element specifies the count of non-exported
packets contained in flow records of the flow recording process.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD7
Status: Proposed
Units: packets
7.2.6. FsFrec_UnexportedBytesInFrecCount
Description:
This Information Element specifies the count of non-exported bytes
contained in flow records of the flow recording process.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD8
Peluso, et al. Expires June 4, 2011 [Page 18]
Internet-Draft Flow Selection Techniques December 2010
Status: Proposed
Units: bytes
7.3. Flow exporting process related (TBD9-TBD14)
Information Elements presented in this section are related to Flow
Selection performed during the Flow Exporting Process.
+-------+--------------------------------+
| ID | Name |
+-------+--------------------------------+
| TBD9 | FsExp_PacketInDroppedRecsCount |
+-------+--------------------------------+
| TBD10 | FsExp_ByteInDroppedRecsCount |
+-------+--------------------------------+
| TBD11 | FsExp_FrecDroppedCount |
+-------+--------------------------------+
| TBD12 | FsExp_UnexportedCount |
+-------+--------------------------------+
| TBD13 | FsExp_UnexportedPacketCount |
+-------+--------------------------------+
| TBD14 | FsExp_UnexportedByteInExpCount |
+-------+--------------------------------+
7.3.1. FsExp_PacketInDroppedRecsCount
Specifies the count of non-exported packets that were contained in
the flow records eliminated from the flow recording process because
of resource limitations/policies in the exporting process. It is
defined in terms of:
TsFirst: timestamp of the first non-exported packet belonging to an
eliminated flow record (Type: dateTime)
TsLast: timestamp of the last non-exported packet belonging to an
eliminated flow record (Type: dateTime)
7.3.2. FsExp_ByteInDroppedRecsCount
Description:
This Information Element specifies the count of non-exported bytes
that were contained in the flow records eliminated from the flow
recording process because of resource limitations/policies in the
exporting process.
Abstract Data Type: unsigned64
Peluso, et al. Expires June 4, 2011 [Page 19]
Internet-Draft Flow Selection Techniques December 2010
Data Type Semantics: quantity
ElementId: TBD10
Status: Proposed
Units: bytes
7.3.3. FsExp_FrecDroppedCount
Specifies the count of flow records containing non-exported packets
eliminated from the flow recording process because of resource
limitations/policies in the exporting process. It is defined in
terms of:
TsFirst: timestamp of the first flow record elimination event
occurring during the flow recording process (Type: dateTime)
TsLast: timestamp of the last flow record elimination event occurring
during the flow recording process (Type: dateTime)
7.3.4. FsExp_UnexportedCount
Description:
This Information Element specifies the count of the flow records
currently existing in the flow recording process, containing non-
exported traffic and not being exported because of exporting
process resource limitations/policies.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD12
Status: Proposed
Units: flow records
7.3.5. FsExp_UnexportedPacketCount
Description:
This Information Element specifies the count of non-exported
packets contained in the flow records of the flow recording
process not being exported because of exporting process resource
limitations/policies.
Peluso, et al. Expires June 4, 2011 [Page 20]
Internet-Draft Flow Selection Techniques December 2010
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD13
Status: Proposed
Units: packets
7.3.6. FsExp_UnexportedByteInExpCount
Description:
This Information Element specifies the count of non-exported bytes
contained in the flow records of the flow recording process not
being exported because of exporting process resource limitations/
policies.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD14
Status: Proposed
Units: bytes
8. Implementation requirements
In order to implement the described information model counters for
non-exported packets and bytes have to be inserted in the flow
records as flow metrics. Sometimes these counters are referred to as
delta counts. An implementation may also keep absolute counts for
purposes not being specified in this information model (both delta
and absolute counters can be exported in the IPFIX information model,
see [RFC5102]). In addition, to fully support this information
model, it would be required to keep in a flow record the timestamps
of the first and last non-exported packets. An implementation may
need to keep timestamps of the first and last exported packets for
other purposes than those of this information model, or to join the
two timers for the last exported and first exported packets, or even
to approximate them with the time of the exporting event.
Peluso, et al. Expires June 4, 2011 [Page 21]
Internet-Draft Flow Selection Techniques December 2010
9. Information Model for Configuration of Flow Selection Techniques
This section aims at describing the representative parameters of the
flow selection techniques presented above. To this regard, it
provides the basis of an information model to be adopted in order to
configure the flow selection process within an IPFIX device. The
information elements herein introduced are defined in accordance with
the IPFIX information model [RFC5102]. Furthermore, the data types
used to represent the Flow Selection-related information elements are
those defined in section 3.1 of the IPFIX information model [RFC
2051].
List of additional Flow Selection information elements:
+-------+-------------------------+-------+-----------------------+
| ID | Name | ID | Name |
+-------+-------------------------+-------+-----------------------+
| TBD15 | selectorMethod | TBD18 | flowRecordPacketsSize |
+-------+-------------------------+-------+-----------------------+
| TBD16 | flowMaxAdmitFlowRecords | TBD19 | flowInactivityTime |
+-------+-------------------------+-------+-----------------------+
| TBD17 | flowRecordBytesSize | | |
+-------+-------------------------+-------+-----------------------+
9.1. selectorMethod
Description:
This Information Element identifies the flow selection method that
is applied by the Flow Selection process, in accordance with what
is described in section 5 of this document.
Some of these methods may have parameters in order to fully
support the selected technique. For that reason, further
Information Elements are defined in the following subsections.
Flow selection methods identifiers are herein defined:
+----+-----------------+--------------------------------------------+
| ID | Method | Parameters |
+----+-----------------+--------------------------------------------+
| 1 | Selection based | flowMaxAdmitFlowRecords |
| | on flow size | flowRecordBytesSize flowRecordPacketsSize |
| | count | |
+----+-----------------+--------------------------------------------+
Peluso, et al. Expires June 4, 2011 [Page 22]
Internet-Draft Flow Selection Techniques December 2010
+----+-----------------+--------------------------------------------+
| 2 | Selection based | flowMaxAdmitFlowRecords |
| | on flow content | flowIPAddressVersion flowSourceIPv4Address |
| | property match | flowDestinationIPv4Address |
| | | flowSourceIPv6Address |
| | | flowDestinationIPv6Address |
| | | flowDestinationPort |
+----+-----------------+--------------------------------------------+
| 3 | Selection based | flowMaxAdmitFlowRecords flowInactivityTime |
| | on flow record | |
| | arrival time or | |
| | sequence | |
+----+-----------------+--------------------------------------------+
| 4 | Selection based | flowMaxAdmitFlowRecords |
| | on external | |
| | events | |
+----+-----------------+--------------------------------------------+
Abstract Data Type: unsigned16
Data Type Semantics: identifier
ElementId: TBD15
Status: Proposed
9.2. flowMaxAdmitFlowRecords
Description:
This Information Element specifies the maximum number of eligible
flow records which might be created in the flow cache. It is used
by the Selector Process in order to identify the time when flow
selection should be triggered. A value of 0 means that the Flow
Selection State related to the memory space available for flow
recording must be used to estimate the max flow cache size.
For example, this Information Element may be used to set the
configuration of a flow size count Flow Selector.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD16
Status: Proposed
Peluso, et al. Expires June 4, 2011 [Page 23]
Internet-Draft Flow Selection Techniques December 2010
Units: flow records
9.3. flowRecordBytesSize
Description:
This Information Element specifies the minimum number of bytes
contained in a flow record to be considered as not eligible for
removal. It may be used for elephant flows identification.
For example, this Information Element may be used to describe the
configuration of a flow size count Flow Selector.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD17
Status: Proposed
Units: bytes
9.4. flowRecordPacketsSize
Description:
This Information Element specifies the minimum number of packets
contained in a flow record to be considered as not eligible for
removal. It may be used for elephant flows identification.
For example, this Information Element may be used to describe the
configuration of a flow size count Flow Selector.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD18
Status: Proposed
Units: packets
9.5. flowInactivityTime
Description:
Peluso, et al. Expires June 4, 2011 [Page 24]
Internet-Draft Flow Selection Techniques December 2010
This Information Element specifies the time interval in
microseconds during which the corresponding flow record may be
considered as still active. It is used by the metering process
and/or the flow recording process in order to make the decision on
whether to discard an existing flow to make room for a new one.
For example, this Information Element may be used to describe the
configuration of a flow arrival time Flow Selector.
Abstract Data Type: dateTimeMicroseconds
Data Type Semantics: quantity
ElementId: TBD19
Status: Proposed
Units: microseconds
9.6. flowIPAddressVersion
Description:
This Information Element specifies the IP version field of the
packets belonging to flow records which should be considered as
not eligible for removal.
Abstract Data Type: unsigned8
Data Type Semantics: identifier
ElementId: 60
Status: current
Reference: see [RFC5102] for the definition of the ipVersion
information element and its Id.
9.7. flowIPv4SourceAddress
Description:
This Information Element specifies the IPv4 source address which
is the parameter to be used as flow key to access the flow record
not eligible for removal.
Abstract Data Type: ipv4Address
Peluso, et al. Expires June 4, 2011 [Page 25]
Internet-Draft Flow Selection Techniques December 2010
Data Type Semantics: identifier
ElementId:8
Status: current
Reference: see [RFC5102] for the definition of the sourceIPv4Address
information element and its Id.
9.8. flowIPv4DestinationAddress
Description:
This Information Element specifies the IPv4 destination address
which is the parameter to be used as flow key to access the flow
record not eligible for removal.
Abstract Data Type: ipv4Address
Data Type Semantics: identifier
ElementId:12
Status: current
Reference: see [RFC5102] for the definition of the
destinationIPv4Address information element and its Id.
9.9. flowIPv6SourceAddress
Description:
This Information Element specifies the IPv6 source address which
is the parameter to be used as flow key to access the flow record
not eligible for removal.
Abstract Data Type: ipv6Address
Data Type Semantics: identifier
ElementId:27
Status: current
Reference: see [RFC5102] for the definition of the sourceIPv6Address
information element and its Id.
Peluso, et al. Expires June 4, 2011 [Page 26]
Internet-Draft Flow Selection Techniques December 2010
9.10. flowIPv6DestinationAddress
Description:
This Information Element specifies the IPv6 destination address
which is the parameter to be used as flow key to access the flow
record not eligible for removal.
Abstract Data Type: ipv6Address
Data Type Semantics: identifier
ElementId:28
Status: current
Reference: see [RFC5102] for the definition of the
destinationIPv6Address information element and its Id.
10. IANA Considerations
This document introduces several new information elements as an
extension to the IPFIX information model. Values TBD1-TBD19 in this
document should be replaced with the assigned numbers by IANA.
11. References
11.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
11.2. Informative References
[DuLT01a] Duffield, N., Lund, C., and M. Thorup, "Charging from
Sampled Network Usage", ACM Internet Measurement Workshop
IMW 2001, San Francisco, USA, November 2001.
[DuLT01b] Duffield, N., Lund, C., and M. Thorup, "Properties and
Prediction of Flow Statistics from Sampled Packet
Streams", ACM SIGCOMM Internet Measurement Workshop 2002,
November 2002.
[DuLT01c] Duffield, N., Lund, C., and M. Thorup, "Learn More, sample
less: control of volume and variance in network
measurement", IEEE Transactions on Information Theory,
Peluso, et al. Expires June 4, 2011 [Page 27]
Internet-Draft Flow Selection Techniques December 2010
May 2005.
[DuLT01d] Duffield, N., Lund, C., and M. Thorup, "Flow Sampling
under Hard Resource Constraints", ACM IFIP Conference on
Measurement and Modeling of Computer Systems SIGMETRICS,
June 2004.
[EsVa01] Estan, C. and G,. Varghese, "New Directions in Traffic
Measurement and Accounting: Focusing on the Elephants,
Ignoring the Mice", ACM SIGCOMM Internet Measurement
Workshop 2001, San Francisco (CA), November 2001.
[FeGL98] Feldmann, A., Rexford, J., and R. Caceres, "Efficient
Policies for Carrying Web Traffic over Flow-Switched
Networks", IEEE/ACM Transaction on Networking,
December 1998.
[IPFIX-ARCH]
Sadasivan, G., Bownlee, N., Claise, B., and J. Quittek,
"Architecture for IP Flow Information Export", Internet
Draft draft-ietf-ipfix-architecture-12.txt, work in
progress, September 2006.
[IPFIX-REQ]
Quittek, J., Zseby, T., Claise, B., and S. Zander,
"Requirements for IP Flow Information Export",
RFC3917 Requirements for IP Flow Information Export
(IPFIX), July 2008.
[KuXW04] Kumar, K., Xu, J., Wang, J., Spatschek, O., and L. Li,
"Space-code bloom filter for efficient per-flow traffic
measurement", INFOCOM 2004 Twenty-third AnnualJoint
Conference of the IEEE Computer and Communications
Societies, March 2004.
[Moli03] Molina, M., "A scalable and efficient methodology for flow
monitoring in the Internet", International Teletraffic
Congress (ITC-18), Berlin, September 2003.
[PSAMP-TECH]
Zseby, T., Molina, M., Raspall, F., Duffield, N., and S.
Niccolini, "Sampling and Filtering techniques for IP
Packet Selection", Internet
Draft draft-ietf-psamp-sample-tech-11.txt, work in
progress, July 2008.
[RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
Meyer, "Information Model for IP Flow Information Export",
Peluso, et al. Expires June 4, 2011 [Page 28]
Internet-Draft Flow Selection Techniques December 2010
RFC 5102, January 2008.
Authors' Addresses
Lorenzo Peluso
University of Napoli
Via Claudio 21
Napoli 80125
Italy
Phone: +39 081 7683821
Email: lorenzo.peluso@unina.it
Tanja Zseby
Fraunhofer Institute FOKUS
Kaiserin-Augusta-Allee 31
Berlin 10589
Germany
Phone: +49 30 3463 7153
Email: tanja.zseby@fokus.fraunhofer.de
Salvatore D'Antonio
CINI Consortium/University of Napoli "Parthenope"
Monte S.Angelo, Via Cinthia
Napoli 80126
Italy
Phone: +39 081 679944
Email: saldanto@unina.it
Christian Henke
Fraunhofer Institute FOKUS
Kaiserin-Augusta-Allee 31
Berlin 10589
Germany
Phone: +49 30 3463 7366
Email: christian.henke@fokus.fraunhofer.de
Peluso, et al. Expires June 4, 2011 [Page 29]
