Internet Engineering Task Force S. D'Antonio
Internet-Draft CINI Consortium/University of
Intended status: Standards Track Napoli "Parthenope"
Expires: November 24, 2011 T. Zseby
Fraunhofer Institute FOKUS
C. Henke
Technische Universitat Berlin
L. Peluso
University of Napoli
May 23, 2011
Flow Selection Techniques
draft-ietf-ipfix-flow-selection-tech-06.txt
Abstract
Flow selection is the process of selecting a subset of flows from all
flows observed at an observation point. Flow selection reduces the
effort of post-processing flow data and transferring flow records.
This document describes motivations for flow selection and presents
flow selection techniques. It provides an information model for
configuring flow selection techniques and discusses what information
about a flow selection process should be exported.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 24, 2011.
D'Antonio, et al. Expires November 24, 2011 [Page 1]
Internet-Draft Flow Selection Techniques May 2011
Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
D'Antonio, et al. Expires November 24, 2011 [Page 2]
Internet-Draft Flow Selection Techniques May 2011
Table of Contents
1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Difference between Flow Selection and Packet Selection . . . . 6
4. Flow selection as Function in the IPFIX Architecture . . . . . 7
4.1. Flow selection in the Metering Process before
Aggregation . . . . . . . . . . . . . . . . . . . . . . . 9
4.2. Flow selection in the Metering Process after
Aggregation . . . . . . . . . . . . . . . . . . . . . . . 9
4.3. Flow selection during the Exporting Process . . . . . . . 9
4.4. Flow selection as a function of the IPFIX Mediator . . . . 10
5. Flow Selection Techniques . . . . . . . . . . . . . . . . . . 10
5.1. Flow Filtering . . . . . . . . . . . . . . . . . . . . . . 10
5.1.1. Property Match Filtering . . . . . . . . . . . . . . . 10
5.1.2. Hash-based Flow Filtering . . . . . . . . . . . . . . 11
5.1.3. Flow State Dependent Flow Filtering . . . . . . . . . 11
5.2. Flow Sampling . . . . . . . . . . . . . . . . . . . . . . 12
5.2.1. Systematic Sampling . . . . . . . . . . . . . . . . . 12
5.2.2. Random Sampling . . . . . . . . . . . . . . . . . . . 12
5.3. Flow-state Dependent Packet Selection . . . . . . . . . . 13
6. Information Model for Configuration of Flow Selection
Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.1. Description of Flow Filtering Techniques . . . . . . . . . 15
6.2. Description of Flow Sampling Techniques . . . . . . . . . 16
6.3. Description of Flow State Dependent Packet Selection . . . 17
7. Information Model for Flow Selection Reporting . . . . . . . . 18
7.1. fsFlowRecordTotalCount . . . . . . . . . . . . . . . . . . 19
7.2. fsFlowRecordSelectedCount . . . . . . . . . . . . . . . . 19
7.3. fsCurrentFlowEntries . . . . . . . . . . . . . . . . . . . 19
7.4. fsMaxFlowEntries . . . . . . . . . . . . . . . . . . . . . 20
7.5. fsFlowEntryTotalCount . . . . . . . . . . . . . . . . . . 20
7.6. fsFlowEntrySelectedCount . . . . . . . . . . . . . . . . . 20
7.7. fsPacketTotalCount . . . . . . . . . . . . . . . . . . . . 21
7.8. fsFlowEntrySelectedCount . . . . . . . . . . . . . . . . . 21
7.9. fsOctetTotalCount . . . . . . . . . . . . . . . . . . . . 21
7.10. fsOctetSelectedCount . . . . . . . . . . . . . . . . . . . 22
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
9. Security Considerations . . . . . . . . . . . . . . . . . . . 22
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23
10.1. Normative References . . . . . . . . . . . . . . . . . . . 23
10.2. Informative References . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24
D'Antonio, et al. Expires November 24, 2011 [Page 3]
Internet-Draft Flow Selection Techniques May 2011
1. Scope
This document describes flow selection techniques for traffic
measurements. A flow is defined as a set of packets with common
properties as described in [RFC5101]. Flow selection can be done to
limit the resource demands for capturing, storing, exporting and
post-processing of flow records. It also can be used to select a
particular set of flows that are of interest to a specific
application. This document provides a cathegorization of flow
selection techniques and describes configuration and reporting
parameters for them. In order to be compliant with this document, at
least one of proposed flow selection schemes MUST be implemented.
That means that the configuration parameters as well as the reporting
information elements for this particular scheme MUST be supported.
This document also addresses configuration and reporting parameters
for flow-state dependent packet selection as described in [RFC5475],
although the technique is categorized as packet selection. The
reason is, thta flow-state dependent packet selection techniques
often aim at the reduction of resources for flow capturing and flow
processing. Furthermore, they were only briefly discussed in
[RFC5475]. Therefore we included configuration and reporting
considerations for such techniques in this document.
2. Terminology
This document is consistent with the terminology introduced in
[RFC5101], [RFC5470], [RFC5475] and [RFC3917]. As in [RFC5101] and
[RFC5476], the first letter of each IPFIX-specific and PSAMP-specific
term is capitalized along with the flow selection specific terms
defined here.
* Classification
Classification is a process in which packets are mapped to
specific flow records based on packet properties. These
properties make up the flow key (e.g. header information, packet
content, AS number). In case a flow record for a specific flow
key already exists the flow record is updated, otherwise a new
flow record is created.
* Flow Selection Process
A Flow Selection Process takes classified packets, flow cache
entries or flow records as its input and selects a subset of that
set as its output. A Flow Selection Process MAY run on several
instances within the IPFIX architecture. A Flow Selection Process
D'Antonio, et al. Expires November 24, 2011 [Page 4]
Internet-Draft Flow Selection Techniques May 2011
MAY be part of an IPFIX metering process, exporting process or as
an Intermediate Selection Process running on an IPFIX Mediator.
* Flow Selection State
A Flow Selection Process SHOULD maintain state information for use
by the Flow Selector. At a given time, the Flow Selection State
may depend on flows and packets observed at and before that time,
as well as other variables. Examples include:
(i) sequence number of packets and accounted flow records;
(ii) number of selected flows;
(iii) number of observed flows;
(iv) current flow cache occupancy;
(v) flow specific counters, lower und upper bounds
(vi) flow selection timeout intervals
* Flow Selector
A Flow Selector defines the action of a Flow Selection Process on
a single flow of its input. The Flow Selector can make use of the
following information in order to establish whether a flow has to
be selected or not:
(i) the content of the flow record;
(ii) any state information related to the metering or exporting
process;
(iii) any Flow Selection State that may be maintained by the
Flow Selection Process.
* Complete Flow
A Complete Flow consists of all packets within the flow time-Out
interval that enter the Flow Selection Process and belong to the
same flow as defined by the flow definition. For this definition
only packets are considered that arrive at the Flow Selection
Process. That means, packets that are not observed at the Flow
Selection Process because of prior packet selection or packet loss
are not considered as belonging to the Complete Flow.
* Flow Filtering
D'Antonio, et al. Expires November 24, 2011 [Page 5]
Internet-Draft Flow Selection Techniques May 2011
Flow Filtering selects flows based on a deterministic function on
the flow record content, flow state, external properties (e.g.
ingress interface) or external events (e.g violated Access Control
List). If the relevant parts of the flow record content can be
already observed at packet level (e.g. flow keys from packet
header fields) Flow Filtering can be performed at packet level by
property match packet filtering as described in [RFC5475].
* Flow Sampling
Flow Sampling selects flows based on flow record sequence or
arrival times (e.g. position in flow cache, arrival time at
exporter or mediator). The selection can be systematic (e.g.
every n-th record) or based on a random functions (e.g. select
each record with probability p, or randomly select n out of N
records).
* Aggregation Process
In the IPFIX metering process the aggregation process aggregates
packet data into flow data and forms the flow cache entries or
flow records. After the aggregation step only the aggregated flow
information is available. Information about individual packets is
lost.
3. Difference between Flow Selection and Packet Selection
Flow selection differs from packet selection described in [RFC5475].
Packet selection techniques consider packets as basic element and the
parent population consists of all packets observed at an observation
point. In contrast to this the basic elements in flow selection are
the flows. The parent population consists of all observed flows and
the selection process operates on the flows. The major
characteristics of flow selection are the following:
- Flow selection takes flows as basic elements. For packet
selection, packets are considered as basic elements.
- Flow selection can only take place after classification,
because the classification rules determine to which flow a
packet belongs. Packet selection can be applied before or
after classification.
- Flow selection operates on complete flows. That means that
after the flow selection process either all packets of the
flow are kept or all packets of the flow are discarded. All
packets of the flow here means all packets that enter the
D'Antonio, et al. Expires November 24, 2011 [Page 6]
Internet-Draft Flow Selection Techniques May 2011
flow selection process. That means that if the flow
selection is preceded by a packet selection process the
complete flow consists only of the packets that where not
discarded during the packet selection.
There are some techniques that are difficult to unambiguously
categorize into one of the categories. We here give some guidance
how to categorize such techniques:
- Techniques that can be considered as both, packet and flow
selection: Some packet selection techniques result in the
selection of whole flows and therefore can be considered as
packet or as flow selection at the same time. An example is
property match filtering of all packets to a specific
destination address. If flows are defined based on
destination addresses, such a packet selection also results
in a flow selection and can be considered as packet or flow
selection.
- Flow-state dependent packet selection (as described in
[RFC5475]): There exist techniques that select packets based
on the flow state, e.g. based on the number of already
observed packets belonging to the flow. Examples of these
techniques from the literature are "Sample and Hold" [EsVa01]
"Fast Filtered Sampling" [MSZC10] or the "Sticky Sampling"
algorithm presented in [MaMo02]. Such techniques can be used
to influence which flows are captured (e.g. increase the
selection of packets belonging to large flows) and reduce the
number of flows that need to be stored in the flow cache.
Nevertheless, such techniques do not necessarily select
Complete Flows, because it is not ensured that all packets of
a selected flow are captured. Therefore flow-state dependent
packet selection methods that do not ensure that either all
or no packets of a flow are selected strictly speaking have
to be considered as packet selection technique and not as
flow selection.
4. Flow selection as Function in the IPFIX Architecture
Figure 1 shows the IPFIX reference model as defined in [RFC5470], and
extends it by introducing the functional components where flow
selection can take place.
Packet(s) coming in to Observation Point(s)
| |
v v
+----------------+---------------------------+ +-----+-------+
D'Antonio, et al. Expires November 24, 2011 [Page 7]
Internet-Draft Flow Selection Techniques May 2011
| Metering Process | | |
| | | |
| packet header capturing | | |
| | |...| Metering |
| timestamping | | Process N |
| | | | |
| packet selection | | |
| | | | |
| classification | | |
| | | | |
| flow state dependent packet selection | | |
| | | | |
| flow selection before aggregation (*) | | |
| | | | |
| aggregation | | |
| | | | |
| flow selection after aggregation (*) | | |
+--------|-----------------------------------+ +-----|-------+
Flow Records Flow Records
| |
+----------------------+----------------------+
|
+----------------------|-----------------+
| Exporting Process | |
| v |
| flow selection before export(*) |
| | |
| v |
| flow export |
+----------------------+-----------------+
| IPFIX (Flow Records)
v
+-------------------------|-----------------------+
| IPFIX Mediator | |
| v |
| Collecting Process(es) |
| | |
| Intermediate Flow Selection Process (*) |
| | |
| Exporting Process(es) |
+-------------------------|-----------------------+
v
IPFIX
(*) indicates where flow selection can take place.
Figure 1: Flow selection in the IPFIX Architecture
D'Antonio, et al. Expires November 24, 2011 [Page 8]
Internet-Draft Flow Selection Techniques May 2011
In contrast to packet selection, flow selection is always applied
after the packets are classified into flows. Flows can be selected
at different stages of the measurement chain:
1. during Metering Process before aggregation
2. during Metering Process after aggregation;
3. during Exporting Process
4. in an Intermediate Selection Process on a Mediator
4.1. Flow selection in the Metering Process before Aggregation
In the aggregation process the packet information is used to update
the flow entries in the flow cache. Flow selection that is applied
before aggregation equals a packet selection process. The flow still
consists of individual packets. Those are then selected based on the
classification information, i.e. based on the flow they belong to.
Flow selection before aggregation can be based on the fields of the
flow key (also on a hash value over these fields), but not based on
characteristics that are only available after aggregation (e.g. flow
size, flow duration). Flow selection before aggregation is applied
to reduce resources for all succeeding processes (aggregation,
exporting process) or select specific flows of interest in case such
flow characteristics are already observable at packet level (e.g.
flows to specific IPs). In contrast, flow state dependent packet
selection is a packet selection method, because it does not
necessarily select Complete Flows. Flow selection before aggregation
and flow state dependent packet selection can be applied in arbitrary
order.
4.2. Flow selection in the Metering Process after Aggregation
Flow selection after aggregation is usually applied to reduce the
flows to those that are of interest to a particular application and
to unload flow export and flow postprocessing. Since the flow cache
entries are already generated by the aggregation process flow
selection after aggregation can also depend on flow characteristics
that are only visible after the aggregation of packets, such as flow
size and flow duration.
4.3. Flow selection during the Exporting Process
The Exporting Process may implement policies for exporting only a
subset of the flow records which have been stored in the system
memory. Flow selection in the exporting process may select only the
subset of flow records which are of interest to the users
D'Antonio, et al. Expires November 24, 2011 [Page 9]
Internet-Draft Flow Selection Techniques May 2011
application, or select only as many flow records than can be handled
by the available resources ( e.g. limited flow cache size and export
link capacity).
4.4. Flow selection as a function of the IPFIX Mediator
As shown in Figure 1, flow selection can be performed as an
intermediate process within an IPFIX Mediator [RFC6183]. The
Intermediate Selection Process takes a flow record stream as its
input and retrieves a record stream. The Intermediate Selection
Process can again apply a flow selection technique to obtain flows of
interest for the application. Further the Intermediate Selection
Process can base its selection decision on the correlation of data
from different observation points, e.g by only selecting flows that
were at least recorded on two observation points.
5. Flow Selection Techniques
A flow selection technique selects either all packets or none of a
flow, otherwise the technique has to be considered as packet
selection. We distinguish between Flow Filtering and Flow Sampling.
5.1. Flow Filtering
Flow Filtering is a deterministic function on the IPFIX flow record
content. In case that the relevant flow characteristics are already
observable at packet level (e.g. flow keys) Flow Filtering can be
applied before aggregation at packet level.
5.1.1. Property Match Filtering
Flow Filtering can be done similarly to Property Match Filtering for
packet selection described in [RFC5475]. The difference is that,
instead of packet fields, flow record fields are here used to derive
the selection decision. Property Match Filtering is typically used
to select a specific subset of the flows that are of interest to a
particular application (e.g. all flows to a specific destination, all
large flows, etc.). Properties on which the filtering is based can
be for example flow keys, the flow size in bytes, the number of
packets in the flow, the observation time of the first or last
packet, or the maximum packet length. The selection criteria can be
a specific value or an interval. Property match filtering can be
applied before aggregation in case the properties are already
observable at the packet level (e.g. flow key fields).
There are content based Property Match filtering techniques that
require a compution on the current flow cache. An example is the
D'Antonio, et al. Expires November 24, 2011 [Page 10]
Internet-Draft Flow Selection Techniques May 2011
selection of the k largest flows or a percentage of flows with the
longest livetime. This type of Property Match Filtering is also used
in flow selection techniques that react on external events (e.g.
resource constraint). For example in case the flow cache is full,
the flow cache entry with the lowest flow volume per current flow
live time is deleted.
5.1.2. Hash-based Flow Filtering
Hash-based Flow Filtering uses a Hash Function h to map the flow key
c onto a Hash Range R. A flow is selected if the hash value h(c) is
within the Hash Selection Range S, which is a subset of R. Hash-based
Flow Filtering can be used to emulate a random sampling process but
still enable the correlation between selected flow subset at
different observation points. Hash-based Flow Filtering is similar
to Hash-based Packet Selection, and in fact is identical when Hash-
based Packet Selection uses the flow key that define the flow as the
Hash Input. Nevertheless there MAY be the incentive to apply Hash-
based Flow Selection not on the packet level before aggregation, for
example when the size of the Selection Range and therefore the
sampling probability is dependent on the number of observed flows.
5.1.3. Flow State Dependent Flow Filtering
Flow state dependent filtering does not base the selection decision
on fields of the current flow record content but on the flow state
which may be kept additionally for each of the flows. External
processes may update counters, bounds and timers for each of the flow
records and the flow selection process utilises this information for
the selection decision. A review of flow state dependent filtering
techniques that aim at the selection of the most frequent items by
keeping additional flow state information can be found in [CoHa08].
Flow state dependent flow filtering can only be applied after
aggregation, when a packet has been assigned to a flow cache. The
selection process then decides based upon the flow state for each
flow if it is kept in the flow cache or not. Two flow dependent flow
filtering techniques are here described:
The Frequent Algorithm [KaPS03] is a technique that aims at the
selection of all flows that at least exceed a 1/k fraction of the
observed packet stream. The algorithm has only a flow cache of size
k-1 and each flow in the cache has an additional counter. The
counter is incremented each time a packet belonging to the flow in
the flow cache is observed. In case the observed packet does not
belong to any flow all counters are decremented and if any of the
flow counters have a value of zero the flow is replaced with the new
flow.
D'Antonio, et al. Expires November 24, 2011 [Page 11]
Internet-Draft Flow Selection Techniques May 2011
Lossy Counting is a selection technique that identifies all flows
whose packet count exceeds a certain percentage of the whole observed
packet stream (e.g. 5% of all packets) with a certain estimation
error e. Lossy Counting seperates the observed packet stream in
windows of size N=1/e, where N is an amount of consecutive packets.
For each observed flow an additional counter will be held in the flow
state. The counter is incremented each time a packet belonging to
the flow is observed and all counters are decremented at the end of
each window and all flows with a counter of zero will be removed from
the flow cache.
5.2. Flow Sampling
Flow sampling operates on flow record sequence or arrival times. It
can use a systematic or a random functions for the selection process.
Flow sampling usually aims at the selection of a representative
subset of all flows in order to estimate characteristics of the whole
set (e.g. mean flow size in the network).
5.2.1. Systematic Sampling
Systematic sampling is a deterministic selection function.
Systematic sampling may be a periodic selection of the k-th flow
record which arrives at the exporting or mediator process.
Systematic Sampling can also be applied before aggregation. An
example would be to use an additional data structure that saves the
flow keys of the not selected flows. Then one can create a flow
cache entry for the k-th observed packet that has yet no flow cache
entry and is not within the data structure containing the not
selected flows.
Systematic sampling can also be time-based. Systematic Sampling is
applied by only creating flows that are observed between time-based
start and stop triggers. The time interval may be applied at packet
level or after aggregation level, e.g. by selecting every k seconds a
flow arriving at the export process.
5.2.2. Random Sampling
Random flow sampling is based on a random process which requires the
calculation of random numbers. One can differentiate between n-out-N
and probabilistic sampling. The sampling probability of individual
flows records MAY be adjusted according to the flow record content or
external events like the available export resources. Non-uniform
random sampling approaches can be applied similar to the ones defined
in [RFC5475]. An example would be to prefer large volume flows over
small volume flows. Random flow sampling can also be applied before
aggregation when additional flow state about non selected flows is
D'Antonio, et al. Expires November 24, 2011 [Page 12]
Internet-Draft Flow Selection Techniques May 2011
kept.
5.3. Flow-state Dependent Packet Selection
As explained above Flow-state Dependent Packet Selection is not a
Flow Selection Technique but a packet selection. Nevertheless we
will describe configuration and reporting parameters for this
technique in this document. An example is the the "Sample and Hold"
algorithm [EsVa01] that tries to prefer large volume flows in the
selection. When a packet arrives it is selected when already a flow
cache entry for this packet exists. In case there is no flow cache
entry, the packet is selected by a certain probability that is
dependent on the packet size.
6. Information Model for Configuration of Flow Selection Techniques
This section describes the configuration parameters of the flow
selection techniques presented above. It provides the basis of an
information model to be adopted in order to configure the flow
selection process within an IPFIX device. The following table gives
an overview of the defined selection techniques, where they can be
applied and what are their input parameters. Dependent on where the
flow selection techniques are applied different input parameters can
be configured.
Overview of Flow Selection Techniques:
D'Antonio, et al. Expires November 24, 2011 [Page 13]
Internet-Draft Flow Selection Techniques May 2011
+------------------+-----------------+------------------------------+
| Location | Selection | Selection Input |
| | Method | |
+------------------+-----------------+------------------------------+
| before | Flow State | packet sampling |
| aggregation | Dependent | probabilities, flow state, |
| | Packet | packet properties |
| | Selection | |
+------------------+-----------------+------------------------------+
| | Property Match | flow key fields, filter |
| | Flow Filtering | function |
+------------------+-----------------+------------------------------+
| | Hash-Based Flow | selection range, hash |
| | Filtering | function, flow key |
+------------------+-----------------+------------------------------+
| | Time-based | flow position (derived from |
| | Systematic Flow | arrival time of packets), |
| | Sampling | flow state |
+------------------+-----------------+------------------------------+
| | Sequence-based | flow position (derived from |
| | Systematic Flow | packet position), flow state |
| | Sampling | |
+------------------+-----------------+------------------------------+
| | Random Flow | random number generator or |
| | Sampling | list and packet position, |
| | | flow state |
+------------------+-----------------+------------------------------+
| after | Property Match | flow record content, filter |
| aggregation | Flow Filtering | function |
+------------------+-----------------+------------------------------+
| | Hash-Based Flow | selection range, hash |
| | Filtering | function, hash input (flow |
| | | keys and other flow |
| | | properties) |
+------------------+-----------------+------------------------------+
| | Flow State | flow state parameters |
| | Dependent Flow | |
| | Selection | |
+------------------+-----------------+------------------------------+
| | Time-based | flow arrival time, flow |
| | Systematic Flow | state |
| | Sampling | |
+------------------+-----------------+------------------------------+
| | Sequence-based | flow position, flow state |
| | Systematic Flow | |
| | Sampling | |
+------------------+-----------------+------------------------------+
D'Antonio, et al. Expires November 24, 2011 [Page 14]
Internet-Draft Flow Selection Techniques May 2011
+------------------+-----------------+------------------------------+
| | Random Flow | random number generator or |
| | Sampling | list and flow position, flow |
| | | state |
+------------------+-----------------+------------------------------+
| during Exporting | Property Match | flow record content, filter |
| Process or in | Flow Filtering | function |
| the Mediator | | |
+------------------+-----------------+------------------------------+
| | Hash-Based Flow | selection range, hash |
| | Filtering | function, flow key |
+------------------+-----------------+------------------------------+
| | Time-based | flow record arrival time |
| | Systematic Flow | |
| | Sampling | |
+------------------+-----------------+------------------------------+
| | Sequence-based | flow record position |
| | Systematic Flow | |
| | Sampling | |
+------------------+-----------------+------------------------------+
| | Random Flow | random number generator or |
| | Sampling | list and flow position |
+------------------+-----------------+------------------------------+
| | Flow State | flow state parameters |
| | Dependent Flow | |
| | Selection | |
+------------------+-----------------+------------------------------+
A flow selection configuration consists of FS_SELECTOR_ID, FS_TYPE,
FS_SELECTOR PARAMETERS.
FS_SELECTOR ID: Unique ID for the flow sampler
FS_TYPE: Defines which algorithm is used.
FS_SELECTOR_PARAMETERS: Defines the input parameter for the flow
selection methods
6.1. Description of Flow Filtering Techniques
In this section, we define what elements are needed to describe the
most common Flow Filtering techniques.
D'Antonio, et al. Expires November 24, 2011 [Page 15]
Internet-Draft Flow Selection Techniques May 2011
+----------------+----------------------------------------+
| FS_SELECTOR_ID | FS_TYPE |
+----------------+----------------------------------------+
| 1 | fs_property_matching |
+----------------+----------------------------------------+
| 2 | fs_hashing |
+----------------+----------------------------------------+
| 3 | fs_flow_state_dependent_flow_selection |
+----------------+----------------------------------------+
FS_SELECTOR_PARAMETERS:
case fs_property_matching:
- Information Element (from [RFC5102])
- Value or Value Interval
case fs_hashing:
- Hash Domain (input bits from packet) - can be specified for IPv4
or IPv6 or both
- Hash Function Name
- Hash Selection Range
- optional parameters (e.g. random seed)
case fs_flow_state_dependent_flow_selection:
- accuracy paramter
- frequency threshold (in per cent of observed packets)
The above list of parameters for flow dependent flow selection
techniques is suitable for the presented Frequent Item and Lossy
Counting Algorithm. Nevertheless there exist a variety of techniques
with very specific parameters which are not defined here.
6.2. Description of Flow Sampling Techniques
In this section, we define what elements are needed to describe the
most common Flow Sampling techniques.
D'Antonio, et al. Expires November 24, 2011 [Page 16]
Internet-Draft Flow Selection Techniques May 2011
+----------------+---------------------------+
| FS_SELECTOR_ID | FS_TYPE |
+----------------+---------------------------+
| 5 | fs_systematic_count-based |
+----------------+---------------------------+
| 5 | fs_systematic_time-based |
+----------------+---------------------------+
| 6 | fs_n-out-of-N |
+----------------+---------------------------+
| 7 | fs_probabilistic |
+----------------+---------------------------+
FS_SELECTOR_PARAMETERS:
case systematic count-based:
- Interval length (number of new observed flows)
- Spacing (number of new observed flows)
case fs_systematic_time-based:
- Interval length (in usec)
- Spacing (in usec)
case fs_random n-out-of-N:
- Population Size N
- Sample size n
case fs_probabilistic:
- Sampling probability p
6.3. Description of Flow State Dependent Packet Selection
The configuration of flow dependent packet selection has not been
described in [RFC5475] therefore the paramaters are defined here:
SELECTOR_TYPE: flow_dependent_packet_selection
SELECTOR_PARAMETERS:
D'Antonio, et al. Expires November 24, 2011 [Page 17]
Internet-Draft Flow Selection Techniques May 2011
- packet selection probability per possible flow state interval
- additional parameters (e.g. packet properties as in [EsVa01])
7. Information Model for Flow Selection Reporting
In this section we describe Information Elements (IEs) that SHOULD be
exported by a flow selection process in order to support the
interpretation of measurement results from flow measurements where
only some flows are selected. The information is mainly used to
report how many packets and flows have been observed in total and how
many of them where selected. This helps for instance to calculate
the attained sampling fraction, which is an important parameter to
provide an accuracy statement. The IEs can provide reporting
information about flow records, flow cache entries, packets or bytes.
The reported metrics are number of total and the number of selected
elements. From this the number of dropped elements can be derived.
All counters are delta counters and SHOULD be exported and reset when
a new measurement interval starts. Additional IEs may be useful for
future flow selection techniques. Those can be defined additionally
if needed.
List of additional Flow Selection information elements:
+-------+---------------------------+
| ID | Name |
+-------+---------------------------+
| TBD1 | fsFlowRecordTotalCount |
+-------+---------------------------+
| TBD2 | fsFlowRecordSelectedCount |
+-------+---------------------------+
| TBD3 | fsCurrentFlowEntries |
+-------+---------------------------+
| TBD4 | fsMaxFlowEntries |
+-------+---------------------------+
| TBD5 | fsFlowEntryTotalCount |
+-------+---------------------------+
| TBD6 | fsFlowEntrySelectedCount |
+-------+---------------------------+
| TBD7 | fsPacketTotalCount |
+-------+---------------------------+
| TBD8 | fsPacketSelectedCount |
+-------+---------------------------+
| TBD9 | fsOctetTotalCount |
+-------+---------------------------+
| TBD10 | fsOctetSelectedCount |
+-------+---------------------------+
D'Antonio, et al. Expires November 24, 2011 [Page 18]
Internet-Draft Flow Selection Techniques May 2011
7.1. fsFlowRecordTotalCount
Description:
This Information Element specifies the current number of all Flow
Records that form the parent population as input to the Flow
Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD1
Status: Proposed
Units: Flow Records
7.2. fsFlowRecordSelectedCount
Description:
This Information Element specifies the current number Flow Records
that were selected during the Flow Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD2
Status: Proposed
Units: Flow Records
7.3. fsCurrentFlowEntries
Description:
This Information Element specifies the current number of flow
entries in the flow cache.
Abstract Data Type: unsigned64
ElementId: TBD3
Status: Proposed
Units: Flow Entries
D'Antonio, et al. Expires November 24, 2011 [Page 19]
Internet-Draft Flow Selection Techniques May 2011
7.4. fsMaxFlowEntries
Description:
This Information Element specifies the maximum number of flow
entries in the flow cache.
Abstract Data Type: unsigned64
ElementId: TBD4
Status: Proposed
Units: Flow Entries
7.5. fsFlowEntryTotalCount
Description:
This Information Element specifies the current number of all Flow
Entries that form the parent population as input to the Flow
Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD5
Status: Proposed
Units: Flow Entries
7.6. fsFlowEntrySelectedCount
Description:
This Information Element specifies the current number Flow entries
that were selected during the Flow Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD6
Status: Proposed
Units: Flow Entries
D'Antonio, et al. Expires November 24, 2011 [Page 20]
Internet-Draft Flow Selection Techniques May 2011
7.7. fsPacketTotalCount
Description:
This Information Element specifies the current number of packets
in all flows that form the parent population as input to the Flow
Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD7
Status: Proposed
Units: Packets
7.8. fsFlowEntrySelectedCount
Description:
This Information Element specifies the current number packets in
all flows that were selected during the Flow Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD8
Status: Proposed
Units: Packets
7.9. fsOctetTotalCount
Description:
This Information Element specifies the current number of all bytes
in all flows that form the parent population as input to the Flow
Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD9
Status: Proposed
Units: Bytes
D'Antonio, et al. Expires November 24, 2011 [Page 21]
Internet-Draft Flow Selection Techniques May 2011
7.10. fsOctetSelectedCount
Description:
This Information Element specifies the current number of bytes in
all flows that were selected during the Flow Selection Process.
Abstract Data Type: unsigned64
ElementId: TBD10
Status: Proposed
Units: Bytes
8. IANA Considerations
This document introduces several new information elements as an
extension to the IPFIX information model. Values TBD1-TBD10 in this
document should be replaced with the assigned numbers by IANA.
9. Security Considerations
In this section security issues concerning an IPFIX device performing
flow selection are pointed out. In case the flow selection function
is activated an IPFIX device might be exposed to security threats.
Since flow selection implies analysing flow packets, associating them
to a specific traffic flow and selecting flow records, a malicious
user who was able to gain control of an IPFIX device might access
both packet and flow data, thus violating their confidentiality.
Furthermore, the intruder might be attracted by the possibility of
altering the flow selection process by modifying the criteria used to
select flow records. In this case, the IPFIX device would export
flow data which are different from the ones that the Collector
expects to receive.
It is apparent that these security threats can be mitigated by
authenticating entities that interact with the IPFIX device and
keeping information for flow selection configuration confidential.
10. References
D'Antonio, et al. Expires November 24, 2011 [Page 22]
Internet-Draft Flow Selection Techniques May 2011
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
10.2. Informative References
[CoHa08] Cormode, G. and M. Hadjieleftheriou, "Finding frequent
items in data streams", Journal, Proceedings of the Very
Large DataBase Endowment VLDB Endowment, Volume 1 Issue 2,
August 2008, August 2008.
[DuLT01a] Duffield, N., Lund, C., and M. Thorup, "Charging from
Sampled Network Usage", ACM Internet Measurement Workshop
IMW 2001, San Francisco, USA, November 2001.
[DuLT01b] Duffield, N., Lund, C., and M. Thorup, "Properties and
Prediction of Flow Statistics from Sampled Packet
Streams", ACM SIGCOMM Internet Measurement Workshop 2002,
November 2002.
[EsVa01] Estan, C. and G,. Varghese, "New Directions in Traffic
Measurement and Accounting: Focusing on the Elephants,
Ignoring the Mice", ACM SIGCOMM Internet Measurement
Workshop 2001, San Francisco (CA), November 2001.
[KaPS03] Karp, R., Papadimitriou, C., and S. S. Shenker, "A simple
algorithm for finding frequent elements in sets and
bags.", ACM Transactions on Database Systems, Volume 28,
51-55, 2003, March 2003.
[KuXW04] Kumar, K., Xu, J., Wang, J., Spatschek, O., and L. Li,
"Space-code bloom filter for efficient per-flow traffic
measurement", INFOCOM 2004 Twenty-third AnnualJoint
Conference of the IEEE Computer and Communications
Societies, March 2004.
[MSZC10] Mai, J., Sridharan, A., Zang, H., and C. Chuah, "Fast
Filtered Sampling", Computer Networks Volume 54, Issue 11,
Pages 1885-1898, ISSN 1389-1286, January 2010.
[MaMo02] Manku, G. and R. Motwani, "Approximate Frequency Counts
over Data Streams", Proceedings of the Internation
Conference on Very large DataBases (VLDB) pages 346--357,
2002, Hong Kong, China, 2002.
[Moli03] Molina, M., "A scalable and efficient methodology for flow
monitoring in the Internet", International Teletraffic
D'Antonio, et al. Expires November 24, 2011 [Page 23]
Internet-Draft Flow Selection Techniques May 2011
Congress (ITC-18), Berlin, September 2003.
[RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander,
"Requirements for IP Flow Information Export (IPFIX)",
RFC 3917, October 2004.
[RFC5101] Claise, B., "Specification of the IP Flow Information
Export (IPFIX) Protocol for the Exchange of IP Traffic
Flow Information", RFC 5101, January 2008.
[RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
Meyer, "Information Model for IP Flow Information Export",
RFC 5102, January 2008.
[RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
"Architecture for IP Flow Information Export", RFC 5470,
March 2009.
[RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F.
Raspall, "Sampling and Filtering Techniques for IP Packet
Selection", RFC 5475, March 2009.
[RFC5476] Claise, B., Johnson, A., and J. Quittek, "Packet Sampling
(PSAMP) Protocol Specifications", RFC 5476, March 2009.
[RFC6183] Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi,
"IP Flow Information Export (IPFIX) Mediation: Framework",
RFC 6183, April 2011.
Authors' Addresses
Salvatore D'Antonio
CINI Consortium/University of Napoli "Parthenope"
Monte S.Angelo, Via Cinthia
Napoli 80126
Italy
Phone: +39 081 679944
Email: salvatore.dantonio@parthenope.it
D'Antonio, et al. Expires November 24, 2011 [Page 24]
Internet-Draft Flow Selection Techniques May 2011
Tanja Zseby
Fraunhofer Institute FOKUS
Kaiserin-Augusta-Allee 31
Berlin 10589
Germany
Phone: +49 30 3463 7153
Email: tanja.zseby@fokus.fraunhofer.de
Christian Henke
Technische Universitat Berlin
Strasse des 17. Juni 135
Berlin 10623
Germany
Phone: +49 30 3463 7366
Email: c.henke@tu-berlin.de
Lorenzo Peluso
University of Napoli
Via Claudio 21
Napoli 80125
Italy
Phone: +39 081 7683821
Email: lorenzo.peluso@unina.it
D'Antonio, et al. Expires November 24, 2011 [Page 25]
