Internet Engineering Task Force L. Peluso
Internet-Draft University of Napoli
Intended status: Standards Track T. Zseby
Expires: December 3, 2010 Fraunhofer Institute FOKUS
S. D'Antonio
CINI Consortium/University of
Napoli "Parthenope"
M. Molina
DANTE
June 01, 2010
Flow Selection Techniques
draft-ietf-ipfix-flow-selection-tech-02.txt
Abstract
Flow selection is the process of selecting a subset of flows from all
flows observed at an observation point. The objective of flow
selection is to reduce the effort for post-processing flow data and
for transferring flow records. The flow selection process can be
enabled at different stages of the measurement process. It can be
applied directly after classification or at recording/exporting time
by limiting the number of flows to be stored and/or exported to the
collecting process. This document describes motivations for flow
selection and presents flow selection techniques. It furthermore
provides an information model for configuring flow selection
techniques and discusses what information about a flow selection
process is beneficial to be exported by adopting a suitable
information model.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
Peluso, et al. Expires December 3, 2010 [Page 1]
Internet-Draft Flow Selection Techniques June 2010
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 3, 2010.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
Peluso, et al. Expires December 3, 2010 [Page 2]
Internet-Draft Flow Selection Techniques June 2010
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Flow selection as a function of the IPFIX Exporter . . . . . . 5
4.1. Flow selection in the metering process . . . . . . . . . . 7
4.2. Flow selection in the flow recording process . . . . . . . 7
4.3. Flow selection in the exporting process . . . . . . . . . 8
5. Flow selection as a function of the IPFIX Mediator . . . . . . 9
6. Flow selection techniques . . . . . . . . . . . . . . . . . . 11
6.1. Flow selection based on flow record content . . . . . . . 11
6.2. Flow selection based on flow record arrival time or
sequence . . . . . . . . . . . . . . . . . . . . . . . . . 11
6.3. Flow selection on external events . . . . . . . . . . . . 11
7. Information model for flow selection information exporting . . 12
7.1. Meter process related (TBD1-TBD2) . . . . . . . . . . . . 13
7.1.1. FsMeter_UnmeasPacketCount . . . . . . . . . . . . . . 14
7.1.2. FsMeter_UnmeasBytesCount . . . . . . . . . . . . . . . 14
7.2. Flow recording process related (TBD3-TBD8) . . . . . . . . 14
7.2.1. FsFrec_PacketInDroppedRecsCount . . . . . . . . . . . 15
7.2.2. FsFrec_ByteInDroppedRecsCount . . . . . . . . . . . . 15
7.2.3. FsFrec_FrecDroppedCount . . . . . . . . . . . . . . . 15
7.2.4. FsFrec_UnexportedFrecCount . . . . . . . . . . . . . . 16
7.2.5. FsFrec_UnexportedPacketInFrecCount . . . . . . . . . . 16
7.2.6. FsFrec_UnexportedBytesInFrecCount . . . . . . . . . . 16
7.3. Flow exporting process related (TBD9-TBD14) . . . . . . . 17
7.3.1. FsExp_PacketInDroppedRecsCount . . . . . . . . . . . . 17
7.3.2. FsExp_ByteInDroppedRecsCount . . . . . . . . . . . . . 17
7.3.3. FsExp_FrecDroppedCount . . . . . . . . . . . . . . . . 18
7.3.4. FsExp_UnexportedCount . . . . . . . . . . . . . . . . 18
7.3.5. FsExp_UnexportedPacketCount . . . . . . . . . . . . . 18
7.3.6. FsExp_UnexportedByteInExpCount . . . . . . . . . . . . 19
8. Requirements put on implementations . . . . . . . . . . . . . 19
9. Information Model for Configuration of Flow Selection
Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 20
9.1. selectorMethod . . . . . . . . . . . . . . . . . . . . . . 20
9.2. flowMaxAdmitFlowRecords . . . . . . . . . . . . . . . . . 21
9.3. flowRecordBytesSize . . . . . . . . . . . . . . . . . . . 21
9.4. flowRecordPacketsSize . . . . . . . . . . . . . . . . . . 22
9.5. flowInactivityTime . . . . . . . . . . . . . . . . . . . . 22
10. Security Considerations . . . . . . . . . . . . . . . . . . . 23
11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23
12.1. Normative References . . . . . . . . . . . . . . . . . . . 23
12.2. Informative References . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25
Peluso, et al. Expires December 3, 2010 [Page 3]
Internet-Draft Flow Selection Techniques June 2010
1. Introduction
This document describes flow selection techniques for traffic
measurements. As stated in [PSAMP-TECH], packet selection is the
process of selecting a subset of packets. The element on which this
selection mechanism is performed is a packet and the selection
decision is based on packet properties. In contrast to this, flow
selection techniques consider flows as the basic elements on which a
selection process is performed In the IPFIX architecture the element
on which the selection process is performed is the IPFIX flow record.
For several applications it makes sense to select only the flows of
interest in case resources are scarce. Examples are accounting or
attack detection applications. Maintaining and exporting all flow
records to the collecting process would increase resource demands
with the result that data is randomly discarded. A better solution
would be to export only a representative subset of flows. Another
example of application which would benefit from the capability of
selecting only the flows of interest is accounting In many networks
few large flows contribute to the majority of the overall traffic
volume [DuLT01a], [DuLT01b]. This phenomenon is also referred to as
"Quasi-Zipf-Law" [KuXW04] or as "elephant and mice phenomenon". For
accounting purposes it could be useful to concentrate on the so-
called "heavy hitter" flows to cope with a limited flow cache size or
limited transmission capacity in times when resources are scarce.
2. Scope
This document describes flow selection techniques and their
parameters. It addresses the configuration of flow selection
techniques and defines which information should be reported by
devices that perform flow selection. It only describes processes
directly acting on traffic flows during the metering phase and/or the
exporting phase. Therefore it is assumed that flow selection is
performed after packets are classified into flows. This document
does not address the flow selection effects that might result from
the sampling or filtering of packets in the metering process before
the classification process is performed. Such packet selection
techniques are described in [PSAMP-TECH] and, therefore, outside the
scope of this document.
3. Terminology
This document uses the terminology introduced in [IPFIX-ARCH] and
[PSAMP-TECH] In this section, some additional terms are presented
which extend the terminology introduced in [PSAMP-TECH].
Peluso, et al. Expires December 3, 2010 [Page 4]
Internet-Draft Flow Selection Techniques June 2010
* Flow Selection Process
A Flow Selection Process takes a set of Flow Records as its input
and selects a subset of that set as its output.
* Flow Selection State
A Flow Selection Process may maintain state information for use by
the Flow Selection Process. At a given time, the Flow Selection
State may depend on flows observed at and before that time, and
other variables. Examples include:
(i) number of accounted flow records;
(ii) memory space available for flow recording;
(iii) state of the pseudorandom number generators;
(iv) hash values calculated during selection.
* Flow Selector
A Flow Selector defines the action of a Flow Selection Process on
a single flow of its input. The Flow Selector can make use of the
following information in determining whether a flow is selected:
(i) the content of the flow record;
(ii) any information state related to the flow recording;
(iii) any selection state that may be maintained by the Flow
Selection Process.
4. Flow selection as a function of the IPFIX Exporter
Figure 1 shows the IPFIX reference model as defined in [IPFIX-ARCH],
and extends it by introducing the functional components where flow
selection can take place.
Peluso, et al. Expires December 3, 2010 [Page 5]
Internet-Draft Flow Selection Techniques June 2010
Packet(s) coming in to Observation Point(s)
| |
v v
+----------------+---------------------------+ +-----+-------+
| Metering Process on an | | |
| Observation Point | | |
| | | |
| packet header capturing | | |
| | |...| Metering |
| timestamping | | Process N |
| | | | |
| packet selection | | |
| | | | |
| classification | | |
| | | | |
| flow state dependent packet sampling (*) | | |
| | | | |
| aggregation | | |
| | | | |
| flow recording (*) | | |
| | | | |
| | Timing out Flows | | |
| | Handle resource overloads | | |
+--------|-----------------------------------+ +-----|-------+
| |
Flow Records (selected by Observation Domain) Flow Records
| |
+----------------------+----------------------+
|
+----------------------|---------------+
| Exporting Process v |
| +---------------+-----------+ |
| | flow export (*) | |
| +---------------+-----------+ |
| | |
+----------------------+---------------+
|
v
IPFIX export packet to Collector
(*) indicates where flow selection can take place.
Figure 1: Flow selection as a function of the IPFIX Exporter
In contrast to packet selection, flow selection is always applied
after the packets are classified into flows. Flows can be selected
at different stages of the measurement chain:
Peluso, et al. Expires December 3, 2010 [Page 6]
Internet-Draft Flow Selection Techniques June 2010
1. during metering [PSAMP-TECH];
2. during flow recording;
3. during flow export exporting.
4.1. Flow selection in the metering process
The main reason for applying flow state dependent sampling during the
metering process is that flow recording process may not have, at a
certain point in time, enough memory positions to record all
observable flows. Another reason may be that there might not be
enough processing resources to create and manage a new flow record.
To overcome these limitations, a number of possible policies can be
applied, the simplest one being to discard new packets which cannot
be assigned to existing flow records (i.e. that would require the
creation of a new flow record). More complex policies are however
possible, mainly aimed at detecting the so called elephant flows,
i.e. to prioritize flows carrying higher traffic volume in the flow
recording process . For instance, [EsVa01] proposes criteria to
define a packet eligible to create a new flow record (sample and
hold, multistage filters). Regardless of specific algorithms, we are
concerned about identifying what information about the flow state
dependent packet sampling is worth keeping and making available to
applications (by exporting it out of an IPFIX device). An option
could be to keep a cumulative counter of the total number of packets
and bytes that were not considered for measurement because of flow
state dependent sampling. Furthermore, it is possible to keep a
timestamp for the first and last of these discarded packets. In
practice, this implies aggregating all these packets in a single
macro flow, and keeping track of its volume and duration. Storing
more detailed information about packets which have not been measured
because of flow state dependent sampling would contradict the fact
that the sampling is done because of lack of memory and/or processing
resources.
4.2. Flow selection in the flow recording process
As described in the previous section, because of lack of memory
positions in the flow recording process some incoming packets might
be discarded if they lead to the creation of a new flow record.
However, under certain circumstances, it may be advantageous to
discard an existing flow record during the flow recording process in
order to make room for a new one which has been created at the
arrival of a new packet. For example, an algorithm for making the
decision whether to discard the new arriving packet or an existing
flow record is described in [Moli03]. In this section we focus on
the selection of the information to be stored concerning the record
Peluso, et al. Expires December 3, 2010 [Page 7]
Internet-Draft Flow Selection Techniques June 2010
removal rather than on the details of the decision making algorithm.
For the reasons we mentioned above, it does not make sense to store
separate information for each discarded flow record, as it would
contradict the motivation why discarding is done (i.e. lack of memory
resources). The information that can be kept with a limited overhead
is the cumulative counter of the total number of not yet exported
packets and bytes belonging to flow records that were removed during
the flow recording process. Ideally, we would like to keep also a
timestamp for the first (T_fd) and last (T_ld) not yet exported
packets belonging to every discarded flow record. This would mean
aggregating all these packets in a macro flow, and keeping track of
its volume and duration. To do so, we would need to maintain a
timestamp for the first and last non-exported packets in each flow
record, check the values of such timestamps whenever a record is
discarded in order to verify whether they are smaller or larger than
T_fd and T_ld, respectively, and if so update them. Another
information that can be easily maintained is the number of discarding
actions, along with the timestamps of the first and last action.
This information should not be used by applications to re-normalize
their received per flow statistics (because a flow may be discarded
and re-created multiple times) but rather to monitor and control the
good functioning of the implemented policy. Note that we consider a
discarding event only when the discarded flow record contains data
about traffic which has not been exported. Otherwise, the removal of
a record whose traffic was exported (after a timeout or after the
arrival of specific packets, e.g. TCP FIN or RST) is part of the
normal functioning of an IPFIX flow metering system. Note also that
we consider only the case when an elimination of a flow record during
the flow recording process leads to the complete loss of all the
information contained in the flow record itself. If another policy
is implemented, such as immediate exporting of the flow record before
elimination, or freezing of the flow record and moving it in another
area of memory for later exporting, this case is not considered as an
elimination and therefore is out of the scope of this document.
Along with the information about the number of discarded flow records
and associated packets and bytes, it is useful to keep cumulative
information about the number of flow records containing not yet
exported traffic and being currently handled by the flow recording
process, as well as the cumulative number of not exported packets and
bytes contained in them.
4.3. Flow selection in the exporting process
The exporting process may implement policies for exporting only a
subset of the flow records which have been stored in the system
memory. The decision to export only a subset of the flow records can
be motivated by the existence of an explicit policy which filters out
the flow records to be exported. An example of such a policy could
Peluso, et al. Expires December 3, 2010 [Page 8]
Internet-Draft Flow Selection Techniques June 2010
be to export only the flow records associated to flows whose
accounted traffic is below a certain threshold, or a more complex
mechanism such as the one described in [DuLT01a] or [DuLT01b].
Another motivation which might bring to the exporting of a subset of
stored flow records is resource limitation. For example, the
exporting process has been assigned a limited time slot to operate or
it exports only a predefined number of packets. Hybrid cases can
happen where the exporting of a subset of the flow records is
motivated by the co-existence of resource limitations and ad-hoc
policies which are applied in order to optimize the exporting process
(e.g. given that the exporting process applies to a subset of the
flow records, such subset is selected so that the overall number of
exported packets and bytes belonging to the subset is maximized).
Selecting flow records during the exporting process raises the issue
of identifying the information which is worth keeping about the flow
selection process. Two different scenarios cab be envisaged. If a
flow record is not exported and then it does not feed the flow
recording process, the scenario is the same as when the deletion of
the flow record is caused by the need to make room to another record.
The metrics to be kept are cumulative packets and bytes associated
with not exported flow records, timestamps of the first and last
packets belonging to non exported flow records, counter of dropping
events and timestamp of first and last dropping event. If a record
eligible for exporting is not exported and it enters the flow
recording process it has a chance of being exported in the future.
It would be beneficial for an application to get information, in
terms of number of packets and bytes about the flow records which are
not being exported due to the existence of exporting policies and/or
resource limitations. This, is intended to make it possible to
detect possible pathologic conditions, like the missing exporting of
a large number of flow records and/or associated traffic, or the
growing number of flow records being involved in the flow recording
process. The selection of the flow records to be exported implies
performing a complete scanning of the memory area where flow
information is stored, thus jeopardizing the efficiency of the
overall exporting process. For this reason, flow exporting protocol
specification does not include as flow selection during the recording
process as a mandatory function even if the information model has
been designed to enable such function.
5. Flow selection as a function of the IPFIX Mediator
As shown in Figure 2, flow selection can be performed as an
intermediate process within an IPFIX Mediator. This process selects
the flow records from a sequence which meet pre-defined criteria and
exports them to an IPFIX Collector. This selection function can be
seen as a more fine-grained process with respect to the selection
Peluso, et al. Expires December 3, 2010 [Page 9]
Internet-Draft Flow Selection Techniques June 2010
performed by an IPFIX Exporter. The criteria used to drive the
selection process at Mediator's level might be applied to the set of
flow records coming from the IPFIX Exporter, thus triggering a
further flow selection process.
Packet(s) coming in to Observation Point(s)
|
IPFIX Original |
Exporter v
+------------------+-------------------+
| |
| Metering Process on an |
| Observation Point |
| | |
| Flow metering and selection |
| | |
| Flow recording and selection |
| | |
| Flow exporting and selection |
| | |
+------------------+-------------------+
|
v
Flow Records (selected by Observation Domain)
|
IPFIX Mediator v
+------------------+-------------------+
| |
| Collecting process |
| | |
| Flow selection (*) |
| | |
| Exporting process |
| | |
+------------------+-------------------+
|
v
Flow Records
Figure 2: Flow selection as a function of the IPFIX Mediator
As an example, if an IPFIX Mediator interacts with a set of IPFIX
Collectors, flow records arriving at the IPFIX Mediator might be
selected based on the IPFIX Collector requesting flow information.
As described in previous sections, flow selection can take place
during metering, recording, and exporting processes of an IPFIX
exporter depending on the policies which are implemented to meet
Peluso, et al. Expires December 3, 2010 [Page 10]
Internet-Draft Flow Selection Techniques June 2010
application requirements. In case flow selection is performed at
Mediator's level, we envisage the use of flow selection techniques as
a step of the exporting process aimed to identify the flow records to
be exported among those stored in the system's memory. This is
because the lighter is the intermediate selection process the better
is the performance of the mediation framework.
6. Flow selection techniques
We can distinguish the following selection techniques:
1. based on flow record content (i.e. all reported flow
characteristics);
2. based on flow record arrival time;
3. based on external events like the exhaustion of local resources.
6.1. Flow selection based on flow record content
Flow selection can be done based on fields in an IPFIX flow record.
This can be done analogous to field match filtering for packet
selection described in [PSAMP-TECH]. The difference here is that
instead of packets here field of the flow record content are used for
the selection decision. An example would be to select flow records
with regard to the flow size in bytes or number of packets. Another
application would be to select flow records based on flow start time
or on flow keys (IP addresses, ports) of the stored flow record.
6.2. Flow selection based on flow record arrival time or sequence
Flow records can be selected based on their arrival time at the
exporting process. An example would be to select a number of flow
records for certain periods of time. Another option is to select
flow records based on the order at which they arrive at the exporting
process. With this one can select systematically every kth record or
select randomly a set of flow records.
6.3. Flow selection on external events
The selection of flow records can be also triggered by external
events. An example would be router state like number of entries in
flow cache.
Peluso, et al. Expires December 3, 2010 [Page 11]
Internet-Draft Flow Selection Techniques June 2010
7. Information model for flow selection information exporting
We formally define the elements to contain the information described
in the previous section. Some elements have an associated couple of
timestamps, which we reference for brevity (when it is not ambiguous)
as Tfirst and Tlast (instead of element_nameTfirst,
element_nameTlast). Note that all the following information elements
are aimed at describing macro flows (e.g. the total number of packets
and bytes contained in all dropped or not created flow records).
Some of these macro flows are additive only, in the sense that they
only add contributions to them, but never subtract. E.g. the macro
flow of the packets contained in flow records that are discarded from
the flow reporting process receives a contribution when a flow record
is discarded, and this contribution can never be subtracted. On the
contrary, some of the macro flows can dynamically receive and loose
contributions. E.g. the macro flows of packets not yet exported
receives a contribution when a new packets arrives, and looses some
contribution when there is an exporting event. Associating a
timestamp for the oldest and most recent contributions to additive
only flow is easy, while for the others is not (would require to
maintain full state) and that is why we did not define timestamps for
these information elements.
The information elements here introduced are defined in accordance
with the IPFIX information model [RFC5102] to which reference should
be made for more detailed information. Furthermore, the data types
used to formally rappresent the Flow Selection related information
elements are those defined in section 3.1 of the IPFIX information
model [RFC 2051]. For that reason, they are not redefined in this
section.
List of additional Flow Selection information elements:
Peluso, et al. Expires December 3, 2010 [Page 12]
Internet-Draft Flow Selection Techniques June 2010
+-------+------------------------------------+
| ID | Name |
+-------+------------------------------------+
| TBD1 | FsMeter_UnmeasPacketCount |
+-------+------------------------------------+
| TBD2 | FsMeter_UnmeasBytesCount |
+-------+------------------------------------+
| TBD3 | FsFrec_PacketInDroppedRecsCount |
+-------+------------------------------------+
| TBD4 | FsFrec_ByteInDroppedRecsCount |
+-------+------------------------------------+
| TBD5 | FsFrec_FrecDroppedCount |
+-------+------------------------------------+
| TBD6 | FsFrec_UnexportedFrecCount |
+-------+------------------------------------+
| TBD7 | FsFrec_UnexportedPacketInFrecCount |
+-------+------------------------------------+
| TBD8 | FsRec_UnexportedBytesInFrecCount |
+-------+------------------------------------+
| TBD9 | FsExp_PacketInDroppedRecsCount |
+-------+------------------------------------+
| TBD10 | FsExp_BytesInDroppedRecsCount |
+-------+------------------------------------+
| TBD11 | FsExp_FrecDroppedCount |
+-------+------------------------------------+
| TBD12 | FsExp_UnexportedCount |
+-------+------------------------------------+
| TBD13 | FsExp_UnexportedPacketCount |
+-------+------------------------------------+
| TBD14 | FsExp_UnexportedByteInExpCount |
+-------+------------------------------------+
7.1. Meter process related (TBD1-TBD2)
Information Elements in this section are related to Flow Selection at
the Matering Process.
+------+---------------------------+
| ID | Name |
+------+---------------------------+
| TBD1 | FsMeter_UnmeasPacketCount |
+------+---------------------------+
| TBD2 | FsMeter_UnmeasBytesCount |
+------+---------------------------+
Peluso, et al. Expires December 3, 2010 [Page 13]
Internet-Draft Flow Selection Techniques June 2010
7.1.1. FsMeter_UnmeasPacketCount
Contains the count of packets that were not measured because of flow
state dependent sampling, in terms of:
TsFirst: timestamp of the first packet not measured because of flow
state dependent sampling (Type: dateTime)
TsLast: timestamp of the last packet not measured because of flow
state dependent sampling (Type: dataTime)
7.1.2. FsMeter_UnmeasBytesCount
Description:
This Information Elements contains the count of bytes that were
not measured because of flow state dependent sampling
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD2
Status: Proposed
Units: bytes
7.2. Flow recording process related (TBD3-TBD8)
Information Elements in this section are related to Flow Selection at
the Flow Recording Process if present.
+------+------------------------------------+
| ID | Name |
+------+------------------------------------+
| TBD3 | FsFrec_PacketInDroppedRecsCount |
+------+------------------------------------+
| TBD4 | FsFrec_ByteInDroppedRecsCount |
+------+------------------------------------+
| TBD5 | FsFrec_FrecDroppedCount |
+------+------------------------------------+
| TBD6 | FsFrec_UnexportedFrecCount |
+------+------------------------------------+
| TBD7 | FsFrec_UnexportedPacketInFrecCount |
+------+------------------------------------+
| TBD8 | FsFrec_UnexportedBytesInFrecCount |
+------+------------------------------------+
Peluso, et al. Expires December 3, 2010 [Page 14]
Internet-Draft Flow Selection Techniques June 2010
7.2.1. FsFrec_PacketInDroppedRecsCount
Contains the count of non exported packets that were contained in
flow records eliminated from the flow recording process because of
resource limitations/policies in the flow recording process. It is
defined in terms of:
TsFirst: timestamp of the first non-exported packet belonging to a
eliminated flow record (Type: dateTime)
TsLast: timestamp of the last non-exported packet belonging to a
eliminated flow record (Type: dateTime)
7.2.2. FsFrec_ByteInDroppedRecsCount
Description:
This Information Elements contains the count of non exported bytes
that were contained in flow records eliminated from the flow
recording process because of resource limitations/policies in the
flow recording process.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD4
Status: Proposed
Units: bytes
7.2.3. FsFrec_FrecDroppedCount
Contains the count of flow records containing non exported packets
eliminated from the flow recording process because of resources
limitations/policies in the flow recording process. It is defined in
terms of:
TsFirst: timestamp of the first flow record elimination event from
the flow recording process (Type: dateTime)
TsLast: timestamp of the last flow record elimination event from the
flow recording process (Type: dateTime)
Peluso, et al. Expires December 3, 2010 [Page 15]
Internet-Draft Flow Selection Techniques June 2010
7.2.4. FsFrec_UnexportedFrecCount
Description:
This Information Elements contains the count of the flow records
currently existing in the flow recording process containing at
least one non exported packet.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD6
Status: Proposed
Units: flow records
7.2.5. FsFrec_UnexportedPacketInFrecCount
Description:
This Information Elements contains the count of non exported
packets contained in flow records of the flow recording process.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD7
Status: Proposed
Units: packets
7.2.6. FsFrec_UnexportedBytesInFrecCount
Description:
This Information Elements contains the count of non exported bytes
contained in flow records of the flow recording process.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD8
Peluso, et al. Expires December 3, 2010 [Page 16]
Internet-Draft Flow Selection Techniques June 2010
Status: Proposed
Units: bytes
7.3. Flow exporting process related (TBD9-TBD14)
Information Elements in this section are related to Flow Selection at
the Flow Exporting Process.
+-------+--------------------------------+
| ID | Name |
+-------+--------------------------------+
| TBD9 | FsExp_PacketInDroppedRecsCount |
+-------+--------------------------------+
| TBD10 | FsExp_ByteInDroppedRecsCount |
+-------+--------------------------------+
| TBD11 | FsExp_FrecDroppedCount |
+-------+--------------------------------+
| TBD12 | FsExp_UnexportedCount |
+-------+--------------------------------+
| TBD13 | FsExp_UnexportedPacketCount |
+-------+--------------------------------+
| TBD14 | FsExp_UnexportedByteInExpCount |
+-------+--------------------------------+
7.3.1. FsExp_PacketInDroppedRecsCount
Contains the count of non exported packets that were contained in
flow records eliminated from the flow recording process because of
resource limitations/policies in the exporting process. It is
defined in terms of:
TsFirst: timestamp of the first non exported packet belonging to a
eliminated flow record (Type: dateTime)
TsLast: timestamp of the last non exported packet belonging to a
eliminated flow record (Type: dateTime)
7.3.2. FsExp_ByteInDroppedRecsCount
Description:
This Information Elements contains the count of non exported bytes
that were contained in flow records eliminated from the flow
recording process because of resource limitations/policies in the
exporting process.
Abstract Data Type: unsigned64
Peluso, et al. Expires December 3, 2010 [Page 17]
Internet-Draft Flow Selection Techniques June 2010
Data Type Semantics: quantity
ElementId: TBD10
Status: Proposed
Units: bytes
7.3.3. FsExp_FrecDroppedCount
Contains the count of flow records containing non exported packets
eliminated from the flow recording process because of resource
limitations/policies in the exporting process. It is defined in
terms of:
TsFirst: timestamp of the first flow record elimination event from
the flow recording process (Type: dateTime)
TsLast: timestamp of the last flow record elimination event from the
flow recording process (Type: dateTime)
7.3.4. FsExp_UnexportedCount
Description:
This Information Elements contains the count of the flow records
currently existing in the flow recording process containing non-
exported traffic and not being exported because of exporting
process resource lmitations/policies.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD12
Status: Proposed
Units: flow records
7.3.5. FsExp_UnexportedPacketCount
Description:
This Information Elements contains the count of non exported
packets contained in flow records of the flow recording process
not being exported because of exporting process resource
limitations/policies.
Peluso, et al. Expires December 3, 2010 [Page 18]
Internet-Draft Flow Selection Techniques June 2010
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD13
Status: Proposed
Units: packets
7.3.6. FsExp_UnexportedByteInExpCount
Description:
This Information Elements contains the count of non exported bytes
contained in flow records of the flow recording process not being
exported because of exporting process resource limitations/
policies.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD14
Status: Proposed
Units: bytes
8. Requirements put on implementations
To support the described information model an implementation must
keep, in the flow records, counts for non-exported packets and bytes.
Sometimes these are referred as delta counts. An implementation may
also keep absolute counts for scopes not specified in this
information model (it appears that both delta and absolute counters
can be exported in the IPFIX information model, see [RFC5102]). In
addition, to fully support this information model, it would be
required to keep in a flow record a timestamp for the first and last
non-exported packets. An implementation may need to keep timestamps
for the first and last exported packets as well for scopes not
specified in this information model, or to join the two timers for
the last exported and first exported packets (which is of course an
approximation) or to approximate them with the time of the exporting
event.
Peluso, et al. Expires December 3, 2010 [Page 19]
Internet-Draft Flow Selection Techniques June 2010
9. Information Model for Configuration of Flow Selection Techniques
This section aims at describing the representative parameters of the
above presented flow selection techniques. To this regard, it
provides the basis for an information model to adopt in order to
configure the flow selection process at an IPFIX device. The
information elements here introduced are defined in accordance with
the IPFIX information model [RFC5102] to which reference should be
made for more detailed information. Furthermore, the data types used
to formally rappresent the Flow Selection related information
elements are those defined in section 3.1 of the IPFIX information
model [RFC 2051]. For that reason, they are not redefined in this
section.
List of additional Flow Selection information elements:
+-------+-------------------------+-------+-----------------------+
| ID | Name | ID | Name |
+-------+-------------------------+-------+-----------------------+
| TBD15 | selectorMethod | TBD18 | flowRecordPacketsSize |
+-------+-------------------------+-------+-----------------------+
| TBD16 | flowMaxAdmitFlowRecords | TBD19 | flowInactivityTime |
+-------+-------------------------+-------+-----------------------+
| TBD17 | flowRecordBytesSize | ... | ... |
+-------+-------------------------+-------+-----------------------+
9.1. selectorMethod
Description:
This Information Element identifies the flow selection method that
are applied by the Flow Selection process, in accordance to what
described in the section 5 of this document.
Same of these methods may have parameters in order to fully
support the selected technique. For that reason, further
Information Elements are defined in the following subsections.
The following flow selection methods identifiers are defined here:
+----+----------------------------+---------------------------------+
| ID | Method | Parameters |
+----+----------------------------+---------------------------------+
| 1 | Selection based on flow | flowMaxAdmitFlowRecords |
| | size count | flowRecordBytesSize |
| | | flowRecordPacketsSize |
+----+----------------------------+---------------------------------+
Peluso, et al. Expires December 3, 2010 [Page 20]
Internet-Draft Flow Selection Techniques June 2010
+----+----------------------------+---------------------------------+
| 2 | Selection based on flow | flowMaxAdmitFlowRecords |
| | content property match | ........... |
+----+----------------------------+---------------------------------+
| 3 | Selection based on flow | flowMaxAdmitFlowRecords |
| | record arrival time or | flowInactivityTime |
| | sequence | |
+----+----------------------------+---------------------------------+
| 4 | Selection based on | flowMaxAdmitFlowRecords |
| | external events | ........... |
+----+----------------------------+---------------------------------+
Abstract Data Type: unsigned16
Data Type Semantics: identifier
ElementId: TBD15
Status: Proposed
9.2. flowMaxAdmitFlowRecords
Description:
This Information Element specifies the maximum number of elegible
flow records which might be created in to the flow cache. It is
used by the Selector Process in order to identify the time when
flow selection should be triggered. A value of 0 means that the
Flow Selection State related to the memory space available for
flow recording must be used to estimate the max flow cache size.
For example, this Information Element may be used to describe the
configuration of a flow size count Flow Selector.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD16
Status: Proposed
Units: flow records
9.3. flowRecordBytesSize
Description:
Peluso, et al. Expires December 3, 2010 [Page 21]
Internet-Draft Flow Selection Techniques June 2010
This Information Element specifies the minimum number of bytes
contained in a flow record to be considered not elegible for
removal. It may be used in order to identify elephant flows.
For example, this Information Element may be used to describe the
configuration of a flow size count Flow Selector.
Abstract Data Type: unsigned64
Data Type Semantics: quantity
ElementId: TBD17
Status: Proposed
Units: bytes
9.4. flowRecordPacketsSize
Description:
This Information Element specifies the minimum number of packets
contained in a flow record to be considered not elegible for
removal. It may be used in order to identify elephant flows.
For example, this Information Element may be used to describe the
configuration of a flow size count Flow Selector.
Abstract Data Type: unsigned32
Data Type Semantics: quantity
ElementId: TBD18
Status: Proposed
Units: packets
9.5. flowInactivityTime
Description:
This Information Element specifies the time interval in
microseconds during which the corresponding flow record may be
considered still active. It is used by the metering process
and/or the flow recording process in order to take the decision
whether to discard an existing flow to make room for a new one.
Peluso, et al. Expires December 3, 2010 [Page 22]
Internet-Draft Flow Selection Techniques June 2010
For example, this Information Element may be used to describe the
configuration of a flow arrival time Flow Selector.
Abstract Data Type: dateTimeMicroseconds
Data Type Semantics: quantity
ElementId: TBD19
Status: Proposed
Units: microseconds
10. Security Considerations
This document descirbes methods for flow selection techniques that
are applied in network measurements. If users know or can guess the
selection policies they may craft flows in a way to avoid beeing
selected. Furthermore network measurements are often used for the
detecction of network attacks. Therefore it has to be taken into
account that flow selection may remove flows that are of interest for
the detection taks. [more here]
11. IANA Considerations
This document introduces several new information elements as an
extension to the IPFIX information model. IANA assignments should be
created for the information elements described in this document.
12. References
12.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
12.2. Informative References
[DuLT01a] Duffield, N., Lund, C., and M. Thorup, "Charging from
Sampled Network Usage", ACM Internet Measurement Workshop
IMW 2001, San Francisco, USA, November 2001.
[DuLT01b] Duffield, N., Lund, C., and M. Thorup, "Properties and
Prediction of Flow Statistics from Sampled Packet
Streams", ACM SIGCOMM Internet Measurement Workshop 2002,
Peluso, et al. Expires December 3, 2010 [Page 23]
Internet-Draft Flow Selection Techniques June 2010
November 2002.
[DuLT01c] Duffield, N., Lund, C., and M. Thorup, "Learn More, sample
less: control of volume and variance in network
measurement", IEEE Transactions on Information Theory,
May 2005.
[DuLT01d] Duffield, N., Lund, C., and M. Thorup, "Flow Sampling
under Hard Resource Constraints", ACM IFIP Conference on
Measurement and Modeling of Computer Systems SIGMETRICS,
June 2004.
[EsVa01] Estan, C. and G,. Varghese, "New Directions in Traffic
Measurement and Accounting: Focusing on the Elephants,
Ignoring the Mice", ACM SIGCOMM Internet Measurement
Workshop 2001, San Francisco (CA), November 2001.
[FeGL98] Feldmann, A., Rexford, J., and R. Caceres, "Efficient
Policies for Carrying Web Traffic over Flow-Switched
Networks", IEEE/ACM Transaction on Networking,
December 1998.
[IPFIX-ARCH]
Sadasivan, G., Bownlee, N., Claise, B., and J. Quittek,
"Architecture for IP Flow Information Export", Internet
Draft draft-ietf-ipfix-architecture-12.txt, work in
progress, September 2006.
[KuXW04] Kumar, K., Xu, J., Wang, J., Spatschek, O., and L. Li,
"Space-code bloom filter for efficient per-flow traffic
measurement", INFOCOM 2004 Twenty-third AnnualJoint
Conference of the IEEE Computer and Communications
Societies, March 2004.
[Moli03] Molina, M., "A scalable and efficient methodology for flow
monitoring in the Internet", International Teletraffic
Congress (ITC-18), Berlin, September 2003.
[PSAMP-TECH]
Zseby, T., Molina, M., Raspall, F., Duffield, N., and S.
Niccolini, "Sampling and Filtering techniques for IP
Packet Selection", Internet
Draft draft-ietf-psamp-sample-tech-11.txt, work in
progress, July 2008.
[RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
Meyer, "Information Model for IP Flow Information Export",
RFC 5102, January 2008.
Peluso, et al. Expires December 3, 2010 [Page 24]
Internet-Draft Flow Selection Techniques June 2010
Authors' Addresses
Lorenzo Peluso
University of Napoli
Via Claudio 21
Napoli 80125
Italy
Phone: +39 081 7683821
Email: lorenzo.peluso@unina.it
Tanja Zseby
Fraunhofer Institute FOKUS
Kaiserin-Augusta-Allee 31
Berlin 10589
Germany
Phone: +49 30 3463 7153
Email: tanja.zseby@fokus.fraunhofer.de
Salvatore D'Antonio
CINI Consortium/University of Napoli "Parthenope"
Monte S.Angelo, Via Cinthia
Napoli 80126
Italy
Phone: +39 081 679944
Email: saldanto@unina.it
Maurizio Molina
DANTE
Hill Road 126-130
Cambridge CB2 1PQ
United Kingdom
Phone: +44 1223 371300
Email: maurizio.molina@dante.org.uk
Peluso, et al. Expires December 3, 2010 [Page 25]
