IPFIX working group
Internet Draft L. Deri
Document: draft-deri-impl-00.txt NETikos S.p.A.
Expires: January 2005 July 2004
IPFIX Implementation Notes
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
Providing an early implementation report is important in order to
help validating a standard before its finalization. This document
describes the lessons learnt while implementing both an IPFIX probe
and collector. Its goals are manifold and include feedback to the
IPFIX working group, in addition to suggestions for tackling open
protocol issues that can be addressed in future IPFIX drafts.
Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119.
Table of Contents
1. Introduction...................................................2
2. IPFIX Implementation...........................................2
2.1 General Implementation Notes...............................3
2.2 Vendor-specific Extensions.................................4
2.3 SCTP Protocol Implementation...............................4
3. Implementation Evaluation......................................6
Deri Expires - January 2005 [Page 1]
IPFIX Implementation Notes July 2004
4. Open IPFIX Issues..............................................7
5. Final Remarks..................................................9
6. Code Availability..............................................9
References........................................................9
Acknowledgments..................................................10
Author's Addresses...............................................10
1. Introduction
The IETF IPFIX working group [ipfix-wg] is working towards the
definition of a standard for the export of IP flows. At present a
draft of the IPFIX protocol specification has been released.
The interest of the author for IPFIX is because in the past years he
has developed two open source applications:
- A passive traffic monitoring application named ntop [ntop] that
can also act as a NetFlow collector.
- A libpcap-based [pcap] NetFlow v5/v9 probe named nProbe [nprobe].
In order to both evaluate IPFIX and provide feedback to the working
group, the author has extended both applications with IPFIX support
so that it is possible to have control over both the probe and the
collector. The author believes that this is a typical scenario at
least for early IPFIX applications, as they will likely be derived
from existing NetFlow applications, considering that IPFIX is very
similar to NetFlow v9. For this reason the paper does not cover
NetFlow or general flow-based implementation issues. Instead it
focuses on the effort required to enhance an existing NetFlow
application so that it can also support IPFIX.
Since IPFIX uses SCTP [sctp] as default transport, in order to ease
the development cycle until SCTP will be supported in all the main
operating systems, Linux has been used as reference platform. However
this report should be considered platform-neutral, as both probe and
collector do not use Linux-only facilities.
The author is not aware of any IPFIX implementations so
interoperability issues have not been addressed here.
2. IPFIX Implementation
Implementing IPFIX is rather straightforward if the starting point is
a NetFlow v9 application. This is because NetFlow v9 has been used as
base for the IPFIX protocol [ipfix-eval]. In a nutshell [ipfix-nf-
eval] IPFIX is basically NetFlow v9 over SCTP with little differences
such as templates definition when non-IETF defined flow fields are
used.
Deri Expires - January 2005 [Page 2]
IPFIX Implementation Notes July 2004
As many NetFlow implementations do not support v9 yet, it is worth to
spend a few words about the complexity required to enhance an
existing NetFlow application (e.g. based on v5) to v9 [nfv9-spec].
The main difference between v5 and v9 is the template concept, in
addition to other minor issues such as little changes in the flow
header (e.g. the sequence number). Implementing templates is a major
issue and it definitively costs more, in terms of code complexity, to
move from NetFlow v5 to v9 than from v9 to IPFIX. This is for several
reasons:
- The probe cannot guess in advance the template formats so all the
possible fields combinations are possible.
Due to the dynamic flow size, the probe must carefully allocate
buffers and make sure that exported flows can fit in a single UDP
packet.
- Optimizing template transmission is a complex activity in
particular when the probe needs to send out flows to various
collectors. For this reasons it is recommended that the probe does
not send more that a few different flow/template formats otherwise
the code complexity can increase significantly.
- From the performance point of view, the probe must be able to
collect all the possible flow information even if very few fields
can be emitted. For instance VLAN tags were not supported in
NetFlow v5 so the VLAN tag was ignored. Instead VLANs are supported
in V9, so the probe must take them into account even if the VLAN
tag will not be exported in the flow. Optimizing these kind of
activities is very time consuming and often the results are not
that great in term of performance and memory gain.
- Collectors must be prepared to handle all the possible flows
without breaking their logic. For instance a collector should not
assume that the source/destination IP address are contained in the
received flow as with previous NetFlow versions.
The following sections describe the effort required to implement
IPFIX starting from NetFlow v9 according to the role, either probe or
collector.
2.1 General Implementation Notes
This section contains general implementation notes that are applied
to both probe and collector. For both sides implementing IPFIX
support basically means to:
- Implement vendor-specific extensions.
- Support the SCTP protocol.
Deri Expires - January 2005 [Page 3]
IPFIX Implementation Notes July 2004
IPFIX support has been implemented according to the following
documents: [ipfix-info] and [ipfix-spec]. Technical documentation
about IPFIX is rather straightforward to undertstand in particular
for those who are familiar with NetFlow as IPFIX is often a
1:1 mapping with the corresponding NetFlow v9 specification.
2.2 Vendor-specific Extensions
Contrary to NetFlow v9, IPFIX vendor-specific extensions are
specified by means of the vendor specified field type defined using
the IANA enterprise number (PEN) [pen]. Although the use of PEN
numbers is general (e.g. it has been used into the SNMP MIB II for
the sysObjectID [mib2]), it has the drawback of requiring a different
template format from IETF-defined fields where a numeric identifier
is used. This difference requires that both probe and collector
support both formats, with and without the PEN. It should be
investigated whether it is feasible to merge both template formats
(IETF and vendor-specific extensions) into one, so that the
implementation complexity is reduced. For instance the IETF format
can be replaced with the vendor-specific one, using an enterprise
number for IETF. Although this is a minor issue, it definitively
simplifies the development work and troubleshooting.
2.3 SCTP Protocol Implementation
The SCTP protocol is a reliable, congestion-aware transport protocol
based on IP. It sports several features such as acknowledged error-
free retransmission of messages, detection of data corruption and
duplication. It is basically a general-purpose transport for message-
oriented applications such as flow-based applications based on IPFIX.
The flow peers handle SCTP very similar to TCP as they need to
- Bind to a port.
- Connect with a peer.
- Start sending data.
- Close the connection.
The main difference with TCP is that in case of network events such
as disconnection of the remote peer, the SCTP layer continues to work
even if it reports the problem to the sender. This means that an
application does not have to handle reconnection with the remote peer
as this is done transparently by the transport when the remote peer
is available although some emitted flows might be lost. Still the
application must register with SCTP in order to receive events about
the underlying network connection. This is because in case of
reconnection, the application has to retransmit the template records
so it is necessary to be informed about the state of the network
transport.
From the programming point of view, SCTP is very similar to TCP/UDP
so little code needs to be changed while migrating existing
Deri Expires - January 2005 [Page 4]
IPFIX Implementation Notes July 2004
applications to SCTP. The main issues with SCTP support are related
to the shift from a pure message oriented protocol such as UDP to a
connection-oriented protocol. In fact UDP-based applications do not
have to keep track of per-connection information nor handle transport
messages. This has definitively a major impact on application
complexity and performance that depends on the role of the
application as described below.
Probe Side
Probes need to keep track of peer connections that means:
- Templates need to be retransmitted per-connection every time a
connection is reestablished; using a connectionless protocol such
as UDP, this was a periodic operation as there is no simple way to
detect whether a collector disconnected. Hence using SCTP, template
sending can be optimized, as it is not really necessary to
periodically send templates, because it is now possible to detected
whether the collector has disconnected.
On the other hand as a probe can send flows to several collectors
(e.g. in round-robin or as a reflector) it must keep track of per-
collector templates transmission. This means that if collector X
reconnects, the probe must send the template only to this collector
and not to all collectors.
- The IPFIX specification allows a probe to open a connection for
sending templates and another for sending flows, making the whole
setup even trickier.
- Transmitting templates is a time consuming activity for both the
application and the transport, as they need to be sent quite often.
For instance Cisco IOS sends NetFlow v9 templates every 10 flow
packets (ip flow-export template refresh-rate IOS parameter
[nfv9]). Sending templates using SCTP is a little more costly than
with UDP, in particular if IPSec is used, so every transmission
that is not necessary should be carefully avoided. Nevertheless,
sending templates only when there is a reconnection it
significantly reduces the network traffic with respect to a
connectionless transport.
Collector Side
There are no major changes to be supported from the transport point
of view with respect to NetFlow v9, beside little coding necessary
to support SCTP.
As stated before, SCTP is currently supported only on some selected
OSs such as Linux and Solaris. At the moment it is not clear whether
both popular operating systems such as Windows will ever support it
natively (Microsoft has no plan to support SCTP in the current and
next Windows version, codename Longhorn), and network manufacturers
will implement it on their routers. In addition, being SCTP a totally
Deri Expires - January 2005 [Page 5]
IPFIX Implementation Notes July 2004
new protocol, it is unclear what kind of problems it will suffer when
deployed in production environments or whether it will be vulnerable
to network attacks. This makes the IPFIX future not too bright, as
SCTP is the mandatory transport protocol. It is clear that the IPFIX
WG is betting a lot on this protocol, however making it mandatory is
probably a very strong position that will prevent IPFIX diffusion at
least in its early days.
3. Implementation Evaluation
From the coding point of view, implementing IPFIX without vendor-
specific extensions is not much different from NetFlow v9 support. In
fact the only major difference is just the transport that has to be
handled slightly differently from UDP but that has the advantage of
reducing template periodic transmission.
In order to evaluate differences between transports, some packets
(the sample contained 50'000 packets sent from the same address
towards random destinations) have been captured and reproduced using
nProbe. In order to evaluate the two transports regardless of the
template transmission, templates are sent periodically on both
transports, although templates transmission affects the test very
little. Note that as explained before, periodic template transmission
is not really necessary when SCTP is used, unless there is a
transport disconnection/reconnection. The test outcome is shown in
the following table.
+-------------+-----------+-----------------+
| | UDP | SCTP |
+-------------+-----------+-----------------+
| Packets | 1'667 | 2'514 [+150%] |
+-------------+-----------+-----------------+
| Bytes | 2'509'540 | 2'595'952 [+3%] |
+-------------+-----------+-----------------+
Table 1 - UDP vs. SCTP Network Usage
Basically with SCTP the number of packets is significantly increased
with respect to UDP. This is because with SCTP the flow collector
acknowledges almost immediately all the received packets. However as
these acknowledge packets are rather small (62 bytes) they have very
little impact on the total amount of data transfer but just on the
number of packets sent. Furthermore it is worth to remark that if
there is no traffic (i.e. flows are not sent) for some time (the
default is 30 seconds in the current Linux 2.6 implementation), the
SCTP stack on the probe side sends a heartbeat message that is
acknowledged by the collector side. Heartbeat messages are very small
(86 bytes) however they influence the amount of traffic exchanged.
Deri Expires - January 2005 [Page 6]
IPFIX Implementation Notes July 2004
From the performance point of view, IPFIX is basically as fast as
NetFlow v9. This is because there are not many differences between
the two protocols. However it has to be remarked that the use of
templates has some little performance drawbacks with respect to
'static' NetFlow as v5. This is because:
- Applications (probes and collectors) know in advance the data
format so they can be optimized and tuned in advance.
- Several routines that handle templates (definition, transmission,
check) are avoided, so there are less per-flow CPU cycles needed.
4. Open IPFIX Issues
As stated already, IPFIX is very similar to NetFlow v9. On one hand
this is an advantage as it allows existing NetFlow v9 applications to
be made IPFIX aware with little effort, however on the other hand
this is a serious limitation. In fact the NetFlow design has evolved
very little since its introduction in 90s, and most of the changes
are minor and often due to the need to support new features while
maintaining a wide compatibility with past versions. This explains
for instance why NetFlow v7 is very similar to v1, and because the
first Cisco IOS implementation of v9 is basically a "port" of v5 over
templates.
On the other hand, IPFIX is a brand new effort to define a standard
flow-based protocol that should capitalize on experience and lessons
learnt after ten years of NetFlow. This obviously did not fully
happen for many reasons, as this protocol is far too similar to v9
and not really introduces new features. The following list summarizes
some of the missing features that according to the author should be
introduced before the protocol is finalized:
- As with NetFlow, the IPFIX flow format is fixed. Consider the
following scenario: a user requires in-depth ICMP flow analysis
(e.g. ICMP code and type). There are basically two solutions:
either a probe/collector agree on two different templates (one for
ICMP and one for non-ICMP flows) or they agree on a common template
for all protocols with the outcome that some flow fields are empty
when non-ICMP flows are emitted. The ideal solution to this kind of
scenario is dynamic flows, where emitted flows can have all the
template fields or a subset of them according to the flow kind.
This means that non-ICMP flows will not have ICMP code and type,
that instead will be present on ICMP flows that will lack source
and destination ports, meaningless for ICMP. Unfortunately due to
the way templates and flows are defined, it is not possible to
simply handle dynamic flows, unless an "inline" template is
embedded in flows as with nFlow [nflow]. nFlow has no concept of
template and it embeds in flows the field identifier. This practice
slightly increases the flow size, however thanks to both on-the-fly
Deri Expires - January 2005 [Page 7]
IPFIX Implementation Notes July 2004
flow compression and lack of sending templates, the volume of data
transmitted is significantly reduced. This feature will save
processing time and bandwidth while enabling probes and collectors
to have a rich template format.
- IPFIX templates specify just the format of flows and cannot
modify the flow header. This means that it is not possible to
define flow header attributes. For instance it is not possible to
define attributes like MD5 flow message checksum for non-
repudiation of flows, or attributes that can be transported in
piggy-back instead of putting them into the options template.
Adding this capability would enable developers to have the same
level of freedom in both header and flow content.
- The SCTP protocol is definitively an enhancement over UDP,
although it has some severe drawbacks in terms of number of packets
transmitted. The main issue is that every packet is acknowledged
almost immediately, that can for instance significantly amplify the
effect of a DoS attack. Note that NetFlow amplifies the effects of
a DoS when small packets are used, because NetFlow flows are larger
than packets used for the attack. It should be explored whether
SCTP ACKs can be reduced for instance using mechanisms like those
used by TCP, in order to reduce the (almost) 1:1 traffic flow (one
ACK for each flow received) from the collector to the probe.
Finally it should be explored whether it still makes sense to have
a flow identifier in the flow header when SCTP is taking care of
issues such as flow retransmission or duplicate discard.
- IPFIX does not specify a complete, two-way protocol but just a
one-way (from probe to collector) format for exported flows. The
drawback is that the collector has no way to query the probe
configuration or to modify it. This statement is true unless an
offline protocol (e.g. via the CLI interface) is used. The probe to
inform the collector uses the options template/flow, although this
is not an effective solution as:
- It does not allow the collector to query the probe but it is the
probe that decides when to inform the collector and what to
report.
- It is used to periodically send information that could be
discarded by the collector. It would make more sense to allow the
collector to query the probe when needed.
It is not rich enough to let the probe inform the collector about
issues such as lack of memory for handling incoming flows.
A possible solution could either be the approach used in other flow
protocols (e.g. sFlow [sflow]) that defines an SNMP MIB implemented
by the probe, or enrich the IPFIX specification with a real two-way
protocol.
Deri Expires - January 2005 [Page 8]
IPFIX Implementation Notes July 2004
5. Final Remarks
This document reported about implementing IPFIX in both a probe and a
collector. It covers issues encountered during the implementation and
describes open issues that can be tackled before IPFIX is released
that include:
- Limited innovation, if any, with respect to existing technologies
such as NetFlow.
- Use of different data formats for IETF and non-IETF information.
- Use of SCTP requires additional coding and is not supported in
all platforms.
- Lack of configuration and monitoring that can be solved through
an SNMP MIB.
This paper has also proved that the IPFIX protocol is feasible to
implement in existing NetFlow applications and shown two open source
applications that can be used by those interested in evaluating IPFIX
and developing IPFIX-aware applications.
6. Code Availability
The applications described in this document are distributed under the
GPL2 license and available from the ntop home page
(http://www.ntop.org/).
References
[ipfix-wg] IETF, "IP Flow Information Export Working Group",
http://www.ietf.org/html.charters/ipfix-charter.html.
[ipfix-eval] S. Leinen, "Evaluation of Candidate Protocols for IP Flow
Information Export (IPFIX)", draft-leinen-ipfix-eval-contrib-02.txt,
January 2003.
[ipfix-info] P. Calato and others, "Information Model for IP Flow
Information Export", draft-ietf-ipfix-info-03.txt, February 2004.
[ipfix-nf-eval] B. Claise, "Evaluation Of NetFlow Version 9 Against
IPFIX Requirements", draft-claise-ipfix-eval-netflow-04.txt, February
2003.
Deri Expires - January 2005 [Page 9]
IPFIX Implementation Notes July 2004
[ipfix-spec] B. Claise and others, "IPFIX Protocol Specifications",
draft-ietf-ipfix-protocol-03.txt, January 2003.
[mib2] K. McCloghrie, and M. Rose, "Management Information Base for
Network Management of TCP/IP-based internets: MIB-II", RFC 1213,
March 1991.
[nfv9] Cisco Systems Inc., "NetFlow v9 Data Format",
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/
120newft/120limit/120s/120s24/nfv9expf.htm, 2003.
[nfv9-spec] B. Claise, "Cisco Systems NetFlow Services Export Version
9", draft-claise-netflow-9-08.txt, April 2004.
[ntop] L. Deri and others, "Monitoring Networks using ntop",
Proceedings of IM 2001, Seattle, May 2001.
[nprobe] L. Deri, "nProbe: an Open Source NetFlow Probe for Gigabit
Networks", Proceedings of Terena TNC 2003, Zagreb, May 2003.
[nflow] L. Deri, "nFlow", http://www.ntop.org/nFlow/, March 2003.
[pen] IANA, "Private Enterprise Numbers Registry",
http://www.iana.org/assignments/enterprise-numbers
[sctp] R. Stewart and others, "Stream Control Transmission Protocol
(SCTP)", RFC 2960, October 2000.
[sflow] P. Phaal and others, "sFlow: A Method for Monitoring Traffic
in Switched and Routed Networks", RFC 3176, September 2001.
Acknowledgments
This work was supported in part by the IST project SCAMPI (IST-2001-
32404) funded by the European Union. The author would like to thank
Fulvio Risso <fulvio.risso@polito.it> and Simon Leinen
<simon@switch.ch> for the valuable discussions and
suggestions.
Author's Addresses
Luca Deri
NETikos S.p.A.
Via del Brennero km 4
Deri Expires - January 2005 [Page 10]
IPFIX Implementation Notes July 2004
56123 Pisa
Italy
Phone: +39 050 80091
Email: luca.deri@netikos.com
Deri Expires - January 2005 [Page 11]