Internet Draft Expiration: August 2003 Martin Djernaes Document: draft-djernaes-netflow-9-transport-00.txt Cisco Systems Category: Informational February 2003 Cisco Systems NetFlow Services Export Version 9 Transport Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsolete by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract NetFlow Export Version 9 Export is defined in the internet draft draft-claise-netflow-9 [NFv9], which describes how to transport NetFlow over UDP [RFC768]. This document describes how NetFlow would use SCTP [RFC2960] as a transport protocol. Traditionally NetFlow was transmitted over UDP to a physically closely located collector. As the networks grow this is not necessarily the case any longer, so issues like congestion avoidance and reliability are becoming an increasing issue for applications using NetFlow. This document addresses these issues in a simple way using the SCTP protocol. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document is to be interpreted as described in RFC 2119. Djernaes Draft [Page 1]
Cisco NetFlow v9 transport February 2003 Table of Contents 1. Introduction...................................................2 1.1 Overview...................................................2 1.2 Congestion Avoidance.......................................2 1.3 Reliability................................................2 2. NetFlow data...................................................3 2.1 Packet size................................................3 2.2 Source ID..................................................3 3. Exporter.......................................................3 3.1 Observation Domain.........................................3 3.2 Association................................................4 3.3 Stream.....................................................4 3.4 Template...................................................4 3.5 Stream usage...............................................5 4. Collector......................................................5 4.1 Listen.....................................................5 4.2 Receive....................................................5 5. References.....................................................6 6. Acknowledgments................................................6 7. Authors Addresses..............................................6 1. Introduction 1.1 Overview This document describes how Cisco NetFlow version 9[NFv9] can be transported over SCTP[RFC2960] using traditional reliable mode, but also use the partial reliable or unreliable mode[PR-SCTP]. Weight is especially put on the freedom to use SCTP as a reliable, single stream transport, as well as multiple streams with different properties, for example in terms of reliability, carrying different data types dependant on their importance for the system. This document does not describe how to classify importance of data, and does therefore not describe how an exporter selects what kind of stream to use for a given set of data. 1.2 Congestion Avoidance The SCTP transport protocol provides the required level of congestion avoidance by design. 1.3 Reliability The SCTP transport protocol is by default reliable, but has the capability to operate in unreliable and partially reliable modes [PR-SCTP]. Using reliable SCTP streams for NetFlow v9 export is not in itself a guarantee that all records are delivered. If there is congestion on the link from the exporter to the collector, or if a significant Djernaes Draft [Page 2]
Cisco NetFlow v9 transport February 2003 amount of retransmissions are needed, the send queues on the exporter may fill up. In that case it's up to the exporter to decide what to do. It may either halt export (buffer the data until there is space in the send queues again) or just throw export packets away instead of inserting them into the send queue. If any data is not inserted into the send queues, the sequence numbers used for export must reflect the loss of data. Unreliable or partial reliability may be chosen for one or more streams inside an association. Unreliable transport may be preferred where large amount of data is to be exported and keeping send queues is either an unnecessary overhead or impractical. Partial reliability may be chosen where a small amount of queuing is possible. 2. NetFlow data 2.1 Packet size When transmitting NetFlow data it should be formatted as described in the NetFlow draft[NFv9] with one Packet Header and a number of Flow Sets per export packet. Each NetFlow export packet should be equal to or less than the local MTU in size. Where a NetFlow packet is transmitted over a network with an MTU smaller than the local MTU, IP fragmentation may be used. 2.2 Source ID The NetFlow draft states that each NetFlow export packet must contain a Packet Header, which includes a source id (SID). The SID indicate from which Observation Domain inside the exporter the data is being exported, and should be keept unique for each such domain. 3. Exporter 3.1 Observation Domain 3.1.1 Single Observation Domain If an exporter consists of a single observation domain, a single SID value must be used for all export packets. The exporter will typically open one association to the collector, but more are possible, in which one or more streams can be used. The exporter has the choice of transmitting parts of the export data in separate streams or all data in one stream. Djernaes Draft [Page 3]
Cisco NetFlow v9 transport February 2003 3.1.2 Multiple Observation Domains If an exporter consists of multiple observation domains, one SID value for each observation domain must be used. The exporter will typically open one association, but more are possible, in which at least one stream per observation domain is used. The exporter have the choice of using more than one stream per observation domain, but data from multiple observation domains should not be transmitted over the same stream. 3.2 Association The exporter may create one or more associations (connection "bundle" in SCTP terminology) from the NetFlow exporter to the NetFlow collector. The NetFlow collector may not initiate the connection. Inside each association one or more streams may be requested by the exporter. If the collector can not support the requested number of streams, it may choose to refuse the connection and the exporter should try to reduce, if possible, the number of streams needed to perform the export. 3.3 Stream An Observation Domain must use at least one stream, but may use multiple streams, to export NetFlow data. The Observation Domain must use the same SID value for all streams used. An exporter must not transmit packets with different SID values in one stream, the collector should however verify that the SID values are the expected values. 3.4 Template Since the SCTP association is connection oriented the available templates must be transmitted from each Observation Domain to the collector immediately after the association is established. As a minimum the templates must be transmitted immediately after they start to exist on the exporter and should preferably be transmitted before any data, using the new template, have been transmitted. The collector should however accept data without a template and as with UDP it should store the data until the template arrives. When using a reliable mode for template export, or if the exporter knows that the export packet containing the templates was positively acknowledged by the SCTP layer, it is not necessary to periodically export the templates as described in the NetFlow v9 Export draft [NFv9]. Djernaes Draft [Page 4]
Cisco NetFlow v9 transport February 2003 3.5 Stream usage The decision of what to send over what kind of stream is an application choice and outside the scope of this document. Naturally it is better to send templates over a reliable stream and send the data on an unreliable (or partial reliable) stream. When an exporter handles data with different properties it might even be preferable to send them over different streams according to those properties. 3.5.1 Example: One stream for everything If an exporter contain only one Observation Domain the NetFlow v9 export packets can be inserted unmodified onto a single SCTP stream. This stream can either be reliable, partial reliable or just unreliable. Using an unreliable stream gives only few advantages over a partial reliable stream. 3.5.2 Example: Two streams per Observation Domain An exporter can use two streams per Observation Domain. A reliable stream could be used for exporting templates, to reduce the likelihood of loss and to remove the need for blind retransmissions, and a partial or unreliable stream for data, to avoid buffering of large amounts of data. 4. Collector 4.1 Listen The collector should listen for a new association request from the exporter. The exporter will request a number of streams to use for export. If the collector doesn't support the number of streams inside the association, the collector must refuse the connection and continue listen for a new request. 4.2 Receive When data is received from an association, the collector must correlate data, with the same SID value, from multiple streams into one export flow from an Observation Domain. This allow the Observation Domain to use separate streams for different types of information. The collector should verify that the received export packets inside one stream does not have diverting SID values. The exporter must not export packets inside one stream with multiple SID values. The correlated export flow are then treated like a normal export flow as described in [NFv9]. Djernaes Draft [Page 5]
Cisco NetFlow v9 transport February 2003 5. References [NFv9] Claise, B, "Cisco Systems NetFlow Services Export Version 9", draft-claise-netflow-9-01.txt [RFC768] Postel, J. (ed.), "User Datagram Protocol", STD 6, RFC 768, August 1980. [RFC2960] Stewart, R. (ed.) "Stream Control Transmission Protocol", RFC 2960, October 2000 [PR-SCTP] Stewart, R, "SCTP Partial Reliability Extension", draft-stewart-tsvwg-prsctp-01.txt 6. Acknowledgments Thanks to Benoit Claise, Vamsi Valluri, Randall Stewart and Stewart Bryant for their valuable comments. 7. Authors Addresses Martin Djernaes Cisco Systems, Inc. 170 W Tasman Drive San Jose, CA 95134 USA Phone: +1 (408) 853-1676 Email: djernaes@cisco.com Djernaes Draft [Page 6]