IPFIX Working Group J. Meyer
Internet-Draft Hewlett-Packard
Expires: February 18, 2003 August 20, 2002
Reliable Streaming Internet Protocol Detail Records
draft-meyer-ipdr-streaming-00
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on February 18, 2003.
Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved.
Abstract
This memo defines a mechanism to deliver streams of network usage
information which follow the Internet Protocol Detail Records (IPDR)
format. This format provides an extensible means to exchange usage
information between systems. The model of extension is based on the
use of a subset of the XML Schema specification language, in
conjunction with a well defined mapping to a binary format based on
the eXtensible Data Record (XDR).
Meyer Expires February 18, 2003 [Page 1]
Internet-Draft Streaming IPDR August 2002
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Trivial TCP Delivery . . . . . . . . . . . . . . . . . . . . . 4
2.1 Protocol Procedures . . . . . . . . . . . . . . . . . . . . . 4
2.2 Illustration . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Acknowledged TCP Delivery . . . . . . . . . . . . . . . . . . 7
3.1 NDM-U 3.1 IDL extension . . . . . . . . . . . . . . . . . . . 7
3.2 Procedures for Reliable Streaming . . . . . . . . . . . . . . 9
3.3 Illustration . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4 Failover Procedure . . . . . . . . . . . . . . . . . . . . . . 12
4. Security Considerations . . . . . . . . . . . . . . . . . . . 13
References . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Author's Address . . . . . . . . . . . . . . . . . . . . . . . 14
Full Copyright Statement . . . . . . . . . . . . . . . . . . . 15
Meyer Expires February 18, 2003 [Page 2]
Internet-Draft Streaming IPDR August 2002
1. Introduction
The NDM-U 3.1 [4] specification from ipdr.org defines an information
modeling technique based on a well defined subset of XML-Schema [1].
Models which are developed according to these techniques are called
Service Specifications in NDM-U.
The resulting information models in turn allow the creation of
documents representing sets of usage information. These documents
can be written in either XML or in an efficient binary format.
Although the focus of NDM-U to date has been on the exchange of usage
information between collecting mediation systems and Business Support
Systems (BSS), the specification acknowledges that Service Elements
(SEs) themselves may directly produce information in NDM-U format.
The direct use of NDM-U information modelling and encoding by Service
Elements is desirable, because it reduces the translation overhead
associated between collecting information and delivering it to its
ultimate consumer(s).
The use of a common, well defined, machine readable information model
such as that defined by NDM-U 3.1 is also desirable. This would
greatly improve the ability of vendors to concisely define new
information elements produced by their Service Elements and increase
the likelihood of interoperability existing collection systems.
By carrying the binary format on top of a reliable transport, the
NDM-U mechanism can be extended to a record oriented delivery system
in addition to its existing document oriented approach.
Meyer Expires February 18, 2003 [Page 3]
Internet-Draft Streaming IPDR August 2002
2. Trivial TCP Delivery
The design of NDM-U's compact format specification allows for
streamed delivery of IPDR records over a TCP connection, without
change to the definition. The compact format is defined in section
4.3 of the NDM-U 3.1 specification.
The underlying encoding for the compact format is based on the
Extensible Data Records (XDR) [5]. The encoding rules differ on one
point, which is the use of "indefinite length" for two structures.
The XDR encoding model benefits from being an IDL driven encoding.
This allows for a machine readable, concise and familiar writing of
the structures to be encoded.
2.1 Protocol Procedures
The SE initiates a TCP connection to a configured destination and
port (it may make use of SVRLOC or DNS, in the same manner
described by Diameter [9]).
The SE should recognize ICMP no port or TCP rejections and
timeouts as indicating the server is unavailable. In this case it
may try alternate configured servers.
Upon connection establishment, the SE must deliver header and any
record descriptors known at this point.
For each service usage event to be recorded, provide an IPDR
record of a type compliant with a previously delivered Descriptor.
The SE will continue delivering usage until either: a. The
underlying TCP connection indicates a failure. b. Some
administrative control function, not defined in this spec, is
delivered to shutdown the SE's usage reporting function.
During graceful termination the SE will deliver a DocEnd prior to
closing the TCP connection.
2.2 Illustration
An XDR IPDR document is structured as binary data elements as
follows:
Meyer Expires February 18, 2003 [Page 4]
Internet-Draft Streaming IPDR August 2002
+----------------+
| Header |
+----------------+
| Descriptor |
+----------------+
| Usage Event |
+----------------+
...
+----------------+
| Usage Event |
+----------------+
| Doc End |
+----------------+
Rather than writing these into a file, the information is
written onto a TCP connection. The actual network communication
may be structured as:
Time t0:
<-- TCP Conn Established -->
+----------------+
| Header |
+----------------+ -- transmitted -->
| Descriptor |
+----------------+
Time t1 (one or more usage event(s) recorded):
+----------------+
| Usage Event | -- transmitted -->
+----------------+
...
+----------------+
| Usage Event |
+----------------+
Time t2 (more usage event(s) recorded):
+----------------+
| Usage Event | -- transmitted -->
+----------------+
...
+----------------+
| Usage Event |
+----------------+
Meyer Expires February 18, 2003 [Page 5]
Internet-Draft Streaming IPDR August 2002
Continues until some terminating control message arrives
and finally at time tn:
+----------------+
| Doc End | -- transmitted -->
+----------------+
<-- TCP Connection Release -->
Note that additional descriptor stream elements MAY be transmitted in
the stream along with Usage Events at any time.
Meyer Expires February 18, 2003 [Page 6]
Internet-Draft Streaming IPDR August 2002
3. Acknowledged TCP Delivery
A key issue with the trivial scheme is that if the connection is not
terminated gracefully, there is no information about which IPDR
Records may have been lost in transit.
Some IPDR record producing entities will want to reliably deliver the
records to another entity for persistence. In the event of
communication failure with one persistence entity, it is important to
identify records which were sent, but need to be retransmitted to
another store.
The entity producing these records should be prepared to hold onto
the records until receiving an acknowledgement from the receiving
entity.
3.1 NDM-U 3.1 IDL extension
In order to support these additional requirements on the B-interface,
the existing XDR specification can be trivially modified to:
Reflect a new version of the protocol (version 4).
Add two new "StreamElement" objects to the current set:
RecordDescriptor
IPDRRecord
IPDRDocEnd
IPDRAck /* introduced in v4 */
IPDRDisconnect /* introduced in v4 */
The new Ack and Disconnect elements contain the following
information:
BEGIN IDL FRAGMENT
------------------
/* IPDRAck (Version 4 StreamElement)
*
* An IPDRAck element may be sent by either the producer or
* consumer to acknowledge the delivery of IPDR records and
* to distinguish between idle channels and broken connections.
*/
struct IPDRAck {
hyper curTime; /* 64-bit time representation, indicates
* the current GMT time at the time sent.
*/
int lastSeqNum; /* The last record sequence number seen
Meyer Expires February 18, 2003 [Page 7]
Internet-Draft Streaming IPDR August 2002
* (for consumer) or sent (for producer).
* A value of -1 indicates no sequence
* numbers seen.
*/
int timeoutHint; /* Specifies a hint as to how often some
* acknowledgement or record should be sent
* in the stream to enable faster detection
* of disconnect. Expressed in msecs.
* A value of -1 indicates no ack is
* requested. A value of 0 indicates a
* request for a single Ack to be
* transmitted by the peer, ASAP.
*/
int ipdrCountHint; /* Used only by a producing system to
* indicate how many outstanding records
* may go unacknowledged. A value of 0
* indicates no acks are requested.
*/
};
/* IPDRDisconnect (Version 4 StreamElement)
*
* An IPDRDisconnect element is similar to the DocEnd element,
* but is more appropriate when communicating over a network
* connection. It may be sent by either the producer or consumer
* and provides additional information about the status of the
* connection.
*/
struct IPDRDisconnect {
hyper curTime; /* 64-bit time representation, indicates
* the current GMT time at the time sent.
*/
int lastSeqNum; /* The last record sequence number seen
* (for consumer) or sent (for producer).
* A value of -1 indicates no sequence
* numbers seen.
*/
int reasonCode; /* An integer code representing the cause
* of the disconnect.
*
* Reusing the HTTP response code scheme:
* 100-199 - informational
Meyer Expires February 18, 2003 [Page 8]
Internet-Draft Streaming IPDR August 2002
* 200-299 - client request successful
* 300-399 - redirection
* 400-499 - client request error
* 500-599 - server errors
*
* Reason codes defined are:
* 200 - Normal shutdown request.
* 201 - Disconnect in response to peer
* disconnect request.
* 400 - Restart connection request.
* 401 - Stream integrity check error.
* 402 - Timeout.
* 500 - Server error, shutdown request.
*/
UTF8String reasonString; /* A string providing additional
* information about the disconnect.
* May be empty.
*/
};
END IDL FRAGMENT
----------------
3.2 Procedures for Reliable Streaming
The same TCP connection model is used as in Trivial Streaming.
The Ack stream element is written by the producer after initial
output of header and descriptors to indicate its desired frequency
of response acks.
The server contacted should, after seeing an initial ack in the
input stream, begin sending a special "IPDRDoc" back to the IPDR
Recorder on the same TCP connection.
The server IPDRDoc is used only for control information. It does
not contain any IPDR records.
The header information in the server generated IPDRDoc will be
restricted as follows:
The version shall be.
The IPDRRecorderInfo shall use a URI to describe the system
which is generating this control IPDRDoc.
Meyer Expires February 18, 2003 [Page 9]
Internet-Draft Streaming IPDR August 2002
The defaultNameSpaceURI, otherNameSpaces and
serviceDefinitionURIs shall all be empty as indicated by having
their length fields set to zero.
The docId shall match the docId sent by the IPDRRecorder
initiating the connection.
After sending the header an initial Ack should be sent. This
should indicate any preferences the server has for "heartbeat"
acks. These will indicate that the producing system is still
present during idle periods. It will also allow the detection of
inactivity timeouts outside of the underlying TCP mechanism. TCP
traditionally takes a very forgiving timeout policy, with system
defaults often taking many minutes to mark the TCP connection as
failed. By enabling applications a mechanism under their control,
operators can more directly control fail over policy.
The timeoutHint and ipdrCountHint sent by the IPDR Recorder
suggest values which the consuming application should use to
determine the frequency of Ack's being delivered in the control
document.
If the receiving entity has not sent an Ack within the time period
in the timeoutHint, the receiving entity SHOULD send an Ack with
the current sequence number and system time sent. The previously
used hint values should be repeated.
If the receiving entity has not sent an Ack since receiving
ipdrCountHit number of records, it SHOULD send an ack with the
current sequence number and system time sent. The previously
used hint values should be repeated.
Note that the lastSeqNum field is dependent on the use of the
seqNum attribute in the IPDRRecords. seqNum is defined in the
IPDRDoc base schema, but is optional. A system wishing to
maintain records for retransmission on failure must use the seqNum
field in each IPDR record in order to identify potentially lost
records.
As the sequence number approaches the value of 2^31-1
(2524971007), the IPDR Recorder should close and re-establish the
connection to avoid rollover. A new document id should be used on
the subsequent connection.
3.3 Illustration
Meyer Expires February 18, 2003 [Page 10]
Internet-Draft Streaming IPDR August 2002
Time t0:
<-- TCP Conn Established -->
+----------------+
| Header |
+----------------+ -- transmitted -->
| Descriptor |
+----------------+
| Ack |
+----------------+
+----------------+
| Header |
<-- transmitted +----------------+
| Ack |
+----------------+
Time t1 (one or more usage event(s) recorded):
+----------------+
| Usage Event | -- transmitted -->
+----------------+
...
+----------------+
| Usage Event |
+----------------+
Time t2 (either time or volume hint triggers Ack):
+----------------+
<-- transmitted | Ack |
+----------------+
Time t3 (idle period has exceeded hint):
+----------------+
| Ack | -- transmitted -->
+----------------+
Basic scenario continues until some terminating control message
arrives and finally at time tn:
+----------------+
| Disconnect | -- transmitted -->
+----------------+
+----------------+
Meyer Expires February 18, 2003 [Page 11]
Internet-Draft Streaming IPDR August 2002
<-- transmitted | Disconnect |
+----------------+
<-- TCP Connection Release -->
Note that additional descriptor stream elements MAY be transmitted in
the stream along with Usage Events at any time.
3.4 Failover Procedure
If an IPDR Recorder determines that it has lost communication with
the current peer, it SHOULD:
send a DISCONNECT message with a reason code of 402 and close the
current TCP connection
establish a TCP connection with an alternate server.
send a document header using the SAME documentId sent on the
previous failed connection.
send the necessary descriptors and Ack stream elements
write the records which were not acknowledged on the previous
connection, repeating the sequence numbers used on the previous
delivery.
write a DISCONNECT message with reasonCode of 200 into the stream.
follow standard connect procedures to create a new docId and TCP
connection to deliver subsequent messages. Note that this
connection MAY be created at any time, it may even have been
created prior to failure detection, as a "hot standby".
Meyer Expires February 18, 2003 [Page 12]
Internet-Draft Streaming IPDR August 2002
4. Security Considerations
Security can be accomplished by using either IPSec [7] or TLS [8]
mechanisms when establishing the underlying TCP connection. This is
the same security policy used by Diameter [9], sec (13.1 and 13.2)
Because the use of IPSec and TLS are transparent to the from the
perspective of TCP connection endpoints, the protocol specified here
is unchanged.
Meyer Expires February 18, 2003 [Page 13]
Internet-Draft Streaming IPDR August 2002
References
[1] World Wide Web Consortium, "Extensible Markup Language (XML)
1.0", W3C XML, February 1998, <http://www.w3.org/TR/1998/REC-
xml-19980210>.
[2] World Wide Web Consortium, "XML Schema Part 1: Structures", W3C
XML, May 2001, <http://www.w3.org/TR/2001/REC-xmlschema-1-
20010502/>.
[3] World Wide Web Consortium, "XML Schema Part 2: Datatypes", W3C
XML, May 2001, <http://www.w3.org/TR/2001/REC-xmlschema-2-
20010502/datatypes.html>.
[4] Internet Protocol Detail Record Organization, "Network Data
Management - Usage (NDM-U) For IP-Based Services Version 3.1",
April 2002, <http://www.ipdr.org/documents/NDM-U_3.1.pdf>.
[5] Srinivasan, R., "XDR: External Data Representation Standard",
RFC 2026, August 1995, <http://www.ietf.org/rfc/rfc1832.txt>.
[6] Brownlee, N. and A. Blount, "Accounting Attributes and Record
Formats", RFC 2924, Sept. 2000, <http://www.ietf.org/rfc/
rfc2924.txt>.
[7] Kent, S., "Security Architecture for the Internet Protocol",
RFC 2401, Nov. 1998, <http://www.ietf.org/rfc/rfc2401.txt>.
[8] Dierks, T., "The TLS Protocol, Version 1.0", RFC 2246, Jan.
1999, <http://www.ietf.org/rfc/rfc2246.txt>.
[9] Calhoun, P., Loughney, J., Guttman, E., Zorn, G. and J. Arkko,
"Diameter Base Protocol", draft Work in progress, Jul. 2002,
<http://search.ietf.org/internet-drafts/draft-ietf-aaa-diameter-
12.txt>.
Author's Address
Jeff Meyer
Hewlett-Packard
19420 Homestead Rd.
Cupertino, CA 95014
US
Phone: +1 408 447-3477
EMail: jeff_meyer2@hp.com
URI: http://www.hp.com
Meyer Expires February 18, 2003 [Page 14]
Internet-Draft Streaming IPDR August 2002
Full Copyright Statement
Copyright (C) The Internet Society (2002). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
Meyer Expires February 18, 2003 [Page 15]