Flow Aggregation for the IP Flow Information Export (IPFIX) Protocol
draft-ietf-ipfix-a9n-08

[Ballot comment]
  The definition of an aggregated flow in Section 2 reads:

    Aggregated Flow:  A Flow, as defined by
    [I-D.ietf-ipfix-protocol-rfc5101bis], derived from a set of zero
    or more original Flows within a defined Aggregation Interval.  The
    primary difference between a Flow and an Aggregated Flow in the
    general case is that the time interval (i.e., the two-tuple of
    start and end times) of a Flow is derived from information about
    the timing of the packets comprising the Flow, while the time
    interval of an Aggregated Flow is often externally imposed.  Note
    that an Aggregated Flow is defined in the context of an
    Intermediate Aggregation Process only.  Once an Aggregated Flow is
    exported, it is essentially a Flow as in
    [I-D.ietf-ipfix-protocol-rfc5101bis] and can be treated as such.

  The way the second phrase is written confuses me. If 'the time
  interval of an Aggregated Flow' is  externally imposed, this
  means that there are exceptions? In which cases? And in these cases
  what is the specific difference that makes that Flow to be Aggregated?


  Section 5.1.1 says:

    Each counter for an Original Flow is divided by the
    number of time _units_ the Original Flow covers, to derive a mean
    count rate.

  What is the meaning of _units_?

[Ballot comment]
I'll follow up on Stephen's Comment and ask about
the requirements language.  There is very little
RFC 2119 language, and some of it seems unnecessary:

  Additional such Information Elements
  SHOULD be registered with IANA on an as-needed basis.

Is the RFC 2119 language really needed anywhere?

[Ballot comment]
About half way through I read the other AD discuss/comment positions that were already entered and noted that I was starting to feel the same way Stephen did in his first comment.  At 4.3 pages/requirement, the draft seems rather long winded to standardize aggregation of IPFIX flow data.  I guess the community finds this approach acceptable so that's good enough for me.

[Ballot comment]


- I was a bit puzzled as to what exactly is being specified
here. It seems like there's little normative text here and most
of that is in draft-ietf-ipfix-mediation-protocol, so I
wondered it it wouldn't have been better for this to be an
informational guide and for all the normative stuff to be in
the mediation protocol draft that a developer might be more
likely to read?  To be clear: I'm not fussed about whether this
is standards track or informational nor about the use of 2119
language, but more about whether all the stuff that a developer
might need is in the right document(s). (57 pages of mostly
advice is the kind of thing that'll get ignored:-) Maybe if you
added a sentence in the intro saying that only bits of section
5 and section 7 are normative (or whatever is the case) that
might help the reader?

- 5.1.1 defines a bunch of methods, one of which is the
"simulated process" method of distributing counters over
aggregated flow intervals, however there is nothing in the
description of the simulated process that could be directly
implemented that I can see.  That's puzzling so I don't know
what code might be written to do this, that'd result in
interop. The "Direct" method seems equally vague btw.

- 5.1.1, "SHOULD accept ... in any reasonable order" seems
overly vague, esp. when you say that the meaning of "reasonable
order" might not be clear when writing code.

- 6.4, are the "should" statements here meant as 2119 SHOULDs
or not?

- the examples look great! thanks

[Ballot discuss]

  In Section 5.3.1 the URL quoted for the IPFIX Information Elements
  registry http://www.iana.org/assignments/ipfix/ipfix.html does not
  exist.

  The IANA Note in section 7.2.4 is supposed to be left in the text? If
  not, is it necessary to mention someplace else the issue of backwards
  compatibility with the IE value in NetFlow version 9?

  Same question for the IANA note in Section 10.

[Ballot comment]

  The definition of an aggregated flow in Section 2 reads:

    Aggregated Flow:  A Flow, as defined by
    [I-D.ietf-ipfix-protocol-rfc5101bis], derived from a set of zero
    or more original Flows within a defined Aggregation Interval.  The
    primary difference between a Flow and an Aggregated Flow in the
    general case is that the time interval (i.e., the two-tuple of
    start and end times) of a Flow is derived from information about
    the timing of the packets comprising the Flow, while the time
    interval of an Aggregated Flow is often externally imposed.  Note
    that an Aggregated Flow is defined in the context of an
    Intermediate Aggregation Process only.  Once an Aggregated Flow is
    exported, it is essentially a Flow as in
    [I-D.ietf-ipfix-protocol-rfc5101bis] and can be treated as such.

  The way the second phrase is written confuses me. If 'the time
  interval of an Aggregated Flow' is  externally imposed, this
  means that there are exceptions? In which cases? And in these cases
  what is the specific difference that makes that Flow to be Aggregated?


  Section 5.1.1 says:

    Each counter for an Original Flow is divided by the
    number of time _units_ the Original Flow covers, to derive a mean
    count rate.

  What is the meaning of _units_?

[Ballot comment]
A couple of minor comments; no email response necessary:

-- Sections 5.3.1 and 12.2 --
You use three slightly different URIs in the document for the IANA registry:
http://www.iana.org/assignments/ipfix/ipfix.html  (in Section 5.3.1)
http://www.iana.org/assignments/ipfix  (in Section 10)
http://www.iana.org/assignments/ipfix/ipfix.xml  (in Section 12.2)

The second is the form that IANA prefers, and it will redirect to the correct page; please change them all to use that form.  (The third will currently work; the first does not (because the registry was converted from HTML to XML)... but the second will continue to work even if they reformat the page in JSON or whatever.)

-- Section 7.2.4 --
  [IANA NOTE: This Information Element is compatible with Information
  Element 3 as used in NetFlow version 9.]

Is this meant to remain after publication?  Or should you say that the RFC Editor should remove this note?

IANA has reviewed draft-ietf-ipfix-a9n-07 and has the following comments:

IANA has a question about the IANA actions requested in this document

IANA understands that, upon approval of this document, there are twelve
actions which IANA must complete.

All of the actions are additions of IPFIX Information Elemenets to the IPFIX
Information Element registry located at:

http://www.iana.org/assignments/ipfix/ipfix.xml

Currently the IPFIX Information Elements registry is maintained through expert
review as defined in RFC 5226.

IANA Question -> has the document been reviewed by the IPFIX Information
Elements  registry expert?

ElementID: [ tbd ]
Name: originalFlowsPresent
Data Type: unsigned64
Data Type Semantics: deltaCount
Status:
Description: The non-conservative count of Original Flows contributing to this
Aggregated Flow.  Non-conservative counts need not sum to the original count on
re-aggregation.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: originalFlowsInitiated
Data Type: unsigned64
Data Type Semantics: deltaCount
Status:
Description: The conservative count of Original Flows whose first packet is
represented within this Aggregated Flow.  Conservative counts must sum to the
original count on re-aggregation.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: originalFlowsCompleted
Data Type: unsigned64
Data Type Semantics: deltaCount
Status:
Description: The conservative count of Original Flows whose last packet is
represented within this Aggregated Flow.  Conservative counts must sum to the
original count on re-aggregation.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: 3
Name: deltaFlowCount
Data Type: unsigned64
Data Type Semantics: deltaCount
Status:
Description: The conservative count of Original Flows contributing to this
Aggregated Flow; may be distributed via any of the methods expressed by the
valueDistributionMethod Information Element.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: distinctCountOfSourceIPAddress
Data Type: unsigned64
Data Type Semantics: totalCount
Status:
Description: The count of distinct source IP address values for Original Flows
contributing to this Aggregated Flow, without regard to IP version.  This
Information Element is preferred to the IP-version-specific counters, unless it
is important to separate the counts by version.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name:
Data Type:
Data Type Semantics:
Status:
Description:
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: distinctCountOfDestinationIPAddress
Data Type: unsigned64
Data Type Semantics: totalCount
Status:
Description: The count of distinct destination IP address values for Original
Flows contributing to this Aggregated Flow, without regard to IP version.  This
Information Element is preferred to the version-specific counters below, unless
it is important to separate the counts by version.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: distinctCountOfSourceIPv4Address
Data Type: unsigned32
Data Type Semantics: totalCount
Status:
Description: The count of distinct source IPv4 address values for Original Flows
contributing to this Aggregated Flow.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: distinctCountOfDestinationIPv4Address
Data Type: unsigned32
Data Type Semantics: totalCount
Status:
Description: The count of distinct destination IPv4 address values for Original
Flows contributing to this Aggregated Flow
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: distinctCountOfSourceIPv6Address
Data Type: unsigned64
Data Type Semantics: totalCount
Status:
Description: The count of distinct source IPv6 address values for Original Flows
contributing to this Aggregated Flow.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: distinctCountOfDestinationIPv6Address
Data Type: unsigned64
Data Type Semantics: totalCount
Status:
Description: The count of distinct destination IPv6 address values for Original
Flows contributing to this Aggregated Flow.
Units:
Range:
Reference: [ RFC-to-be ]

ElementID: [ tbd ]
Name: valueDistributionMethod
Data Type: unsigned8
Data Type Semantics:
Status: Current
Description: A description of the method used to distribute the counters from
Contributing Flows into the Aggregated Flow records described by an associated
scope, generally a Template.  The method is deemed to apply to all the non-key
Information Elements in the referenced scope for which value distribution is a
valid operation; if the originalFlowsInitiated and/or originalFlowsCompleted
Information Elements appear in the Template, they are not subject to this
distribution method, as they each infer their own distribution method.  This is
intended to be a complete set of possible value distribution methods; it is
encoded as follows:

+-------+-----------------------------------------------------------+
| Value | Description                                              |
+-------+-----------------------------------------------------------+
| 0    | Unspecified: The counters for an Original Flow are        |
|      | explicitly not distributed according to any other method  |
|      | defined for this Information Element; use for arbitrary  |
|      | distribution, or distribution algorithms not described by |
|      | any other codepoint.                                      |
|      | --------------------------------------------------------- |
|      |                                                          |
| 1    | Start Interval: The counters for an Original Flow are    |
|      | added to the counters of the appropriate Aggregated Flow  |
|      | containing the start time of the Original Flow.  This    |
|      | should be assumed the default if value distribution      |
|      | information is not available at a Collecting Process for  |
|      | an Aggregated Flow.                                      |
|      | --------------------------------------------------------- |
|      |                                                          |
| 2    | End Interval: The counters for an Original Flow are added |
|      | to the counters of the appropriate Aggregated Flow        |
|      | containing the end time of the Original Flow.            |
|      | --------------------------------------------------------- |
|      |                                                          |
| 3    | Mid Interval: The counters for an Original Flow are added |
|      | to the counters of a single appropriate Aggregated Flow  |
|      | containing some timestamp between start and end time of  |
|      | the Original Flow.                                        |
|      | --------------------------------------------------------- |
|      |                                                          |
| 4    | Simple Uniform Distribution: Each counter for an Original |
|      | Flow is divided by the number of time intervals the      |
|      | Original Flow covers (i.e., of appropriate Aggregated    |
|      | Flows sharing the same Flow Key), and this number is      |
|      | added to each corresponding counter in each Aggregated    |
|      | Flow.                                                    |
|      | --------------------------------------------------------- |
|      |                                                          |
| 5    | Proportional Uniform Distribution: Each counter for an    |
|      | Original Flow is divided by the number of time _units_    |
|      | the Original Flow covers, to derive a mean count rate.    |
|      | This mean count rate is then multiplied by the number of  |
|      | time units in the intersection of the duration of the    |
|      | Original Flow and the time interval of each Aggregated    |
|      | Flow.  This is like simple uniform distribution, but      |
|      | accounts for the fractional portions of a time interval  |
|      | covered by an Original Flow in the first and last time    |
|      | interval.                                                |
|      | --------------------------------------------------------- |
| 6    | Simulated Process: Each counter of the Original Flow is  |
|      | distributed among the intervals of the Aggregated Flows  |
|      | according to some function the Intermediate Aggregation  |
|      | Process uses based upon properties of Flows presumed to  |
|      | be like the Original Flow.  This is essentially an        |
|      | assertion that the Intermediate Aggregation Process has  |
|      | no direct packet timing information but is nevertheless  |
|      | not using one of the other simpler distribution methods.  |
|      | The Intermediate Aggregation Process specifically makes  |
|      | no assertion as to the correctness of the simulation.    |
|      | --------------------------------------------------------- |
|      |                                                          |
| 7    | Direct: The Intermediate Aggregation Process has access  |
|      | to the original packet timings from the packets making up |
|      | the Original Flow, and uses these to distribute or        |
|      | recalculate the counters.                                |
+-------+-----------------------------------------------------------+

IANA understands that these are the only actions requied to be completed
upon approval of this document.

Note:  The actions requested in this document will not be completed
until the document has been approved for publication as an RFC.

(1) What type of RFC is being requested (BCP, Proposed Standard,
    Internet Standard, Informational, Experimental, or Historic)?
    Why is this the proper type of RFC?  Is this type of RFC
    indicated in the title page header?

  Standards Track.  Needs to be so that all implementations will
  aggregate IPFIX flow data in the same way.  Yes, its header says
  'Standards Track.'

(2) The IESG approval announcement includes a Document Announcement
    Write-Up. Please provide such a Document Announcement Write-Up.
    Recent examples can be found in the "Action" announcements for
    approved documents. The approval announcement contains the
    following sections:

  Technical Summary
    This document provides a common implementation-independent basis
    for the interoperable application of the IP Flow Information
    Export (IPFIX) Protocol to the handling of Aggregated Flows, which
    are IPFIX Flows representing packets from multiple Original Flows
    that share some set of common properties.

  Working Group Summary
    This draft attracted real discussion on the IPFIX list, and took
    time to reach consensus on its final approach, i.e. "through a
    detailed terminology and a descriptive Intermediate Aggregation
    Process architecture, including a specification of methods for
    Original Flow counting and counter distribution across intervals."

  Document Quality
    Are there existing implementations of the protocol? Have a
    significant number of vendors indicated their plan to
    implement the specification?
    I'm not aware of any, so far.

    Are there any reviewers that merit special mention as having
    done a thorough review ... ?
    Rahul Patel was a strong contributor to the discussion,
    Paul Aitken provided a very thorough review.

  Personnel
    Who is the Document Shepherd?          Nevil Brownlee
    Who is the Responsible Area Director?  Benoit Claise

(3) Briefly describe the review of this document that was performed
    by the Document Shepherd.  If this version of the document is not
    ready for publication, please explain why the document is being
    forwarded to the IESG.

  I have followed this draft carefully from its -00 version, particularly
  it's notion of of 'spatial aggregation,' a topic which needs careful
  definition of its terms!  I feel that it is ready for publication.

(4) Does the document Shepherd have any concerns about the depth or
    breadth of the reviews that have been performed?

  No.

(5) Do portions of the document need review from a particular or from
    broader perspective, e.g., security, operational complexity, AAA,
    DNS, DHCP, XML, or internationalization? If so, describe the
    review that took place.

  No.  This draft follows tha IPFIX naming conventions; those are
  its only external dependencies.

(6) Describe any specific concerns or issues that the Document
    Shepherd has with this document that the Responsible Area
    Director and/or the IESG should be aware of? For example, perhaps
    he or she is uncomfortable with certain parts of the document, or
    has concerns whether there really is a need for it. In any event,
    if the WG has discussed those issues and has indicated that it
    still wishes to advance the document, detail those concerns here.

  No (but see section 8).

(7) Has each author confirmed that any and all appropriate IPR
    disclosures required for full conformance with the provisions of
    BCP 78 and BCP 79 have already been filed. If not, explain why.

Yes (see section 8 for more detail)

(8) Has an IPR disclosure been filed that references this document?
    If so, summarize any WG discussion and conclusion regarding the
    IPR disclosures.

  We (the IPFIX WG) were unaware of any IPR for this draft until it
  was in WG Last Call.  Carter Bullard raised the question, we now
  have two IPR decparations (Avaya and Cisco) and no others have
  come forward.

(9) How solid is the WG consensus behind this document? Does it
    represent the strong concurrence of a few individuals, with
    others being silent, or does the WG as a whole understand and
    agree with it?

  The WG as a whole has reached strong consensus on this draft.

(10) Has anyone threatened an appeal or otherwise indicated extreme
    discontent? If so, please summarise the areas of conflict in
    separate email messages to the Responsible Area Director. (It
    should be in a separate email because this questionnaire is
    publicly available.)

  No.

(11) Identify any ID nits the Document Shepherd has found in this
    document. (See http://www.ietf.org/tools/idnits/ and the
    Internet-Drafts Checklist). Boilerplate checks are not enough;
    this check needs to be thorough.

  ID-nits says "0 errors"

(12) Describe how the document meets any required formal review
    criteria, such as the MIB Doctor, media type, and URI type
    reviews.

  No formal reviews required.

(13) Have all references within this document been identified as
    either normative or informative?

  Yes.

(14) Are there normative references to documents that are not ready
    for advancement or are otherwise in an unclear state? If such
    normative references exist, what is the plan for their
    completion?

  Normative reference to I-D 5101-bis.
  IPFIX 5101bis (and 5102bis) are close to WG Last Call,
  that's now the WG's highest-priority activity.


(15) Are there downward normative references references (see RFC
    3967)?  If so, list these downward references to support the Area
    Director in the Last Call procedure.

  No.

(16) Will publication of this document change the status of any
    existing RFCs? Are those RFCs listed on the title page header,
    listed in the abstract, and discussed in the introduction? If the
    RFCs are not listed in the Abstract and Introduction, explain
    why, and point to the part of the document where the relationship
    of this document to the other RFCs is discussed. If this
    information is not in the document, explain why the WG considers
    it unnecessary.

  No.

(17) Describe the Document Shepherd's review of the IANA
    considerations section, especially with regard to its consistency
    with the body of the document. Confirm that all protocol
    extensions that the document makes are associated with the
    appropriate reservations in IANA registries.  Confirm that any
    referenced IANA registries have been clearly identified. Confirm
    that newly created IANA registries include a detailed
    specification of the initial contents for the registry, that
    allocations procedures for future registrations are defined, and
    a reasonable name for the new registry has been suggested (see
    RFC 5226).

  The draft's IANA Considerations only requires some new IPFIX
  Information Elements to be assigned, its instructions for that
  are clear and consistent.

(18) List any new IANA registries that require Expert Review for
    future allocations. Provide any public guidance that the IESG
    would find useful in selecting the IANA Experts for these new
    registries.

  No new registries.

(19) Describe reviews and automated checks performed by the Document
    Shepherd to validate sections of the document written in a formal
    language, such as XML code, BNF rules, MIB definitions, etc.

  No sections in a formal language.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Flow Aggregation for the IP Flow Information Export (IPFIX) Protocol) to Proposed Standard


The IESG has received a request from the IP Flow Information Export WG
(ipfix) to consider the following document:
- 'Flow Aggregation for the IP Flow Information Export (IPFIX) Protocol'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2012-11-13. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document provides a common implementation-independent basis for
  the interoperable application of the IP Flow Information Export
  (IPFIX) Protocol to the handling of Aggregated Flows, which are IPFIX
  Flows representing packets from multiple Original Flows sharing some
  set of common properties.  It does this through a detailed
  terminology and a descriptive Intermediate Aggregation Process
  architecture, including a specification of methods for Original Flow
  counting and counter distribution across intervals.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-ipfix-a9n/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-ipfix-a9n/ballot/


The following IPR Declarations may be related to this I-D:

  http://datatracker.ietf.org/ipr/1715/
  http://datatracker.ietf.org/ipr/1726/