IP Flow Information Accounting and Export Benchmarking Methodology
draft-ietf-bmwg-ipflow-meth-10

The following are Benoit's DISCUSS comments:

 - Introduction
    The most significant performance parameter is the rate at which IP
    flows are created and expired in the network device's memory and
    exported to a collector

One or multiple different rates? I guess different ones (but reading the
document further will tell). So:
    The most significant performance parameters
are the rates at which IP
    flows are created, expired in the network device's
memory, and
    exported to a collector
However, looking at the terminology
section, it seems that you have only one benchmark metric: "Flow Monitoring
Throughput".
BMWG is about black box testing, but it doesn't mean that we don't
have 3 different rates. The section 3.1 about "the Flow Monitoring Throughput"
proves I'm right. Please improve the text.

- See email "No active/inactive timeout definitions in any IPFIX RFCs? Idle
versus inactive terminology? (part of draft-ietf-bmwg-ipflow-meth-09 review)"
sent to the IPFIX WG.

- Section 4.3.1
   The (*) in Figure 2 designates the Observation Points in the default
   configuration. Other DUT Observation Points might be configured
   depending on the specific measurement needs as follows:

   a. ingress port/ports only
   b. egress port/ports only
   c. both ingress and egress

If I refer to figure 2, there is no return traffic to the "traffic sender".
Therefore, how could it be b. or c.?
Am I dreaming or you had in the past a
similar figure that explains that the return traffic could come back to the
"traffic sender"?
Figure 2 should be updated, or a new figure added, because,
for egress, the traffic analysis must happen also on the "traffic sender"

- Section 4.3.3
  The Exporting Process SHOULD be configured with IPFIX
[RFC5101] as
   the protocol to use to format the Flow Export data
You want a
MUST here, as IP Flow = IPFIX at the IETF.
Same remark for this sentence in 4.3
The DUT MUST support the Flow monitoring architecture as specified by
[RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
   results
comparison due to the standardized export protocol.
Same remark for this
sentence in 4.4
   However if the Collector is also used to decode the Flow
Export data
   then it SHOULD support IPFIX [RFC5101] for meaningful results
However, looking at figure 1, you mention NetFlow, others. So you want to add
that additional export mechanism MAY use the same benchmarking mechanism, i.e.
NetFlow v9 [RFC3954]
Comment (2012-04-10) 

- Introduction
    Monitoring of IP flows (Flow monitoring) is defined in the
    Architecture for IP Flow Information Export [RFC5470] and related
    IPFIX documents.

Which documents? Do we expect the BMWG community to know about the relevant
IPFIX documents. Please refer to "them"

- Abstract mentions

   This document provides a methodology and framework for quantifying
   the performance impact of monitoring of IP flows on a network device
   and export of this information to a collector. It identifies the rate
   at which the IP flows are created, expired, and successfully exported
   as a new performance metric in combination with traditional
   throughput. The metric is only applicable to the devices compliant
   with the Architecture for IP Flow Information Export [RFC5470].

However, the introduction mentions:
   This document provides a methodology for measuring Flow
    monitoring performance so that network operators have a framework
    for measurements of impact on the network and network equipment.

So if this document covers both "impact on the network and network equipment",
it should be clearly mentioned in the abstract.
Maybe this is what you mean by
"export of this information to a collector", but it can be understood in
different ways: impact on the Exporter, and/or Collector, and/or network.


-
    A more
    complete understanding of the stress points of a particular device
    can be attained using this internal information and the tester MAY 
    choose to gather this information during the measurement iterations. 

replace "device" by "DUT"

-

2.1 Existing Terminology -> I would refer to RFC5101 instead of RFC5470 when
possible.
Because RFC5101 will be updated by RFC5101bis. And one important
change in RFC5101bis will be the new "Flow" definition, which will be removed
"IP". RFC5470 will most likely not have a bis version.

- 2.2.5 Flow Export Rate
      
   Definition:
      The number of Cache entries that expire from the Cache (as defined
      by the Flow Expiration term) and are exported to the Collector
      within a measurement time interval. There SHOULD NOT be any export
      filtering, so that all the expired cache entries are exported. If
      there is export filtering and it can't be disabled, this needs to
      be noted.

If you use to use RFC2119 terms in the definition, be consistent: replace "this
needs to
      be noted. " by this MUST be noted".


- 3.2 Device Applicability

   The Flow monitoring performance metric is applicable to network
   devices that implement [RFC5470] architecture.

Replace the end with something similar to: "that implement
RFC5101 and RFC5102, according to [RFC5470] architecture."
After reading the entire draft, I see the sentence I was looking in section 4.3
   The DUT MUST support the Flow monitoring architecture as specified by
   [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
   results comparison due to the standardized export protocol.
You should have something similar in section 3.2

- NetFlow is mentioned. At least refer to RFC3954.

- "The Cache
   entries are expired from the Cache depending on the Cache
   configuration (ie, the Active and Inactive Timeouts, number of Cache
   entries and the Cache Size)"
The cache entries is not a configuration parameter.

-  The DUT's export interface (connecting the Collector) MUST NOT be
   used for forwarding the test traffic but only for the Flow Export
   data containing the Flow Records. In all measurements, the export
   interface MUST have enough bandwidth to transmit Flow Export data
   without congestion. In other words, the export interface MUST NOT be
   a bottleneck during the measurement.

I guess that the "the collector (interface) MUST NOT be a bottleneck during the
measurement".
Exactly like you wrote for the traffic receiver " The traffic
receiver MUST have sufficient resources to measure all
   test traffic
transferred successfully by the DUT."
After reading section 4.4, I see this
sentence:
  The Collector MUST be capable of capturing the export packets sent
from the DUT at the full rate without losing any of them.
This is a source of
confusion for me in this draft. It seems that the information is fragmented
throughout the draft... Therefore, the reading flow is sometimes not easy...

- what is the difference between "Any such feature configuration MUST be part of
the measurement
   report." and "All configurations MUST be fully documented."
Should the latter say "All configurations MUST be fully documented in the
measurement report"?

-    The DUT configuration and any existing Cache MUST be erased before
   application of any new configuration for the currently executed
   measurement.

replace Cache by Cache entries?


- Section 4.3.2
   The Cache Size available to the DUT MUST be known and taken into
   account when designing the measurement as specified in section 5.

What about Metering Process features that increase the cache size. Is this
allowed? Should this be documented in the measurement report? Or should the
cache size be set up to its maximum before starting the measurement?

- Section 4.3.2
  The configuration of the Metering Process MUST be recorded.
-> MUST be included in the measurement report?

- Section 4.3.3
   The templates used by the tested implementations SHOULD
   be analysed and reported as part of the measurement report. Ideally
   only tests with same templates layout should be compared.
"template layout" = Template Record in RFC5101.
Please add to the terminology and use the term.

- Section 4.3.3
   Only benchmarks with the
   same transport layer protocol should be compared.
should -> SHOULD

- Section 4.3.4.
For all the examples such as the following, please include the
IPFIX Information Element.
      Flow Keys:
               Source IP address
Destination IP address
               MPLS label (for MPLS traffic type only)
Transport layer source port
               Transport layer destination port
IP protocol number (IPv6 next header)
               IP type of service (IPv6
traffic class)
Rational:
    - what does the MPLS label mean? In IPFIX IANA, we
report mplsTopLabelStackSection, which is a combination of "The Label, Exp, and
S fields".  Note: clarify this as well in the section 4.3.6
    - packet
counters, byte counters: we have multiple IEs for those. Which one should the
DUT chose?
    - IP addresses: IPv4 and/or IPv6?

- 4.6 Frame Formats

   Flow monitoring itself is not dependent in any way on the media used
   on the input and output ports. Any media can be used as supported by
   the DUT and the test equipment.

What about the export interface?

- section 4.8
    The used packet size SHOULD be part of the measurement
   report
Why not a MUST?

- section 5.1
I read multiple times the following sentences, and still could not
understand the link between the first two.
     The number of unique Flow Keys
sets that the traffic
      generator (sender) provides should be multiple times
larger than
      the Cache Size. This ensures that the existing Cache entries
are
      never updated before Flow Expiration and Flow Export. The Cache
Size MUST be known in order to define the measurement
      circumstances
properly.

- Section 7 Flow Monitoring Accuracy
Don't you have to say a few words about the
accuracy of DUT that also does forwarding, i.e. the traffic analysis at the
"traffic receiver" in figure 1 must also check that no packets were lost in the
case of ingress monitoring. Note: there is also the case of egress monitoring.
Interestingly, bidirectional is mentioned in the "Appendix A: Recommended Report
Format", but not a single time in the draft.

- Appendix A:
    would be great to mention which entries are SHOULD and MUST
Note: not sure why it's an appendix, as there is normative text referring to
each entries

- Appendix A
    could be very helpful to poll the IPFIX-MIB
(http://tools.ietf.org/html/draft-ietf-ipfix-rfc5815bis-03, currently in RFC-
editor queue) and IPFIX-CONF (http://tools.ietf.org/html/draft-ietf-ipfix-
configuration-model-10, currently in RFC-editor queue) as MAY in the measurement
report.
Note: [IPFIX-CONF] is almost ready, simply waiting for PSAMP-MIB. I
believe it makes sense to wait for this RFC.

- Appendix B.
Not sure whether the use of "SHOULD" in the appendix is
appropriate. To be checked.

- Appendix B.6 Tests With Bidirectional Traffic
Not sure why this is an appendix. The recommended report mentions
 Traffic Direction                 unidirectional, bidirectional
  Direction                         ingress, egress, both
... and the bidirectionality is only mentioned in the appendix?

-  Section 5.2
   Traffic Generation
      The traffic generator needs to increment the Flow Keys values with
      each sent packet. This way each packet represents one Cache entry
      in the DUT Cache.

Here is a comment I made to you years ago. You will find surprises if you
increment the IP addresses by one in your generator, as opposed to random
traffic, as the hash function are not optimized for incremental IP addresses.
You should say a few words about this.