Skip to main content

IP Flow Information Accounting and Export Benchmarking Methodology
RFC 6645

Revision differences

Document history

Date Rev. By Action
2016-11-30
10 (System) Closed request for Last Call review by GENART with state 'Unknown'
2015-10-14
10 (System) Notify list changed from bmwg-chairs@ietf.org, draft-ietf-bmwg-ipflow-meth@ietf.org to (None)
2012-07-11
10 (System) RFC published
2012-04-24
10 (System) IANA Action state changed to No IC
2012-04-24
10 Amy Vezza State changed to RFC Ed Queue from Approved-announcement sent
2012-04-23
10 Amy Vezza State changed to Approved-announcement sent from Approved-announcement to be sent
2012-04-23
10 Amy Vezza IESG has approved the document
2012-04-23
10 Amy Vezza Closed "Approve" ballot
2012-04-23
10 Amy Vezza Ballot approval text was generated
2012-04-23
10 Amy Vezza Ballot writeup was changed
2012-04-23
10 Ron Bonica State changed to Approved-announcement to be sent from IESG Evaluation::AD Followup
2012-04-23
10 Ron Bonica [Ballot Position Update] Position for Ronald Bonica has been changed to Yes from Discuss
2012-04-23
10 (System) Sub state has been changed to AD Followup from Revised ID Needed
2012-04-23
10 Jan Novak New version available: draft-ietf-bmwg-ipflow-meth-10.txt
2012-04-12
09 Cindy Morgan State changed to IESG Evaluation::Revised ID Needed from Waiting for AD Go-Ahead
2012-04-12
09 Benoît Claise
[Ballot comment]
Discussing with Ron, since I was an author for 2 versions of the individual draft, I will recuse myself.

==========================================================================

Introduction
    …
[Ballot comment]
Discussing with Ron, since I was an author for 2 versions of the individual draft, I will recuse myself.

==========================================================================

Introduction
    Monitoring of IP flows (Flow monitoring) is defined in the
    Architecture for IP Flow Information Export [RFC5470] and related
    IPFIX documents.

Which documents? Do we expect the BMWG community to know about the relevant IPFIX documents. Please refer to "them"

- Abstract mentions

  This document provides a methodology and framework for quantifying
  the performance impact of monitoring of IP flows on a network device
  and export of this information to a collector. It identifies the rate
  at which the IP flows are created, expired, and successfully exported
  as a new performance metric in combination with traditional
  throughput. The metric is only applicable to the devices compliant
  with the Architecture for IP Flow Information Export [RFC5470].

However, the introduction mentions:
  This document provides a methodology for measuring Flow
    monitoring performance so that network operators have a framework
    for measurements of impact on the network and network equipment.

So if this document covers both "impact on the network and network equipment", it should be clearly mentioned in the abstract.
Maybe this is what you mean by  "export of this information to a collector", but it can be understood in different ways: impact on the Exporter, and/or Collector, and/or network.


-
    A more
    complete understanding of the stress points of a particular device
    can be attained using this internal information and the tester MAY
    choose to gather this information during the measurement iterations.

replace "device" by "DUT"

-

2.1 Existing Terminology -> I would refer to RFC5101 instead of RFC5470 when possible.
Because RFC5101 will be updated by RFC5101bis. And one important change in RFC5101bis will be the new "Flow" definition, which will be removed "IP". RFC5470 will most likely not have a bis version.



- 2.2.5 Flow Export Rate
     
  Definition:
      The number of Cache entries that expire from the Cache (as defined
      by the Flow Expiration term) and are exported to the Collector
      within a measurement time interval. There SHOULD NOT be any export
      filtering, so that all the expired cache entries are exported. If
      there is export filtering and it can't be disabled, this needs to
      be noted.

If you use to use RFC2119 terms in the definition, be consistent: replace "this needs to
      be noted. " by this MUST be noted".


- 3.2 Device Applicability

  The Flow monitoring performance metric is applicable to network
  devices that implement [RFC5470] architecture.

Replace the end with something similar to: "that implement
RFC5101 and RFC5102, according to [RFC5470] architecture."
After reading the entire draft, I see the sentence I was looking in section 4.3
  The DUT MUST support the Flow monitoring architecture as specified by
  [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
  results comparison due to the standardized export protocol.
You should have something similar in section 3.2

- NetFlow is mentioned. At least refer to RFC3954.

- "The Cache
  entries are expired from the Cache depending on the Cache
  configuration (ie, the Active and Inactive Timeouts, number of Cache
  entries and the Cache Size)"
The cache entries is not a configuration parameter.

-  The DUT's export interface (connecting the Collector) MUST NOT be
  used for forwarding the test traffic but only for the Flow Export
  data containing the Flow Records. In all measurements, the export
  interface MUST have enough bandwidth to transmit Flow Export data
  without congestion. In other words, the export interface MUST NOT be
  a bottleneck during the measurement.

I guess that the "the collector (interface) MUST NOT be a bottleneck during the measurement".
Exactly like you wrote for the traffic receiver " The traffic receiver MUST have sufficient resources to measure all
  test traffic transferred successfully by the DUT."
After reading section 4.4, I see this sentence:
  The Collector MUST be capable of capturing the export packets sent
  from the DUT at the full rate without losing any of them.
This is a source of confusion for me in this draft. It seems that the information is fragmented throughout the draft... Therefore, the reading flow is sometimes not easy...

- what is the difference between "Any such feature configuration MUST be part of the measurement
  report." and "All configurations MUST be fully documented."
Should the latter say "All configurations MUST be fully documented in the measurement report"?

-    The DUT configuration and any existing Cache MUST be erased before
  application of any new configuration for the currently executed
  measurement.

replace Cache by Cache entries?


- Section 4.3.2
  The Cache Size available to the DUT MUST be known and taken into
  account when designing the measurement as specified in section 5.

What about Metering Process features that increase the cache size. Is this allowed? Should this be documented in the measurement report? Or should the cache size be set up to its maximum before starting the measurement?

- Section 4.3.2
  The configuration of the Metering Process MUST be recorded.
-> MUST be included in the measurement report?

- Section 4.3.3
  The templates used by the tested implementations SHOULD
  be analysed and reported as part of the measurement report. Ideally
  only tests with same templates layout should be compared.
"template layout" = Template Record in RFC5101.
Please add to the terminology and use the term.

- Section 4.3.3
  Only benchmarks with the
  same transport layer protocol should be compared.
should -> SHOULD

- Section 4.3.4.
For all the examples such as the following, please include the IPFIX Information Element.
      Flow Keys:
              Source IP address
              Destination IP address
              MPLS label (for MPLS traffic type only)
              Transport layer source port
              Transport layer destination port
              IP protocol number (IPv6 next header)
              IP type of service (IPv6 traffic class)
Rational:
    - what does the MPLS label mean? In IPFIX IANA, we report mplsTopLabelStackSection, which is a combination of "The Label, Exp, and S fields".  Note: clarify this as well in the section 4.3.6
    - packet counters, byte counters: we have multiple IEs for those. Which one should the DUT chose?
    - IP addresses: IPv4 and/or IPv6?

- 4.6 Frame Formats

  Flow monitoring itself is not dependent in any way on the media used
  on the input and output ports. Any media can be used as supported by
  the DUT and the test equipment.

What about the export interface?

- section 4.8
    The used packet size SHOULD be part of the measurement
  report
Why not a MUST?

- section 5.1
I read multiple times the following sentences, and still could not understand the link between the first two.
    The number of unique Flow Keys sets that the traffic
      generator (sender) provides should be multiple times larger than
      the Cache Size. This ensures that the existing Cache entries are
      never updated before Flow Expiration and Flow Export. The Cache
      Size MUST be known in order to define the measurement
      circumstances properly.

- Section 7 Flow Monitoring Accuracy
Don't you have to say a few words about the accuracy of DUT that also does forwarding, i.e. the traffic analysis at the "traffic receiver" in figure 1 must also check that no packets were lost in the case of ingress monitoring. Note: there is also the case of egress monitoring.
Interestingly, bidirectional is mentioned in the "Appendix A: Recommended Report Format", but not a single time in the draft.

- Appendix A:
    would be great to mention which entries are SHOULD and MUST
    Note: not sure why it's an appendix, as there is normative text referring to each entries

- Appendix A
    could be very helpful to poll the IPFIX-MIB (http://tools.ietf.org/html/draft-ietf-ipfix-rfc5815bis-03, currently in RFC-editor queue) and IPFIX-CONF (http://tools.ietf.org/html/draft-ietf-ipfix-configuration-model-10, currently in RFC-editor queue) as MAY in the measurement report.
Note: [IPFIX-CONF] is almost ready, simply waiting for PSAMP-MIB. I believe it makes sense to wait for this RFC.

- Appendix B.
Not sure whether the use of "SHOULD" in the appendix is appropriate. To be checked.

- Appendix B.6 Tests With Bidirectional Traffic
Not sure why this is an appendix. The recommended report mentions
Traffic Direction                unidirectional, bidirectional
  Direction                        ingress, egress, both
... and the bidirectionality is only mentioned in the appendix?

-  Section 5.2
  Traffic Generation
      The traffic generator needs to increment the Flow Keys values with
      each sent packet. This way each packet represents one Cache entry
      in the DUT Cache.

Here is a comment I made to you years ago. You will find surprises if you increment the IP addresses by one in your generator, as opposed to random traffic, as the hash function are not optimized for incremental IP addresses. You should say a few words about this.
2012-04-12
09 Benoît Claise [Ballot Position Update] Position for Benoit Claise has been changed to Recuse from Discuss
2012-04-12
09 Ron Bonica
[Ballot comment]
The following are Benoit's DISCUSS comments:

- Introduction
    The most significant performance parameter is the rate at which IP
    …
[Ballot comment]
The following are Benoit's DISCUSS comments:

- Introduction
    The most significant performance parameter is the rate at which IP
    flows are created and expired in the network device's memory and
    exported to a collector

One or multiple different rates? I guess different ones (but reading the
document further will tell). So:
    The most significant performance parameters
are the rates at which IP
    flows are created, expired in the network device's
memory, and
    exported to a collector
However, looking at the terminology
section, it seems that you have only one benchmark metric: "Flow Monitoring
Throughput".
BMWG is about black box testing, but it doesn't mean that we don't
have 3 different rates. The section 3.1 about "the Flow Monitoring Throughput"
proves I'm right. Please improve the text.

- See email "No active/inactive timeout definitions in any IPFIX RFCs? Idle
versus inactive terminology? (part of draft-ietf-bmwg-ipflow-meth-09 review)"
sent to the IPFIX WG.

- Section 4.3.1
  The (*) in Figure 2 designates the Observation Points in the default
  configuration. Other DUT Observation Points might be configured
  depending on the specific measurement needs as follows:

  a. ingress port/ports only
  b. egress port/ports only
  c. both ingress and egress

If I refer to figure 2, there is no return traffic to the "traffic sender".
Therefore, how could it be b. or c.?
Am I dreaming or you had in the past a
similar figure that explains that the return traffic could come back to the
"traffic sender"?
Figure 2 should be updated, or a new figure added, because,
for egress, the traffic analysis must happen also on the "traffic sender"

- Section 4.3.3
  The Exporting Process SHOULD be configured with IPFIX
[RFC5101] as
  the protocol to use to format the Flow Export data
You want a
MUST here, as IP Flow = IPFIX at the IETF.
Same remark for this sentence in 4.3
The DUT MUST support the Flow monitoring architecture as specified by
[RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
  results
comparison due to the standardized export protocol.
Same remark for this
sentence in 4.4
  However if the Collector is also used to decode the Flow
Export data
  then it SHOULD support IPFIX [RFC5101] for meaningful results
However, looking at figure 1, you mention NetFlow, others. So you want to add
that additional export mechanism MAY use the same benchmarking mechanism, i.e.
NetFlow v9 [RFC3954]
Comment (2012-04-10)

- Introduction
    Monitoring of IP flows (Flow monitoring) is defined in the
    Architecture for IP Flow Information Export [RFC5470] and related
    IPFIX documents.

Which documents? Do we expect the BMWG community to know about the relevant
IPFIX documents. Please refer to "them"

- Abstract mentions

  This document provides a methodology and framework for quantifying
  the performance impact of monitoring of IP flows on a network device
  and export of this information to a collector. It identifies the rate
  at which the IP flows are created, expired, and successfully exported
  as a new performance metric in combination with traditional
  throughput. The metric is only applicable to the devices compliant
  with the Architecture for IP Flow Information Export [RFC5470].

However, the introduction mentions:
  This document provides a methodology for measuring Flow
    monitoring performance so that network operators have a framework
    for measurements of impact on the network and network equipment.

So if this document covers both "impact on the network and network equipment",
it should be clearly mentioned in the abstract.
Maybe this is what you mean by
"export of this information to a collector", but it can be understood in
different ways: impact on the Exporter, and/or Collector, and/or network.


-
    A more
    complete understanding of the stress points of a particular device
    can be attained using this internal information and the tester MAY
    choose to gather this information during the measurement iterations.

replace "device" by "DUT"

-

2.1 Existing Terminology -> I would refer to RFC5101 instead of RFC5470 when
possible.
Because RFC5101 will be updated by RFC5101bis. And one important
change in RFC5101bis will be the new "Flow" definition, which will be removed
"IP". RFC5470 will most likely not have a bis version.

- 2.2.5 Flow Export Rate
     
  Definition:
      The number of Cache entries that expire from the Cache (as defined
      by the Flow Expiration term) and are exported to the Collector
      within a measurement time interval. There SHOULD NOT be any export
      filtering, so that all the expired cache entries are exported. If
      there is export filtering and it can't be disabled, this needs to
      be noted.

If you use to use RFC2119 terms in the definition, be consistent: replace "this
needs to
      be noted. " by this MUST be noted".


- 3.2 Device Applicability

  The Flow monitoring performance metric is applicable to network
  devices that implement [RFC5470] architecture.

Replace the end with something similar to: "that implement
RFC5101 and RFC5102, according to [RFC5470] architecture."
After reading the entire draft, I see the sentence I was looking in section 4.3
  The DUT MUST support the Flow monitoring architecture as specified by
  [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
  results comparison due to the standardized export protocol.
You should have something similar in section 3.2

- NetFlow is mentioned. At least refer to RFC3954.

- "The Cache
  entries are expired from the Cache depending on the Cache
  configuration (ie, the Active and Inactive Timeouts, number of Cache
  entries and the Cache Size)"
The cache entries is not a configuration parameter.

-  The DUT's export interface (connecting the Collector) MUST NOT be
  used for forwarding the test traffic but only for the Flow Export
  data containing the Flow Records. In all measurements, the export
  interface MUST have enough bandwidth to transmit Flow Export data
  without congestion. In other words, the export interface MUST NOT be
  a bottleneck during the measurement.

I guess that the "the collector (interface) MUST NOT be a bottleneck during the
measurement".
Exactly like you wrote for the traffic receiver " The traffic
receiver MUST have sufficient resources to measure all
  test traffic
transferred successfully by the DUT."
After reading section 4.4, I see this
sentence:
  The Collector MUST be capable of capturing the export packets sent
from the DUT at the full rate without losing any of them.
This is a source of
confusion for me in this draft. It seems that the information is fragmented
throughout the draft... Therefore, the reading flow is sometimes not easy...

- what is the difference between "Any such feature configuration MUST be part of
the measurement
  report." and "All configurations MUST be fully documented."
Should the latter say "All configurations MUST be fully documented in the
measurement report"?

-    The DUT configuration and any existing Cache MUST be erased before
  application of any new configuration for the currently executed
  measurement.

replace Cache by Cache entries?


- Section 4.3.2
  The Cache Size available to the DUT MUST be known and taken into
  account when designing the measurement as specified in section 5.

What about Metering Process features that increase the cache size. Is this
allowed? Should this be documented in the measurement report? Or should the
cache size be set up to its maximum before starting the measurement?

- Section 4.3.2
  The configuration of the Metering Process MUST be recorded.
-> MUST be included in the measurement report?

- Section 4.3.3
  The templates used by the tested implementations SHOULD
  be analysed and reported as part of the measurement report. Ideally
  only tests with same templates layout should be compared.
"template layout" = Template Record in RFC5101.
Please add to the terminology and use the term.

- Section 4.3.3
  Only benchmarks with the
  same transport layer protocol should be compared.
should -> SHOULD

- Section 4.3.4.
For all the examples such as the following, please include the
IPFIX Information Element.
      Flow Keys:
              Source IP address
Destination IP address
              MPLS label (for MPLS traffic type only)
Transport layer source port
              Transport layer destination port
IP protocol number (IPv6 next header)
              IP type of service (IPv6
traffic class)
Rational:
    - what does the MPLS label mean? In IPFIX IANA, we
report mplsTopLabelStackSection, which is a combination of "The Label, Exp, and
S fields".  Note: clarify this as well in the section 4.3.6
    - packet
counters, byte counters: we have multiple IEs for those. Which one should the
DUT chose?
    - IP addresses: IPv4 and/or IPv6?

- 4.6 Frame Formats

  Flow monitoring itself is not dependent in any way on the media used
  on the input and output ports. Any media can be used as supported by
  the DUT and the test equipment.

What about the export interface?

- section 4.8
    The used packet size SHOULD be part of the measurement
  report
Why not a MUST?

- section 5.1
I read multiple times the following sentences, and still could not
understand the link between the first two.
    The number of unique Flow Keys
sets that the traffic
      generator (sender) provides should be multiple times
larger than
      the Cache Size. This ensures that the existing Cache entries
are
      never updated before Flow Expiration and Flow Export. The Cache
Size MUST be known in order to define the measurement
      circumstances
properly.

- Section 7 Flow Monitoring Accuracy
Don't you have to say a few words about the
accuracy of DUT that also does forwarding, i.e. the traffic analysis at the
"traffic receiver" in figure 1 must also check that no packets were lost in the
case of ingress monitoring. Note: there is also the case of egress monitoring.
Interestingly, bidirectional is mentioned in the "Appendix A: Recommended Report
Format", but not a single time in the draft.

- Appendix A:
    would be great to mention which entries are SHOULD and MUST
Note: not sure why it's an appendix, as there is normative text referring to
each entries

- Appendix A
    could be very helpful to poll the IPFIX-MIB
(http://tools.ietf.org/html/draft-ietf-ipfix-rfc5815bis-03, currently in RFC-
editor queue) and IPFIX-CONF (http://tools.ietf.org/html/draft-ietf-ipfix-
configuration-model-10, currently in RFC-editor queue) as MAY in the measurement
report.
Note: [IPFIX-CONF] is almost ready, simply waiting for PSAMP-MIB. I
believe it makes sense to wait for this RFC.

- Appendix B.
Not sure whether the use of "SHOULD" in the appendix is
appropriate. To be checked.

- Appendix B.6 Tests With Bidirectional Traffic
Not sure why this is an appendix. The recommended report mentions
Traffic Direction                unidirectional, bidirectional
  Direction                        ingress, egress, both
... and the bidirectionality is only mentioned in the appendix?

-  Section 5.2
  Traffic Generation
      The traffic generator needs to increment the Flow Keys values with
      each sent packet. This way each packet represents one Cache entry
      in the DUT Cache.

Here is a comment I made to you years ago. You will find surprises if you
increment the IP addresses by one in your generator, as opposed to random
traffic, as the hash function are not optimized for incremental IP addresses.
You should say a few words about this.
2012-04-12
09 Ron Bonica Ballot comment text updated for Ronald Bonica
2012-04-12
09 Ron Bonica
[Ballot discuss]
I am taking the very strange action of posting a DISCUSS on my own document. Benoit has some significant issues that need to …
[Ballot discuss]
I am taking the very strange action of posting a DISCUSS on my own document. Benoit has some significant issues that need to be worked out with the author and the WG. However, he must recuse because he was once a co-author of this draft. So, I will carry his DISCUSS and work towards resolving it.
2012-04-12
09 Ron Bonica [Ballot Position Update] Position for Ronald Bonica has been changed to Discuss from Yes
2012-04-12
09 Gonzalo Camarillo [Ballot Position Update] New position, No Objection, has been recorded for Gonzalo Camarillo
2012-04-12
09 Stewart Bryant [Ballot comment]
On the basis of a quick read and complete confidence that Benoit will work with the authors to make the draft perfect.
2012-04-12
09 Stewart Bryant [Ballot Position Update] New position, No Objection, has been recorded for Stewart Bryant
2012-04-12
09 Pete Resnick
[Ballot comment]
As usual, I find most of the uses of 2119 language in a document of this sort to be bizarre: I don't understand …
[Ballot comment]
As usual, I find most of the uses of 2119 language in a document of this sort to be bizarre: I don't understand how a statement like "MUST be part of the measurement report" constitutes something "required for interoperation or to limit behavior which has the potential for causing harm", and is not simply trying "to impose a particular method on implementors where the method is not required for interoperability." I'm hard pressed to find any occurrence of 2119 language in this document that is used as 2119 intended.
2012-04-12
09 Pete Resnick [Ballot Position Update] New position, No Objection, has been recorded for Pete Resnick
2012-04-11
09 Wesley Eddy [Ballot Position Update] New position, No Objection, has been recorded for Wesley Eddy
2012-04-11
09 Sean Turner [Ballot Position Update] New position, No Objection, has been recorded for Sean Turner
2012-04-11
09 Robert Sparks [Ballot Position Update] New position, No Objection, has been recorded for Robert Sparks
2012-04-11
09 Stephen Farrell [Ballot Position Update] New position, No Objection, has been recorded for Stephen Farrell
2012-04-11
09 Brian Haberman [Ballot Position Update] New position, No Objection, has been recorded for Brian Haberman
2012-04-10
09 Ralph Droms
[Ballot comment]
A statement like this in section 1 begs for a little more explanation:

    The most significant performance parameter is the rate …
[Ballot comment]
A statement like this in section 1 begs for a little more explanation:

    The most significant performance parameter is the rate at which IP
    flows are created and expired in the network device's memory and
    exported to a collector.

Is there a reference or some other justification for this statement?
Could this statement simply be elided without losing the importance of
the document?

And here are a couple of minor editorial or clarification suggestions:

In section 3.4.2:

  Mainly the Flow Export
  Rate caused by the test traffic during an [RFC2544] measurement MUST
  be known and reported.

s/Mainly/Most importantly/  ??

In section 4.9.1 and 4.9.2, is it intended that the destination IP
address recycles to the address for stream 1 after stream 10000?
2012-04-10
09 Ralph Droms [Ballot Position Update] New position, No Objection, has been recorded for Ralph Droms
2012-04-10
09 Benoît Claise
[Ballot discuss]
- Introduction
    The most significant performance parameter is the rate at which IP
    flows are created and expired in …
[Ballot discuss]
- Introduction
    The most significant performance parameter is the rate at which IP
    flows are created and expired in the network device's memory and
    exported to a collector

One or multiple different rates? I guess different ones (but reading the document further will tell). So:
    The most significant performance parameters are the rates at which IP
    flows are created, expired in the network device's memory, and
    exported to a collector
However, looking at the terminology section, it seems that you have only one benchmark metric: "Flow Monitoring Throughput".
BMWG is about black box testing, but it doesn't mean that we don't have 3 different rates. The section 3.1 about "the Flow Monitoring Throughput" proves I'm right. Please improve the text.

- See email "No active/inactive timeout definitions in any IPFIX RFCs? Idle versus inactive terminology? (part of draft-ietf-bmwg-ipflow-meth-09 review)" sent to the IPFIX WG.

- Section 4.3.1
  The (*) in Figure 2 designates the Observation Points in the default
  configuration. Other DUT Observation Points might be configured
  depending on the specific measurement needs as follows:

  a. ingress port/ports only
  b. egress port/ports only
  c. both ingress and egress

If I refer to figure 2, there is no return traffic to the "traffic sender". Therefore, how could it be b. or c.?
Am I dreaming or you had in the past a similar figure that explains that the return traffic could come back to the "traffic sender"?
Figure 2 should be updated, or a new figure added, because, for egress, the traffic analysis must happen also on the "traffic sender"

- Section 4.3.3
  The Exporting Process SHOULD be configured with IPFIX [RFC5101] as
  the protocol to use to format the Flow Export data
You want a MUST here, as IP Flow = IPFIX at the IETF.
Same remark for this sentence in 4.3
  The DUT MUST support the Flow monitoring architecture as specified by
  [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
  results comparison due to the standardized export protocol.
Same remark for this sentence in 4.4
  However if the Collector is also used to decode the Flow Export data
  then it SHOULD support IPFIX [RFC5101] for meaningful results
However, looking at figure 1, you mention NetFlow, others. So you want to add that additional export mechanism MAY use the same benchmarking mechanism, i.e. NetFlow v9 [RFC3954]
2012-04-10
09 Benoît Claise
[Ballot comment]
- Introduction
    Monitoring of IP flows (Flow monitoring) is defined in the
    Architecture for IP Flow Information Export [ …
[Ballot comment]
- Introduction
    Monitoring of IP flows (Flow monitoring) is defined in the
    Architecture for IP Flow Information Export [RFC5470] and related
    IPFIX documents.

Which documents? Do we expect the BMWG community to know about the relevant IPFIX documents. Please refer to "them"

- Abstract mentions

  This document provides a methodology and framework for quantifying
  the performance impact of monitoring of IP flows on a network device
  and export of this information to a collector. It identifies the rate
  at which the IP flows are created, expired, and successfully exported
  as a new performance metric in combination with traditional
  throughput. The metric is only applicable to the devices compliant
  with the Architecture for IP Flow Information Export [RFC5470].

However, the introduction mentions:
  This document provides a methodology for measuring Flow
    monitoring performance so that network operators have a framework
    for measurements of impact on the network and network equipment.

So if this document covers both "impact on the network and network equipment", it should be clearly mentioned in the abstract.
Maybe this is what you mean by  "export of this information to a collector", but it can be understood in different ways: impact on the Exporter, and/or Collector, and/or network.


-
    A more
    complete understanding of the stress points of a particular device
    can be attained using this internal information and the tester MAY
    choose to gather this information during the measurement iterations.

replace "device" by "DUT"

-

2.1 Existing Terminology -> I would refer to RFC5101 instead of RFC5470 when possible.
Because RFC5101 will be updated by RFC5101bis. And one important change in RFC5101bis will be the new "Flow" definition, which will be removed "IP". RFC5470 will most likely not have a bis version.



- 2.2.5 Flow Export Rate
     
  Definition:
      The number of Cache entries that expire from the Cache (as defined
      by the Flow Expiration term) and are exported to the Collector
      within a measurement time interval. There SHOULD NOT be any export
      filtering, so that all the expired cache entries are exported. If
      there is export filtering and it can't be disabled, this needs to
      be noted.

If you use to use RFC2119 terms in the definition, be consistent: replace "this needs to
      be noted. " by this MUST be noted".


- 3.2 Device Applicability

  The Flow monitoring performance metric is applicable to network
  devices that implement [RFC5470] architecture.

Replace the end with something similar to: "that implement
RFC5101 and RFC5102, according to [RFC5470] architecture."
After reading the entire draft, I see the sentence I was looking in section 4.3
  The DUT MUST support the Flow monitoring architecture as specified by
  [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
  results comparison due to the standardized export protocol.
You should have something similar in section 3.2

- NetFlow is mentioned. At least refer to RFC3954.

- "The Cache
  entries are expired from the Cache depending on the Cache
  configuration (ie, the Active and Inactive Timeouts, number of Cache
  entries and the Cache Size)"
The cache entries is not a configuration parameter.

-  The DUT's export interface (connecting the Collector) MUST NOT be
  used for forwarding the test traffic but only for the Flow Export
  data containing the Flow Records. In all measurements, the export
  interface MUST have enough bandwidth to transmit Flow Export data
  without congestion. In other words, the export interface MUST NOT be
  a bottleneck during the measurement.

I guess that the "the collector (interface) MUST NOT be a bottleneck during the measurement".
Exactly like you wrote for the traffic receiver " The traffic receiver MUST have sufficient resources to measure all
  test traffic transferred successfully by the DUT."
After reading section 4.4, I see this sentence:
  The Collector MUST be capable of capturing the export packets sent
  from the DUT at the full rate without losing any of them.
This is a source of confusion for me in this draft. It seems that the information is fragmented throughout the draft... Therefore, the reading flow is sometimes not easy...

- what is the difference between "Any such feature configuration MUST be part of the measurement
  report." and "All configurations MUST be fully documented."
Should the latter say "All configurations MUST be fully documented in the measurement report"?

-    The DUT configuration and any existing Cache MUST be erased before
  application of any new configuration for the currently executed
  measurement.

replace Cache by Cache entries?


- Section 4.3.2
  The Cache Size available to the DUT MUST be known and taken into
  account when designing the measurement as specified in section 5.

What about Metering Process features that increase the cache size. Is this allowed? Should this be documented in the measurement report? Or should the cache size be set up to its maximum before starting the measurement?

- Section 4.3.2
  The configuration of the Metering Process MUST be recorded.
-> MUST be included in the measurement report?

- Section 4.3.3
  The templates used by the tested implementations SHOULD
  be analysed and reported as part of the measurement report. Ideally
  only tests with same templates layout should be compared.
"template layout" = Template Record in RFC5101.
Please add to the terminology and use the term.

- Section 4.3.3
  Only benchmarks with the
  same transport layer protocol should be compared.
should -> SHOULD

- Section 4.3.4.
For all the examples such as the following, please include the IPFIX Information Element.
      Flow Keys:
              Source IP address
              Destination IP address
              MPLS label (for MPLS traffic type only)
              Transport layer source port
              Transport layer destination port
              IP protocol number (IPv6 next header)
              IP type of service (IPv6 traffic class)
Rational:
    - what does the MPLS label mean? In IPFIX IANA, we report mplsTopLabelStackSection, which is a combination of "The Label, Exp, and S fields".  Note: clarify this as well in the section 4.3.6
    - packet counters, byte counters: we have multiple IEs for those. Which one should the DUT chose?
    - IP addresses: IPv4 and/or IPv6?

- 4.6 Frame Formats

  Flow monitoring itself is not dependent in any way on the media used
  on the input and output ports. Any media can be used as supported by
  the DUT and the test equipment.

What about the export interface?

- section 4.8
    The used packet size SHOULD be part of the measurement
  report
Why not a MUST?

- section 5.1
I read multiple times the following sentences, and still could not understand the link between the first two.
    The number of unique Flow Keys sets that the traffic
      generator (sender) provides should be multiple times larger than
      the Cache Size. This ensures that the existing Cache entries are
      never updated before Flow Expiration and Flow Export. The Cache
      Size MUST be known in order to define the measurement
      circumstances properly.

- Section 7 Flow Monitoring Accuracy
Don't you have to say a few words about the accuracy of DUT that also does forwarding, i.e. the traffic analysis at the "traffic receiver" in figure 1 must also check that no packets were lost in the case of ingress monitoring. Note: there is also the case of egress monitoring.
Interestingly, bidirectional is mentioned in the "Appendix A: Recommended Report Format", but not a single time in the draft.

- Appendix A:
    would be great to mention which entries are SHOULD and MUST
    Note: not sure why it's an appendix, as there is normative text referring to each entries

- Appendix A
    could be very helpful to poll the IPFIX-MIB (http://tools.ietf.org/html/draft-ietf-ipfix-rfc5815bis-03, currently in RFC-editor queue) and IPFIX-CONF (http://tools.ietf.org/html/draft-ietf-ipfix-configuration-model-10, currently in RFC-editor queue) as MAY in the measurement report.
Note: [IPFIX-CONF] is almost ready, simply waiting for PSAMP-MIB. I believe it makes sense to wait for this RFC.

- Appendix B.
Not sure whether the use of "SHOULD" in the appendix is appropriate. To be checked.

- Appendix B.6 Tests With Bidirectional Traffic
Not sure why this is an appendix. The recommended report mentions
Traffic Direction                unidirectional, bidirectional
  Direction                        ingress, egress, both
... and the bidirectionality is only mentioned in the appendix?

-  Section 5.2
  Traffic Generation
      The traffic generator needs to increment the Flow Keys values with
      each sent packet. This way each packet represents one Cache entry
      in the DUT Cache.

Here is a comment I made to you years ago. You will find surprises if you increment the IP addresses by one in your generator, as opposed to random traffic, as the hash function are not optimized for incremental IP addresses. You should say a few words about this.
2012-04-10
09 Benoît Claise [Ballot Position Update] New position, Discuss, has been recorded for Benoit Claise
2012-04-09
09 Russ Housley [Ballot Position Update] New position, No Objection, has been recorded for Russ Housley
2012-04-06
09 Adrian Farrel [Ballot Position Update] New position, No Objection, has been recorded for Adrian Farrel
2012-04-03
09 Amanda Baber We understand that this document doesn't require any IANA actions.
2012-04-03
09 Amanda Baber We understand that this document doesn't require any IANA actions.
2012-03-26
09 (System) State changed to Waiting for AD Go-Ahead from In Last Call
2012-03-16
09 Samuel Weiler Request for Last Call review by SECDIR is assigned to Ondřej Surý
2012-03-16
09 Samuel Weiler Request for Last Call review by SECDIR is assigned to Ondřej Surý
2012-03-15
09 Jean Mahoney Request for Last Call review by GENART is assigned to Joel Halpern
2012-03-15
09 Jean Mahoney Request for Last Call review by GENART is assigned to Joel Halpern
2012-03-12
09 Amy Vezza Last call sent
2012-03-12
09 Amy Vezza
State changed to In Last Call from Last Call Requested

The following Last Call Announcement was sent out:

From: The IESG

To: IETF-Announce

CC:

Reply-To: …
State changed to In Last Call from Last Call Requested

The following Last Call Announcement was sent out:

From: The IESG

To: IETF-Announce

CC:

Reply-To: ietf@ietf.org

Subject: Last Call:  (IP Flow Information Accounting and Export Benchmarking Methodology) to Informational RFC





The IESG has received a request from the Benchmarking Methodology WG

(bmwg) to consider the following document:

- 'IP Flow Information Accounting and Export Benchmarking Methodology'

  as an Informational RFC



The IESG plans to make a decision in the next few weeks, and solicits

final comments on this action. Please send substantive comments to the

ietf@ietf.org mailing lists by 2012-03-26. Exceptionally, comments may be

sent to iesg@ietf.org instead. In either case, please retain the

beginning of the Subject line to allow automated sorting.



Abstract





  This document provides a methodology and framework for quantifying

  the performance impact of monitoring of IP flows on a network device

  and export of this information to a collector. It identifies the rate

  at which the IP flows are created, expired, and successfully exported

  as a new performance metric in combination with traditional

  throughput. The metric is only applicable to the devices compliant

  with the Architecture for IP Flow Information Export [RFC5470].









The file can be obtained via

http://datatracker.ietf.org/doc/draft-ietf-bmwg-ipflow-meth/



IESG discussion can be tracked via

http://datatracker.ietf.org/doc/draft-ietf-bmwg-ipflow-meth/ballot/





No IPR declarations have been submitted directly on this I-D.





2012-03-12
09 Ron Bonica Placed on agenda for telechat - 2012-04-12
2012-03-12
09 Ron Bonica Last call was requested
2012-03-12
09 Ron Bonica Last call announcement was generated
2012-03-12
09 Ron Bonica State changed to Last Call Requested from AD Evaluation::AD Followup
2012-03-12
09 Ron Bonica Ballot has been issued
2012-03-12
09 Ron Bonica Ballot approval text was generated
2012-03-12
09 Ron Bonica [Ballot Position Update] New position, Yes, has been recorded for Ronald Bonica
2012-03-12
09 Ron Bonica Ballot writeup was changed
2012-03-12
09 Ron Bonica Created "Approve" ballot
2012-03-12
09 Jan Novak New version available: draft-ietf-bmwg-ipflow-meth-09.txt
2012-03-09
08 (System) Sub state has been changed to AD Followup from Revised ID Needed
2012-03-09
08 Jan Novak New version available: draft-ietf-bmwg-ipflow-meth-08.txt
2012-03-05
07 Ron Bonica State changed to AD Evaluation::Revised ID Needed from Publication Requested
2012-02-27
07 Ron Bonica Ballot writeup was generated
2012-02-09
07 Al Morton Changed protocol writeup
2012-02-09
07 Al Morton Submitted Jan 31
2012-02-09
07 Al Morton IETF state changed to Submitted to IESG for Publication from WG Document
2012-02-09
07 Al Morton Publication Requested
2012-02-09
07 Al Morton Annotation tag Other - see Comment Log set. Annotation tag Revised I-D Needed - Issue raised by WGLC cleared.
2012-01-30
07 Cindy Morgan
This is a Publication Request from bmwg:

Document Title: IP Flow Information Accounting and Export
Benchmarking Methodology
Filename:        draft-ietf-bmwg-ipflow-meth-07.txt
Intended Status: Informational …
This is a Publication Request from bmwg:

Document Title: IP Flow Information Accounting and Export
Benchmarking Methodology
Filename:        draft-ietf-bmwg-ipflow-meth-07.txt
Intended Status: Informational


  (1.a) Who is the Document Shepherd for this document? Has the
        Document Shepherd personally reviewed this version of the
        document and, in particular, does he or she believe this
        version is ready for forwarding to the IESG for publication?
Al Morton has reviewed the memo, it is ready (+/- a few minor edits,
see the end of this write-up).
This will be a difficult read for those not familiar with at least
one of the areas covered here (IPFIX and Benchmarking), but the intended
audience of testers should be adequately served.

  (1.b) Has the document had adequate review both from key WG members
        and from key non-WG members? Does the Document Shepherd have
        any concerns about the depth or breadth of the reviews that
        have been performed?
The Current draft reflects extensive feedback, beginning with its
first review at a session in Dublin. Throughout the process, non-wg
people have been involved in reviews and WG last calls, especially
the IPFIX WG (obviously).

  (1.c) Does the Document Shepherd have concerns that the document
        needs more review from a particular or broader perspective,
        e.g., security, operational complexity, someone familiar with
        AAA, internationalization or XML?
No.

  (1.d) Does the Document Shepherd have any specific concerns or
        issues with this document that the Responsible Area Director
        and/or the IESG should be aware of? For example, perhaps he
        or she is uncomfortable with certain parts of the document, or
        has concerns whether there really is a need for it. In any
        event, if the WG has discussed those issues and has indicated
        that it still wishes to advance the document, detail those
        concerns here. Has an IPR disclosure related to this document
        been filed? If so, please include a reference to the
        disclosure and summarize the WG discussion and conclusion on
        this issue.
No concerns and No IPR disclosures.

  (1.e) How solid is the WG consensus behind this document? Does it
        represent the strong concurrence of a few individuals, with
        others being silent, or does the WG as a whole understand and
        agree with it?
Quite a few bmwg participants and ipfix participants have given this a look
and now concur with the results. It took several WGLC before the version
reached consensus (with a few minor editorial changes).
Examples of Test Implementation and Results were presented
during development, which is compelling evidence of practicality.
There were WGLCs yielding long lists of comments/issues to deal with,
and this was finally accomplished.

  (1.f) Has anyone threatened an appeal or otherwise indicated extreme
        discontent? If so, please summarise the areas of conflict in
        separate email messages to the Responsible Area Director. (It
        should be in a separate email because this questionnaire is
        entered into the ID Tracker.)
No Appeals Threatened.

  (1.g) Has the Document Shepherd personally verified that the
        document satisfies all ID nits? (See the Internet-Drafts Checklist
        and http://tools.ietf.org/tools/idnits/). Boilerplate checks are
        not enough; this check needs to be thorough. Has the document
        met all formal review criteria it needs to, such as the MIB
        Doctor, media type and URI type reviews?
All nits appear to be satisfied, with a few minor editorial changes needed.

  (1.h) Has the document split its references into normative and
        informative? Are there normative references to documents that
        are not ready for advancement or are otherwise in an unclear
        state? If such normative references exist, what is the
        strategy for their completion? Are there normative references
        that are downward references, as described in [RFC3967]? If
        so, list these downward references to support the Area
        Director in the Last Call procedure for them [RFC3967].
The Refs are split and the Normative Refs are stable.
No DownRefs.

  (1.i) Has the Document Shepherd verified that the document IANA
        consideration section exists and is consistent with the body
        of the document? If the document specifies protocol
        extensions, are reservations requested in appropriate IANA
        registries? Are the IANA registries clearly identified? If
        the document creates a new registry, does it define the
        proposed initial contents of the registry and an allocation
        procedure for future registrations? Does it suggest a
        reasonable name for the new registry? See [RFC5226]. If the
        document describes an Expert Review process has Shepherd
        conferred with the Responsible Area Director so that the IESG
        can appoint the needed Expert during the IESG Evaluation?
IANA section exists, making no IANA requests, as is usual in bmwg.

  (1.j) Has the Document Shepherd verified that sections of the
        document that are written in a formal language, such as XML
        code, BNF rules, MIB definitions, etc., validate correctly in
        an automated checker?
NA

  (1.k) The IESG approval announcement includes a Document
        Announcement Write-Up.

      Technical Summary
For inter-networking devices that perform routing or switching as
their primary function, the likely reduction in traffic-handling
capacity when traffic monitoring is active continues to be a relevant
question many years after it was first asked ("What happens when you
turn-on Netflow?").

This document provides a methodology and framework for quantifying
the performance impact of monitoring of IP flows on a network device
and export of this information to a collector. It identifies the rate
at which the IP flows are created, expired, and successfully exported
as a new performance metric in combination with traditional
throughput. The metric is only applicable to the devices compliant
with the Architecture for IP Flow Information Export [RFC5470].

The methods are applicable to both inter-networking devices that
forward traffic and other devices that simply monitor traffic with
non-intrusive access to transmission facilities.
The Forwarding Plane and Monitoring Plane represent two separate
functional blocks, each with its own performance capability. The
Forwarding Plane handles user data packets and is fully characterised
by the metrics defined by [RFC2544].
The Monitoring Plane handles Flows which reflect the analysed
traffic. The metric for Monitoring Plane performance is Flow Export
Rate, and the benchmark is the Flow Monitoring Throughput.


      Working Group Summary
Quite a few bmwg participants and ipfix participants have given this a look
and now concur with the results.
Examples of Test Implementation and Results were presented
during development, which is compelling evidence of practicality.
There were WGLCs yielding long lists of comments/issues to deal with,
and this was finally accomplished. It took several WGLCs before this version
reached consensus (with a few minor editorial changes).


      Document Quality
All would agree that Paul Aitken provided very careful and complete
reviews throughout the development process; he left no stone unturned.

-=-=-=-=-=-=-=-=-=-=-=-

Minor Editorial Points:

Sec3.3
s/each with it's own performance/each with its own performance/

Sec3.4.2, last para
s/The details how/The details of how/

Sec4
s/4. Measurement Set Up/4. Measurement Set-up/


Section 6.4 and 6.5
>>> since these are options of 6.3, it makes more sense if
they are numbered 6.3.1 and 6.3.2:
s/6.4/6.3.1/
s/6.5/6.3.5/ everywhere on pages 24 and 25

Sec 7
s/c. all the possible Flow Record fields values/c. all the possible
Flow Record field values/

Sec 8
s/Packet per flow/Packets per flow/
and
s/Be required to process/be required to process/



2012-01-30
07 Cindy Morgan Draft added in state Publication Requested
2012-01-30
07 Cindy Morgan [Note]: 'Al Morton (acmorton@att.com) is the document shepherd.' added
2012-01-30
07 (System) New version available: draft-ietf-bmwg-ipflow-meth-07.txt
2012-01-22
06 (System) New version available: draft-ietf-bmwg-ipflow-meth-06.txt
2011-12-06
05 (System) New version available: draft-ietf-bmwg-ipflow-meth-05.txt
2011-12-05
07 Al Morton IETF state changed to WG Document from In WG Last Call
2011-12-05
07 Al Morton 5 Issues need to be closed
2011-12-05
07 Al Morton Annotation tag Revised I-D Needed - Issue raised by WGLC set.
2011-10-02
04 (System) New version available: draft-ietf-bmwg-ipflow-meth-04.txt
2011-07-11
03 (System) New version available: draft-ietf-bmwg-ipflow-meth-03.txt
2011-06-21
02 (System) New version available: draft-ietf-bmwg-ipflow-meth-02.txt
2011-05-30
07 Al Morton LC ends June 20, 2011
2011-05-30
07 Al Morton IETF state changed to In WG Last Call from WG Document
2011-04-15
01 (System) New version available: draft-ietf-bmwg-ipflow-meth-01.txt
2010-12-13
00 (System) New version available: draft-ietf-bmwg-ipflow-meth-00.txt