IP Flow Information Accounting and Export Benchmarking Methodology
RFC 6645
Yes
No Objection
Recuse
Note: This ballot was opened for revision 09 and is now closed.
(Ron Bonica; former steering group member) (was Discuss, Yes) Yes
The following are Benoit's DISCUSS comments:
- Introduction
The most significant performance parameter is the rate at which IP
flows are created and expired in the network device's memory and
exported to a collector
One or multiple different rates? I guess different ones (but reading the
document further will tell). So:
The most significant performance parameters
are the rates at which IP
flows are created, expired in the network device's
memory, and
exported to a collector
However, looking at the terminology
section, it seems that you have only one benchmark metric: "Flow Monitoring
Throughput".
BMWG is about black box testing, but it doesn't mean that we don't
have 3 different rates. The section 3.1 about "the Flow Monitoring Throughput"
proves I'm right. Please improve the text.
- See email "No active/inactive timeout definitions in any IPFIX RFCs? Idle
versus inactive terminology? (part of draft-ietf-bmwg-ipflow-meth-09 review)"
sent to the IPFIX WG.
- Section 4.3.1
The (*) in Figure 2 designates the Observation Points in the default
configuration. Other DUT Observation Points might be configured
depending on the specific measurement needs as follows:
a. ingress port/ports only
b. egress port/ports only
c. both ingress and egress
If I refer to figure 2, there is no return traffic to the "traffic sender".
Therefore, how could it be b. or c.?
Am I dreaming or you had in the past a
similar figure that explains that the return traffic could come back to the
"traffic sender"?
Figure 2 should be updated, or a new figure added, because,
for egress, the traffic analysis must happen also on the "traffic sender"
- Section 4.3.3
The Exporting Process SHOULD be configured with IPFIX
[RFC5101] as
the protocol to use to format the Flow Export data
You want a
MUST here, as IP Flow = IPFIX at the IETF.
Same remark for this sentence in 4.3
The DUT MUST support the Flow monitoring architecture as specified by
[RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
results
comparison due to the standardized export protocol.
Same remark for this
sentence in 4.4
However if the Collector is also used to decode the Flow
Export data
then it SHOULD support IPFIX [RFC5101] for meaningful results
However, looking at figure 1, you mention NetFlow, others. So you want to add
that additional export mechanism MAY use the same benchmarking mechanism, i.e.
NetFlow v9 [RFC3954]
Comment (2012-04-10)
- Introduction
Monitoring of IP flows (Flow monitoring) is defined in the
Architecture for IP Flow Information Export [RFC5470] and related
IPFIX documents.
Which documents? Do we expect the BMWG community to know about the relevant
IPFIX documents. Please refer to "them"
- Abstract mentions
This document provides a methodology and framework for quantifying
the performance impact of monitoring of IP flows on a network device
and export of this information to a collector. It identifies the rate
at which the IP flows are created, expired, and successfully exported
as a new performance metric in combination with traditional
throughput. The metric is only applicable to the devices compliant
with the Architecture for IP Flow Information Export [RFC5470].
However, the introduction mentions:
This document provides a methodology for measuring Flow
monitoring performance so that network operators have a framework
for measurements of impact on the network and network equipment.
So if this document covers both "impact on the network and network equipment",
it should be clearly mentioned in the abstract.
Maybe this is what you mean by
"export of this information to a collector", but it can be understood in
different ways: impact on the Exporter, and/or Collector, and/or network.
-
A more
complete understanding of the stress points of a particular device
can be attained using this internal information and the tester MAY
choose to gather this information during the measurement iterations.
replace "device" by "DUT"
-
2.1 Existing Terminology -> I would refer to RFC5101 instead of RFC5470 when
possible.
Because RFC5101 will be updated by RFC5101bis. And one important
change in RFC5101bis will be the new "Flow" definition, which will be removed
"IP". RFC5470 will most likely not have a bis version.
- 2.2.5 Flow Export Rate
Definition:
The number of Cache entries that expire from the Cache (as defined
by the Flow Expiration term) and are exported to the Collector
within a measurement time interval. There SHOULD NOT be any export
filtering, so that all the expired cache entries are exported. If
there is export filtering and it can't be disabled, this needs to
be noted.
If you use to use RFC2119 terms in the definition, be consistent: replace "this
needs to
be noted. " by this MUST be noted".
- 3.2 Device Applicability
The Flow monitoring performance metric is applicable to network
devices that implement [RFC5470] architecture.
Replace the end with something similar to: "that implement
RFC5101 and RFC5102, according to [RFC5470] architecture."
After reading the entire draft, I see the sentence I was looking in section 4.3
The DUT MUST support the Flow monitoring architecture as specified by
[RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
results comparison due to the standardized export protocol.
You should have something similar in section 3.2
- NetFlow is mentioned. At least refer to RFC3954.
- "The Cache
entries are expired from the Cache depending on the Cache
configuration (ie, the Active and Inactive Timeouts, number of Cache
entries and the Cache Size)"
The cache entries is not a configuration parameter.
- The DUT's export interface (connecting the Collector) MUST NOT be
used for forwarding the test traffic but only for the Flow Export
data containing the Flow Records. In all measurements, the export
interface MUST have enough bandwidth to transmit Flow Export data
without congestion. In other words, the export interface MUST NOT be
a bottleneck during the measurement.
I guess that the "the collector (interface) MUST NOT be a bottleneck during the
measurement".
Exactly like you wrote for the traffic receiver " The traffic
receiver MUST have sufficient resources to measure all
test traffic
transferred successfully by the DUT."
After reading section 4.4, I see this
sentence:
The Collector MUST be capable of capturing the export packets sent
from the DUT at the full rate without losing any of them.
This is a source of
confusion for me in this draft. It seems that the information is fragmented
throughout the draft... Therefore, the reading flow is sometimes not easy...
- what is the difference between "Any such feature configuration MUST be part of
the measurement
report." and "All configurations MUST be fully documented."
Should the latter say "All configurations MUST be fully documented in the
measurement report"?
- The DUT configuration and any existing Cache MUST be erased before
application of any new configuration for the currently executed
measurement.
replace Cache by Cache entries?
- Section 4.3.2
The Cache Size available to the DUT MUST be known and taken into
account when designing the measurement as specified in section 5.
What about Metering Process features that increase the cache size. Is this
allowed? Should this be documented in the measurement report? Or should the
cache size be set up to its maximum before starting the measurement?
- Section 4.3.2
The configuration of the Metering Process MUST be recorded.
-> MUST be included in the measurement report?
- Section 4.3.3
The templates used by the tested implementations SHOULD
be analysed and reported as part of the measurement report. Ideally
only tests with same templates layout should be compared.
"template layout" = Template Record in RFC5101.
Please add to the terminology and use the term.
- Section 4.3.3
Only benchmarks with the
same transport layer protocol should be compared.
should -> SHOULD
- Section 4.3.4.
For all the examples such as the following, please include the
IPFIX Information Element.
Flow Keys:
Source IP address
Destination IP address
MPLS label (for MPLS traffic type only)
Transport layer source port
Transport layer destination port
IP protocol number (IPv6 next header)
IP type of service (IPv6
traffic class)
Rational:
- what does the MPLS label mean? In IPFIX IANA, we
report mplsTopLabelStackSection, which is a combination of "The Label, Exp, and
S fields". Note: clarify this as well in the section 4.3.6
- packet
counters, byte counters: we have multiple IEs for those. Which one should the
DUT chose?
- IP addresses: IPv4 and/or IPv6?
- 4.6 Frame Formats
Flow monitoring itself is not dependent in any way on the media used
on the input and output ports. Any media can be used as supported by
the DUT and the test equipment.
What about the export interface?
- section 4.8
The used packet size SHOULD be part of the measurement
report
Why not a MUST?
- section 5.1
I read multiple times the following sentences, and still could not
understand the link between the first two.
The number of unique Flow Keys
sets that the traffic
generator (sender) provides should be multiple times
larger than
the Cache Size. This ensures that the existing Cache entries
are
never updated before Flow Expiration and Flow Export. The Cache
Size MUST be known in order to define the measurement
circumstances
properly.
- Section 7 Flow Monitoring Accuracy
Don't you have to say a few words about the
accuracy of DUT that also does forwarding, i.e. the traffic analysis at the
"traffic receiver" in figure 1 must also check that no packets were lost in the
case of ingress monitoring. Note: there is also the case of egress monitoring.
Interestingly, bidirectional is mentioned in the "Appendix A: Recommended Report
Format", but not a single time in the draft.
- Appendix A:
would be great to mention which entries are SHOULD and MUST
Note: not sure why it's an appendix, as there is normative text referring to
each entries
- Appendix A
could be very helpful to poll the IPFIX-MIB
(http://tools.ietf.org/html/draft-ietf-ipfix-rfc5815bis-03, currently in RFC-
editor queue) and IPFIX-CONF (http://tools.ietf.org/html/draft-ietf-ipfix-
configuration-model-10, currently in RFC-editor queue) as MAY in the measurement
report.
Note: [IPFIX-CONF] is almost ready, simply waiting for PSAMP-MIB. I
believe it makes sense to wait for this RFC.
- Appendix B.
Not sure whether the use of "SHOULD" in the appendix is
appropriate. To be checked.
- Appendix B.6 Tests With Bidirectional Traffic
Not sure why this is an appendix. The recommended report mentions
Traffic Direction unidirectional, bidirectional
Direction ingress, egress, both
... and the bidirectionality is only mentioned in the appendix?
- Section 5.2
Traffic Generation
The traffic generator needs to increment the Flow Keys values with
each sent packet. This way each packet represents one Cache entry
in the DUT Cache.
Here is a comment I made to you years ago. You will find surprises if you
increment the IP addresses by one in your generator, as opposed to random
traffic, as the hash function are not optimized for incremental IP addresses.
You should say a few words about this.
(Adrian Farrel; former steering group member) No Objection
(Brian Haberman; former steering group member) No Objection
(Gonzalo Camarillo; former steering group member) No Objection
(Pete Resnick; former steering group member) No Objection
As usual, I find most of the uses of 2119 language in a document of this sort to be bizarre: I don't understand how a statement like "MUST be part of the measurement report" constitutes something "required for interoperation or to limit behavior which has the potential for causing harm", and is not simply trying "to impose a particular method on implementors where the method is not required for interoperability." I'm hard pressed to find any occurrence of 2119 language in this document that is used as 2119 intended.
(Ralph Droms; former steering group member) No Objection
A statement like this in section 1 begs for a little more explanation:
The most significant performance parameter is the rate at which IP
flows are created and expired in the network device's memory and
exported to a collector.
Is there a reference or some other justification for this statement?
Could this statement simply be elided without losing the importance of
the document?
And here are a couple of minor editorial or clarification suggestions:
In section 3.4.2:
Mainly the Flow Export
Rate caused by the test traffic during an [RFC2544] measurement MUST
be known and reported.
s/Mainly/Most importantly/ ??
In section 4.9.1 and 4.9.2, is it intended that the destination IP
address recycles to the address for stream 1 after stream 10000?
(Robert Sparks; former steering group member) No Objection
(Russ Housley; former steering group member) No Objection
(Sean Turner; former steering group member) No Objection
(Stephen Farrell; former steering group member) No Objection
(Stewart Bryant; former steering group member) No Objection
On the basis of a quick read and complete confidence that Benoit will work with the authors to make the draft perfect.
(Wesley Eddy; former steering group member) No Objection
(Benoît Claise; former steering group member) (was Discuss) Recuse
Discussing with Ron, since I was an author for 2 versions of the individual draft, I will recuse myself.
==========================================================================
Introduction
Monitoring of IP flows (Flow monitoring) is defined in the
Architecture for IP Flow Information Export [RFC5470] and related
IPFIX documents.
Which documents? Do we expect the BMWG community to know about the relevant IPFIX documents. Please refer to "them"
- Abstract mentions
This document provides a methodology and framework for quantifying
the performance impact of monitoring of IP flows on a network device
and export of this information to a collector. It identifies the rate
at which the IP flows are created, expired, and successfully exported
as a new performance metric in combination with traditional
throughput. The metric is only applicable to the devices compliant
with the Architecture for IP Flow Information Export [RFC5470].
However, the introduction mentions:
This document provides a methodology for measuring Flow
monitoring performance so that network operators have a framework
for measurements of impact on the network and network equipment.
So if this document covers both "impact on the network and network equipment", it should be clearly mentioned in the abstract.
Maybe this is what you mean by "export of this information to a collector", but it can be understood in different ways: impact on the Exporter, and/or Collector, and/or network.
-
A more
complete understanding of the stress points of a particular device
can be attained using this internal information and the tester MAY
choose to gather this information during the measurement iterations.
replace "device" by "DUT"
-
2.1 Existing Terminology -> I would refer to RFC5101 instead of RFC5470 when possible.
Because RFC5101 will be updated by RFC5101bis. And one important change in RFC5101bis will be the new "Flow" definition, which will be removed "IP". RFC5470 will most likely not have a bis version.
- 2.2.5 Flow Export Rate
Definition:
The number of Cache entries that expire from the Cache (as defined
by the Flow Expiration term) and are exported to the Collector
within a measurement time interval. There SHOULD NOT be any export
filtering, so that all the expired cache entries are exported. If
there is export filtering and it can't be disabled, this needs to
be noted.
If you use to use RFC2119 terms in the definition, be consistent: replace "this needs to
be noted. " by this MUST be noted".
- 3.2 Device Applicability
The Flow monitoring performance metric is applicable to network
devices that implement [RFC5470] architecture.
Replace the end with something similar to: "that implement
RFC5101 and RFC5102, according to [RFC5470] architecture."
After reading the entire draft, I see the sentence I was looking in section 4.3
The DUT MUST support the Flow monitoring architecture as specified by
[RFC5470]. The DUT SHOULD support IPFIX [RFC5101] to allow meaningful
results comparison due to the standardized export protocol.
You should have something similar in section 3.2
- NetFlow is mentioned. At least refer to RFC3954.
- "The Cache
entries are expired from the Cache depending on the Cache
configuration (ie, the Active and Inactive Timeouts, number of Cache
entries and the Cache Size)"
The cache entries is not a configuration parameter.
- The DUT's export interface (connecting the Collector) MUST NOT be
used for forwarding the test traffic but only for the Flow Export
data containing the Flow Records. In all measurements, the export
interface MUST have enough bandwidth to transmit Flow Export data
without congestion. In other words, the export interface MUST NOT be
a bottleneck during the measurement.
I guess that the "the collector (interface) MUST NOT be a bottleneck during the measurement".
Exactly like you wrote for the traffic receiver " The traffic receiver MUST have sufficient resources to measure all
test traffic transferred successfully by the DUT."
After reading section 4.4, I see this sentence:
The Collector MUST be capable of capturing the export packets sent
from the DUT at the full rate without losing any of them.
This is a source of confusion for me in this draft. It seems that the information is fragmented throughout the draft... Therefore, the reading flow is sometimes not easy...
- what is the difference between "Any such feature configuration MUST be part of the measurement
report." and "All configurations MUST be fully documented."
Should the latter say "All configurations MUST be fully documented in the measurement report"?
- The DUT configuration and any existing Cache MUST be erased before
application of any new configuration for the currently executed
measurement.
replace Cache by Cache entries?
- Section 4.3.2
The Cache Size available to the DUT MUST be known and taken into
account when designing the measurement as specified in section 5.
What about Metering Process features that increase the cache size. Is this allowed? Should this be documented in the measurement report? Or should the cache size be set up to its maximum before starting the measurement?
- Section 4.3.2
The configuration of the Metering Process MUST be recorded.
-> MUST be included in the measurement report?
- Section 4.3.3
The templates used by the tested implementations SHOULD
be analysed and reported as part of the measurement report. Ideally
only tests with same templates layout should be compared.
"template layout" = Template Record in RFC5101.
Please add to the terminology and use the term.
- Section 4.3.3
Only benchmarks with the
same transport layer protocol should be compared.
should -> SHOULD
- Section 4.3.4.
For all the examples such as the following, please include the IPFIX Information Element.
Flow Keys:
Source IP address
Destination IP address
MPLS label (for MPLS traffic type only)
Transport layer source port
Transport layer destination port
IP protocol number (IPv6 next header)
IP type of service (IPv6 traffic class)
Rational:
- what does the MPLS label mean? In IPFIX IANA, we report mplsTopLabelStackSection, which is a combination of "The Label, Exp, and S fields". Note: clarify this as well in the section 4.3.6
- packet counters, byte counters: we have multiple IEs for those. Which one should the DUT chose?
- IP addresses: IPv4 and/or IPv6?
- 4.6 Frame Formats
Flow monitoring itself is not dependent in any way on the media used
on the input and output ports. Any media can be used as supported by
the DUT and the test equipment.
What about the export interface?
- section 4.8
The used packet size SHOULD be part of the measurement
report
Why not a MUST?
- section 5.1
I read multiple times the following sentences, and still could not understand the link between the first two.
The number of unique Flow Keys sets that the traffic
generator (sender) provides should be multiple times larger than
the Cache Size. This ensures that the existing Cache entries are
never updated before Flow Expiration and Flow Export. The Cache
Size MUST be known in order to define the measurement
circumstances properly.
- Section 7 Flow Monitoring Accuracy
Don't you have to say a few words about the accuracy of DUT that also does forwarding, i.e. the traffic analysis at the "traffic receiver" in figure 1 must also check that no packets were lost in the case of ingress monitoring. Note: there is also the case of egress monitoring.
Interestingly, bidirectional is mentioned in the "Appendix A: Recommended Report Format", but not a single time in the draft.
- Appendix A:
would be great to mention which entries are SHOULD and MUST
Note: not sure why it's an appendix, as there is normative text referring to each entries
- Appendix A
could be very helpful to poll the IPFIX-MIB (http://tools.ietf.org/html/draft-ietf-ipfix-rfc5815bis-03, currently in RFC-editor queue) and IPFIX-CONF (http://tools.ietf.org/html/draft-ietf-ipfix-configuration-model-10, currently in RFC-editor queue) as MAY in the measurement report.
Note: [IPFIX-CONF] is almost ready, simply waiting for PSAMP-MIB. I believe it makes sense to wait for this RFC.
- Appendix B.
Not sure whether the use of "SHOULD" in the appendix is appropriate. To be checked.
- Appendix B.6 Tests With Bidirectional Traffic
Not sure why this is an appendix. The recommended report mentions
Traffic Direction unidirectional, bidirectional
Direction ingress, egress, both
... and the bidirectionality is only mentioned in the appendix?
- Section 5.2
Traffic Generation
The traffic generator needs to increment the Flow Keys values with
each sent packet. This way each packet represents one Cache entry
in the DUT Cache.
Here is a comment I made to you years ago. You will find surprises if you increment the IP addresses by one in your generator, as opposed to random traffic, as the hash function are not optimized for incremental IP addresses. You should say a few words about this.