Methodology for Benchmarking IPsec Devices
draft-ietf-bmwg-ipsec-meth-05

idnits seems to not like the way you have handled references. Although the RFC Editor can sort this out, I think you should have a go first.

You'll also need a null IANA section. I don't see any IANA email about this I-D.

In talking to Ron about this - it sounds like in practice only a subset of these test would be run on a given device. The resolves the concern I had and I would like to see if we can update the working in the draft to reflect that.

I have reviewed draft-ietf-bmwg-ipsec-meth-05, and have couple of
concerns/questions that I'd like to discuss before recommending
approval of the document:

- Section 5/9.1/10.1/11.1: These sections suggest that the scope of
this document is limited to IPsec devices that also work as ordinary
routers, and these benchmarks can't be used with e.g. remote access
IPsec VPN gateways (that would not usually support "without IPsec"
mode at all). Is this the intent? (If it is, a short explanation in
Section 5 would be in order.)

- Section 7.6.1: This section requires testing transport mode; would
this mean IPsec devices that are specifically intended for gateway use
(and thus may not support transport mode at all) cannot be benchmarked
by this methodology?

- Section 8.2: This text doesn't seem to distinguish between
"maximum number of IPsec SAs on a device" and "maximum number of IPsec
SAs per IKE_SA/user". The methodology is clearly measuring the latter,
which could be very different from the former?

- Sections 9.3/9.4/11.2/11.3/11.4: these test methodologies talk about
counting the frames to detect packet loss -- but if fragmentation
occurs somewhere, the number of frames sent in and number of frames
coming out would be different even without packet loss?

- Section 10.3: Since the DUT will encrypt the frames, how would
the tester see the tags?

- Section 12.1: It seems this test is measuring the average duration
of one tunnel setup, but you can't calculate the tunnel setup rate
from this value? (it seems with this methodology, the DUT would be
mostly sitting idle, and nowhere near its maximum SAs-per-second
limit...) (Also applies to 12.2/12.3)

- Finally, any changes to address my comments about bmwg-ipsec-term 
probably require changes in this document, too.

I am picking up Pasi's DISCUSS on this document.

I have reviewed draft-ietf-bmwg-ipsec-meth-05, and have couple of
concerns/questions that I'd like to discuss before recommending
approval of the document:

- Section 5/9.1/10.1/11.1: These sections suggest that the scope of
this document is limited to IPsec devices that also work as ordinary
routers, and these benchmarks can't be used with e.g. remote access
IPsec VPN gateways (that would not usually support "without IPsec"
mode at all). Is this the intent? (If it is, a short explanation in
Section 5 would be in order.)

- Section 7.6.1: This section requires testing transport mode; would
this mean IPsec devices that are specifically intended for gateway use
(and thus may not support transport mode at all) cannot be benchmarked
by this methodology?

- Section 8.2: This text doesn't seem to distinguish between
"maximum number of IPsec SAs on a device" and "maximum number of IPsec
SAs per IKE_SA/user". The methodology is clearly measuring the latter,
which could be very different from the former?

- Sections 9.3/9.4/11.2/11.3/11.4: these test methodologies talk about
counting the frames to detect packet loss -- but if fragmentation
occurs somewhere, the number of frames sent in and number of frames
coming out would be different even without packet loss?

- Section 10.3: Since the DUT will encrypt the frames, how would
the tester see the tags?

- Section 12.1: It seems this test is measuring the average duration
of one tunnel setup, but you can't calculate the tunnel setup rate
from this value? (it seems with this methodology, the DUT would be
mostly sitting idle, and nowhere near its maximum SAs-per-second
limit...) (Also applies to 12.2/12.3)

- Finally, any changes to address my comments about bmwg-ipsec-term 
probably require changes in this document, too.