MINUTES

Benchmarking Methodology WG (BMWG)
Thursday August 2, 2012
1730-1830  Afternoon Session III
Regency A       	OPS 	bmwg      

This report is arranged in 2 parts, a summary and detailed minutes.
The report was prepared by Al Morton, using notes from Matt Zekauskas
as official note taker.  Mike Hamilton provided a Skpe connection for
Carol Davids, and Bill Cerveny monitored jabber.

SUMMARY

BMWG met with 13 people in attendance, in a short (1 hr) session
late on Thursday afternoon.

Most comments have been addressed in the SIP Benchmarking drafts,
but revised drafts are needed for this work to advance.
The Content-Aware benchmarking method has been advancing quickly
with many reviews and comments exchanged on the list,
and now we need the terminology draft to catch-up.
Next Step is WG Last Call for IMIX Genome draft.
Some co-authors believe we have the RFC 2544-AS draft LastCall
comments resolved, and the text was shared on the list (revised draft coming).
The BGP convergence benchmarking will need to be adopted on
the list, as only one person had read the draft.
The WG will pursue support for new and currently chartered work,
and revise milestones for approval. In particular,
the power benchmarking will be read by several WG members
and it should be shared with the new Energy Efficiency working group
here at IETF.

The BMWG also reviewed a Liaison from the MEF on Service Activation
Testing, and will plan a response in time for MEF's Oct meeting.

ACTION ITEMS

1. Re-run the call for adoption of the Basic BGP Convergence.

2. Collect comments on the Data Center Bridge Draft and call for adoption

3. Solicit the Power Benchmarking work for comment in EERG.

4. Prepare a response to MEF on Service Activation testing, if comments emerge.

5. Looking for Revised drafts on most WG items.

6. Revise milestones and charter, if needed.


DETAILED MINUTES

0. Agenda Bashing
Al pre-bashed the agenda to include a Liaison from MEF.

1. WG Status and Milestones

Published : RFC 6645
IP Flow Information Accounting and Export Benchmarking Methodology

Drafts not presented at this meeting:
1B. IETF Last Call comments on: <draft-ietf-bmwg-protection-meth-10.txt>
    from Joel M. Halpern and Gregory Mirsky and Jia He
Status:  Waiting for Revised Draft

last charter 2010, which is why it needs revision
suplimental page http://home.comcast.net/~acmacm/BMWG

Also covered related drafts.  For the most part, one that
note data center bridgeing draft.  on charter, but no progress to date
power useage seems important.

Standard security paragraph reminder presented, used to show security
area what should do.
(stopped deck at slide 7, rest later)


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

2. SIP performance benchmarking terms and methods
   Presenter: Vijay Gurbani
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-04   
http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-04
WGLC comments, including Expert review. Revised drafts needed.

Carol Davids on skype.  Vijay called away.

Reminder of scope - SIP throughput.  Just the *signalling* throughput.

WGLC.  received a few comments.  discussion on list.  changes described
will be done by next time.  here are changes.

RAI reviewer detailed comments.  Good feedback from everyone, though.
Dale Worley was RAI reviewer, did excellent work.

<see technical comment slide for complete detail, just mentioning more
detail mentioned verbally here>

Confusion about why defining the character of media as condition for test,
but not measuring quality of media.  Reason again is SIP throughput, which
could be affected by stress of processing media.  Clarifying.

Concern about RE-INVITES being out of scope... revisiting that now to make
sure out of scope is the correct thing. 

on sending responses immediately, this is done to try to get to performance
edge as quickly as possible -- want to characterize the edges.

--
In general, need full editorial review for consistency.

Have a new lab implementation.  Planning to show results on commercial
products, and not just asterisk and <chamilio> respond too before next
IETF.



 
3. Benchmarking Methodology for Content-Aware Network Devices
   Presenter: Mike Hamilton
There are some updates in the draft in response to Tom A's comments
Some comments were not yet addressed in the previous version due to 
time constraints.

Lots of activity and discussion on email list.

Spinning wheels on how to define individual flow.  Almost to the point
where need to have people that build the equipment generate the flows
[document?].

Matt Z: is there some parameters to characterization you can give to
manufacturers so that the flows are properly documented?

There are specifications in the appendix that have examples.

If there's some list of features that MUST be reported, that might
be the most useful.

On traffic mix, have somthing kicking around.   Use same mix Sandvine reported.

Al: There's a parallel to IMIX Genome specification of sizes, if we teach
people how to go fishing, can catch their own fish in future (as it relates to
the list of applications tested.

Next steps:

incorporate recent feedback.
now that methodology is tighter, submit terminology draft.
then try last call.

goal: next couple of weeks for methodology and terminology.


4. IMIX Genome
   Presenter: Al
   Comments on methods to determine the mix (Aamer) were addressed

See side by side diff between -02 and -01 for changes.

Goal: specification for test streams that have variable sizes.  Everyone 
does IMIX in some form.  Want to standardize what that means.

Ahmer looking for a way to teach new users to fish.  Added comment to
say, use IPFIX as a way to characterize what you have, so that you
can then describe it to test sets.  This is now in the text.

Next step, WGLC, unless comments here and now. (no)


5. RFC 2544 Applicability Statement: 
   Presenter: Al
   IETF Last Call Comments, and discussion for resolution (if not yet achieved)

IETF-wide last call held on applicability statement.  Have AD-level comments
and addressed them.  But addressing the comments, caused a lot of scrutiny.
Found an ITU spec that used RFC2544 in a way inconsistent with this draft.
Throwing mud at ITU-T was deemed inappropriate.

All co-authors were chairs and past chairs, so Bill C is doc shepherd.

Feedback during LC, advisory was too strong.  Some kinds of production
networks where people could do this, with  no shared resources, tdm basis,
wouldn't do damage.

This meth will conclude congestion even if there is none if you try
to run it on a live network because the background loss will cause problems.
Hence, added text.  Presence of loss on live networks will not give
you repeatable results.  If you change, you should write a new standard

Revised draft needed, the on to IESG.


6. Basic BGP Convergence Benchmarking Methodology status
   Presenter: 
draft-papneja-bgp-basic-dp-convergence
draft-varlashkin-router-conv-bench
Status: Call for review and adoption...

Al presented for authors.

Teriminology took a long time, complete in 2005.

Methodology has been going on since IETF80.

And focus has moved from control to data plane tesitng.          

Problem noted with fib autoconvergence(?); Ilya Varlashkin
working on that.  He didn't want his draft accepted yet due to problem.
But seemed to sqelch progress on the BGP dataplane draft.
Will likely re-ask on list.

slide4: laundry list of feedback.
--
current status & action items.

want to push dp-convergence draft.

<only one person in the room had read draft, so can't call for adoption now.

for router-conv, still looking for input.  This work is on the charter, want to
finish up soon.

7. Liaison from MEF on Service Activation Testing
          
Contacted MEF co-chair, and now being sent to statements@ietf.org
will get officially published.

MEF has sent this to the world (many standards bodies)

Working on service activitation testing for Ethernet service.
At a meeting last week, approved a draft (working draft to share, not
a final thing); asking bmwg to review document and provide feedback.

metroethernetform.urg/liaison_login.htm
username mef
pass: 3030

if folks would like to comment on this, BMWG will assemble comments.

Believe Service Activation is out of scope for bmwg.  This is production networks.
Want to see if consistent with 2544 applicability statement, though.



8. Milestone Review

Completed one, but rest is in the red.  Needs to be revised.

Think that cvoers it.

billc: will look at energy, huge in cellular, getting in to that.
benoit will look too.

mike will read both of the drafts.



9. New Work Proposal Review/Matrix

Al identifies Two new drafts to look at.

Data Center networking is a big thing.   Timmons Player and David Newman
id that 2544 loss signal in overload won't work in DC with "pause" protocols.
Characterize that differently.  have a draft.  Please read
draft-player-dcb-benchmarking

Benchmarking power utilization.  draft-manral-bmwg-power-usage-02.txt
Compare power along with througput.

Benoit: mainly FYI, yesterday evening there was an energy efficient
research group.  Not a RG yet, but behave like RG for year, if activity
then would become research group.  Interesting to have metrics for energy
because most of research papers say "save 10%".  DC has "PUE" to test
efficiency of data center.  No metric for networks.

eerg@ietf is a source of comments, contact them.

--

security effectiveness no updates, but not dead yet.
Sw Update Time is ooking like it might drop off list.

Time Synch benchmarking... champion retired from job, think it's dead.


LAST. AOB  Meeting closed after 55 of 60 minute slot.