Enhanced Remote Direct Memory Access (RDMA) Connection Establishment
RFC 6581

[Ballot discuss]
The Gen-ART Review by Elwyn Davies on 26-Sept-2011 raises some
  concerns that deserve a response.  Please find the review at
  http://www.ietf.org/mail-archive/web/gen-art/current/msg06754.html.

State changed to IESG Evaluation::Revised ID Needed from IESG Evaluation::AD Followup.
I reviewed draft-ietf-storm-mpa-peer-connect-08 and have a few
questions, related to the gen-art review
http://www.ietf.org/mail-archive/web/gen-art/current/msg06754.html

active/passive OK

RTR frame definition - where is this?

private - I see you added that other specs could add private data
formatting. I think the suggestion was to specify how flag ordering
and data chunk ordering was important. Nothing is said about whether
the private data for this extension should be first, or anything. The
suggestion does not seem to be addressed.

s10: p1 and p4 seem contradictory, unless it is only the initiator
that has a choice of enhanced versus unenhanced (i.e, the responder
has no choice in the matter). If this is the case, that should be
stated explicitly.

State changed to IESG Evaluation::Revised ID Needed from IESG Evaluation.
need to create registry for MPS codes and error codes.
other editorial fixes from IESG comments and Gen-ART review.

[Ballot comment]
Readable document, thank you.

RDMA is not a "well known" acronym at
http://www.rfc-editor.org/rfc-style-guide/abbrev.expansion.txt

You should expand it in the Abstract and Introduction.

(You might also want to lobby the RFC Editor to get it made "well known"

MPA will also need expansion in the Abstract.

---

FULPDU:  Framed Upper Layer Protocol PDU

How many letter Ps?

[Ballot comment]
- The abstract uses the asconyms MPA and RDMA without expansion,
it'd be better not to do that and generally many acronyms are
used before being expanded - a pass to fix that would be useful

- The security considerations section says that this changes
nothing compared to RFC 5044, however, I guess that a peer
could try a DoS against another peer by setting large xRD
values, but I don't know if that's significant enough to
warrant a mention or not. Is it? If it were, then I guess
just a warning that implementations ought to have some kind of
sanity checking on those inputs would suffice.

- Just out of curiosity - RFCs 5043 and 5044 seem to say: "if
you want security, do IPsec" - is that how things are actually
deployed?

[Ballot discuss]
The Gen-ART Review by Elwyn Davies on 26-Sept-2011 raises some
  concerns that deserve a response.  Please find the review at
  http://www.ietf.org/mail-archive/web/gen-art/current/msg06754.html.

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Enhanced RDMA Connection Establishment) to Proposed Standard


The IESG has received a request from the STORage Maintenance WG (storm)
to consider the following document:
- 'Enhanced RDMA Connection Establishment'
  as a Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-09-26. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document updates RFC5043 and RFC5044 by extending MPA
  negotiation for RDMA connection establishment.  The first enhancement
  extends RFC5044, enabling peer-to-peer connection establishment over
  MPA/TCP.  The second enhancement extends both RFC5043 and RFC5044, by
  providing an option for standardized exchange of RDMA-layer
  connection configuration.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-storm-mpa-peer-connect/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-storm-mpa-peer-connect/


No IPR declarations have been submitted directly on this I-D.

AD Review update for -06-

all issues resolved except the following.
I think an additional revised ID is called for.

> 1) idnits complains about unreachable reference:
>
http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMA
> C.pdf

I can reach this, but idnits seems to have a problem parsing it.

> 5) It would be helpful to explain why taking the bit from RES is
> backwards compatible.
> Explain why older implementation will not have a problem with the
new
> bit, and why new implemntations will be able to benefit from the new
> bit.

fixed. text is still a bit rough though.
I think it is meant to say:

When the S bit is set to zero, when no additional private data is used
for enhanced RDMA connection 
establishment, the resulting the MPA request/reply frame is identical
to the unenhanced protocol.

> 11) What is "enhanced DDP stream management"?

I didn't see this addressed, but it might be clear enough. We'll see
if IESG has concerns over this terminology.

> 17) in section 9, the responder SHOULD NOT set its IRD higher than
> Initiator ORD. Why is this not a MUST? Under what conditions is it
> acceptable to set this higher than Initiator ORD? What are the
> implications of doing so?
> the responder SHOULD set its ORD lower. What are the valid
exceptions
> to this?

in -06-,
Responder SHOULD set its IRD at least to Initiator ORD
"at least" seems to indicate that setting it higher than or equal to
Initiator ORD is expected.
But the next sentence says "Responder SHOULD NOT set its IRD higher
than Initiator ORD."
But the next sentence says "Responder MAY set IRD to one if Initiator
ORD is zero if ..."
so the logic seems unclear.
I think this paragraph should be rewritten to be much clearer than the
current text.

It is possiblr that the text is actually a clear description of the
way things should be processed.
If that is true, then you might want to see if you could simplify the
processing.

> 19) "Control flag for using" doesn't discuss whatthe values are.
Does
> this mean that if A has a value 1, then FULPDU has a zero length?
> This section might be better organized by defining what the fields
are
> (control flags and depths), then describing how the values are set.
> You do that for A,B,C but describe IRD and ORD in-line.

I am not sure how you interpeted my suggestion. In -04-, you had the
descriptions of the flags AFTER defining the fields,
but you had the descriptions of IRD and ORD in line. Now you seem to
have moved the descriptions for IRD and ORD out of line, but put the
descriptions for the flags inline. I recommend that for ALL of these
fields, you define the field, and then put the descriptions of the
values AFTER the field definitions.

-06- describes the meanings of the flags differently than -04-
Is this consistent with WG consensus?

The description says to set the flags to TRUE, but doesn't define
TRUE.
These flags are all 1-bit, so the choice seems to be 0 or 1.
Is TRUE equal to 1? Then why not say 1 rather than TRUE?
If TRUE has some special meaning for RDMA, please provide a reference

"Control flag for using a zero length FULPDU as the RTR message."
This is a little ambiguous.
Please describe what the values mean for each flag, as you do for flag
A.

> 21) "Setting no Control Flags in the MPA Reply indicates that an ....

wow. this text has become way more complicated than the -04- version.
Can this be simplified?
Can you start by saying
"If the value of A is 0, then:
B,C,D flags must be set to zero and ignored by the reciever.
processing steps for client-server mode are:
and then provide bullets for the processing steps for
client-server support?
"If the value of A is 1, then:"
and then provide bullets for the processing steps for
peer-to-peer support?

> 25) section11 This document defines error codes; so does RFC5044.

I repeat my question. Should there be a registry for the error codes
so there is a consistent place to look for which error codes are
defined, and their values and meanings?

AD Review of draft-ietf-storm-mpa-peer-connect
(actually done 5/20, but no comment was logged; this is logged only for background info.

I have reviewed this document and have comments. I think this work is
important, but the way this document explains it could be better.

My goal as AD is to have this document go through IETF Last Call and
IESG review without any issues delaying advancement.

Some comments are technical/process comments that refer to text that
will likely cause delays during IESG Evaluation.

Some are stylistic, such as capitalization and wording consistency,
that are distracting and are likely to make readers wonder "is there
something special here that I need to understand?" and to waste time
trying to determine if there is or is not something special that led
to your capitalizing the text, or using different wording than
elsewhere. These could lead to IESG members raising unnecessary
questions that would delay advancement.

Others are editorial in nature, mostly about writing clear and
unambiguous text. Generally, if it is simply a small editorial thing,
I ignored it, expecting the RFC Editor to clean up the text a bit. But
sometimes, the lack of clarity could lead to differing interpretations
of the text, or lack of understanding of the text, and those should be
fixed before I bring this to the IESG.

In some cases, I have asked a question. If I had to ask a question,
then the document apparently didn't explain it clearly enough. The
answer to the question usually needs to be explained "in the
document", not just in a private email to me.

1) idnits complains about unreachable reference:
http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMA
C.pdf

2) abstracts cannot contain references.

3) in 4.3.2, the conclusion could be stated more clearly, such as "If
a nop approach was used, it would require lower layers to know the
usage model, and would disrupt the calculations ..."

4) in 4.3.3, I have no idea what this text is saying or why it is
relevant to the problem at hand. I don't know what an untagged message
is; I don't know why a WC can only be generated by an untagged
message. Can you expand on this so readers that are not RDMA experts
understand what you're talking about? The IESG will need to review and
approve this document, and to my knowledge none of them are RDMA
experts.

5) It would be helpful to explain why taking the bit from RES is
backwards compatible.
Explain why older implementation will not have a problem with the new
bit, and why new implemntations will be able to benefit from the new
bit.

6) Rev "MUST be set to two"; would this be better as "two or higher?"
to allow for a version three+ that would be backward compatible with
the enhanced features of version two?

7) I don't understand "The Enhanced Reject function code SHOULD be
used to indicate a configuration that would have been accepted." Would
have been accepted except was not because of what condition? Why is
this only a SHOULD and not a MUST?
8) How can the private data be zero length if it must begin with
Enhanced RDMA Connection establishment data?
9) sometimes "Enhanced RDMA Connection Establishment Data" is
capitalized, sometimes not. Be consistent.
10) Received extended ... message SHOULD be reported to the ULP. Why
is this not a MUST? Anytime there is a SHOULD, the text should
identify when the rule is not mandatory - what are the valid
exceptions that make this not a MUST?
11) What is "enhanced DDP stream management"?
12) "It should be noted that" - you're noting it, so this lead-in
phrase is totally unnecessary. If you leave it as "It should be noted
that", then explain WHY it should be noted. If "it is especially
important to recognize that" then say that, and explain WHY it is
important to recognize.
It appears that it is important only because you've made a case for
peer-to-peer needing to initiate from either side, and the SCTP
adaption layer allows for that already. So you should probably explain
why you're duplicating that capability.
13) additional legal sequences - but then you don't seem to define any
**sequences**. Is "Legal Sequences" a special term? If so, please
explain where it is defined. if not, why is it capitalized?
14) I cannot parse "The protocol layers on which RDMA connection
establishment is layered upon [RFC5040] and [RFC5041] define layers
and error types."
15) if 5043 and 5044 do not define error codes, then how are the error
codes used? are they carried in messages? which messages?
16) In looking for the answer to 15, I found that error codes ARE
defined in RFC5044 section 8. So your text is apparently wrong.
17) in section 9, the responder SHOULD NOT set its IRD higher than
Initiator ORD. Why is this not a MUST? Under what conditions is it
acceptable to set this higher than Initiator ORD? What are the
implications of doing so?
the responder SHOULD set its ORD lower. What are the valid exceptions
to this?
18) the Initiator MUST set its IRD to a value at least as large as the
responder ORD if it has sufficient resources
"MUST if" means this is a SHOULD, and lack of resources is the
exception. If this is truly a MUST (which the following sentences seem
to say, requiring a termination if ti doesn't do so, then I would
remove the if clause from the first sentence, making it a definite
MUST.
19) "Control flag for using" doesn't discuss whatthe values are. Does
this mean that if A has a value 1, then FULPDU has a zero length?
This section might be better organized by defining what the fields are
(control flags and depths), then describing how the values are set.
You do that for A,B,C but describe IRD and ORD in-line.
20) In section 9, "If there is no Standard RDMAP Configuration Data in
the MPA Reply Frame, and the Enhanced Connection Establishment version
number is used, it is the equivalent of setting 'A', 'B' and 'C'. "
Why do we need this extra complexity? Why not require that these be
set ONE way, not two? I also have concerns about "THE enhanced
connectiosn establshment version number"; will this document need to
be updated if a version three is ever developed?
21) "Setting no Control Flags in the MPA Reply indicates that an RDMA
Send message will be required. As this option will require the
initiator ULP to be involved it SHOULD NOT be used unless necessary. "
Please define what necessary is. I think this would be better as a
MUST NOT.
22) "Initiator or Responder SHOULD generate the TERM message" - why is
thi snot a MUST? What are the valid exceptions to this?
in section 10:
23) "An initiator SHOULD NOT use the Enhanced DDP Connection
Establishment formats or function codes when no enhanced functionality
is desired. " what are the valid exceptions that make this a SHOULD
rather than a MUST?
24) the paragraph containing "If a responder does not support received
extended MPA message, then it MUST close the TCP connection and exit
MPA since MPA frame is improperly formatted for it as stated in
[RFC5044]." needs a bit of English cleanup.
25) section11 This document defines error codes; so does RFC5044.
Implementers and operators will need to hunt through documents to find
the various error codes. This document apparently overlooks the error
codes defined in 5044 section 8. Should there be a registry,
maintained by IANA, for the MPA error codes?
section 12:
26) I am always concerned when a security coinsiderations section says
this document does not introduce any new security considerations. This
is also a red flag during IESG Evaluation. I suggest you document WHY
these changes do not impact security compared to 5043 and 5044.
Demonstrate that the WG actually **considered** the security issues.

27) expand acronyms on first use (MPA, RDMA, RDDP, ULP, iWARP, etc.)

28) references for sctp, infiniband,

29) in 4.3.1, "required only for one transport" - mention the one
transport?

30) an explanation/definition of MPA Fencing would help readers. Maybe
a terminology section would be helpful, given all the acronyms, etc.

31) The information in 4.4 would be helful in the Introduction
section. This would help to organize the document into "Here's how it
works now; here is why the client/server model is undesirable in a P2P
environment; and this is how to modify the existing standard to
address this problem."

32) Is section 5 a description of how things currently work, or the
way this document proposes they work? Section 5 mentions "Enhanced
Connection Setup"; is that the same as "Enhanced RDMA Connection
Establishment"? if so, then naming section 5 "Enhanced RDMA Connection
Establishment" or "Enhanced MPA Connection Setup" might be helful. Use
terminology consistently, please.

I hope these suggestions help improve the document so we can advance
it through the approval process quickly,

PROTO writeup:
                Enhanced RDMA Connection Establishment
                draft-ietf-storm-mpa-peer-connect-04.txt

Requested Publication Status: Proposed Standard
PROTO shepherd: David L. Black (STORM WG Co-Chair)
------------------------------------------------------------------------

  (1.a) Who is the Document Shepherd for this document? Has the
        Document Shepherd personally reviewed this version of the
        document and, in particular, does he or she believe this
        version is ready for forwarding to the IESG for publication?

David L. Black (david.black@emc.com) is the Document Shepherd.  The
Document Shepherd has reviewed this version of the document and believes
that it is ready for forwarding to the IESG for publication.

  (1.b) Has the document had adequate review both from key WG members
        and from key non-WG members? Does the Document Shepherd have
        any concerns about the depth or breadth of the reviews that
        have been performed?

The document has had sufficient review from key WG members.  The Document
Shepherd is satisifed that this document has been sufficiently reviewed
by members of the community that uses this protocol.

  (1.c) Does the Document Shepherd have concerns that the document
        needs more review from a particular or broader perspective,
        e.g., security, operational complexity, someone familiar with
        AAA, internationalization or XML?

No.

  (1.d) Does the Document Shepherd have any specific concerns or
        issues with this document that the Responsible Area Director
        and/or the IESG should be aware of? For example, perhaps he
        or she is uncomfortable with certain parts of the document, or
        has concerns whether there really is a need for it. In any
        event, if the WG has discussed those issues and has indicated
        that it still wishes to advance the document, detail those
        concerns here. Has an IPR disclosure related to this document
        been filed? If so, please include a reference to the
        disclosure and summarize the WG discussion and conclusion on
        this issue.

No.

  (1.e) How solid is the WG consensus behind this document? Does it
        represent the strong concurrence of a few individuals, with
        others being silent, or does the WG as a whole understand and
        agree with it?

The WG consensus of the WG members who are familiar with this technology is
solid.  The storm (STORage Maintenance) WG conducts maintenance on multiple
storage protocols, and different WG members have differing levels of
interest and expertise across the protocols that the WG maintains.

  (1.f) Has anyone threatened an appeal or otherwise indicated extreme
        discontent? If so, please summarise the areas of conflict in
        separate email messages to the Responsible Area Director. (It
        should be in a separate email because this questionnaire is
        entered into the ID Tracker.)

No.

  (1.g) Has the Document Shepherd personally verified that the
        document satisfies all ID nits? (See the Internet-Drafts Checklist
        and http://tools.ietf.org/tools/idnits/). Boilerplate checks are
        not enough; this check needs to be thorough. Has the document
        met all formal review criteria it needs to, such as the MIB
        Doctor, media type and URI type reviews?

Yes, the document satisfies ID nits.  No other formal review criteria apply.

  (1.h) Has the document split its references into normative and
        informative? Are there normative references to documents that
        are not ready for advancement or are otherwise in an unclear
        state? If such normative references exist, what is the
        strategy for their completion? Are there normative references
        that are downward references, as described in [RFC3967]? If
        so, list these downward references to support the Area
        Director in the Last Call procedure for them [RFC3967].

The references have been split, and there are no downward references or
normative references to work-in-progress documents.

  (1.i) Has the Document Shepherd verified that the document IANA
        consideration section exists and is consistent with the body
        of the document? If the document specifies protocol
        extensions, are reservations requested in appropriate IANA
        registries? Are the IANA registries clearly identified? If
        the document creates a new registry, does it define the
        proposed initial contents of the registry and an allocation
        procedure for future registrations? Does it suggest a
        reasonable name for the new registry? See [RFC5226]. If the
        document describes an Expert Review process has Shepherd
        conferred with the Responsible Area Director so that the IESG
        can appoint the needed Expert during the IESG Evaluation?

The IANA Considerations section exists and states that no IANA actions
are required by this document.  There are some values defined in this
document that may be appropriate to be move into IANA registries
if future extensions should occur, but creation of IANA registries is
not necessary at this juncture (this document suffices as a reference).

  (1.j) Has the Document Shepherd verified that sections of the
        document that are written in a formal language, such as XML
        code, BNF rules, MIB definitions, etc., validate correctly in
        an automated checker?

N/A.

  (1.k) The IESG approval announcement includes a Document
        Announcement Write-Up. Please provide such a Document
        Announcement Write-Up? Recent examples can be found in the
        "Action" announcements for approved documents. The approval
        announcement contains the following sections:

    Technical Summary

This document extends iWARP (rddp) RDMA connection establishment
with two functions that apply to the adaptation layer between RDMA
functionality and the transport protocol.  The first extension broadens
MPA (adaptation to TCP) to enable connection establishment without
initial data to send in support of applications structured as a
collection of peers.  The second extension improves connection setup
for both MPA/TCP and the SCTP adaptation by adding support for
standardized exchange of resource availability (queue depth) information.

    Working Group Summary

This document makes small additions to existing protocols.  There
has been clear WG recognition that this functionality is needed to
match the usage of these protocols by an important class of applications,
and no significant WG dissent from the design in this document.

    Document Quality

There are multiple existing implementations of the iWARP (rddp) RDMA
protocols that have plans to add the functionality specified in
this document.  Hemal Shah reviewed the near-final version of this
draft and found some important corrections that needed to be made.