Remote Direct Memory Access (RDMA) Protocol Extensions
draft-ietf-storm-rdmap-ext-10

[Ballot comment]
Editorial nits: Please don't use [IB], [RSOCKETS], [OFAVERBS], etc., as nouns. Instead, on first use, say "Infiniband [IB]" and the like, and then use the word (not the reference) throughout the rest. Also, an expansion and/or reference for iWARP and RNIC would be helpful in the intro.

4.1 & 4.2: s/MUST be used/are used. What else would an implementation do?

4.1:

  Figure 2 also defines when the STag, Tagged Offset, and Queue Number
  fields MUST be provided for the RDMA Messages defined in this
  specification.

I'm confused. Doesn't Figure 2 say that STag and TO are N/A for these messages? That's not a MUST. How about this instead:

  As shown in Figure 2, STag and Tagged Offset are not applicable for
  the RDMA Messages defined in this specification. Figure 2 also shows
  the appropriate Queue Number for each Opcode.

5:

  An RNIC that supports Atomic
  Operations as specified in this document MUST implement all Atomic
  Operation Codes defined in Figure 5.
 
Do you really mean Figure 5? If so, say "both" instead of "all", or even better, try:

  An RNIC that supports Atomic Operations as specified in this document
  MUST implement both the FetchAdd operation as specified in section
  5.1.1 and CmpSwap operation as specified in section 5.1.2.
 
There are three requirements stated in the last paragraph of this section: MUST use Untagged Buffer model with QN=3, MUST use queue number 3, and MUST use MSN. Whenever I see MUST requirements, I always ask "Why MUST I do that?" If there's a good answer to that question, the explanation (like, "If you don't do this, the implementation will blow up because the other end will be expecting you to handle buffers bigger that you probably thought you needed") should probably be stated if it's not clear why. If the answer is, "Because if you're not doing that, you're not implementing the protocol", then the MUST is silly and should be replaced with "will" or "is". If the answer is, "No reason; we just think you should", then MUST isn't appropriate at all. So, for these three requirements, which answer is correct? (Probably useful to look at other requirements in the document and ask the same questions.)

5.2.1:

AOpCode: The "MUST" is redundant. Already stated above.

Remote Tagged Offset: It says "MAY start at an arbitrary offset". But I thought above it was stated that it MUST be 64-bit aligned. Did I misunderstand.

Add or Swap Mask, Compare Data and Mask: s/MUST be set/is set

[Ballot comment]

- Section 1 and elsewhere: There are a number of acronyms
not expanded on 1st use.  Doing so would be good, e.g iWARP.
It'd also be nice to not use the reference as part of the
sentence in at least some cases here, w.g. "...support in
[IB]" assumes I can tell what is meant from the two letters
IB. (This reader cannot:-)

- I wondered if these operations (and RDMA generally) has
been examined for potential timing side-channel attacks.  If
all RDMAP messages were sent via an encrypted channel, how
much could I deduce from the timing of the messages and
responses? (The pseudo code in section 5 suggested this
question.)

Document Writeup

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up.

Changes are expected over time. This version is dated 24 February 2012.

STORM WG
draft-ietf-storm-rdmap-ext
January 10, 2014

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

  Proposed Standard.

  The RFC type is appropriate because the protocol specifies extensions
  to RFC5040, which is also classified Proposed Standard. The extensions are
  upwardly compatible, not experimental, and request that IANA commit the
  extension's opcodes and two new registries to the existing RDDP Registries.

  The RFC Type in the draft's title page header reads: "Intended status: Standards Track".

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  This document specifies extensions to the IETF Remote Direct Memory
  Access Protocol (RDMAP [RFC5040]). RDMAP provides read and write
  services directly to applications and enables data to be transferred
  directly into Upper Layer Protocol (ULP) Buffers without
  intermediate data copies. The extensions specified in this document
  provide the following capabilities and/or improvements: Atomic
  Operations and Immediate Data.

Working Group Summary

  The extensions add Atomic Operations and Immediate Data to the RDMAP
  Protocol.  Other RDMA transport protocols define the functionality added by
  these extensions leading to differences in RDMA applications and/or
  Upper Layer Protocols. Removing these differences in the transport
  protocols simplifies these applications and ULPs.

  The STORM Working Group chartered this work in mid-2011, and while
  no significant dissent was encountered, the document spent an extended
  time in the WG owing to other WG priorities. When work resumed in 2013,
  the document received substantial attention and review. Four update
  cycles resulted in the final text. There was no controversy and consensus
  was easily reached.

Document Quality

  The document is of high quality and is well written. The authors represent
  companies which deliver industry implementations of the RDDP protocol, and
  discussion in the STORM WG by developers of upper layers and applications
  indicated strong interest in using the extensions.

Personnel

  Document Shepherd: Tom Talpey (storm WG co-chair, ttalpey@microsoft.com)
  Responsible Area Director: Martin Stiemerling (Transport, mls.ietf@gmail.com)

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

  The Document Shepherd personally reviewed the document numerous times
  during the Working Group cycle, and as a contributor to all the RDDP RFCs,
  considers the document to be of high quality and the extensions well architected.
  Many other WG contributors also reviewed, and all comments were addressed
  in a thorough manner. The Shepherd considers the document fully ready for
  publication.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed? 

  No, as detailed above.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

  No, the extensions are specific to the RDMAP RFC5040 protocol and do
  not introduce nor depend on external WGs or from broader perspectives.
  Security issues were carefully reviewed in the WG to ensure the extensions
  aligned with existing RDMAP security considerations and with DDP/RDMAP
  Security (RFC5042).

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

  The Document Shepherd has no concerns, and additionally is not aware of
  any concerns from any other individual.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

  Yes, the Document Shepherd has obtained recent direct responses from
  all five named authors to this question. All indicated that they were aware
  of no IPR requiring disclosure, and had no intention to file any.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

  No IPR disclosures have been filed, according to the tools page as of
  this date. No discussion of potential IPR was brought up in the WG.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

  The consensus is solid, representing the concurrence of numerous
  active WG contributors well-versed in the RDDP protocols. No concerns
  have been raised.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

  No appeals have been threatened or filed.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

  Two nits are reported by the tool. A mention of RFC5040 in the abstract
  appears within brackets and triggers a warning, which in the opinion of
  the Shepherd is harmless. A second reference to "RFCXXXX" is a note to
  the RFC Editor. Both these can and will be easily corrected by the RFC
  Editor before publication, as needed.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

  No formal review is necessary, as the document contains no MIB, media
  or URI types.

(13) Have all references within this document been identified as
either normative or informative?

  Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

  No such references are present in the document.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

  No downward normative references are present in the document.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

  The document extends RFC5040, and will not change its status.
  RFC5040 is listed in the abstract and discussed in the introduction.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

  The IANA Considerations section was carefully reviewed and its
  Proposed contents discussed by the WG. The Shepherd agrees with
  the actions it proposes, which include adding 4 values to an existing
  registry, and adding two newly-defined registries to the existing "RDDP
  registries" set. The "RDMAP Queue Numbers" addition represents a
  prior omission, and is therefore a required improvement.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

  The new IANA registries will not require future Expert Review beyond
  the normal diligence applied to review of such requests.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

  Not applicable.

IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-storm-rdmap-ext-08.  Authors should review the comments and/or questions below.  Please report any inaccuracies and respond to any questions as soon as possible.

IANA has questions for one of the IANA actions for this document.

We received the following comments/questions from the IANA's reviewer:

IANA understands that, upon approval of this document, there are three actions which IANA must complete.

First, in the RDMAP Message Operation Codes subregistry of the RDDP Registry located at:

http://www.iana.org/assignments/rddp/

The following four operation codes are to be added as follows:

RDMAP Message Operation Code: 0x8
Message Type: Immediate Data
Reference: [ RFC-to-be ]

RDMAP Message Operation Code: 0x9
Message Type: Immediate Data with Solicited Event
Reference: [ RFC-to-be ]

RDMAP Message Operation Code: 0xA
Message Type: Atomic Request
Reference: [ RFC-to-be ]

RDMAP Message Operation Code: 0xB
Message Type: Atomic Response
Reference: [ RFC-to-be ]

Second, a new registry is to be created called the RDMAP Message Atomic Operation Subcodes and located in the RDDP Registry located at:

http://www.iana.org/assignments/rddp/

Subcodes are 4-bit values. Fields to be recorded in the new registry are: RDMAP Message Atomic Operation Subcode, Atomic Operation, RFC Reference.

The registry is to be maintained through Standards Action as defined by RFC 5226.  All other values are Unassigned and available to IANA for assignment.

There are two initial values for the new registry as follows:

RDMAP Message Atomic Operation Subcode: 0x0
Atomic Operation: FetchAdd
Reference: [ RFC-to-be ]

RDMAP Message Atomic Operation Subcode: 0x2
Atomic Operation: CmpSwap
Reference: [ RFC-to-be ]


Third, a new registry is to be created called the RDMAP Queue Numbers and also located in the RDDP Registry located at:

http://www.iana.org/assignments/rddp/

Subcodes are 32-bit values. Fields to be recorded in the new registry are: RDMAP DDP Untagged Queue Numbers, Atomic Operation, RFC Reference.

The registry is to be maintained through Standards Action as defined by RFC 5226.  All other values are Unassigned and available to IANA for assignment.

There are four initial values for the new registry as follows:

  0x00000000, Queue 0 (Send operation Variants), [RFC5040]

  0x00000001, Queue 1 (RDMA Read Request operations), [RFC5040]

  0x00000002, Queue 2 (Terminate operations), [RFC5040]

  0x00000003, Queue 3 (Atomic Response operations), [RFCXXXX]


QUESTIONS:
1. It appears that you define three fields:

RDMAP DDP Untagged Queue Numbers, Atomic Operation, RFC Reference.

But, according to the "Initial registry contents", there are four
types/fields: 0x00000000, Queue 0 (Send operation Variants), [RFC5040].
What is the field name for 0x00000000?
What is the field name for Queue #?
What is the field name for 'Send operation Variants'?

2. Should the value 2^32-1 be made available for assigment?

IANA understands that these three actions are the only ones required to be completed upon approval of this document.


Note:  The actions requested in this document will not be completed
until the document has been approved for publication as an RFC.
This message is only to confirm what actions will be performed.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (RDMA Protocol Extensions) to Proposed Standard


The IESG has received a request from the STORage Maintenance WG (storm)
to consider the following document:
- 'RDMA Protocol Extensions'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2014-02-19. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document specifies extensions to the IETF Remote Direct Memory
  Access Protocol (RDMAP [RFC5040]). RDMAP provides read and write
  services directly to applications and enables data to be transferred
  directly into Upper Layer Protocol (ULP) Buffers without
  intermediate data copies. The extensions specified in this document
  provide the following capabilities and/or improvements: Atomic
  Operations and Immediate Data.






The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-storm-rdmap-ext/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-storm-rdmap-ext/ballot/


No IPR declarations have been submitted directly on this I-D.

Document Writeup

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up.

Changes are expected over time. This version is dated 24 February 2012.

STORM WG
draft-ietf-storm-rdmap-ext
January 10, 2014

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

  Proposed Standard.

  The RFC type is appropriate because the protocol specifies extensions
  to RFC5040, which is also classified Proposed Standard. The extensions are
  upwardly compatible, not experimental, and request that IANA commit the
  extension's opcodes and two new registries to the existing RDDP Registries.

  The RFC Type in the draft's title page header reads: "Intended status: Standards Track".

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  This document specifies extensions to the IETF Remote Direct Memory
  Access Protocol (RDMAP [RFC5040]). RDMAP provides read and write
  services directly to applications and enables data to be transferred
  directly into Upper Layer Protocol (ULP) Buffers without
  intermediate data copies. The extensions specified in this document
  provide the following capabilities and/or improvements: Atomic
  Operations and Immediate Data.

Working Group Summary

  The extensions add Atomic Operations and Immediate Data to the RDMAP
  Protocol.  Other RDMA transport protocols define the functionality added by
  these extensions leading to differences in RDMA applications and/or
  Upper Layer Protocols. Removing these differences in the transport
  protocols simplifies these applications and ULPs.

  The STORM Working Group chartered this work in mid-2011, and while
  no significant dissent was encountered, the document spent an extended
  time in the WG owing to other WG priorities. When work resumed in 2013,
  the document received substantial attention and review. Four update
  cycles resulted in the final text. There was no controversy and consensus
  was easily reached.

Document Quality

  The document is of high quality and is well written. The authors represent
  companies which deliver industry implementations of the RDDP protocol, and
  discussion in the STORM WG by developers of upper layers and applications
  indicated strong interest in using the extensions.

Personnel

  Document Shepherd: Tom Talpey (storm WG co-chair, ttalpey@microsoft.com)
  Responsible Area Director: Martin Stiemerling (Transport, mls.ietf@gmail.com)

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

  The Document Shepherd personally reviewed the document numerous times
  during the Working Group cycle, and as a contributor to all the RDDP RFCs,
  considers the document to be of high quality and the extensions well architected.
  Many other WG contributors also reviewed, and all comments were addressed
  in a thorough manner. The Shepherd considers the document fully ready for
  publication.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed? 

  No, as detailed above.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

  No, the extensions are specific to the RDMAP RFC5040 protocol and do
  not introduce nor depend on external WGs or from broader perspectives.
  Security issues were carefully reviewed in the WG to ensure the extensions
  aligned with existing RDMAP security considerations and with DDP/RDMAP
  Security (RFC5042).

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

  The Document Shepherd has no concerns, and additionally is not aware of
  any concerns from any other individual.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

  Yes, the Document Shepherd has obtained recent direct responses from
  all five named authors to this question. All indicated that they were aware
  of no IPR requiring disclosure, and had no intention to file any.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

  No IPR disclosures have been filed, according to the tools page as of
  this date. No discussion of potential IPR was brought up in the WG.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

  The consensus is solid, representing the concurrence of numerous
  active WG contributors well-versed in the RDDP protocols. No concerns
  have been raised.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

  No appeals have been threatened or filed.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

  Two nits are reported by the tool. A mention of RFC5040 in the abstract
  appears within brackets and triggers a warning, which in the opinion of
  the Shepherd is harmless. A second reference to "RFCXXXX" is a note to
  the RFC Editor. Both these can and will be easily corrected by the RFC
  Editor before publication, as needed.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

  No formal review is necessary, as the document contains no MIB, media
  or URI types.

(13) Have all references within this document been identified as
either normative or informative?

  Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

  No such references are present in the document.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

  No downward normative references are present in the document.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

  The document updates and extends RFC5040, but will not change its status.
  RFC5040 is listed in the abstract and discussed in the introduction.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

  The IANA Considerations section was carefully reviewed and its
  Proposed contents discussed by the WG. The Shepherd agrees with
  the actions it proposes, which include adding 4 values to an existing
  registry, and adding two newly-defined registries to the existing "RDDP
  registries" set. The "RDMAP Queue Numbers" addition represents a
  prior omission, and is therefore a required improvement.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

  The new IANA registries will not require future Expert Review beyond
  the normal diligence applied to review of such requests.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

  Not applicable.