NNTP Extensions Working Group J. Vinocur Internet Draft Cornell University Updates: 2970 (if approved) K. Murchison Expires: April 2005 Oceana Matrix Ltd. October 2004 NNTP Extension for Streaming Feeds draft-ietf-nntpext-streaming-02 Status of this memo By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.html. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (2004). Abstract This memo defines an extension to the Network News Transport Protocol [NNTP] to provide asynchronous transfer of articles. This allows servers to transfer articles to other servers with much greater efficiency. Section 1 of [NNTP-COMMON] summarizes some ad-hoc transport extensions currently used in the NNTP protocol. This document Vinocur Expires April 2005 [Page 1]
Internet Draft NNTP Extension for Streaming Feeds October 2004 updates and formalizes the CHECK and TAKETHIS commands and deprecates the MODE STREAM command. Table of Contents 0. Changes from Previous Version ............................ 2 1. Introduction ............................................. 3 1.1. Conventions Used in this Document ................... 3 2. The STREAMING Extension .................................. 4 2.1. Advertising the STREAMING Extension ................. 4 2.2. Asynschronous Article Transfer ...................... 4 2.3. MODE STREAM Command ................................. 5 2.3.1. Usage .......................................... 5 2.3.2. Description .................................... 5 2.3.3. Examples ....................................... 6 2.4. CHECK Command ....................................... 6 2.4.1. Usage .......................................... 6 2.4.2. Description .................................... 6 2.4.3. Examples ....................................... 7 2.5. TAKETHIS Command .................................... 7 2.5.1. Usage .......................................... 7 2.5.2. Description .................................... 7 2.5.3. Examples ....................................... 8 3. Augmented BNF Syntax for the STREAMING Extension ......... 9 3.1. Commands ............................................ 9 3.2. LIST EXTENSIONS responses ........................... 10 4. Summary of Response Codes ................................ 10 5. Security Considerations .................................. 11 6. IANA Considerations ...................................... 11 7. References ............................................... 12 7.1. Normative References ................................ 12 7.2. Informative References .............................. 12 8. Author's Address ......................................... 12 9. Acknowledgements ......................................... 12 10. Intellectual Property Rights ............................ 13 11. Copyright ............................................... 13 0. Changes from Previous Version New: o Added Ken as co-author. o Added 432 response code as a temporary failure for TAKETHIS (with example). o "Summary of Response Codes" section. o "Intellectual Property Rights" section (as required). Changed: o Updated boilerplate to conform to new requirements. Vinocur Expires April 2005 [Page 2]
Internet Draft NNTP Extension for Streaming Feeds October 2004 o Changed layout to be more consistent with the base, AUTHINFO, and STARTTLS documents. o Updated Abstract, referencing NNTP-COMMON. o Split pipelining discussion into separate section. o Updated CHECK and TAKETHIS description, utilizing text from [NNTP]. o Merged two TAKETHIS examples to illustrate pipelining. o Updated IANA section to be more complete. o Updated acknowledgments. Clarified: o Behavior of LIST EXTENSIONS after MODE STREAM (with extended example). o MODE STREAM is deprecated and only used for compatibility with legacy implementations. o 439 response code is a rejection (permanent failure). Other: o Assorted updates of phrasing and typographical varieties. 1. Introduction According to the NNTP specification [NNTP], a peer uses the IHAVE command to query whether a server wants a particular article. Because the IHAVE command cannot be pipelined, the need to stop and wait for the remote end's response greatly restricts the throughput that can be achieved. The alternative method of server-to-server article transfer described in this document permits a more consistent use of network bandwidth. 1.1. Conventions Used in this Document The notational conventions used in this document are the same as those in [NNTP] and any term not defined in this document has the same meaning as in that one. The key words "REQUIRED", "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", "MAY", and "OPTIONAL" in this document are to be interpreted as described in "Key words for use in RFCs to Indicate Requirement Levels" [KEYWORDS]. This document assumes you are familiar with NNTP [NNTP]. In general, the connections described below are from one server to another (i.e., feeding articles to a peer), but we will continue to use "client" to mean the initiator of the NNTP connection, and "server" to mean the other endpoint. In the examples, commands from the client are indicated with [C], Vinocur Expires April 2005 [Page 3]
Internet Draft NNTP Extension for Streaming Feeds October 2004 and responses from the server are indicated with [S]. 2. The STREAMING Extension This extension provides three new commands: MODE STREAM, CHECK, and TAKETHIS. The label for this extension is STREAMING. 2.1. Advertising the STREAMING Extension A server supporting the streaming commands described in this document will advertise the "STREAMING" extension label in response to the LIST EXTENSIONS command. The server MUST continue to advertise this extension label after a client has issued the MODE STREAM command. This extension label MAY be advertised both before and after any use of MODE READER, with the same semantics. Example: [C] LIST EXTENSIONS [S] 202 Extensions supported: [S] NNTPv2 [S] STREAMING [S] . [C] MODE STREAM [S] 203 Streaming permitted [C] LIST EXTENSIONS [S] 202 Extensions supported: [S] NNTPv2 [S] STREAMING [S] . [C] MODE READER [S] 200 Posting allowed [C] LIST EXTENSIONS [S] 202 Extensions supported: [S] NNTPv2 [S] HDR [S] . 2.2. Asynschronous Article Transfer The STREAMING extension provides the same functionality as the IHAVE command ([NNTP] section 6.3.2) but splits the query and transfer functionality into the CHECK and TAKETHIS commands respectively. This allows the CHECK and TAKETHIS commands to be pipelined (unlike IHAVE) and provides for asynschronous article transfer. A streaming client will often pipeline many CHECK commands and use the responses to construct a list of articles to be sent by a Vinocur Expires April 2005 [Page 4]
Internet Draft NNTP Extension for Streaming Feeds October 2004 pipelined sequence of TAKETHIS commands, thus increasing the fraction of time spent transferring articles. The CHECK and TAKETHIS commands utilize distinct response codes so that these commands can be intermingled in a pipeline and the response to any single command can be definitively identified by the client. The client MAY send articles via TAKETHIS without first querying the server with CHECK. The client SHOULD NOT send every article in this fashion unless explicitly configured to do so by the site administrator based on out-of-band information. However, the client MAY use an adaptive strategy where it initially sends CHECK commands for all articles, but switches to using TAKETHIS without CHECK if most articles are being accepted (over 95% acceptance may be a reasonable metric in some configurations). If the client uses such a strategy, it SHOULD also switch back to using CHECK on all articles if the acceptance rate ever falls much below the threshold. 2.3. MODE STREAM Command This command is deprecated and is only documented for compatibility with legacy implementations of the STREAMING extension. New client and server implementations are not required to support this command but they are encouraged to do so. 2.3.1. Usage This command MUST NOT be pipelined. Syntax MODE STREAM Responses 203 Streaming permitted 2.3.2. Description A client issues the MODE STREAM command to indicate that it wishes to use the CHECK and TAKETHIS commands instead of, or in addition to, IHAVE. If the server supports the CHECK and TAKETHIS commands a 203 response MUST be returned. If a server advertises the STREAMING extension label in the LIST EXTENSIONS response, the MODE STREAM command is not required and a client SHOULD NOT issue it. If a legacy server does not support the LIST EXTENSIONS command or does not advertise the NNTPv2 extension label, a client SHOULD issue the MODE STREAM command prior to using the CHECK or TAKETHIS commands. Vinocur Expires April 2005 [Page 5]
Internet Draft NNTP Extension for Streaming Feeds October 2004 A server MUST NOT require that the MODE STREAM command be issued by the client before accepting the CHECK or TAKETHIS commands. A server SHOULD accept the MODE STREAM command for compatibility with legacy clients which don't use the LIST EXTENSIONS discovery mechanism. 2.3.3. Examples Example of a client requesting the ability to stream articles from a server which does not support this extension: [C] MODE STREAM [S] 501 Unknown MODE variant Example of a client successfully requesting the ability to stream articles: [C] MODE STREAM [S] 203 Streaming permitted 2.4. CHECK Command 2.4.1. Usage Syntax CHECK message-id Responses 238 message-id Send article to be transferred 431 message-id Transfer not possible; try again later 438 message-id Article not wanted Parameters message-id = Article message-id The first parameter of the 238, 431, and 438 responses MUST be the message-id provided by the client as the parameter to CHECK. 2.4.2. Description The CHECK command informs the server that the client has an article with the specified message-id. If the server desires a copy of that article a 238 response MUST be returned, indicating that the client may send the article using the TAKETHIS command. If the server does not want the article (if, for example, the server already has a copy of it), a 438 response MUST be returned, indicating that the article is not wanted. Finally, if the article isn't wanted immediately but the client should retry later if possible (if, for example, another client is in the process of sending the same article to the server), a 431 response MUST be Vinocur Expires April 2005 [Page 6]
Internet Draft NNTP Extension for Streaming Feeds October 2004 returned. Note however, that the responses to CHECK are advisory; the server MUST NOT rely on the client to behave as requested by these responses. 2.4.3. Examples Example of a client checking whether the server would like a set of articles and getting a mixture of responses: [C] CHECK <i.am.an.article.you.will.want@example.com> [S] 238 <i.am.an.article.you.will.want@example.com> [C] CHECK <i.am.an.article.you.have@example.com> [S] 438 <i.am.an.article.you.have@example.com> [C] CHECK <i.am.an.article.you.defer@example.com> [S] 431 <i.am.an.article.you.defer@example.com> Example of pipelining the CHECK commands in the previous example: [C] CHECK <i.am.an.article.you.will.want@example.com> [C] CHECK <i.am.an.article.you.have@example.com> [C] CHECK <i.am.an.article.you.defer@example.com> [S] 238 <i.am.an.article.you.will.want@example.com> [S] 438 <i.am.an.article.you.have@example.com> [S] 431 <i.am.an.article.you.defer@example.com> 2.5. TAKETHIS Command 2.5.1. Usage Syntax TAKETHIS message-id Responses 239 message-id Article transferred OK 432 message-id Transfer failed; try again later 439 message-id Transfer rejected; do not retry Parameters message-id = Article message-id The first parameter of the 239, 432, and 439 responses MUST be the message-id provided by the client as the parameter to TAKETHIS. 2.5.2. Description The TAKETHIS command informs the server that the client has an article with the specified message-id that will be sent immediately following the CRLF at the end of the TAKETHIS command. The client Vinocur Expires April 2005 [Page 7]
Internet Draft NNTP Extension for Streaming Feeds October 2004 MUST send the entire article, including headers and body, in the format defined in Section 3.1 of [NNTP] for multi-line responses (except that there is no initial line containing a response code). Thus a single dot (".") on a line indicates the end of the text, and lines starting with a dot in the original text have that dot doubled during transmission. The server MUST return either a 239 response, indicating that the article was successfully transferred, a 432 response, indicating that the transfer failed but should be tried again later, or a 439 response, indicating that the article was rejected. This function differs from the POST command in that it is intended for use in transferring already-posted articles between hosts. It SHOULD NOT be used when the client is a personal news reading program, since use of this command indicates that the article has already been posted at another site and is simply being forwarded from another host. However, despite this, the server MAY elect not to post or forward the article if, after further examination of the article, it deems it inappropriate to do so. Reasons for such subsequent rejection of an article may include such problems as inappropriate newsgroups or distributions, disc space limitations, article lengths, garbled headers, and the like. These are typically restrictions enforced by the server host's news software and not necessarily the NNTP server itself. The client SHOULD NOT assume that the article has been successfully transferred unless it receives an affirmative response from the server. A lack of response (such as a dropped network connection or a network timeout) SHOULD be treated the same as a 432 response. Because some news server software may not be able immediately to determine whether or not an article is suitable for posting or forwarding, an NNTP server MAY acknowledge the successful transfer of the article (with a 239 response) but later silently discard it. 2.5.3. Examples Example of sending two pipelined articles to another site (the first article is accepted and the second is rejected): [C] TAKETHIS <i.am.an.article.you.will.want@example.com> [C] Path: pathost!demo!somewhere!not-for-mail [C] From: "Demo User" <nobody@example.com> [C] Newsgroups: misc.test [C] Subject: I am just a test article [C] Date: 6 Oct 1998 04:38:40 -0500 [C] Organization: An Example Com, San Jose, CA [C] Message-ID: <i.am.an.article.you.will.want@example.com> Vinocur Expires April 2005 [Page 8]
Internet Draft NNTP Extension for Streaming Feeds October 2004 [C] [C] This is just a test article. [C] . [C] TAKETHIS <i.am.an.article.you.have@example.com> [C] Path: pathost!demo!somewhere!not-for-mail [C] From: "Demo User" <nobody@example.com> [C] Newsgroups: misc.test [C] Subject: I am just a test article [C] Date: 6 Oct 1998 04:38:40 -0500 [C] Organization: An Example Com, San Jose, CA [C] Message-ID: <i.am.an.article.you.have@example.com> [C] [C] This is just a test article. [C] . [S] 239 <i.am.an.article.you.will.want@example.com> [S] 439 <i.am.an.article.you.have@example.com> Example of sending an article to a site where the transfer fails: [C] TAKETHIS <i.am.an.article.you.will.want@example.com> [C] Path: pathost!demo!somewhere!not-for-mail [C] From: "Demo User" <nobody@example.com> [C] Newsgroups: misc.test [C] Subject: I am just a test article [C] Date: 6 Oct 1998 04:38:40 -0500 [C] Organization: An Example Com, San Jose, CA [C] Message-ID: <i.am.an.article.you.will.want@example.com> [C] [C] This is just a test article. [C] . [S] 432 <i.am.an.article.you.will.want@example.com> 3. Augmented BNF Syntax for the STREAMING Extension This section describes the syntax of the STREAMING extension. It extends the syntax in [NNTP], and non-terminals not defined in this document are defined there. 3.1. Commands This syntax extends the non-terminal "command", which represents an NNTP command. command =/ mode-stream-command / check-command / takethis-command mode-stream-command = "MODE" WS "STREAM Vinocur Expires April 2005 [Page 9]
Internet Draft NNTP Extension for Streaming Feeds October 2004 check-command = "CHECK" WS message-id takethis-command = "TAKETHIS" WS message-id 3.2. LIST EXTENSIONS responses This syntax defines the specific LIST EXTENSIONS responses for the STREAMING extension. extension-descriptor =/ streaming-extension streaming-extension = %x53.54.52.45.41.4D.49.4E,47 ; "STREAMING" 4. Summary of Response Codes This section contains a list of every new response code defined in this document, whether it is multi-line, which commands can generate it, what arguments it has, and what its meaning is. Response code 203 Generated by: MODE STREAM Meaning: streaming permitted. Response code 238 Generated by: CHECK Meaning: send article to be transferred. Response code 239 Generated by: TAKETHIS Meaning: article transferred OK. Response code 431 Generated by: CHECK Meaning: transfer not possible; try again later. Response code 432 Generated by: TAKETHIS Meaning: transfer failed; try again later. Response code 438 Generated by: CHECK Meaning: article not wanted. Response code 439 Generated by: TAKETHIS Meaning: transfer rejected; do not retry. Vinocur Expires April 2005 [Page 10]
Internet Draft NNTP Extension for Streaming Feeds October 2004 5. Security Considerations No new security considerations are introduced by this extension, beyond those already described in the core specification [NNTP]. 6. IANA Considerations This section gives a formal definition of the STREAMING extension as required by Section 8 of [NNTP] for the IANA registry. o The STREAMING extension provides for asynchronous transfer of articles. o The extension-label is "STREAMING". o The extension-label has no arguments. o The extension defines three new commands, MODE STREAM, CHECK, and TAKETHIS, whose behaviour, arguments, and responses are defined in Sections 2.2, 2.3, and 2.4 respectively. o The extension does not associate any new responses with pre- existing NNTP commands. o The extension does not affect the behaviour of a server or client other than via the new commands. o The extension does not affect the maximum length of commands and initial response lines. o The extension does not alter pipelining, and the CHECK and TAKETHIS command can be pipelined, but the MODE STREAM command cannot be pipelined. o Use of this extension does not alter the output from LIST EXTENSIONS. o The extension does not cause any pre-existing command to produce a 401, 480, or 483 response. o A server MAY choose to disallow the MODE STREAM, CHECK, TAKETHIS commands after the MODE READER command. o Published Specification: This document. o Author, Change Controller, and Contact for Further Information: Author of this document. Vinocur Expires April 2005 [Page 11]
Internet Draft NNTP Extension for Streaming Feeds October 2004 7. References 7.1. Normative References [ABNF] Crocker, D., Overell, P., "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997. [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, March 1997. [NNTP] Feather, C., "Network News Transport Protocol", draft-ietf-nntpext-base-*.txt, Work in Progress. 7.2. Informative References [NNTP-COMMON] Barber, S., "Common NNTP Extensions", RFC 2980, Academ Consulting Services, October 2000. 8. Author's Address Jeffrey M. Vinocur Department of Computer Science Upson Hall Cornell University Ithaca, NY 14853 EMail: vinocur@cs.cornell.edu Kenneth Murchison Oceana Matrix Ltd. 21 Princeton Place Orchard Park, NY 14127 USA Email: ken@oceana.com 9. Acknowledgements This document is based heavily on the relevant sections of RFC 2980 [NNTP-COMMON], by Stan Barber. Special acknowledgment also goes to Russ Allbery, Clive Feather, and others who commented privately on intermediate revisions of this document, as well as the members of the IETF NNTP Working Group for continual (yet sporadic) insight in discussion. Vinocur Expires April 2005 [Page 12]
Internet Draft NNTP Extension for Streaming Feeds October 2004 10. Intellectual Property Rights The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards-related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. 11. Copyright Copyright (C) The Internet Society (2004). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights." This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Vinocur Expires April 2005 [Page 13]
