[Search] [txt|pdf|bibtex] [Tracker] [Email] [Nits]

Versions: 00 01 02 03 04                                                
Network Working Group                                     Heiko W.Rupp
Experimental                                                29.04.1997

         A Protocol for the Transmission of Net News Articles
                           over IP multicast.

Status of this Memo

   This document is an Internet-Draft.  Internet-Drafts are working
   documents of the Internet Engineering task Force (IETF), its areas,
   and its working groups.  Note that other groups may also distribute
   working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time.  It is inappropriate to use Internet-Drafts
   as reference material or to cite them other than as "work in

   To learn the current status of any Internet-Draft, please check the
   "1id-abstract.txt" listing contained in the Internet-Drafts Shadow
   Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
   munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
   ftp.isi.edu (US West Coast).


   This protocol provides a way to use the IP multicast infrastructure
   to transmit NetNews articles between news servers. Doing so will
   reduce the bandwidth that is actually needed for transmission via
   NNTP. This does not affect how news reading clients communicate with

Overview and Rationale

   NetNews are bulk data that are produced in large quantities every day
   around the world. Distribution of NetNews on the Internet are usually
   distributed with NNTP[1]. In order to get a fast and redundant
   distribution many news servers communicate with many others, thus
   imposing a higher load on the underlying network than necessary.

   Assume the following scenario:

                             +---------  R1
             S -- A ------- B  --------  R2
                              +--------- R3

   A sender S which wants to transmit articles via NNTP to receivers
   R1...R3 will thus transmit them three times across the link from A to

   With IP Multicast[2], an efficient way to distribute datagrams to
   groups of users exists in the Internet. Thus articles would traverse
   the link A to B only once, thus reducing load on that link.

   This cannot be done with existing news transfer technology, as it is

Rupp                                                            [Page 1]

I/D                 NETNEWS OVER IP MULTICAST           29 April 1997

   based on TCP[10] which cannot be multicasted.  The protocol described
   in this memo is designed to put news articles into datagrams and
   distribute them via IP multicast to receivers that are interested in
   the specific newsgroup. For more information about NetNews, refer to
   [7] and [1].

   In the rest of this memo I will refer to the protocol as ``Multicast
   News Transport Protocol'' (Mcntp).

Protocol overview

   This paragraph will show how news articles are propagated with Mcntp.
   Basically, three parties are involved:

   +    Multicast directory service, MD, coordinates the assignment
        between multicast and news groups.

   +    A Multicast sender, MS, that sends news articles over an
        IP multicast infrastructure

   +    A Multicast receiver, MR, gets packets from the IP multicast
        infrastructure and processes them further.

   So this can be seen as follows:

               directory       +---------+    directory
             +------------- >  |         | ----------------+
            \|/                |   MD    |                \|/
         +---------+           |         |             +--------+
         |         |           +---------+             |        |
         |   MS    |                                   |   MR   |
         |         | ------------ articles ------->    |        |
         +---------+                                   +--------+

   MS and MR can be implemented into existing news server software, or
   can be implemented as separate processes that communicate with the
   news servers (e.g. via NNTP); this does not matter to the protocol.

   MD can either be implemented within MS, or as separate processes that
   communicate with each other. A practical way is to have on MD per
   sender host so that communication between MD and MS is fast and
   reliable, while not too many resources are needed.

   The protocol itself consist of two parts that will be presented in
   the next two chapters -- Distribution of articles and the directory

Rupp                                                            [Page 2]

I/D                NETNEWS OVER IP MULTICAST           29 April 1997

Packet format -- Distribution of articles

   To send articles via IP multicast, they have to be encapsulated into
   UDP packets. The following diagram shows how this can be done:

         |                 Magic                 |
         | Ver|Rev |  Comp   |  Crypt  |  Offset |
         |            Original length            |
         |             Length as sent            |
         |                  Crc                  |
         |              Message-id               |
         |               Data                    |

   All entries are in network byte order.  The fields have the following
   meaning and types:

    +    Magic (32-bit): The String ``McNt''

    +    Ver (4-bit): Protocol version -- currently 1

    +    Rev (4-bit): Protocol revision -- currently 1

    +    Comp (8-bit): Compression method used. Currently are only 2
         methods defined:

          0    Article is not compressed

          1    Article is compressed via zlib [8]

    +    Crypt (8-bit): Encryption method used. See below

    +    Offset (8-bit): Offset of article data from packet start

    +    Original length (32-bit): Length of article before compression
         and encryption

    +    Length as sent (32-bit):

    +    Length of the ``Data'' field (see below)

Rupp                                                            [Page 3]

I/D                 NETNEWS OVER IP MULTICAST           29 April 1997

    +    Crc (32-bit): Checksum over the original articles. See also

    +    Message-ID: The message id of the article in the form it is
         defined in RFC 1036 [7], terminated by a null byte.

    +    Data: The article data after possible compression and

   This memo does not specify a encryption method for the case that the
   field ``Crypt'' is set to anything other than 0; the involved parties
   (i.e. the senders of encrypted news and their receivers) have to
   agree on a method they want to use. If encryption and compression is
   used then the article data is first to be compressed and then the
   result to be encrypted.

   The checksum is computed over the original article data (i.e. before
   possible compression and encryption) by the algorithm of IEEE
   Std1003.2-1992 which is also used in 4.4 BSD cksum(1) tool. See also

   To send an article off, it is encapsulated and then just sent to the
   appropriate multicast group. There is no feedback from the receiver
   to the sender when an article is received.

Directory service

   In order to get a relation between newsgroups and multicast groups, a
   directory service exists; this has been referenced as MD above.  When
   a sender MS wants to propagate a news group, it asks the directory
   service for a multicast group it can use to distribute articles,
   waits for the reply, and starts to send. The directory server
   registers this group in its tables and periodically distributes this
   table over IP multicast. For this purpose, the multicast group
   ``mcntp-directory.mcast.net'' has been officially been assigned by
   the IANA. The UDP port which announcements are sent to, has
   officially been assigned by the IANA as UDP port number 5418 with the
   name ``mcntp''.

   Announcements should not be sent too often to keep traffic low, but
   often enough that new receivers don't have to wait to long to be able
   to receive articles. Once a minute is assumed to be a good value
   here.  Announcements can be sent less often if they are transmitted
   immediately after a change in the directory.

   If more than one directory server is involved (e.g. if there is more
   than one sender site), the directory servers have to listen to
   announcement packets on ``mcntp-directory.mcast.net''. If it does not

Rupp                                                            [Page 4]

I/D                 NETNEWS OVER IP MULTICAST           29 April 1997

   receive a packet after five times the waiting period (e.g., five
   minutes) it can consider itself alone on the net and can choose the
   multicast groups as it wishes. If it does receive a packet, it must
   register the contained groups in its tables and has to select new
   groups so that different newsgroups are not mapped on the same
   multicast group.

   When a receiver (MR) wants to receive a newsgroup, it listens on
   ``mcntp-directory.mcast.net'' for announcements, parses them, and
   then joins the appropriate multicast groups.

   Multicast groups that are no longer in use (e.g. because the sender
   has stopped working) must be removed from the announcement.

   The format of those announcement packets is:

         |   Magic   | Vers | Rev | Offset |
         |             Length              |
         |               Crc               |
         |            Sender-ID            |
         +-----------------+------+---+-----------+       -+
         | Multicast group | Port |TTL| Newsgroup |        |
         +-----------------+------+---+-----------+        |
          ...  repeat ...                                  |  NG lines
         +-----------------+------+---+-----------+        |
         | Multicast group | Port |TTL| Newsgroup |        |
         +-----------------+------+---+-----------+       -+

   All numbers are in network byte order.  The fields have the following
   meaning and types:

       +    Magic (16-bit): The Bytes 0xab

       +    Vers (4-bit): Protocol version

       +    Rev (4-bit): Protocol revision (currently 1)

       +    Offset (8-bit): Offset of NG-lines from packet start.

       +    Length (32-bit): Total packet length.

       +    Crc (32-bit): Checksum over the rest of the packet.

Rupp                                                            [Page 5]

I/D               NETNEWS OVER IP MULTICAST           29 April 1997

       +    Sender-ID  : Identification of sender host, terminated by a
            null byte (see below).

       +    Multicast group (32-bit *): The associated multicast group

       +    Port (8-bit): UDP Port to use for this group

       +    TTL (8-bit): Time to live for multicast packets.

       +    Newsgroup: Name of the Newsgroup in wildmat(3) format,
            terminated by a null byte.

   The protocol version (Vers) is currently 1 for IPv4 and 11 for IPv6.
   The multicast group field (*) is 32 bit in size for IPv4 and 128-bit
   for IPv6 in size.

   The lenght field is 32 bit in size to support IPv6 jumbo datagrams.

   The sender-ID is normally the fully qualified domain name of the
   hosts that sends the announcement. As is common practice with
   NetNews, this can also be the (possibly shorter) entry that the host
   puts in the ``Path:'' header when an article passes through it.

   The checksum is computed over all lines with newsgroup to multicast
   group relations in the packet by the algorithm of IEEE Std1003.2-1992
   which is also used in 4.4 BSD cksum(1) tool. See also [11].

   The lines with newsgroup to multicast group relations are repeated as
   often as needed to announce all groups. The TTL can be used by
   clients to find out if packets that come from this source can reach
   them, or if the sender is too far away.  Note that all entries have
   to fit into one UDP packet.

Reliability Considerations

   As UDP is a unreliable service, provisions for reliable distribution
   of articles are needed. There exist some approaches to reliable
   multicast (XTP [4], KLG [5] or RMTP [6]) which all suffer from some
   problem or other. Specifically, additional hard- or software is
   needed and usually requires kernel modification.

   As there is already a reliable transport of NetNews via NNTP, there
   is no need for a reliable transport via IP multicast: articles need
   not be in order, so it is no problem if one is missing in the
   multicast. Since articles need not arrive in order, lost or missing
   articles can easily be transmitted via an additional NNTP feed.

   As UDP packets can be at maximum 64kBytes in size and every Mcntp

Rupp                                                            [Page 6]

I/D                 NETNEWS OVER IP MULTICAST           29 April 1997

   packet has to fit in one UDP packet, there is no provision given to
   distribute news articles larger than about 63kBytes in size (other
   than compressing them). This does not matter much in practise as
   recent research has shown that more than 95% off all news articles
   are smaller 64kBytes [9]. The remaining 5% can still be transferred
   via NNTP. Some hosts may have problems in receiving UDP packets as
   large as 64kBytes, so in practical use article sizes of 16kBytes
   would be appropriate.


   The distribution of NetNews articles via IP multicast can be a way to
   decrease the network bandwidth used to distribute them.  Articles are
   delivered fast via a nonreliable protocol; later, the holes are
   filled via a reliable, already existing protocol. Compression of
   articles can further reduce the network Load. With encryption private
   news groups can be established on a public IP multicast
   infrastructure. A prototype of a reference implementation already
   shows that Mcntp is fast and can be used as an alternative to
   classical transports.

Security Considerations

   This memo touches security only as far as the provision for
   encryption of article data is given in the protocol.

   No further security issues are discussed.


[1]  RFC 977 -- B. Kantor, P. Lapsley, "Network News Transfer Protocol:
     A Proposed Standard for the Stream-Based Transmission of News".

[2]  RFC 1112 -- S. Deering, "Host extensions for IP multicasting",

[3]  RFC 768 -- J. Postel, "User Datagram Protocol", 08/28/1980.

[4]  XTP -- W. T. Strayer, D.J. Dmepsey, B.C. Weaver, "XTP: The Xpress
     Transfer Protocol", Addison-Wesley

[5]  KLG -- M. Hofmann, "Zuverlaessige Kommunikation in heterogenen
     Netzen", Thesis at "Institut fuer Telematik, CS Dept. Univ

[6]  RMTP -- Lin, John C., Paul Sanjoy, "RMTP: A Reliable Multicast
     Transport Protocol".

Rupp                                                            [Page 7]

I/D                 NETNEWS OVER IP MULTICAST           29 April 1997

[7]  RFC 1036 -- M. Horton, R. Adams, "Standard for interchange of
     USENET messages", 12/01/1987.

[8]  RFC 1950 -- L. Deutsch, J. Gailly, "ZLIB Compressed Data Format
     Specification version 3.3", 05/23/1996.

[9]  http://www.xlink.net/~hwr/histo/ -- Some Statistics about size dis-
     tribution of NetNews

[10] RFC 793 -- J. Postel, "Transmission Control Protocol", 09/01/1981.

[11] Dilip V. Sarwate, "Computation of Cyclic Redundancy Checks Via
     Table Lookup", Communications of the ACM, August 1988.

Author's Address

   Heiko W.Rupp
   Gerwigstr. 5
   D-76131 Karlsruhe

   Phone: +49 721 9661524

   EMail: hwr@pilhuhn.de

Rupp                                                            [Page 8]