[Search] [pdf|bibtex] [Tracker] [WG] [Email] [Nits]

Versions: 00                                                            
Network Woking Group                                 Danny Cohen Myricom
Internet Draft                              Craig Lund Mercury Computers
Expire in six months          Tony Skjellum, Thom McMahon, Robert George
                                            Mississippi State University
                                                                May 1998

             Enumerations for the PacketWay Protocol for
          High-Performance Interconnection of Computer Clusters

Status of this Memo

     This document is an Internet-Draft.  Internet-Drafts are working
     documents of the Internet Engineering Task Force (IETF), its
     areas, and its working groups.  Note that other groups may also
     distribute working documents as Internet-Drafts.

     Internet-Drafts are draft documents valid for a maximum of six
     months and may be updated, replaced, or obsoleted by other
     documents at any time.  It is inappropriate to use Internet-
     Drafts as reference material or to cite them other than as
     "work in progress."

     To view the entire list of current Internet-Drafts, please check
     the "1id-abstracts.txt" listing contained in the Internet-Drafts
     Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net
     (Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East
     Coast), or ftp.isi.edu (US West Coast).

Table of Contents:

             This page....................................1
             PacketWay at a Glance ("cheat-sheet")........2
             A note about the PacketWay documents.........3
             (1) PacketWay Packet Types...................5
             (2) RRP messages.............................6
             (3) RRP records..............................6
             (4) Error messages...........................7
             (5) Node capabilities........................7
             (6) Optional header fields types (OH)........8
             (7) Byte order (endianness)..................8
             (8) Symbol types (ST)........................9
             Appendix-A: Glossary........................10
             Appendix-B: Acronyms and Abbreviations......12

Cohen et al.                  Experimental                      [Page 1]

Internet-Draft               PacketWay Enumerations             May 1998

                        PacketWay at a Glance

       2   6    type       24                     16                16
   PH1|V|  P   |     Destination-Type     |  Type-Extension |   Packet-Type   |
   PH2| E | PL|   Data-Length (8B-words)  |h|  RZ  |0     Source-Address      |
        4    3             25              1   7    1           23

                type = 0xxx Physical Address
                       10xx L2RH
                       110x Reserved
                       1110 Logical Address
                       1111 Symbols

        2   6    2   6      8        8        8        8        8        8
L2RH  |V|  P   |10LLLLLL|  SR01  |  SR02  |........|........|........|........|

        2   6     4   4     8        8        8        8        8        8
Symbol|V|  P   |1111ssss|ssssssss|ssssssss| Length |  data  |........|........|
                    <---- Symbol Type --->

        2   6      8        8        8        8        8        8        8
Opt'l +--------+--------+--------+--------+--------+--------+--------+--------+
hdr   |TCtttttt|LLLLLLLL|  data  |........|........|........|........|........|
      T: 0=optional, 1=mandatory;  C: 0=more OH-fields follow, 1=last OH-field

          8        8        8        8        8        8        8        8
RRP   +--------+--------+--------+--------+--------+--------+--------+--------+
Record|  RTyp  |   PL   |       RL        |........|........|........|........|
      RRP-messages: GVL2, L2SR, RDRC, TELL, INFO, HRTO, WRU,  GVRT, RTBL;

Cohen et al.                  Experimental                      [Page 2]

Internet-Draft               PacketWay RRP                      May 1998

                  A note about the PacketWay Documents

The PacketWay protocol is defined by a series of documents:

                * EEP (End-to-End Protocol)
                * RRP-1 (basic Router-to-Router Protocol)
                * RRP-2 (dynamic inter-SAN routing)
                * PacketWay enumerations

Each of these documents should include the same "PacketWay at a Glance
(Cheat=Sheet)", this note, and the Notations page.  They should include
also (as appendices) a copy of the PacketWay glossary of terms and its
acronyms and abbreviations list.

The EEP and the RRP documents will be published first as Internet-Drafts
and later as Proposed-Standards, Draft-Standards, and Standards.

The Enumeration Document will be first published as an
"Informational-RFC" and later will be maintained by IANA.

The enumeration document may be attached to the EEP/RRP documents, as
a matter of convenience.  The enumeration is NOT a part of the
PacketWay standard, just as RFC0739 (the original "Assigned Numbers"
RFC) is not a part of RFC0791, that defines IP.

Similarly, the EEP-documenmt has "Appendix-A: A Recommendation for
PacketWay Address Assignment" which is a recommendation only and NOT a
part of the PacketWay standard, just as IP-address-assignment is not a
part of RFC0791, that defines IP.

Cohen et al.                  Experimental                      [Page 3]

Internet-Draft               PacketWay RRP                      May 1998

8B means "8-byte" (64 bits).

0x indicates hexadecimal values,  e.g., 0x0100 is 2^8=256(decimal).

0b indicates binary values, e.g., 0b0100 is 4(decimal).

xxx indicate a field that is discarded without any checking (e.g., padding).

[fff] indicates that fff is an optional field, when appropriate.

[exp], in equations, is the integral part of `exp`, e.g., [17/8]=2.

All length fields do not include themselves, and therefore may be zero.
Fields lengths are specified either (a) by byte count, with following
padding bytes to fill 8B-words, or (b) by 8B-word count, and PL, the
number of trailing padding bytes (with 0<=PL<8).

Cohen et al.                  Experimental                      [Page 4]

Internet-Draft               PacketWay RRP                      May 1998

                        PacketWay Enumerations

(1) PacketWay Packet Types

The EEP header reserves 4 bytes for signaling from the source node
directly to the destination node.  They are the PACKET TYPE (PT),
and the TYPE EXTENSION (TE), 2 bytes each.

This list defines values for the PACKET-TYPE (PT) 2B-field.  Each
packet-type has its own interpretation of the TE and the h-fields.

    2B-Code     Packet Type
+----------     ----------------------
          0     Illegal
          1     RRP
          2     Embedded PacketWay Packet
          3     MEM-READ
          4     MEM-WRITE
                Higher level protocols:
         21             IP
         22             SNMP
         23             ATM
                Link layer Protocols
         50             Ethernet (E10)
         51             Ethernet (E100)
         52             Ethernet (E1000)
         53             Myrinet
         54             Fibre Channel
         55             RACEway
         56             SCI
         57             VME
                Application level protocols:
         81             MPI
         82             PVM
                Secure Protocols
        121     Secure (1)
        122     Secure (2)
        123     Secure (3)
1,024-2,047     User defined
     65,535     ERR (for Error)

More values will be assigned.  "Ether-types" should be added with
a pointer to those used by the Internet.

Cohen et al.                  Experimental                      [Page 5]

Internet-Draft               PacketWay RRP                      May 1998

Explanation of the MEM-WRITE and MEM-READ (that a memory board
may need):

     PT        Meaning
  ---------    --------------------------------------------------------
  MEM-WRITE -- Treat the first 8 bytes of the Data Block as a local
               memory-address and write the remaining data into memory.

  MEM-READ  -- Treat the data block as 2 8B-memory-addresses and an
               8B-byte count.  Generate a return WRITE packet containing
               the first address, followed by the appropriate data,
               that was read from the second address.

(2) RRP Messages  (Type Extensions of PT="RRP)
  Type  Code Description
+------ ---- ----------------------------------------------------
          0  Illegal
  GVL2   21  Please give me L2-routes from you to node (address)
  L2SR   22  Here are L2-routes to node (address)
  RDRC   23  Re-direct to node (address) via a neighbor HR (address)
  TELL   24  Please tell about node (address | name | capabilities)
  INFO   25  Info about node (address, name, capabilities)
  HRTO   26  Which HR should I use for node (address)
  WRU?   27  Who/what-Are-You?
  GVRT   28  Please give me your RTs
  RTBL   29  Here is an RT

Throughout this document the RRP messages are indicated by their
type (e.g., RDRC for re-direct).  In actual messages the code is used
(e.g., 2 for RDRC).

(3) RRP records

  RTyp  Code Description
+------ ---- ----------------------------------------------------
          0  Illegal
  ADDR   41  Address record for one or many nodes
  NAME   42  Node Name record
  CAPA   43  Node Capability record
  LADR   44  Node Logical Addresses record
  SRQR   45  Source Route record and its Quality (SR, Q)
  MTUR   46  MTU record (for the previous SRQR)

Throughout this document the RRP records are indicated by their RTyp (e.g.,
ADDR for address).  In actual messages the code is used (e.g.,41 for ADDR).

Cohen et al.                  Experimental                      [Page 6]

Internet-Draft               PacketWay RRP                      May 1998

(4) Error Messages

  Subtype     Code      Description
  ---------   ----      ----------------------------------------------
                0       Illegal
  UNK          71       Unknown (address)
  HRDOWN       72       HR-Down (and the links associated with it)
  LINKDOWN     73       Link-Down (between two HRs)
  GENERAL      74       General error message

Throughout this document the error messages are indicated by their
subtype (e.g., LINKDOWN for Link-Down).  In actual messages the code
is used (e.g., 3 for LINKDOW).

(5) PacketWay Node Capabilities

Code  Capability                Parameters
+---  ------------------------  --------------------------------------
  0   Illegal
  1   GP Computing Node
  2   Router                    SAN-IDs, 1+3 Bytes each
  3   PacketWay Server
  4   Network Multicast Server
  5   NFS
  6   NPS (Paging Server)
  7   Floating-point DSP        IEEE word-sizes (in bytes), 1B per size
  8   Fixed-point DSP           word-sizes (in bytes), 1B per size
  9   Printer
250   SRVLOC server
253   Secure PacketWay HR
254   Multicast agent for its SAN
255   SAN

Cohen et al.                  Experimental                      [Page 7]

Internet-Draft               PacketWay RRP                      May 1998

(6) Optional Header Fields Types (OH)

The MSbit of the type field (T) is the type-of-type.  Its assignment is:

   0b0: Optional (may drop this OH if its type, tttttt, is unknown)
   0b1: Mandatory (should not process this OH if its type is unknown)

The next bit is the "Completion bit" (C).  Its assignment is:

   0b0: More options follow
   0b1: This is the last option field

The 6 LSbits, tttttt, are the type field.  Their assignment is:

 0x00:      Illegal
 0x01:      TBD
 0x02:      CRC32 here
 0x03:      CRC32 following in the OT (after the DB)
 0x04:      CRC64 here
 0x05:      CRC64 following in the OT (after the DB)
 0x06:      There is an OT (Optional Trailer)
 0x07-0x3D: TBD
 0x3E:      Cryptographic data

(7) Byte Order (Endianness)

A 4 bit field (E) is used to indicate Endianness, with e being its first
bit (MSbit).

        e=0: Big-Endian order
        e=1: Little-Endian order

The 3 LSbits indicate the size of data chucks (must be the same for the
entire data block) to allow hardware swapping

        e000: don't swap, it's 8-bit data
        e001: swap as if all the data is  16-bit words
        e010: swap as if all the data is  32-bit words
        e011: swap as if all the data is  64-bit words
        e100: swap as if all the data is 128-bit words
        e101: illegal and reserved for future use
        e110: illegal and reserved for future use
        e111: illegal and reserved for future use

Cohen et al.                  Experimental                      [Page 8]

Internet-Draft               PacketWay RRP                      May 1998

(8) Symbol Types (ST)

The format of Symbols is:

|vv000000|1111ssss|ssssssss|ssssssss| Length |  data  |........|........|
              <---- Symbol Type --->

        Code      Symbol Type
        --------  -----------------
        0x00000:  Reserved
        0x00001:  Multicast
        0x00002:  Secure PacketWay

Cohen et al.                  Experimental                      [Page 9]

Internet-Draft               PacketWay RRP                      May 1998

                          Appendix-A: Glossary
                      (Last update: Aug-24, 1997)

Address:        A unique designation of a node (actually an interface to
                that node) or a SAN.

Buddy-HR:       HRs are "buddies" if they are on the same SAN.

Cut-Thru:       See wormhole.

Destination:    The node to which a packet is intended

Dynamic-Routing: Routing according to dynamic information
                (i.e., acquired  at run time, rather than pre-set).

Endianness:     The property of being Big-Endian or Little-Endian
                (transmission order, etc.)

Ethertype:      A 16-bit value designating the type of Level-3 packets
                carried by a Level-2 communication system.

HR:             Half-Router, the part of a router that handles one
                network only.

L2-Forwarding:  Forwarding based on Level-2 (i.e., data-link layer
                of the ISORM) information, e.g., the native technique
                of each SAN or LAN.  Also called "source routing."

L3-Forwarding:  Forwarding based on end-to-end Level-3 (i.e., network
                layer of the ISORM) addresses.  Also called
                "destination routing."

Map:            The topology of a network.

Mapper:         A node on a SAN/LAN that has the map and an RT for that
                network.  It is expected that the mapper dynamically
                updates the map and the RT.

Multi-homed Node: A node with more than one network interface, where each
                interface has another address.

Node:           Whatever can send and receive packets (e.g., a computer,
                an MPP, a software process, etc.)

Node structure: A C-struct (or equivalent) containing values for some
                attributes of a node.

Planned Transfer: Transfer of information, occurs after an initial phase
                in which the sender decides which Level-2 route to use
                for that transfer.

Cohen et al.                  Experimental                     [Page 10]

Internet-Draft               PacketWay RRP                      May 1998

RCVF:           The "Received From" set includes all the physical
                addresses through which an RT was disseminated, starting
                with that of the mapper that created that RT.

Re-direct-message: A message that tells nodes which HR should be used in

                order to get to a certain remote address (or range of).

Router:         The inter-SAN communication device

Security Context: A relationship between 2 (or more) nodes that defines
                how the nodes utilize security services to communicate

Source:         The node that created a packet.

Source-Route:   A Level-2 route that is chosen for a packet by its source.

Symbol:         Data preceeding the EEP header of a PacketWay message,
                interleaving with the L2RHs.

Twin-HR:        Two HRs are twins if they both are parts of the same
                inter-SAN router.

Wormhole-routing: (aka cut-thru routing) forwarding packets out of
                switches as soon as possible, without storing that
                entire packet in the switch (unlike Stop-and-forward).

Zero-copy TCP:  A TCP system that copies data directly between the user
                area and the network device, bypassing OS copies.

Cohen et al.                  Experimental                     [Page 11]

Internet-Draft               PacketWay RRP                      May 1998

                 Appendix-B: Acronyms and Abbreviations
                     (Last update: August 24, 1997)

0bNNNN  The binary number NNNN (e.g., 0b0100 is 4-decimal)
0xNNNN  The hexadecimal number NNNN (e.g., 0x0100 is 256-decimal)
8B      8 byte (64 bits) entity
ADDR    The Address-record of RRP
API     Application/Program Interface
AT      Address Type
ATM     Asynchronous Transmission Mode
B       Byte (e.g., 4B)
b       bit (e.g., 32b)
BC      Byte Count (of parameters)
BER     Bit Error Rate
CAPA    The CAPAbility-record of RRP
CC      Capability Code
CSR     Common Source-Route
DA      Destination Address
DB      Data Block
DL      Data Length (in 8B words)
DSP     Digital Signal Processore
DT      Destination-Type
e       The MSbit of E
E       The Endianness field (in the EEP header)
EEP     End/End Protocol
EI      Error Indication
GP      General Purpose
GVL2    An RRP message, requesting L2 route to a given destination
GVRT    An RRP message asking an HR to give its routing tables
h       Optional header fields flag
HR      Half Router
HRTO    An RRP message asking which HR to use for a given destination
ID      Identification
IGMP    Internet Group Management Protocol
INFO    An RRP message providing information about nodes
IP      The Internet protocol
ISORM   The ISO Reference Model
L       Length field (exclusive of itself)
L2      Level-2 of the ISORM (Link)
L2RH    Level-2 Routing Header
L2SR    Source Route
L3      Level-3 of the ISORM (Network)
LA      Logical Address
LADR    The Logical-addresses-record of RRP
LAN     Local Area Network
LRT     Local Routing Table
LSbit   Least Significant bit
LSbyte  Least Significant byte
MAC     Message Authentication Code / Media Access Control
MPI     Message Passing Interface
MPP     Massively Parallel Processing system

Cohen et al.                  Experimental                     [Page 12]

Internet-Draft               PacketWay RRP                      May 1998

MSbit   Most Significant bit
MSbyte  Most Significant byte
MSU     Mississippi State University
MTU     Maximum Transmission Unit
MTUR    The MTU-record of RRP
M/C     Multicast
NAME    The name-record of RRP
NFS     Network File Server
OH      Optional Header field
OH-TYPE The Type of an Optional Header field
OT      Optional Trailer field
P       The Priority field
PAD     Padding After Data
PBD     Padding Before Data
PCI     The Peripheral Component Interconnect "standard"
PH      PacketWay Header
PL      Padding Length (always in bytes)
PPP     The Point-to-Point Protocol
PROM    Programmable ROM (Read-Only-Memory)
PT      Packet Type (2B)
PVM     Parallel Virtual Machine
PW      The Myrinet Packet Type assigned to PacketWay (PW=0x0300)
Q       Quality (of a path)
RCVF    Received-From list, or the Received-From record of RRP
RDRC    A re-direct message of RRP
RH      Routing Header
RID     Record ID
RL      Record Length (in 8B-words)
RRP     Router/Router Protocol
RT-hd   RT (Routing Table) header
RT      Routing Table
RTBL    An RRP message proving a Routing Table
RTHD    The Routing-Table-Header record of RRP
RTyp    RRP's Record Type
RZ      The Reserved field (in the EEP header)
SA      Source Address
SAN     System Area Network
SAN-ID  The 24-bit PacketWay-address of a SAN
SAR     Segmentation and Reassembly
SN      Serial Number
SNMP    Simple Network Management Protocol
SR      Source Route (always at Level-2)
SRQR    The Source-Route-and-Q-record of RRP
ST      Symbol Type
TAIL    PacketWay EEP Trailer
TE      Type Extension (2B)
TELL    An RRP message requesting information about nodes partially specified
UNK     Unknown
V       Version
WRU?    An RRP message asking its recipient to identify itself
XRT     External Routing Table
xxx     A padding byte

Cohen et al.                  Experimental                     [Page 13]

Internet-Draft               PacketWay RRP                      May 1998

Editor's Address

   Anthony Skjellum
   Computer Science Department
   Box 9637
   Mississippi State University
   Mississippi State, MS 39762-9637

   Phone: 601-325-8435
   Fax:   601-325-8997
   Email: tony@cs.msstate.edu