Network Woking Group Danny Cohen Myricom
Internet Draft Craig Lund Mercury Computers
Expire in six months Tony Skjellum, Thom McMahon, Robert George
Mississippi State University
May 1998
Enumerations for the PacketWay Protocol for
High-Performance Interconnection of Computer Clusters
<draft-ietf-pktway-enumerations-00.txt>
Status of this Memo
This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its
areas, and its working groups. Note that other groups may also
distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-
Drafts as reference material or to cite them other than as
"work in progress."
To view the entire list of current Internet-Drafts, please check
the "1id-abstracts.txt" listing contained in the Internet-Drafts
Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net
(Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East
Coast), or ftp.isi.edu (US West Coast).
Table of Contents:
This page....................................1
PacketWay at a Glance ("cheat-sheet")........2
A note about the PacketWay documents.........3
Notations....................................4
(1) PacketWay Packet Types...................5
(2) RRP messages.............................6
(3) RRP records..............................6
(4) Error messages...........................7
(5) Node capabilities........................7
(6) Optional header fields types (OH)........8
(7) Byte order (endianness)..................8
(8) Symbol types (ST)........................9
Appendix-A: Glossary........................10
Appendix-B: Acronyms and Abbreviations......12
Cohen et al. Experimental [Page 1]
Internet-Draft PacketWay Enumerations May 1998
PacketWay at a Glance
---------------------
2 6 type 24 16 16
PW-Hdr+-+------+-------+--------+---------+--------+--------+--------+--------+
PH1|V| P | Destination-Type | Type-Extension | Packet-Type |
+-+-+---++--------------------------+-+------+--------+-----------------+
PH2| E | PL| Data-Length (8B-words) |h| RZ |0 Source-Address |
+---+---+--------+--------+---------+-+------+--------+--------+--------+
4 3 25 1 7 1 23
type = 0xxx Physical Address
10xx L2RH
110x Reserved
1110 Logical Address
1111 Symbols
2 6 2 6 8 8 8 8 8 8
+--------+--------+--------+--------+--------+--------+--------+--------+
L2RH |V| P |10LLLLLL| SR01 | SR02 |........|........|........|........|
+--------+--------+--------+--------+--------+--------+--------+--------+
Length
2 6 4 4 8 8 8 8 8 8
+--------+--------+--------+--------+--------+--------+--------+--------+
Symbol|V| P |1111ssss|ssssssss|ssssssss| Length | data |........|........|
+--------+--------+--------+--------+--------+--------+--------+--------+
<---- Symbol Type --->
2 6 8 8 8 8 8 8 8
Opt'l +--------+--------+--------+--------+--------+--------+--------+--------+
hdr |TCtttttt|LLLLLLLL| data |........|........|........|........|........|
fields+--------+--------+--------+--------+--------+--------+--------+--------+
T: 0=optional, 1=mandatory; C: 0=more OH-fields follow, 1=last OH-field
8 8 8 8 8 8 8 8
RRP +--------+--------+--------+--------+--------+--------+--------+--------+
Record| RTyp | PL | RL |........|........|........|........|
+--------+--------+--------+--------+--------+--------+--------+--------+
RRP-messages: GVL2, L2SR, RDRC, TELL, INFO, HRTO, WRU, GVRT, RTBL;
RTyp: ADDR, NAME, CAPA, LADR, SRQR, MTUR, RCVF, RTHD;
Cohen et al. Experimental [Page 2]
Internet-Draft PacketWay RRP May 1998
A note about the PacketWay Documents
------------------------------------
The PacketWay protocol is defined by a series of documents:
* EEP (End-to-End Protocol)
* RRP-1 (basic Router-to-Router Protocol)
* RRP-2 (dynamic inter-SAN routing)
* PacketWay enumerations
Each of these documents should include the same "PacketWay at a Glance
(Cheat=Sheet)", this note, and the Notations page. They should include
also (as appendices) a copy of the PacketWay glossary of terms and its
acronyms and abbreviations list.
The EEP and the RRP documents will be published first as Internet-Drafts
and later as Proposed-Standards, Draft-Standards, and Standards.
The Enumeration Document will be first published as an
"Informational-RFC" and later will be maintained by IANA.
The enumeration document may be attached to the EEP/RRP documents, as
a matter of convenience. The enumeration is NOT a part of the
PacketWay standard, just as RFC0739 (the original "Assigned Numbers"
RFC) is not a part of RFC0791, that defines IP.
Similarly, the EEP-documenmt has "Appendix-A: A Recommendation for
PacketWay Address Assignment" which is a recommendation only and NOT a
part of the PacketWay standard, just as IP-address-assignment is not a
part of RFC0791, that defines IP.
Cohen et al. Experimental [Page 3]
Internet-Draft PacketWay RRP May 1998
Notations
---------
8B means "8-byte" (64 bits).
0x indicates hexadecimal values, e.g., 0x0100 is 2^8=256(decimal).
0b indicates binary values, e.g., 0b0100 is 4(decimal).
xxx indicate a field that is discarded without any checking (e.g., padding).
[fff] indicates that fff is an optional field, when appropriate.
[exp], in equations, is the integral part of `exp`, e.g., [17/8]=2.
All length fields do not include themselves, and therefore may be zero.
Fields lengths are specified either (a) by byte count, with following
padding bytes to fill 8B-words, or (b) by 8B-word count, and PL, the
number of trailing padding bytes (with 0<=PL<8).
Cohen et al. Experimental [Page 4]
Internet-Draft PacketWay RRP May 1998
PacketWay Enumerations
----------------------
(1) PacketWay Packet Types
+-------------------------
The EEP header reserves 4 bytes for signaling from the source node
directly to the destination node. They are the PACKET TYPE (PT),
and the TYPE EXTENSION (TE), 2 bytes each.
This list defines values for the PACKET-TYPE (PT) 2B-field. Each
packet-type has its own interpretation of the TE and the h-fields.
2B-Code Packet Type
+---------- ----------------------
0 Illegal
1 RRP
2 Embedded PacketWay Packet
3 MEM-READ
4 MEM-WRITE
Higher level protocols:
21 IP
22 SNMP
23 ATM
Link layer Protocols
50 Ethernet (E10)
51 Ethernet (E100)
52 Ethernet (E1000)
53 Myrinet
54 Fibre Channel
55 RACEway
56 SCI
57 VME
Application level protocols:
81 MPI
82 PVM
Secure Protocols
121 Secure (1)
122 Secure (2)
123 Secure (3)
1,024-2,047 User defined
65,535 ERR (for Error)
More values will be assigned. "Ether-types" should be added with
a pointer to those used by the Internet.
Cohen et al. Experimental [Page 5]
Internet-Draft PacketWay RRP May 1998
Explanation of the MEM-WRITE and MEM-READ (that a memory board
may need):
PT Meaning
--------- --------------------------------------------------------
MEM-WRITE -- Treat the first 8 bytes of the Data Block as a local
memory-address and write the remaining data into memory.
MEM-READ -- Treat the data block as 2 8B-memory-addresses and an
8B-byte count. Generate a return WRITE packet containing
the first address, followed by the appropriate data,
that was read from the second address.
(2) RRP Messages (Type Extensions of PT="RRP)
+---------------------------------------------
RRP-
Type Code Description
+------ ---- ----------------------------------------------------
0 Illegal
GVL2 21 Please give me L2-routes from you to node (address)
L2SR 22 Here are L2-routes to node (address)
RDRC 23 Re-direct to node (address) via a neighbor HR (address)
TELL 24 Please tell about node (address | name | capabilities)
INFO 25 Info about node (address, name, capabilities)
HRTO 26 Which HR should I use for node (address)
WRU? 27 Who/what-Are-You?
GVRT 28 Please give me your RTs
RTBL 29 Here is an RT
Throughout this document the RRP messages are indicated by their
type (e.g., RDRC for re-direct). In actual messages the code is used
(e.g., 2 for RDRC).
(3) RRP records
+--------------
RTyp Code Description
+------ ---- ----------------------------------------------------
0 Illegal
ADDR 41 Address record for one or many nodes
NAME 42 Node Name record
CAPA 43 Node Capability record
LADR 44 Node Logical Addresses record
SRQR 45 Source Route record and its Quality (SR, Q)
MTUR 46 MTU record (for the previous SRQR)
Throughout this document the RRP records are indicated by their RTyp (e.g.,
ADDR for address). In actual messages the code is used (e.g.,41 for ADDR).
Cohen et al. Experimental [Page 6]
Internet-Draft PacketWay RRP May 1998
(4) Error Messages
+-----------------
Subtype Code Description
--------- ---- ----------------------------------------------
0 Illegal
UNK 71 Unknown (address)
HRDOWN 72 HR-Down (and the links associated with it)
LINKDOWN 73 Link-Down (between two HRs)
GENERAL 74 General error message
Throughout this document the error messages are indicated by their
subtype (e.g., LINKDOWN for Link-Down). In actual messages the code
is used (e.g., 3 for LINKDOW).
(5) PacketWay Node Capabilities
+------------------------------
Code Capability Parameters
+--- ------------------------ --------------------------------------
0 Illegal
1 GP Computing Node
2 Router SAN-IDs, 1+3 Bytes each
3 PacketWay Server
4 Network Multicast Server
5 NFS
6 NPS (Paging Server)
7 Floating-point DSP IEEE word-sizes (in bytes), 1B per size
8 Fixed-point DSP word-sizes (in bytes), 1B per size
9 Printer
250 SRVLOC server
253 Secure PacketWay HR
254 Multicast agent for its SAN
255 SAN
Cohen et al. Experimental [Page 7]
Internet-Draft PacketWay RRP May 1998
(6) Optional Header Fields Types (OH)
+------------------------------------
The MSbit of the type field (T) is the type-of-type. Its assignment is:
0b0: Optional (may drop this OH if its type, tttttt, is unknown)
0b1: Mandatory (should not process this OH if its type is unknown)
The next bit is the "Completion bit" (C). Its assignment is:
0b0: More options follow
0b1: This is the last option field
The 6 LSbits, tttttt, are the type field. Their assignment is:
0x00: Illegal
0x01: TBD
0x02: CRC32 here
0x03: CRC32 following in the OT (after the DB)
0x04: CRC64 here
0x05: CRC64 following in the OT (after the DB)
0x06: There is an OT (Optional Trailer)
0x07-0x3D: TBD
0x3E: Cryptographic data
(7) Byte Order (Endianness)
+--------------------------
A 4 bit field (E) is used to indicate Endianness, with e being its first
bit (MSbit).
e=0: Big-Endian order
e=1: Little-Endian order
The 3 LSbits indicate the size of data chucks (must be the same for the
entire data block) to allow hardware swapping
e000: don't swap, it's 8-bit data
e001: swap as if all the data is 16-bit words
e010: swap as if all the data is 32-bit words
e011: swap as if all the data is 64-bit words
e100: swap as if all the data is 128-bit words
e101: illegal and reserved for future use
e110: illegal and reserved for future use
e111: illegal and reserved for future use
Cohen et al. Experimental [Page 8]
Internet-Draft PacketWay RRP May 1998
(8) Symbol Types (ST)
+--------------------
The format of Symbols is:
+--------+--------+--------+--------+--------+--------+--------+--------+
|vv000000|1111ssss|ssssssss|ssssssss| Length | data |........|........|
+--------+--------+--------+--------+--------+--------+--------+--------+
<---- Symbol Type --->
Code Symbol Type
-------- -----------------
0x00000: Reserved
0x00001: Multicast
0x00002: Secure PacketWay
Cohen et al. Experimental [Page 9]
Internet-Draft PacketWay RRP May 1998
Appendix-A: Glossary
---------------------------
(Last update: Aug-24, 1997)
Address: A unique designation of a node (actually an interface to
that node) or a SAN.
Buddy-HR: HRs are "buddies" if they are on the same SAN.
Cut-Thru: See wormhole.
Destination: The node to which a packet is intended
Dynamic-Routing: Routing according to dynamic information
(i.e., acquired at run time, rather than pre-set).
Endianness: The property of being Big-Endian or Little-Endian
(transmission order, etc.)
Ethertype: A 16-bit value designating the type of Level-3 packets
carried by a Level-2 communication system.
HR: Half-Router, the part of a router that handles one
network only.
L2-Forwarding: Forwarding based on Level-2 (i.e., data-link layer
of the ISORM) information, e.g., the native technique
of each SAN or LAN. Also called "source routing."
L3-Forwarding: Forwarding based on end-to-end Level-3 (i.e., network
layer of the ISORM) addresses. Also called
"destination routing."
Map: The topology of a network.
Mapper: A node on a SAN/LAN that has the map and an RT for that
network. It is expected that the mapper dynamically
updates the map and the RT.
Multi-homed Node: A node with more than one network interface, where each
interface has another address.
Node: Whatever can send and receive packets (e.g., a computer,
an MPP, a software process, etc.)
Node structure: A C-struct (or equivalent) containing values for some
attributes of a node.
Planned Transfer: Transfer of information, occurs after an initial phase
in which the sender decides which Level-2 route to use
for that transfer.
Cohen et al. Experimental [Page 10]
Internet-Draft PacketWay RRP May 1998
RCVF: The "Received From" set includes all the physical
addresses through which an RT was disseminated, starting
with that of the mapper that created that RT.
Re-direct-message: A message that tells nodes which HR should be used in
order to get to a certain remote address (or range of).
Router: The inter-SAN communication device
Security Context: A relationship between 2 (or more) nodes that defines
how the nodes utilize security services to communicate
securely.
Source: The node that created a packet.
Source-Route: A Level-2 route that is chosen for a packet by its source.
Symbol: Data preceeding the EEP header of a PacketWay message,
interleaving with the L2RHs.
Twin-HR: Two HRs are twins if they both are parts of the same
inter-SAN router.
Wormhole-routing: (aka cut-thru routing) forwarding packets out of
switches as soon as possible, without storing that
entire packet in the switch (unlike Stop-and-forward).
Zero-copy TCP: A TCP system that copies data directly between the user
area and the network device, bypassing OS copies.
Cohen et al. Experimental [Page 11]
Internet-Draft PacketWay RRP May 1998
Appendix-B: Acronyms and Abbreviations
--------------------------------------
(Last update: August 24, 1997)
0bNNNN The binary number NNNN (e.g., 0b0100 is 4-decimal)
0xNNNN The hexadecimal number NNNN (e.g., 0x0100 is 256-decimal)
8B 8 byte (64 bits) entity
ADDR The Address-record of RRP
API Application/Program Interface
AT Address Type
ATM Asynchronous Transmission Mode
B Byte (e.g., 4B)
b bit (e.g., 32b)
BC Byte Count (of parameters)
BER Bit Error Rate
CAPA The CAPAbility-record of RRP
CC Capability Code
CSR Common Source-Route
DA Destination Address
DB Data Block
DL Data Length (in 8B words)
DSP Digital Signal Processore
DT Destination-Type
e The MSbit of E
E The Endianness field (in the EEP header)
EEP End/End Protocol
EI Error Indication
GP General Purpose
GVL2 An RRP message, requesting L2 route to a given destination
GVRT An RRP message asking an HR to give its routing tables
h Optional header fields flag
HR Half Router
HRTO An RRP message asking which HR to use for a given destination
ID Identification
IGMP Internet Group Management Protocol
INFO An RRP message providing information about nodes
IP The Internet protocol
ISORM The ISO Reference Model
L Length field (exclusive of itself)
L2 Level-2 of the ISORM (Link)
L2RH Level-2 Routing Header
L2SR Source Route
L3 Level-3 of the ISORM (Network)
LA Logical Address
LADR The Logical-addresses-record of RRP
LAN Local Area Network
LRT Local Routing Table
LSbit Least Significant bit
LSbyte Least Significant byte
MAC Message Authentication Code / Media Access Control
MPI Message Passing Interface
MPP Massively Parallel Processing system
Cohen et al. Experimental [Page 12]
Internet-Draft PacketWay RRP May 1998
MSbit Most Significant bit
MSbyte Most Significant byte
MSU Mississippi State University
MTU Maximum Transmission Unit
MTUR The MTU-record of RRP
M/C Multicast
NAME The name-record of RRP
NFS Network File Server
OH Optional Header field
OH-TYPE The Type of an Optional Header field
OT Optional Trailer field
P The Priority field
PAD Padding After Data
PBD Padding Before Data
PCI The Peripheral Component Interconnect "standard"
PH PacketWay Header
PL Padding Length (always in bytes)
PPP The Point-to-Point Protocol
PROM Programmable ROM (Read-Only-Memory)
PT Packet Type (2B)
PVM Parallel Virtual Machine
PW The Myrinet Packet Type assigned to PacketWay (PW=0x0300)
Q Quality (of a path)
RCVF Received-From list, or the Received-From record of RRP
RDRC A re-direct message of RRP
RH Routing Header
RID Record ID
RL Record Length (in 8B-words)
RRP Router/Router Protocol
RT-hd RT (Routing Table) header
RT Routing Table
RTBL An RRP message proving a Routing Table
RTHD The Routing-Table-Header record of RRP
RTyp RRP's Record Type
RZ The Reserved field (in the EEP header)
SA Source Address
SAN System Area Network
SAN-ID The 24-bit PacketWay-address of a SAN
SAR Segmentation and Reassembly
SN Serial Number
SNID SAN-ID
SNMP Simple Network Management Protocol
SR Source Route (always at Level-2)
SRQR The Source-Route-and-Q-record of RRP
ST Symbol Type
TAIL PacketWay EEP Trailer
TE Type Extension (2B)
TELL An RRP message requesting information about nodes partially specified
UNK Unknown
V Version
WRU? An RRP message asking its recipient to identify itself
XRT External Routing Table
xxx A padding byte
Cohen et al. Experimental [Page 13]
Internet-Draft PacketWay RRP May 1998
Editor's Address
Anthony Skjellum
Computer Science Department
Box 9637
Mississippi State University
Mississippi State, MS 39762-9637
Phone: 601-325-8435
Fax: 601-325-8997
Email: tony@cs.msstate.edu
