IPN Research Group V. Cerf
Internet Draft Worldcom/Jet Propulsion Laboratory
May 2001 S. Burleigh
Expires November 2001 A. Hooke
L. Torgerson
NASA/Jet Propulsion Laboratory
R. Durst
K. Scott
The MITRE Corporation
E. Travis
Global Science and Technology
H. Weiss
SPARTA, Inc.
Interplanetary Internet (IPN): Architectural Definition
draft-irtf-ipnrg-arch-00.txt
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
This document describes the Interplanetary Internet: a communication
system to provide Internet-like services across interplanetary
distances in support of deep space exploration. Our approach, which
we refer to as bundling, builds a store-and-forward overlay network
above the transport layers of underlying networks. Bundling uses
many of the techniques of electronic mail, but is directed toward
interprocess communication, and is designed to operate in
environments that have very long speed-of-light delays. We partition
the Interplanetary Internet into IPN Regions, and discuss the
implications that this has on naming and routing. We discuss the way
that bundling establishes dialogs across intermittently connected
internets, and go on to discuss the types of bundle nodes that exist
in the interplanetary internet, followed by a discussion of security
in the IPN, a discussion of the IPN backbone network, and a
discussion of remote deployed internets.
Cerf, et al. [Page 1]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Table of Contents
Status of this Memo................................................1
Abstract...........................................................1
Table of Contents..................................................2
Copyright Notice...................................................2
Desiderata of Interplanetary Internetworking.......................3
Acknowledgments....................................................3
Foreward...........................................................4
Executive Summary..................................................5
1. Introduction....................................................8
1.1. Preliminary Considerations .............................9
1.2. The IPN Operating Environment .........................11
1.3. A "Postal" Communications Model .......................14
2. IPN Architectural Overview.....................................14
3. Inter-Internet Dialogs.........................................15
3.1. Principles of Design ..................................15
3.2. Information Carried by the Bundle Layer ...............19
3.3. Reliability at the Bundle Layer .......................21
3.4. Bandwidth Allocation via Market Mechanisms:
"Starbucks" ...........................................21
4. IPN Nodes......................................................23
4.1. Types of IPN Nodes ....................................24
4.2. Example end-to-end transfer ...........................24
4.3. Error Conditions at the Bundle Layer ..................32
4.4. Support of existing Internet applications .............35
5. Security in the IPN............................................36
5.1. Assumptions Regarding Required IPN Security
Mechanisms ............................................36
5.2. Secure Email Technology ...............................38
5.3. Application of Secure Email Technology to the IPN .....40
5.4. Protecting IPN Data and the IPN Backbone
Infrastructure ........................................41
6. Building a Stable Backbone for the IPN.........................42
6.1. Backbone Design Considerations ........................43
7. Deployed Internets in the IPN..................................46
7.1. Applications of deployed internets in the IPN .........47
7.2. Characteristics of remote deployed internets
in the IPN ............................................48
7.3. Effects of environmental characteristics on
protocols for the IPN RDIs ............................49
7.4. Summary ...............................................53
8. Working Conclusions............................................53
9. Security Considerations........................................56
10. Authors' Addresses............................................57
Copyright Notice
Copyright (C) The Internet Society (2001). All Rights Reserved.
Cerf, et al. Expires November 2001 [Page 2]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Desiderata of Interplanetary Internetworking
Go thoughtfully in the knowledge that all interplanetary
communication derives from the modulation of radiated energy, and
sometimes a planet will be between the source and the destination.
Therefore rely not on end-to-end connectivity at any time, for the
universe does not work that way.
Neither rely on ample bandwidth, for power is scarce out there and
the bit error rates are high. Know too that signal strength drops
off by the square of the distance, and there is a lot of distance.
Consider the preciousness of interplanetary communication links, and
restrict access to them with all your heart. Protect also the
confidentiality of application data or risk losing your customers.
Remember always that launch mass costs money. Think not, then, that
you may require all the universe to adopt at once the newest
technologies. Be backward compatible.
Never confuse patience with inaction. By waiting for acknowledgement
to one message before sending the next, you squander tracking pass
time that will never come to you again in this life. Send as much as
you can, as early as you can, and meanwhile confidently await
responses for as long as they may take to find their way to you.
Therefore be at peace with physics, and expect not to manage the
network in closed control loops -- neither in the limiting of
congestion nor in the negotiation of connection parameters nor even
in on-demand access to transmission bands. Each node must make its
own operating choices in its own understanding, for all the others
are too far away to ask. Truly the solar system is a large place and
each one of us is on his or her own. Deal with it.
S. Burleigh
Acknowledgments
Robert Braden, of USC ISI, Deborah Estrin, of UCLA, and Craig
Partridge, of BBN all contributed useful thoughts and criticisms to
this document.
This work was performed under DOD Contract DAA-B07-00-CC201, DARPA AO
H912; JPL Task Plan No. 80-5045, DARPA AO H870; and NASA Contract
NAS7-1407.
Cerf, et al. Expires November 2001 [Page 3]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Foreward
This document presents the current state of our team's efforts to
define an end-to-end architecture for the Interplanetary Internet.
This is a "living" document, and is frequently updated. In this
version of the document, we introduce a new construct, a tuple
consisting of a topologically significant routing handle and the
administrative name. In this model, the routing handle identifies an
"IPN Region," an area of the Interplanetary Internet in which A) the
administrative name is resolvable, and in which B) a route can be
formed from anywhere within the region to the address returned when
the administrative name is resolved.
On the presumption that the exploration of space will eventually lead
to the need for communication among planets, satellites, asteroids,
robotic spacecraft and crewed vehicles, our project has a heavy focus
on advocating the development of a stable interplanetary backbone
network. We have concluded that simply extending the Internet suite
to operate end-to-end over interplanetary distances is not feasible.
Rather, we envision a "network of internets" _ ordinary internets
that are interconnected by a system of gateways that cooperate to
form the stable backbone across interplanetary space. Each
internet's protocols are terminated at its local gateway, which then
uses a specialized long-haul transport protocol to communicate with
peer gateways. An end-to-end "bundle" protocol will operate above
the transport layer to carry necessary information from one internet
to another. Bundling will, to the extent possible, remove any
"chattiness" from the local protocols, forming atomic units that will
be shipped across the backbone. Bundles may be protected from
unauthorized access and unauthorized modification. The IPN will have
a global namespace that is broken into a number of name-to-address
binding regions, referred to as IPN regions. Names carried in the
bundles consist of a tuple, one element identifying the destination
IPN region and used for routing and a second element that carries the
administrative name of the destination in the namespace that is
relevant within the destination IPN region. The administrative name
will be bound to an address that is routable within the destination
IPN region. Finally, strong authentication and strict access
controls at several levels will protect the IPN from tampering.
In summary, the best way to envision the fundamental architecture of
the Interplanetary Internet is to picture a network of internets.
Ordinary internets (many being wireless in nature) are placed on the
surface of moons and planets as well as in free-flying spacecraft.
These remotely deployed internets run ordinary Internet protocols. A
system of Interplanetary Gateways connected by deep-space
transmission links form a backbone communication infrastructure that
provides connectivity for each of the deployed internets. New long-
haul protocols, some confined to the backbone network and some
operating end-to-end, allow the deployed internets to communicate
with each other.
Cerf, et al. Expires November 2001 [Page 4]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Executive Summary
This document describes the Interplanetary Internet: a communication
system to provide Internet-like services across interplanetary
distances in support of deep space exploration. The communications
environment is characterized by high bandwidth-delay products
resulting from very long signal propagation delays, intermittent
connectivity that results in long periods of network partitioning,
and discontinuities in the capabilities of adjacent networks. Many
of these characteristics are similar to those facing emerging
communication services in the terrestrial Internet. For example,
terabit networks exhibit very high bandwidth-delay products, mobility
can result in partitioning of both nodes and subnetworks, and
interconnecting different physical layer technologies can result in
"impedance mismatches." It is possible to build an end-to-end
solution to address any of these, but difficult to build one that is
adequate for all of them simultaneously.
The long bandwidth-delay products shared by the IPN and very high-
speed terrestrial networks argue in favor of non-chatty communication
protocols. In the IPN, the long delays mean that protocols that use
many round-trips to accomplish some task pay a significant time
penalty. In terrestrial terabit networks where switches can forward
data very fast but take a (relatively) long time to reconfigure, the
penalty is one of lost efficiency in the use of the resources. Both
environments benefit from protocols that pack as much as possible
into each transmission and minimize the number of round-trips needed.
Thus a file transfer protocol that can place the entire file and
associated control information together in a single atomic
transaction completes faster within the IPN and makes more efficient
use of terrestrial high-speed networks.
Most of the problems cited above have existed and been considered,
albeit separately, during the evolution of what is now the Internet.
Many of the solutions, however, represent branches of the Internet's
evolutionary tree that have withered and died as a result of the
availability of infrastructure to mitigate environmental differences.
This effective homogeneity is diminishing, however, with the rapid
deployment of new technologies with fundamentally different
characteristics, such as wireless communication and Dense Wavelength
Division Multiplexing (DWDM) networks. One solution, electronic
mail, provides a means to span very different networks that are not
necessarily always connected. The electronic mail approach has a
number of attractions. First, there is no expectation of continuous
or instantaneous connectivity. Second, electronic mail embodies the
concept of indirection as a means of providing store-and-forward
traversal of different, sometimes-disconnected networks. Finally,
the electronic mail concept is generally considered to be a non-
interactive communications mechanism, potentially well suited to long
delay environments. The electronic mail approach has limitations,
though. Without and end-to-end retransmission mechanism, electronic
mail does not provide true end-to-end reliability. Additionally,
Cerf, et al. Expires November 2001 [Page 5]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
electronic mail is oriented toward human use rather than interprocess
communication. Further, the protocol that typically provides
electronic mail services in the Internet, SMTP, is highly interactive
in its control traffic, even though the electronic mail concept is
only minimally interactive.
Our approach, which we refer to as bundling, builds a store-and-
forward overlay network above the transport layers of underlying
networks. Thus two nodes that are adjacent in bundle space may be
many hops apart in the context of the underlying network topology.
We see the bundle layer as a second "thin waist of the hourglass"
that allows applications built on top of it to communicate across
discontinuities in connectivity, and to communicate efficiently over
a multitude of underlying transport technologies. This discontinuity
in connectivity may result from fluctuations in link availability or
from artificial discontinuities, such as are imposed by firewalls.
For efficient communication, the bundle layer attempts to minimize
interactivity of its control traffic, and expects applications to do
likewise. The bundle layer also provides a level of indirection
between applications and the specific services of the underlying
network protocols. Bundle applications can specify requested
"handling instructions," such as reliability and quality of service
requests that are mapped into the most appropriate mechanisms
available in the underlying networks.
Bundling uses many of the techniques of electronic mail, but is
directed toward interprocess communication. Bundle nodes use the
capabilities of the underlying networks, including transport layer
retransmission protocols, to effect the transfer of bundles between
bundle nodes. Optional end-to-end reliability at the bundle layer
facilitates end-to-end reliability at the application layer. In
addition, the bundle layer allows any bundle node in the path to take
custody of a bundle. When custody is transferred, the receiving
bundle node assumes responsibility for delivering the bundle
according to its handling instructions, and the previous bundle
custodian is allowed to recover its storage resources. The bundle
protocol is designed to function over simplex and half-duplex links,
and custody transfers may occur between non-adjacent bundle nodes.
Finally, because the bundle layer operates over networks that are
often disconnected, the bundle-layer reliability mechanisms adapt the
operation of timers to accommodate this episodic connectivity.
To illustrate how bundling allows connectivity across intermittently
connected networks, one could envision a "strobe light" that
illuminates the portions of the network's topology that are connected
at a given time. Lit portions of the network are available for
bundle forwarding. If one imagines a high-persistence CRT capturing
an ordered sequence of illuminations that proceed from source to
destination, one can envision the available routes for bundle
forwarding. This environment, therefore, requires a mechanism to
route based on both current and expected connectivity. This routing
mechanism exploits predictability, such as that provided by orbital
Cerf, et al. Expires November 2001 [Page 6]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
mechanics or by scheduled network events (possibly including rate
changes, such as "5c Sundays"). It is important to note that this
routing mechanism may choose to defer communication even though a
current path toward the destination exists, if a substantially more
attractive path is expected to become available.
We believe that the bundle layer functionality has utility within the
context of the terrestrial Internet as well. The Internet is
evolving to encompass very different networking technologies,
architectures, and applications. Some of these, such as firewalls,
result in logical partitioning of the Internet, while others, such as
extreme rate mismatches, may result in inefficient use of network
resources. The bundle layer extends the Internet architecture to
provide consistent end-to-end communication in the current and
emerging partitioned environments, facilitating the deployment of new
applications that can operate reliably while allowing networks to
operate efficiently.
Readers may find more about the project at http://www.ipnsig.org/,
the Interplanetary Internet Special Interest Group of the Internet
Society (http://www.isoc.org).
Cerf, et al. Expires November 2001 [Page 7]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
1. Introduction
The exploration of space began with naked-eye observations of the
stars and moon. In more recent centuries, this exploration was aided
by new Earth-based technologies such as optical and radio telescopes.
Even more recently, we have placed observing resources into near
Earth orbit, such as the Hubble Space Telescope. We have also sent
robotic missions to other parts of the Solar system for a closer
look.
It is the latter activity -- the current exploration of the Solar
System by robotic means and possibly later by missions crewed by
people -- that motivates our interest in an Interplanetary Internet.
The new technology of the terrestrial Internet needs to be extended
into space. We believe that the creation and adoption of Internet-
friendly standards for space communication will enhance our ability
to build a common interplanetary communication infrastructure. We
think this infrastructure will be needed to support he expansion of
human intelligence throughout the Solar System. The current
terrestrial Internet and its technology provide a robust basis to
support these missions in an efficient and scalable manner.
Although many missions during the last 40 years of space exploration
have by necessity provided a substantial portion of their own
communication resources, a significant amount of shared
infrastructure is already in place. For example, the multi-mission
Deep Space Network (DSN) provides a complex of large-diameter (up to
70-meter) tracking and data acquisition dishes at three points on the
Earth's surface, each about 120 degrees from each other. Similarly,
the Tracking and Data Relay Satellite System (TDRSS) and a global
network of small ground tracking stations are used to relay data to
and from many near-Earth missions.
In terms of the communications protocols that support space
exploration, the Consultative Committee for Space Data Systems
(CCSDS) has for almost twenty years been developing internationally
agreed standards for the physical and link layers that interconnect
remote spacecraft with their ground control systems. NASA, through
CCSDS, has also been working since 1993 on general application of
terrestrial Internet or Internet-like protocols for space data use.
Such standardization opens up the possibility of re-purposing and re-
using existing and planned communication facilities for multiple
subsequent missions.
Early in 1998, it became apparent to our team that the space
communications research community and the Internet research and
development community were pursuing technology paths that can
potentially lead to a kind of convergence. A few members of the
Internet community began thinking about adaptation of the terrestrial
Internet to deep space communications. The space communications
Cerf, et al. Expires November 2001 [Page 8]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
research community was already trying out variants of the Internet's
TCP/IP protocol suite to support space-based applications. Mutual
recognition led to the formation of a program of work that was aimed
at extending the notion of Internet to interplanetary scale. The
heretofore-independent "rivers" of evolving space technology and
Internet technology are converging in this program.
To realize this convergence, the effort to define the IPN
architecture is being undertaken at the Jet Propulsion Laboratory
(JPL) that is operated by the California Institute of Technology
(Caltech) for the US National Aeronautics and Space Administration
(NASA). Partial funding for the effort comes from the US Defense
Advanced Research Projects Agency (DARPA) that sponsored the original
Internet design work starting in 1973 and its predecessors, such as
the ARPANET, in 1969. NASA supplies in-kind support and staffing
through its standardization program with CCSDS as well as program
involvement from the Mars exploration enterprise.
An Interplanetary Internet Research Group (IPNRG) has been formed
under the auspices of the Internet Research Task Force of the
Internet Society and an Interplanetary Internet Special Interest
Group has also been created as a means of keeping the public informed
as to progress.
Early protocol design phases are underway now and prototype testing
of candidate designs is anticipated within a year. Demonstration of
these protocols in terrestrial environments will likely occur during
2001 and plans for their use in interplanetary contexts are under
consideration as part of the NASA Mars mission in the years beyond
2003. The Mars exploration program is particularly interesting as a
"reference model" for our work, since it includes a "Mars Network"
project that hopes to deploy a series of remote communications
satellites into orbit around the planet. These satellites will be
dedicated to servicing the local communication and positioning
requirements of the other in-orbit and surface observation and
exploration missions that are in the vicinity of the planet. By 2010,
as many as seven communications and navigation satellites could be in
orbit around Mars, most in low Mars orbit (LMO) but with perhaps at
least one in an "areo" synchronous orbit that is analogous to a
geosynchronous orbit around the Earth.
1.1. Preliminary Considerations
The remarkable success and growth of the Earth-bound TCP/IP protocols
of the Internet illustrate the power of communication
standardization. The simplicity of the Internet architecture, with
its layered structure, contributes to its ability to adapt to almost
any underlying communication capability. As with the terrestrial
Internet, the ultimate test of the IPN technology is whether it
successfully supports commercial applications that have a space
component. We expect that space will eventually be commercialized -
not only for communication services, but also for mining the
Cerf, et al. Expires November 2001 [Page 9]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
asteroids, for the operation of space-based hotels, for manufacturing
and medical treatments, and for general tourism. While such
developments may still lie decades in the future, the potential
investment and benefits can be appreciated as we contemplate the
explosion of new markets associated with the commercialization of the
Internet that began only ten years ago, in 1990. We will therefore
architect the Interplanetary Internet in anticipation of possibly
rapid commercialization.
In terms of technologies, the current Internet capabilities work well
on Earth where the propagation delay of light-speed signals is short.
The packets exchanged according to the TCP protocol reach their
destination in fractions of a second, for the most part. The TCP/IP
protocols (a system of over 150 related communication standards), are
therefore expected to work just as well on the surface of other
planets or moons, on space craft and orbiting space stations, all of
which involve data exchange over fairly short distances, subject to
the availability of sufficient power to maintain good signal-to-noise
ratios. However tempting it is to employ similar concepts in
extending the Internet into deep space, there are problems - deep
space communications really still are "rocket science." The distances
between the planets are, well, astronomical. For example, the round-
trip propagation delays - at the speed of light - between Earth and
Mars range from about 8 minutes to over 40 minutes. This makes
"chatty" protocols like TCP relatively unattractive because of their
heavy dependence on near real-time exchanges between the
communicating parties.
These large distances also impair the data rates that can be
sustained because of radio signal degradation and attenuation.
Moreover, the celestial mechanics of the solar system mean that the
distances between the planets change with time. While these changes
are essentially calculable, they still cause variations in delay, in
transmission capacity and occasionally in connectivity due to
occultation of satellites as they orbit a planet, or of ground-based
facilities as planets rotate.
Size, weight and - most of all - power are supreme challenges for
space-based communication systems, as they are for ground-based
mobile systems. Launching mass into interplanetary trajectories,
injecting mass into orbit, and landing mass into the gravity well of
another planet is currently very expensive. Mass translates directly
into the local availability of power. Efficient use of the
communications channel allows more information to be carried per unit
of transmitted power. But power limitations introduce asymmetries in
the communication capacities available between Earth, for example,
and the remote spacecraft and planets. There can be factors of ten or
more differences between the data rate that can be received on Earth
and the rates of reception of off-Earth resources. It is quite common
to be able to receive transmissions from Mars at 100 kilobits/second
while the Mars-based systems may only receive from Earth at 1
kilobits/second.
Cerf, et al. Expires November 2001 [Page 10]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
All of these effects combine to make the design of an interplanetary
backbone communication system a considerable challenge. The Deep
Space Network - the current interplanetary backbone - uses its three
terrestrial communication complexes to communicate with spacecraft,
orbiting satellites and ground-based resources on moons or other
planets of the Solar System. Because these resources must be shared
among many missions, it is necessary to schedule them to be aimed in
particular directions at particular times. Time synchronization is
needed among the various parts of such a system. For example, a
signal from Mars may take 20 minutes to reach Earth, at which time,
the appropriate antenna of the Deep Space network must be aimed
properly to receive the transmission, 20 minutes after it was sent.
This same antenna might then have to be repositioned to send data to
another spacecraft elsewhere in the Solar System, and the receiving
system must be ready to receive that transmission at the right time.
In some ways, this problem is somewhat like the problem of scheduling
trains on railroad tracks; since multiple trains use the tracks, they
must be scheduled to avoid collisions.
1.2. The IPN Operating Environment
There are a number of fundamental differences between the
environments for terrestrial communications and those we envision for
the IPN. These differences include delay, low and asymmetric
bandwidth, intermittent connectivity, and a relatively high bit error
rate. Taking these into account affects the entire model for
communicating; shifting us from the "telephony" model implicit in
current Internet communications to the "Postal", or "Pony Express,"
model. We will first therefore describe the environmental differences
between terrestrial communications and the IPN and gives a brief
accounting of why the standard Internet protocol for reliable
transport, TCP, is unsuitable for end-to-end communications in the
IPN.
The most obvious difference between communicating between points on
Earth and communicating between planets is the delay. While round-
trip times in the terrestrial Internet range from milliseconds to a
few seconds, round-trip times to Mars range from 8 to 40 minutes,
depending on the planet's position, and round-trip times between
Earth and Europa run between 66 and 100 minutes. In addition to the
propagation delay, communicating over interplanetary distances
currently requires special equipment (large antennas, high-
performance receivers, etc.). For most deep-space missions, even
non-NASA ones, these are currently provided by NASA's Deep Space
Network (DSN). The communication resources of the DSN are currently
oversubscribed and will probably continue to be so in the future.
While studies have been done as to the feasibility of upgrading or
replacing the current DSN, the number of deep space missions will
probably continue to grow faster than the terrestrial infrastructure
needed to support them, making over-subscription a persistent
problem.
Cerf, et al. Expires November 2001 [Page 11]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
This over-subscription means that the round-trip times experienced by
packets will be affected not only by the propagation delay, but also
by the scheduling and queuing delays imposed by the Earth-based
resources. Thus packets to a given destination may have to be queued
until the next scheduled contact period, which may be hours, days, or
even weeks away. While queuing and scheduling delays are generally
known well in advance except when missions need emergency service
(such as during landings and maneuvers), the long and highly variable
delays make the design of timers, and retransmission timers in
particular, quite difficult. This again forms a point of departure
from the current Internet model, as IPN-aware applications will
probably need ways to track the status of a communication and to
apprise users of the expected delay before a response can be
expected. This will be complicated once the IPN moves from its
initial Earth-centric approach to a peer-to-peer network, since
notifying users of the progress of their communications will itself
consume precious bandwidth within the network.
The combined effects of large distances, the expense and difficulty
of deploying large antennas to distant planets, and the difficulty in
generating power in space all mean that the available bandwidth for
communications in the IPN will likely be modest compared to
terrestrial systems. Data rates on the order of hundreds of kilobits
per second to a few megabits per second will probably be the norm for
the next few decades. Another characteristic prevalent in today's
deep-space missions is bandwidth asymmetry, where data is transmitted
at different rates in different directions. Current missions are
usually designed with a much higher data return rate (from space to
Earth) than command rate. The reason for the asymmetry is simple:
nobody ever wanted a high-rate command channel, and, all else being
equal, it was deemed better to have a more reliable command channel
than a faster one. This design choice has led to data rate
asymmetries in excess of 100:1, sometimes approaching 1000:1. A
strong desire for a very robust command channel will probably remain,
so that any transport protocol designed for use in the IPN will need
to function with a relatively low bandwidth outbound channel to
spacecraft / landers.
The difficulties of generating power on and around other planets will
also result in relatively high bit error rates. Current deep-space
missions operate with very high bit error rates (on the order of 10e-
1, or one error in ten bits) that are then improved using heavy
coding. The tradeoffs between coding, bit error rate, and
reliability requirements will need to be reexamined in the context of
the IPN.
Finally, interplanetary communications will, at least in the near
future, be characterized by intermittent connectivity between nodes.
As satellites or moons pass behind planets, and as planets pass
behind the sun as seen from Earth, we lose the ability to communicate
with them. This effect adds to the delays experienced by packets,
Cerf, et al. Expires November 2001 [Page 12]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
and could push queuing delays to several weeks or a month if the
source and destination are in opposition (on opposite sides of the
sun). Inter-layer signaling, especially from the link layer to
provide notifications of such breaks in connectivity, will probably
be required.
We see the IPN growing outwards from Earth as we explore more and
more planets, moons, asteroids, and possibly other stars. Thus there
will always be a fringe to the fabric of the IPN, an area without a
rich communications infrastructure. As a result, the data rate,
connectivity, and error characteristics mentioned above will probably
always be an issue somewhere in the IPN. For the more highly
developed core areas of the IPN, it is interesting to note that delay
is the only truly immutable characteristic that differentiates the
IPN from terrestrial communications. Data rates, intermittent
connectivity, and bit error rate can all be mitigated or eliminated
by adding additional infrastructure, in theory if not in practice.
Additional infrastructure can also mitigate the scheduling and
queuing delays mentioned above, but the propagation delays will
remain unless and until we find a way to transmit information faster
than the speed of light.
These environmental characteristics: long delays, low and asymmetric
bandwidth, intermittent connectivity, and relatively high error rate
make using unmodified TCP/IP for end to end communications in the IPN
infeasible. Using the equations from Mathis, et al [ref:
http://www.psc.edu/networking/papers/model_ccr97.ps], we can
calculate an upper bound on the sustainable throughput of a TCP
connection, taking into account TCP's congestion avoidance
mechanisms. Even if only 1 in 100 million packets are lost, a TCP
connection to Mars is limited to just under 250kbps. If we assume
that 1 in 5000 packets is lost (this figure was reported by Paxson as
the packet corruption rate in the Internet ref:
ftp://ftp.ee.lbl.gov/papers/vp-thesis/dis.ps.gz caution: very large
file) then that number falls to around 1,600bps. These values are
upper bounds on steady-state throughput; since the number of packets
in a connection will generally be under 10,000, TCP performance would
be dominated by its behavior during slow-start. Even when Mars is at
its closest approach to Earth, this means that it would take a TCP
nearly 100 minutes to ramp up to a transmission rate of 20kbps. Lab
experiments using a channel emulator and standard applications show
that even if TCP could be pushed to work efficiently at such
distances, many applications either rely on several rounds of
handshaking or have built-in timers that render them non-functional
when the round-trip-time is pushed over a couple of minutes. It
typically takes eight round trips for FTP to get to a state where
data can begin flowing, for example, and an FTP server may time out
and reset the connection after 5 minutes of inactivity. This means
that a conformant standard FTP server could be unusable for
communicating even with the closest planets.
Cerf, et al. Expires November 2001 [Page 13]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
1.3. A "Postal" Communications Model
We have concluded that the standard Internet protocols should be
essentially "terminated" at the Interplanetary Gateways and the
information payloads conveyed through a new set of protocols better
suited to the long distances, variable delays and asymmetric data
rates of the interplanetary backbone network. In essence, the design
is analogous to a kind of postal relay system in which messages are
delivered to the intermediate Interplanetary Gateways, extracted from
their standard Internet protocols, and encapsulated in new link and
transport protocols to be forwarded to the next IPN gateway and
ultimately into the target internet.
Internet electronic mail already works in this fashion but the
transfer of files, the operation of the World Wide Web, and remote
interactive applications do not fit into this model directly. There
are circumstances under which researchers on planet Earth do need an
ability to interact with remote devices, for example to steer wheeled
robotic vehicles around on the surface. But because of the enormous
round-trip delays, such systems must work very indirectly. For
example, to steer the Mars Pathfinder rover, one sends instructions
about intermediate points that the robot must steer past. This is
analogous to automatic airplane pilots that are given a series of
coordinates through which to pass. In effect, a planned itinerary is
sent and the robot vehicle executes the plan, dealing with local
conditions as required.
The "store-and-forward" nature of this communication method is
reminiscent of bucket brigades, except that the contents of the
buckets are actually the payloads (i.e. data) of the applications
that utilize the network. The concept of "custody" is important in
such a system. A sender does not relinquish a copy of a transmission
until it is sure that the next in line has successfully received it.
2. IPN Architectural Overview
We now consider five broad areas that represent areas of significant
research in the Interplanetary Internet architecture:
* The communication conducted between independent internets
(termed "Inter-internet dialogs")
* The architecture and functions of the unique nodes of the
Interplanetary Internet
* A security architecture for meeting anticipated data and
infrastructure protection needs
* The issues in developing a stable backbone network for the
Interplanetary Internet
* The issues in deploying internets on remote planets, asteroids,
and spacecraft
The following five sections of this document consider each of these
areas. Following these sections we present our conclusions to-date.
Cerf, et al. Expires November 2001 [Page 14]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
3. Inter-Internet Dialogs
This section first presents four principles that guide the design of
the Interplanetary Internet. In doing so, we introduce the concept
of the "bundle" layer, a protocol layer providing end-to-end service
in the IPN. The section continues by discussing the information
carried by the bundle layer, and concludes by discussing reliability
at the bundle layer.
3.1. Principles of Design
3.1.1. Name Tuples Consisting of Administrative and Routing Parts are
the Means of Reference
In the (terrestrial) Internet, names are administrative in nature,
and are hierarchically organized. The Domain Name System (DNS) uses
a highly distributed database to translate the name to a numeric
address, and addresses are the common medium used throughout the
network for reference. This scheme has worked well in the Internet,
but the emergence of network address translators and other
partitioning mechanisms have begun to cause some problems with this
scheme.
One of the problems involved with using DNS names across
interplanetary space is the distributed nature of the DNS database.
This means that it is entirely possible for the portion of the
database that can resolve a name to its address to be far (very far,
in the Interplanetary Internet) from the host requesting resolution.
This means that the times required to resolve names to addresses can
become impossibly high, especially when the issues of intermittent
connectivity come into play. The DNS has a mechanism to replicate
portions of its database, using a technique known as zone transfers.
However, this is not a good solution in the Interplanetary Internet,
either. One could easily spend all of the available communication
time transferring the ".com" zone to another planet, rather than
actually transferring data. Clearly, another approach is indicated.
In our initial designs, we considered creating a new top-level
domain, e.g. ".sol", and assigning to it topological significance.
We constructed a scheme by which we routed on the "most significant"
portions of the domain name, such as ".mars.sol", and essentially bet
that these portions would be topologically significant, rather than
administratively significant. This scheme, while attractive from the
standpoint of making use of existing infrastructure, makes use of
that infrastructure in a bad way. We were forced to "grandfather"
existing top-level domain names to be bound to Earth's Internet, so
that ".com" meant ".com ON EARTH." This spawned philosophical
debates within the group regarding sovereignty and the current top-
level domain structure within the internet, but also had the
potential of creating serious technical problems. For example, some
Cerf, et al. Expires November 2001 [Page 15]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
organizations encode geographic information at fairly low levels of
their DNS names (consider "zurich.ibm.com", and then consider
"mars.ibm.com"). If all ".com's" are shipped off to Earth, the
"mars.ibm.com" data is going to take a serious detour. We eventually
concluded that this was not the right approach.
We have concluded that names in the Interplanetary Internet should
consist of a tuple that contains the administrative part plus a
routing part. These names must be carried end-to-end throughout the
Interplanetary Internet. The routing part serves the purpose of the
new top-level domain described above, except that it is not required
to conform to the naming conventions of the Domain Name System (i.e.,
it may be numeric rather than textual), it may be hierarchically
organized, and it must be routed. This requires us to develop the
means for computing and distributing routing information, but
relieves us from a dependence on the relationship between
administrative names and network topology.
3.1.2. The Routing Part of an IPN Tuple Identifies an Internet
The routing portion of the name identifies an IPN Region. We
envision the IPN as a "network of Internets", and the IPN Region, to
some extent, allows us to route to a particular Internet. The use of
an IPN Region is an explicit form of aggregation that is not
otherwise possible using administrative names.
It is necessary and sufficient that the administrative portion of an
IPN name be resolvable to a useful address within its IPN Region. We
are currently treating the structure of the routing part of the name
in a manner similar to the structure of DNS names: the name space of
the routing part is a tree of text labels separated by "dots," with
the root node of the space having a null label. We denote a name in
the IPN in the following manner: {administrative part, routing
part}. So, if "earth.sol" were an IPN Region encompassing the entire
Earth, the web site for the Internet Society's IPN Special Interest
Group would be { www.ipnsig.org, earth.sol}.
In order that any node in the IPN be able to send data to any other
node, the routing part of the name must be interpretable everywhere.
That is, any IPN node, when confronted with any valid IPN Region
name, must be able to identify a transport layer destination that
moves the data toward that region.
3.1.3. The "Bundle Layer" Terminates Local Transport Protocols and
Operates End-to-End
In the IPN, we cannot ever assume that there is direct connectivity
between source and destination. That is, we cannot assume that bits
emitted by a source can travel, delayed only by routing and
transmission delays, to the destination. There may be any number of
reasons for this, ranging from physical (the destination is on the
far side of a distant planet and can't communicate with anything
Cerf, et al. Expires November 2001 [Page 16]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
right now), to schedule-related (a required IPN gateway is currently
serving other customers), to administrative (the source is only on
during the day, the destination only during the night). For the
long-haul links of the backbone, information will almost certainly
have to be stored for some amount of time as the antennas used for
the long-haul links will almost surely be highly directional.
Thus depending on the schedules of the nodes involved and the
possibility of high-priority interrupt traffic, the nodes that make
up the IPN may have to buffer data for hours, days, or weeks before
it can be forwarded. Also, the highly varying communications
environments that will make up the IPN, ranging from optical fiber on
Earth to wireless communications around Mars, to the long-haul links
of the backbone, suggest that different transport protocols will be
needed for the different environments. It makes sense, therefore, for
the IPN nodes to terminate the transport-layer protocols used in the
respective IPN regions, holding data at a higher layer before
forwarding it on, possibly using a different transport-layer
protocol.
We call this higher layer the "bundle layer," and the protocol used
to send data between the various nodes of the IPN the "bundle
protocol." The term "bundle" is used to connote the store-and-
forward aspect of communications where as much interactivity as
possible has been distilled out of the communication. A bundle file
transfer request, for example, might contain the user's
authentication (login/password, e.g.), the location of the file to
get, and where that file should be delivered in the requester's IPN
domain. All of this information would be transmitted as one atomic
"bundle," and the requested file would be returned. We use the term
"bundle" rather than "transaction" to avoid notions of two-phase and
three-phase commitment that are commonly associated with transaction
processing.
In traditional networking terminology it is generally the transport-
layer protocol that operates end-to-end. Since IPN nodes terminate
transport layer protocols in order to buffer data and to enable them
to use a transport protocol appropriate to the IPN region through
which data will be sent, it is the bundle layer in the IPN that
operates end-to-end.
It should be noted that terminating the transport protocols at the
IPN nodes decouples the internets in different IPN regions to a
significant degree. This has the desirable effect of also decoupling
the evolutionary rates of those internets: changes in the Earth's
Internet do not necessarily dictate changes in other internets. This
is important in an environment in which resources are and will
continue to be severely constrained.
Figure 1 illustrates the progression of a bundle of data through the
Interplanetary Internet, from its source at host "A" to the
destination at host "E." Custody transfers are indicated by
Cerf, et al. Expires November 2001 [Page 17]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
asterisks (*), and occur at B, C, D, and E. Host "A" initiates the
bundle transfer, and the bundle is transferred using internet
protocols (i.e., TCP and IP) to bundle node "B", which serves as the
gateway between the left-hand internet and the interplanetary
backbone. The box icon indicates that custody of the bundle is
transferred to that gateway. When conditions permit, the bundle is
forwarded on to the next hop in the store-and-forward chain. In this
case the next hop is another host within the interplanetary backbone
network: host "C".
Also illustrated in Figure 1 is the notion of a "return receipt" sent
by the ultimate destination of the data back to the source. This is
an optional service, much like a return receipt within the postal
system. If the source desires notification of delivery, that is
accomplished by a separate return receipt, which is transmitted as
its own bundle, and is subject to the same custody transfers as the
original transmission (similar to the fact that a postal return
receipt is, in itself, a postcard). It is shown figuratively as
bypassing the forwarding path (E to D to C to B to A), but this is
simply for clarity. The return receipt is forwarded in exactly the
same manner as the original data is.
An Internet The IPN Backbone An Internet
+----------------+ +------------------------+ +----------------+
| | | | | |
| +---+ +---+ +---+ +---+ +---+ |
| / /| / /| / /| / /| / /| |
|+---+ | +---+ | +---+ | +---+ | +---+ | |
|| |=+=====>| |=+=====>| |=+======>| |=+=====>| | + |
|| A |/ | B*|/ | C*|/ | D*|/ | E*|/ |
|+---+ +---+| +---+ +---+| +---+ |
| /\ | | | | || |
| || | | | | || |
| || | | | | || |
+--||-----------+ +-----------------------+ +---------||-----+
|| ||
+=====================================================+
Return Receipt
Figure 1. Custody Transfers
Many aspects of this mode of data transfer resemble the postal
system, or even the Pony Express. The source depends on the store-
and-forward nodes in the chain to operate on its behalf to deliver
data that may not be retained at the source. Source notification
that the destination has received the data is optional, and there is
no guarantee or even any firm expectation on the part of the source
about when the data will be delivered.
Conceptually, the bundle protocol resides above the transport
Cerf, et al. Expires November 2001 [Page 18]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
layer(s), and operates end-to-end between the ultimate source and the
ultimate destination of the data. The bundle layer provides a number
of services to applications using it. For example the bundle layer
carries the source and destination name tuples end-to-end to support
the late binding of the destination name's administrative part to an
address. The bundle layer also carries the users' specifications for
reliability, quality of service, and security. Because the
communication resources are precious, it is desirable to provide
error recovery mechanisms that do not necessitate discarding of
bundles that contain errors. We are considering adding to the
information carried in a bundle some information to assist in
transfer-related error handling. We are thinking in terms of active
networking, using a combination of active packets and active nodes as
a means of specifying appropriate actions to take in the event of
problems in completing the transfer. Additionally, the bundle layer
provides invoking applications with a transfer identifier that is
carried with the bundle, and is used in "return receipts." [Ref:
http://www.comsoc.org/pubs/surveys/1q99issue/psounis.html]
3.2. Information Carried by the Bundle Layer
To effect the end-to-end transfers necessary in the IPN, the bundle
layer must carry some information end-to-end. This section documents
our current thinking on the information that must be carried end-to-
end, and notes which of those data elements may be supplied by the
application using the bundle service.
* Bundle Identifier: this is a monotonically increasing number
that is carried in the bundle, and also returned to the
application to support return receipt processing. It is not
necessarily a sequence number, and as a result, there is no
requirement that the value of Bundle Identifiers increase
consecutively. The requirement for monotonicity derives from a
need to provide robustness against system crashes, and therefore
must be persistent across system crashes.
* Remote entity name: this is the IPN name of the remote bundle
agent, and is a tuple, as described above. It is supplied by
the local application using the bundle service.
* Source entity name: this is the IPN name of the local bundle
agent, and is a tuple. It is supplied by the local bundle
service, since a particular host may have multiple names and one
may be chosen based on routing decisions or other criteria
opaque to the application (as in multihomed hosts). The source
name may be returned to the application to support return
receipt processing.
* Authentication information: this is information, such as a
digital signature, that is passed by the application to the
bundle layer to unambiguously identify the source of the bundle.
(Just what the source of the bundle is, person, place, address,
etc. is still undecided.) This information is checked for
validity at both the source bundle agent and the destination
bundle agent. The authentication information may also be used
Cerf, et al. Expires November 2001 [Page 19]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
for access control purposes within the network.
* Source application instance handle: this is similar in nature to
a source port number in that it identifies the sending
application. Since bundles are inherently non-interactive, the
typical use for this handle is to "reanimate" the source
application when a return receipt arrives. This could be hours,
days, or weeks after the initial transmission, so the handle may
well be a reference to a structure that allows the application
to be reinstantiated with known state. The source application
instance handle may be used at the destination as an identifier,
but may also be redundant with the end-to-end authentication
information for this purpose. This handle is supplied by the
source application.
* Destination application instance handle: this is essentially the
destination port identifier. As with ports, these must be
known to and supplied by the source application. We have not
yet fully explored the implications of port advertisement across
the IPN.
* Size of data: this is a statement of the size of the bundle, in
bytes. It is supplied by the source application, and is used
initially to ensure that sufficient space is available to store
the bundle for its initial transmission. Nodes receiving
transmitted bundles use this information in the same manner: as
a means of making a determination about the availability of
storage early in the bundle handling process. In forming the
initial bundle, the bundle layer at the source may use the size
of data parameter as a consistency check on the amount of data
actually delivered.
* Handling instructions: these are parameters supplied by the user
with the bundle that convey the user's preferences to the
network. Our thoughts on just exactly what these parameters
look like are not yet firm, however, our thoughts are that they
include some or all of the following: priority, quality of
service (although we are still debating what this means in the
context of the IPN), elapsed time after which the content of
this bundle is meaningless (time to live as specified by the
user), reliability requirements, and any error handling
information. For the most part, these are requests, and we
perceive that the bundle layer may either override some or all
of these requests or fail the request if local policy does not
permit that particular user to make that particular request at
that particular time.
* Data Descriptor: This is a reservation token that is generated
by the bundle layer. Its detailed definition and use are still
to be determined.
* Time to Live: This is the time after which this bundle is to be
discarded from the network. It is present to facilitate the
recovery of network resources and to terminate routing loops,
should they occur.
* Loose/Strict Source Route and Record: This information is
provided by the source application to facilitate debugging of
the network. It consists of a list of names of IPN nodes
Cerf, et al. Expires November 2001 [Page 20]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
through which the bundle must pass on its way to the
destination, and our intent is that it should behave similarly
to the corresponding option within IP.
* Current bundle custodian: the bundle protocol supports store-
and-forward operation in which the custody of a bundle (that is,
the responsibility for ensuring reliable delivery) may transfer
from one IPN node to another as the bundle progresses through
the IPN. There is not a requirement for each IPN node
encountered to assume custody of a bundle. As a result, it is
necessary to identify the upstream node that has custody of the
bundle, in order to either request retransmissions or to accept
custody of the bundle.
* User data: this is intended to be all of the data that the
remote entity requires to perform whatever operation is
requested. Since the environmental characteristics of the IPN
make interactivity difficult, the notion is that all of the
information that is required to perform a particular
"transaction" would be provided in a single bundle.
3.3. Reliability at the Bundle Layer
Because no single transport-layer protocol operates end-to-end across
the IPN, end-to-end reliability can only be assured at the bundle
layer. At each node along an end-to-end route, the bundle-layer
protocol entity passes bundle data to the Transport layer for
transmission. Each bundle layer entity is highly confident that the
transport layer will successfully convey the data entrusted to it to
the next bundle-layer protocol entity (which may "take custody" of
the data or merely relay it; a single hop). But failures are
possible (e.g., a host computer does an unplanned reboot). Just in
case the highly unlikely happens and a Transport-layer transmission
fails, the first subsequent node that detects the failure and is
capable of taking custody of the bundle will request that the prior
custodian re-transmit any missing data (again using Transport-level
transmission services across, potentially, one or more relay nodes).
The bundle layer's confidence in the effectiveness of the underlying
Transport-layer protocols is reflected in the design of the timers
for bundle-layer reliability. These timers are highly optimistic _
that is, they expire as late as possible _ in order to give the
Transport protocols every opportunity to complete reliable
transmission. The effect of this optimism is to minimize the chance
of unnecessary bundle-layer retransmission, which could seriously
degrade IPN performance by consuming valuable bandwidth.
3.4. Bandwidth Allocation via Market Mechanisms: "Starbucks"
To promote effective and efficient use of the IPN's scarce
transmission resources, some sort of sophisticated and adaptable
bandwidth allocation system will probably be needed. The scheme
described below is based on a free-market notion of "fare-paying
packets", where at initial transmission each bundle is issued a
Cerf, et al. Expires November 2001 [Page 21]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
"draft" on some small percentage of IPN resources. These resources
might well map back to actual monetary funds provided by the
originator of the bundle to the provider(s) of the IPN service. The
bundle in effect pays its own way across the various legs of end-to-
end transmission. As it traverses each hop, the bundle spends funds
from its original draft until either it is received or else its funds
are exhausted. If a bundle is dropped due to insufficient funds then
the hope is that all available transmission resources were allocated
to bundles that were allotted more funds and therefore were
presumably more important (to somebody).
At the initial Send-bundle service request, the source application
(bundle sender) would specify total funds allocated to getting the
bundle delivered to the destination. Total funds allocation needs to
be a function of (a) the total number of bytes of data and metadata
to be sent, and (b) the prices charged for each "transmission class,"
(something like First Class, Business Class, Economy Class, Steerage,
Overhead Bin; etc.). The allocation should cover the anticipated
costs of traversing all the bundle hops along the anticipated route
to destination. If the bundle must be split up into multiple packets
(bindles, segments), the bundle agent that performs the split also
distributes the bundle's total funds among individual packets. This
distribution will probably be on a prorated basis.
Also, the source application would supply the bundle (and, by
implication, each of its packets) with traveling instructions:
For each time epoch that elapses while awaiting transmission from any
single bundle transmission agent:
* The length of the epoch in units of time (seconds?)
* The queue (transmission class) to wait in over the course of the
epoch
For example: get in the Steerage queue, but if 30 minutes pass and
you still haven't been transmitted then pay for an upgrade to
Business Class; if you still haven't been transmitted after 20
minutes in Business Class, then give up.
At each bundle transmission agent, each packet is handled as follows:
* For each of the packet's authorized time epochs until the packet
is transmitted (that is, initially handed to the underlying
communication layer; retention until successful custody
acquisition by downstream agent is provided at no charge):
- If (epoch duration * packet length * agent's price per unit of
time per byte for this epoch's transmission class, i.e. queue)
is greater than packet's residual funds, then discard the
packet. The packet's residual funds remain unexpended.
- Else, append the packet to the requested queue. Then:
Cerf, et al. Expires November 2001 [Page 22]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
- If epoch expires before packet is de-queued for transmission,
then charge the source application an amount equal to (epoch
duration * packet length * price per unit of time per byte for
queue), reduce packet's residual funds by this amount, and
start the next epoch; if no more authorized epochs, give up.
- Else, upon de-queuing the packet for transmission, charge the
source application an amount equal to (length of time spent in
queue * packet length * price per unit of time per byte for
queue) and reduce packet's residual funds by this amount.
* Transmission classes _ queues _are of varying maximum length
(the more expensive queues are shorter) but packets in all
transmission classes are at the same level of priority; packets
are de-queued in round-robin fashion to ensure that no class is
altogether starved for service. Because First Class is a
shorter queue than Business Class, packets that pay for First
Class spend less time waiting.
* Separately, an Emergency queue is provided for system-critical
packets. Everything in the Emergency queue has higher priority
than everything else, so nothing else gets transmitted until the
Emergency queue is emptied.
Periodically, each bundle protocol agent reports aggregate charge
amounts back to the source applications and also to some central
accounting authority ["the bank", nominally based on Earth]; this is
a separate application-layer protocol. When the central authority
determines that a source application is out of funds, it reports the
source application's bankruptcy to all bundle agents; from that time
on, all service requests and packets received from that source
application are rejected.
Although this particular scheme may ultimately prove too complex to
be workable, we think the general principle of fine-grained bandwidth
allocation could contribute significantly to the viability of the
IPN.
4. IPN Nodes
Nodes within the IPN have a number of responsibilities. As members
in a store-and-forward chain, they have the responsibility for
resource allocation to support bundle transfers. These resources
include, among other things, buffer space and transmission capacity.
Additionally, IPN nodes have the responsibility of actually executing
the bundle transfer. Reliability requirements for bundle transfers
are specified by the using application, and include both reliable and
unreliable transfers (possibly with some intermediate, or partial,
Cerf, et al. Expires November 2001 [Page 23]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
reliability services). The IPN nodes are responsible for using
whatever reliability mechanisms exist in the underlying (transport-
and-below) layers, and augmenting those mechanisms as necessary to
effect the required reliability.
Finally, IPN nodes are responsible for routing bundles between IPN
domains. IPN nodes may depend upon the services of the local
internets (or the IPN backbone) for intra-domain routing.
In this section, we first briefly state the types of IPN nodes that
we have identified, and then we provide a number of exemplary end-to-
end data transfer descriptions. Finally, we list the error
conditions that we have identified that may occur at the bundle layer
during the course of end-to-end data transfer.
4.1. Types of IPN Nodes
We identify three grades of IPN functional capability. In order of
increasing scope, they are: agent capability, relay capability, and
gateway capability. All IPN nodes are able to act as bundle agents;
some bundle agents are additionally able to act as IPN relays; some
IPN relays are additionally able to act as IPN gateways.
* Bundle agents build and consume bundles. These could be stand-
alone proxy devices or could be an operating system service
collocated with an application. The endpoints of an end-to-end
bundle transmission need not be any more than bundle agents,
though they may additionally have relay or even gateway
capability.
* IPN Relays receive bundles and forward them toward their
destinations, either within or between IPN regions.
* IPN Gateways reside at the interface between IPN regions. The IPN
gateways perform routing between the IPN regions.
Orthogonal to these grades of capability is the ability to take
custody of a bundle. The endpoints of a bundle transaction are
typically the initial and final custodians of the bundle. Non-
gateway relays may take custody of the bundles they receive;
alternatively, they may simply provide mitigation of R2 effects
(signal strength losses due to the extreme distances of
interplanetary communication), but provide no custody transfer
capabilities. (In the latter case they are essentially "repeaters.")
Gateways normally take custody but are not required to do so.
4.2. Example end-to-end transfer
We provide the following example of an end-to-end transfer that
crosses multiple IPN Regions: A host on Earth sends a bundle to a
destination on Mars. Figure 2 illustrates the network that is the
subject of this example _ the Interplanetary Internet in this example
Cerf, et al. Expires November 2001 [Page 24]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
has five distinct regions interconnected by four IPN Gateways. This
example highlights the actions taken by the bundle layer and the
interactions of the bundle layer with applications and with
underlying transport protocols.
4.2.1. Backbone connectivity
It is important to have some understanding of the routing that is
required at the IPN Gateways. Unlike terrestrial communications, the
IPN's long-haul communication links are directional, mobile, and
highly scheduled. This is important, because directionality combined
with mobility means that a transmitter and receiver must track each
+-------------------------+
| Earth's Internet |
| IPN Region: earth.sol |
| +---+ |
+-----------/ /|--------+
+---+ |
|G/W| +
+----------| 1 |/---------+
| +---+ |
| The "Backbone" |
| IPN Region: ipn.sol |
+---+ +---+ +---+
/ /|------/ /|-------/ /|
+---+ | +---+ | +---+ |
|G/W| + |G/W| + |G/W| +
+----------------| 3 |/ +---| 4 |/-----+ | 2 |/-------------------+
| +---+ | +---+ | +---+ |
| Venus's Internet | | Jupiter's | | Mars's Internet |
| IPN Region | | Internet | | IPN region: |
| venus.sol | | IPN Region | | mars.sol |
+------------------+ | jupiter.sol | +----------------------+
+--------------+
Figure 2. An Interplanetary Internet of Five IPN Regions
other in order to establish and maintain a communication link. In
the IPN, much of the mobility is due to orbital mechanics and is
therefore relatively predictable. However, this means that nodes
that we would normally consider to be fixed, such as antennas on the
surface of the Earth, are actually highly mobile as a result of the
Earth's rotation and its revolution around the Sun. (In this
example, we confine ourselves to our local solar system, and do not
consider the motion of our sun relative to celestial bodies outside
our solar system.) We can describe the predictable aspects of node
mobility with an ephemeris, a table of the positions of celestial
bodies at specified intervals of time. Both a directional sender and
receiver must know the other's position in order to establish a link
between the pair. In addition, these communication resources are
highly scheduled. It is not sufficient for a receiver to point at a
Cerf, et al. Expires November 2001 [Page 25]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
prospective target and just wait _ for example, a terrestrial node
will typically have to point at several targets sequentially, and an
interplanetary node will typically not have enough power to just wait
for incoming messages. Rather, a schedule of communication
opportunities must be calculated and then refined with planned
communication instances. A communication opportunity establishes
that the endpoints could establish a link if they were pointing at
each other at the proper times. We refer to a planned communication
instance as an agreement (although not irrevocable) between the two
parties to establish contact and communicate for a defined period of
time. The protocols for establishing the schedule of communication
instances between all pairs of possible communicants will evolve --
from something primarily done manually to something more automated as
the Interplanetary Internet grows.
The scheduled nature of connectivity in the Interplanetary Internet,
particularly across the deep-space links, means that at the time of a
bundle's arrival at an IPN Gateway, some or all of the possible
outbound routes may be "down." The gateway must store the bundle
until the appropriate link is available and then transmit the bundle
over that link. One of the fundamental differences between the
Interplanetary Internet and the terrestrial Internet is this inherent
use of store-and-forward mechanisms in routing bundles. With that
in mind, let us consider the routing decisions made at some of the
IPN Gateways in Figure 2.
4.2.2. IPN Gateway routing
Bundle routing at an IPN gateway will typically have to deal with a
mix of continuously available links and intermittently available
links. Routing across a continuously available link is a relatively
straightforward activity. Routing in the presence of intermittently
available links can be significantly more complex, as the gateway
will need to decide not only the next hop destination but also the
time at which to send the bundle. Conditions elsewhere in the
network may make it more desirable to send a bundle to a next-hop
destination that is not yet accessible, even when an alternative
route is currently available. The routing function in IPN Gateways
and other IPN nodes that have intermittent links has three distinct
parts: the contact scheduler, the route evaluation algorithm, and
the dispatcher algorithm.
Figure 3 illustrates the relationship of these functions with each
other and with other elements of the communication protocol suite.
The contact scheduling application establishes the schedule for
communicating with next-hop neighbors. The implications of this
scheduling activity are broad _ orbital mechanics, resource
management onboard the spacecraft, prospective communications loads,
and so forth all play a role. Because spacecraft resources must be
coordinated among all functions (not just communications), this is
typically going to be part of a larger resource management
application (interactions with functions such as pointing and power
Cerf, et al. Expires November 2001 [Page 26]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
control are not shown in Figure 3, but are present nonetheless). It
is very likely that the contact scheduler will *not* be local to the
bundle node, but rather a centralized function that distributes a
contact schedule to IPN nodes that perform routing. This may change
in the future to a distributed contact scheduling algorithm, but for
the foreseeable future, this will not be the case. The product of
the contact scheduler is a schedule of planned contacts, durations,
expected data rates, etc. It is intrinsically link-state in nature.
+-------------------------------+
| User Applications |
+-------------------------------+
+-------------------------------+ +--------+
| +-----------------------+ | | |
+---------+ | | Bundle Transport | | | Policy |
| Contact | | +-----------------------+ | | |
|Scheduler| | +-----------------------+ | +--------+
+---------+ | | | | |
| | | +---------------+ | | |
V | | | Dispatcher |<------------+
/-------\ | | | Algorithm | | | +----------+
<Schedules>----->| | | | | /------\ |Route |
\-------/ | | +---------------+ |<--< Routes ><-|Evaluation|
| | Bundle Routing | | \------/ |Algorithm |
| +-----------------------+ | +----------+
| Bundle Agent Functions |
+-------------------------------+
+-------------------------------+
| +------+ +------+ +------+ |
| | TCP | | UDP | | LTP | |
| +------+ +------+ +------+ |
| +----------------+ +------+ |
| | IP | | - | |
| +----------------+ +------+ |
+-------------------------------+
+-------------------------------+
| Link and Physical Interfaces |
+-------------------------------+
Figure 3. IPN routing applications and their relation to other
communication functions.
The route evaluation application exchanges information with all
first-hop neighbors (we hope this occurs infrequently) to build a
general picture of the IPN beyond the first-hop neighbors. We
currently view this in terms of a distance-vector representation, but
with metrics such as the expected delay to the destination IPN region
and average aggregate data rate to the destination IPN region. The
mechanics for building these metrics are still in development.
Cerf, et al. Expires November 2001 [Page 27]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
The dispatcher algorithms accepts routing requests from the bundle
transport layer and builds a "manifest" for each next-hop contact
that is subsequently consumed by the bundle routing layer. The
dispatcher application consumes some or all of the following in
deciding how to schedule bundle transmissions: the contact schedule,
the routing information, local policy information, and the per-bundle
specifics provided by the bundle transport layer (such as bundle
destination, length, priority, time-to-live, and possibly
"Starbucks"-related information). We envision a family of different
dispatcher algorithms, operating as "plug-ins," that provide
different levels of sophistication in the scheduling function to suit
different needs. The simplest versions of the dispatcher might just
consider destination to schedule the bundle on the soonest-possible
outbound contact. More sophisticated versions might consider
priority, bundle length (to preserve atomicity), and time-to-live
requirements. Others could support the Starbucks model or apply
optimization techniques to attempt to improve the use of each
contact.
The following is a conceptual description of what happens: when a
bundle arrives at the bundle layer and needs to be routed, the bundle
routing function posts a request to the dispatcher, noting the
destination, length, priority, time to live, etc. of the bundle to be
routed. The dispatcher integrates this bundle into its manifest and,
at the bundle's transmission time, informs the bundle routing
function to send the bundle to the appropriate next-hop destination.
4.2.3. Systems participating in example bundle data transfer
Figure 4 is a revision of Figure 2 to highlight those portions of the
Interplanetary Internet that participate directly in the bundle
transfer example. Also shown in Figure 4 are the source and
destination for the bundle data transfer, and the Domain Name System
equivalents in the terrestrial Internet (DNS 1), in the IPN
"Backbone" (DNS 2), and in the Mars Internet (DNS 3). This figure
will serve as the basis for the following bundle data transfer
example.
Table 1 provides the full host names of each of the primary elements
in Figure 4.
Cerf, et al. Expires November 2001 [Page 28]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
+---+
/ /|
+------------------------+---+ |
+---+ Earth's Internet |DNS| +
/ /|IPN Region: earth.sol| 1 |/
+---+ | +---+ +---+
|SRC| +---------/ /|--------+
| |/ +---+ |
+---+ |G/W| +
+----------| 1 |/---------+
| +---+ |
| The "Backbone" |
| IPN Region: ipn.sol |
+---+ +---+ +---+
/ /|-------------------/ /| / /|
+---+ | +---+ | +---+ |
|DNS| + |G/W| + |DST| +
| 2 |/ | 2 |/-------------| |/+
+---+ +---+ +---+ |
| Mars's Internet |
+---+ IPN region: |
/ /| mars.sol |
+---+ |---------------------+
|DNS| +
| 3 |/
+---+
Figure 4. Interplanetary Internet showing principal systems.
Table 1. Host name tuples.
+------------+------------------+---------------------------+
| Host | IPN Regions | Host name tuples |
+------------+------------------+---------------------------+
| SRC | earth.sol | {src.jpl.nasa.gov, |
| | | earth.sol} |
+------------+------------------+---------------------------+
| IPN G/W1 | earth.sol | {ipngw1.jpl.nasa.gov, |
| | | earth.sol} |
| | ipn.sol | {ipngw1.jpl.nasa.gov, |
| | | ipn.sol} |
+------------+------------------+---------------------------+
| IPN G/W2 | ipn.sol | {ipngw2.nasa.mars.org, |
| | | ipn.sol} |
| | mars.sol | {ipngw2.nasa.mars.org, |
| | | mars.sol} |
+------------+------------------+---------------------------+
| DST | mars.sol | {dst.jpl.nasa.gov, |
| | | mars.sol} |
+------------+------------------+---------------------------+
Cerf, et al. Expires November 2001 [Page 29]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
4.2.4. Step 1: Bundle creation and first-hop transmission
An application on the source host in Figure 4 has data that it wishes
to send to the destination on Mars. The exact content of this data
is opaque to the bundle transfer, but assume that it contains all of
the information necessary to accomplish some desired function. That
is, assume that application-specific instructions for storage,
handling, error processing, and disposal accompany whatever data
object is to be operated upon. The application invokes the bundle
agent, supplying it the information shown in Table 2.
Table 2. Information passed from source application to bundle
agent.
+-------------+---------------------+------------------------------+
| Item | Value | Description |
+-------------+---------------------+------------------------------+
| Destination | {dst.jpl.nasa.gov, | IPN Name tuple of the |
| host name | mars.sol} | destination |
+-------------+---------------------+------------------------------+
| Destination | 0x00000008 | Similar to port number. Can|
| application | | be "well-known" (i.e., |
| instance | | identify a daemon) or |
| handle | | "ephemeral" (i.e., identify |
| | | a running, suspended, or |
| | | hibernating process |
+-------------+---------------------+------------------------------+
| Source | 0x1763421A | Value used to identify the |
| application | | appropriate instance of the |
| instance | | source application for |
| handle | | response processing |
+-------------+---------------------+------------------------------+
| Handling | Reliable delivery, | The services requested from |
| instructions| normal priority, | the bundle layer. |
| | data obsolete in 36 | |
| | hours. | |
+-------------+---------------------+------------------------------+
| User data | N/A | |
+-------------+---------------------+------------------------------+
The bundle agent creates a bundle and stores it in persistent storage
(on disk or other non-volatile memory). There are some fields of the
bundle header that the bundle agent must supply: the Bundle
Identifier, the Source Host name tuple, the Custodian name tuple, and
the time to live. (The application may state a time after which the
data are obsolete, but the actual time-to-live field in the bundle
header uses the application's data in combination with network
restrictions on time-to-live to initialize this field properly.) The
bundle agent's routing function requests a route from the dispatcher
application, and receives next-hop destination information and source
information. (Since a host may reside in multiple IPN Regions, the
source host name tuple is a function of the outbound route selected.
Cerf, et al. Expires November 2001 [Page 30]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
The bundle agent uses this information to complete the Source Host
and Custodian name tuples prior to transmission.)
Note: For hosts that have continuously available communication
resources, it is likely that an optimization would be implemented to
reduce the required interaction with the dispatcher application. In
this instance, the bundle routing function might consult a local
routing table before contacting the dispatcher, and only requesting
routing information from the dispatcher if there is no appropriate
entry in the routing table.
The bundle in this example is destined for the "mars.sol" IPN Region
(per Table 1). The dispatcher (or routing table) determines that the
proper next-hop destination for the mars.sol region is
{ipngw1.jpl.nasa.gov, earth.sol}, and that the appropriate transport
protocol to use is TCP. The bundle agent consults the Terrestrial
Domain Name Server to resolve ipngw1.jpl.nasa.gov to an IP address,
establishes a TCP connection with ipngw1, and transfers the bundle to
it. The bundle agent at the source retains a custodial copy of the
bundle in persistent storage.
4.2.5. Step 2: Bundle processing at first-hop destination
When the IPN Gateway {ipngw1.jpl.nasa.gov, earth.sol} receives the
bundle via TCP, it stores it on persistent storage (disk). The
bundle agent consults the dispatcher, and is informed that the
appropriate next-hop destination is in the "ipn.sol" IPN Region:
{ipngw2.nasa.mars.org, ipn.sol}. The dispatcher provides the time at
which the bundle should be transmitted, at which time the contact
scheduler and its associated functionality will ensure that there is
a contact with ipngw2 via the Long Haul Transport Protocol (LTP). In
considering alternative contacts for the bundle, the dispatcher
checks the time-to-live in the bundle, which was 36 hours from the
time of initial submission to the bundle agent at the source, to
ensure that the route selected is consistent with the time-to-live
requirements of the bundle. The bundle transport functionality of
the bundle agent in ipngw1 accepts custody of the bundle, updates
this information in the bundle header, and informs the source that
has done so. The sources bundle agent deletes its custodial copy of
the bundle. When the time indicated by the dispatching function
arrives, the bundle is transmitted via LTP to the IPN Gateway that
connects the ipn.sol Region with the mars.sol Region:
{ipngw2.nasa.mars.org, ipn.sol}.
4.2.6. Step 3: Bundle processing at gateway to destination IPN Region
The Mars gateway, {ipngw2.nasa.mars.org, mars.sol}, receives the
bundle from the Earth gateway via LTP. It stores the bundle in
persistent storage and accepts custody of the bundle, signaling back
to the Earth gateway that it has done so. When the notification of
acceptance reaches the Earth gateway, ipngw1 deletes its custodial
Cerf, et al. Expires November 2001 [Page 31]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
copy. The Mars gateway consults its dispatcher application to find
an outbound contact to forward the bundle over. The dispatcher
returns an indication that the appropriate next hop is the
destination itself, that the proper transport protocol is TCP, and
that the destination is accessible immediately. The gateway verifies
that the time-to-live has not expired, and forwards the bundle to the
destination.
4.2.7. Step 4: Bundle processing at destination
The destination bundle agent receives the bundle via TCP, stores it
on its own persistent storage, and accepts custody of the bundle from
IPN G/W2. The bundle agent "awakens" the destination application
process identified by the Destination Application Instance Handle.
This may involve creating a new instance of a server from a daemon
process, signaling an idle running process, or reinstantiating a
process that has been suspended with its state stored on persistent
storage. (The specifics of this are system-dependent, and may have
to be robust against system restarts, upgrades, and migration of
processes from one host to another.) The bundle agent deletes the
copy of the bundle from persistent storage when the application has
received it. The destination application may generate an
application-layer acknowledgment in a new bundle and send it to the
source's application instance identifier (0x1763421A from Table 2).
4.3. Error Conditions at the Bundle Layer
This section describes the error conditions that may arise at the
bundle layer during bundle creation and transport. When these errors
occur within the sender's IPN domain, it may be possible to conduct a
near-real-time dialog to correct them before the bundle is forwarded.
We say `may be possible' because even if two nodes are in the same
IPN domain, they may not have real-time connectivity. An example of
such a situation would be if a lander were on the opposite side of
the planet from its IPN gateway, and used bundles to communicate with
the gateway through a low altitude orbiter, with the orbiter itself
serving as a bundle agent.
Cerf, et al. Expires November 2001 [Page 32]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Table 3: Error conditions at the bundle layer.
+-------+---------------------------+------------------------------+
| Error | Description | Places where Error can Occur |
+-------+---------------------------+------------------------------+
| 1* | Unknown destination region| Source Bundle Agent |
+-------+---------------------------+------------------------------+
| 2* | Invalid Source App. | Source Bundle Agent |
+-------+---------------------------+------------------------------+
| 3* | Bundle Parameter Syntax | Source Bundle Agent |
| | Error | |
+-------+---------------------------+------------------------------+
| 4* | Bundle Parameter Semantic | Source Bundle Agent |
| | Error | |
+-------+---------------------------+------------------------------+
| 5* | Invalid Node Name in LSRR | Any bundle node |
| | or SSRR | |
+-------+---------------------------+------------------------------+
| 6 | Insufficient buffer space | Any bundle node |
+-------+---------------------------+------------------------------+
| 7 | DNS unreachable | Any bundle node |
+-------+---------------------------+------------------------------+
| 8* | Time exceeded | Any bundle node other than |
| | | the source agent |
+-------+---------------------------+------------------------------+
| 9* | Source Entity Access | Any bundle node other than |
| | Denied | the source agent |
+-------+---------------------------+------------------------------+
| 10* | Invalid Administrative | IPN gateway serving |
| | Destination Name | destination IPN domain |
+-------+---------------------------+------------------------------+
| 11* | Invalid Destination App. | Destination |
+-------+---------------------------+------------------------------+
| 12* | End-to-end Access Denied | Destination |
+-------+---------------------------+------------------------------+
The errors that can occur at the bundle layer are shown in Table 3.
Error numbers marked with an asterisk (*) are reported back to the
sending application.
* Unknown Destination Region: This error occurs when the source
bundle agent is directed to create a bundle destined for an IPN
Region that is not recognized (i.e. one for which there is no
applicable route known to the Dispatcher Application). Note that
only the IPN Region part of the destination name has to be
interpretable outside the destination's IPN Region. In
particular, the administrative part of the destination name need
not be interpretable to the source DNS (assuming the source and
destination are in different IPN Regions), so it cannot
necessarily be checked when the bundle is created.
* Invalid Source Application: If the source application instance
handle supplied by the source application is invalid, the source
Cerf, et al. Expires November 2001 [Page 33]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
bundle agent responds with an Invalid Source Application error.
This might be the case, for instance, if the source application
provided an instance handle referencing a system process for which
the application didn't have privileges.
* Bundle Parameter Syntax Error: The source bundle agent may check
the syntax of some of the bundle-creation parameters (i.e. it may
ensure that the end-to-end and IPN access security certificates
are well-formed, etc.) If a parameter is found to be
syntactically incorrect or obviously and definitely erroneous, the
bundle agent will report a Bundle Parameter Syntax Error back to
the source that includes, at a minimum, the parameter that caused
the error.
* Bundle Parameter Semantic Error: If the source bundle agent can
identify a particular bundle creation parameter as being well-
formed but unserviceable, it will report a Bundle Parameter
Semantic Error to the source application that includes, at a
minimum, the parameter that caused the error.
* Invalid Node Name in LSRR/SSRR: If an invalid node name is
discovered in the loose or strict source route & record, the
bundle agent that detects the error will propagate it back to the
source application. Note that it may be advantageous to have
bundle agents check the validity not only of the next hop in the
source route but as many entries as they can. The value of
checking multiple entries would be in detecting errors as soon as
possible, preferably before the bundle traversed any of the long-
haul links.
* Insufficient Buffer Space: If a bundle agent does not have
sufficient buffer space to accept a bundle, it drops the bundle
and generates an Insufficient Buffer Space error. Note that a
bundle node may choose to drop lower priority bundles in order to
make room for higher priority ones. This error is not propagated
back to the source.
* DNS Unreachable: If a bundle agent needs access to its DNS (or
DNS-equivalent) and cannot obtain information from it, it
generates a DNS Unreachable error. This information is not
propagated back to the source application.
* Time Exceeded: If the time-to-live field (either the source-
provided TTL or the internal bundle TTL) expires, the source is
notified with a Time Exceeded message. These errors are
propagated to the source application.
* Source Entity Access Denied: This error indicates that the source
entity does not have access to a needed resource at an IPN node.
The source might not be authorized to use the node at all, or it
might not have authorization to use a particular interface
required by the bundle. Source Entity Access Denied errors
Cerf, et al. Expires November 2001 [Page 34]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
indicate that the source is not authorized to use a particular
resource; other errors (e.g. Insufficient Buffer Space) indicate
that a particular resource is unavailable to a bundle. For
example, an entity on the surface of a planet might be authorized
to communicate, using the bundle protocol, with another entity on
the other side of the planet via a low-altitude orbiter that is
also an IPN gateway. The sender might not, however, be authorized
to send bundles across interplanetary space. In this case bundles
sent to the orbiter destined for the other side of the planet
would not cause errors, while any bundles with off-planet
destination addresses would. Source Entity Access Denied errors
are propagated back to the source application.
* Invalid Administrative Destination Name: Once a bundle has reached
its destination IPN Region, the administrative part of the
destination name can be verified. If the administrative part of
the destination name is not valid, the source is notified with an
Invalid Administrative Destination Name error message. As with
LSRR/SSRR, it would probably be advantageous to check the
administrative part of the destination name as soon as possible to
avoid propagating misnamed bundles and error messages across the
backbone.
* Invalid Destination Application: If the destination bundle agent
cannot instantiate the destination application (based on the
destination application instance handle in the bundle), it
notifies the source application with an Invalid Destination
Application error message.
* End-to-End Access Denied: If the bundle destination does not
accept the bundle due to an authentication or access-control
error, the source is notified with an End-to-End Access Denied
Message.
4.4. Support of existing Internet applications
There is no clean way to support "legacy" applications in the IPN.
One might think that application-layer proxies could protect
applications from the rigors of interplanetary communications, but it
turns out that this is not the case, even if the appropriate timers
(at both application and transport layers) could be scaled correctly.
As an example, consider a legacy FTP client on Earth trying to get a
file from an FTP server on Mars. If the client's connection request
to {foo.bar.com, mars.sol} is somehow coerced into binding to an
Earth-resident application proxy (as can be done with nonstandard
modifications to the terrestrial DNS) the proxy then has to negotiate
enough information out of the client to form the bundle that will
travel to Mars. This bundle needs to include at least:
* The client's IPN domain name
* The server's name tuple on Mars (Note that if wildcarded A-
records are used to cause the off-planet server name to bind to
the proxy, the proxy will not have a notion of the destination
Cerf, et al. Expires November 2001 [Page 35]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
name. The proxy will have to explicitly request the source name
from the source.)
* The file to be retrieved
* Where to deliver the retrieved file to, since it probably won't be
coming back in this session
While all of this information could in principle be elicited from a
command-line client via appropriate server messages and extensive use
of the `quote' command, there is no guarantee that other clients
(specifically GUI clients) will be able to accomplish this. SMTP (e-
mail) is perhaps the only application that could possibly be tuned to
work over interplanetary distances, but even SMTP has embedded timers
that would have to be altered significantly before it would work.
5. Security in the IPN
We do not have a detailed list of security requirements for the
Interplanetary Internet, primarily because there currently aren't any
"users" of the IPN and few, if any, of the potential users have given
enough thought to security to commit to a set of security
"requirements". However, we know that the Interplanetary Internet's
bandwidth resources will be precious. We can also safely assume that
the IPN will be a prized hacker's target (at least from Earth). We
can also envision that there will be precious, private data flowing
across the IPN. As a result, we assume that various security
mechanisms and services will be required to provide protection for
the bundles traversing the IPN and for the IPN infrastructure itself.
There are two aspects to IPN security: protection of the IPN
infrastructure and protection of the data traversing the IPN. The
protection of the IPN infrastructure is not unlike the protection
required for the Earth-based Internet infrastructure. There will be
a need to exchange routing information securely among the IPN nodes
as well as to securely manage them. For the IPN infrastructure to
protect itself, the IPN nodes need to be mutually suspicious of one
another. That is, the IPN nodes will authenticate themselves to each
other to ensure that they are not being spoofed by an untrustworthy
entity. One might ask if this is overkill if we believe that there
will always be a small, controlled number of IPN nodes (a la the
original ARPANET). However, one could equally envision that there
could be many IPN nodes that are sponsored and controlled by multiple
organizations (a la the current Internet). Since we would like the
IPN to easily scale, we want to build mutual suspicion and security
mechanisms into the IPN architecture from the outset. It should be
noted that the same mechanisms could be used to provide security for
both the IPN infrastructure and the data flowing through it.
5.1. Assumptions Regarding Required IPN Security Mechanisms
The security mechanisms assumed to be required for the IPN are:
* Access control
* Authentication
Cerf, et al. Expires November 2001 [Page 36]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
* Data integrity
* Data privacy
Access controls will be required because the IPN's space-based assets
will have limited resources available and because it will be a likely
target from a hacking perspective. By limiting access to the IPN
resources, we limit the availability of the IPN to only those users
who require its services and do not allow it to be overburdened by
those who are not authorized to access the IPN services.
Authentication of identity will be required to perform access control
mediation. To allow or disallow access to the IPN, an assured
identity of the source of the network traffic will be needed. The
identity might be of an individual (e.g., a payload principal
investigator) or a location (e.g., a science payload control center
or a spacecraft control center).
Data integrity will be required to ensure that data received across
the IPN is the same data that was originally sent. This is true from
both an IPN infrastructure perspective (e.g., management data) as
well as from a user's perspective (e.g., science data, command
sequences). Data integrity assures that any unauthorized modification
of the data will be detectable by the receiver.
Data privacy will be required to ensure that only those who are
authorized to obtain data traversing the IPN or destined to/from the
IPN infrastructure will be able to do so. This mechanism is also
known as confidentiality.
In the network security community, there are two well-known security
paradigms: hop-by-hop security (also known as link security) and end-
to-end security. In the hop-by-hop paradigm, the data to be
transmitted over the network is protected on a hop-by-hop basis.
That is, the data is protected at its source, but in order to be
routed to its final destination, it must be unprotected at a trusted
routing point (e.g., a ground-based gateway, an IPN gateway) in order
to be examined for onward routing. The trusted routing point must
then re-protect the data and forward it on to its next routing point.
Each successive hop point must unprotect and re-protect the data
until it reaches its ultimate destination. A negative aspect of this
security paradigm is that, depending on how the protection is
applied; the data might be completely exposed while in the gateway
(hence the term trusted gateway). This means that the data is
potentially vulnerable to unauthorized modification and unauthorized
disclosure.
The end-to-end paradigm does not employ trusted gateways. Rather, it
assumes that the path between the source of the data and its
destination is hostile and cannot be trusted. As a result, the data
is protected at its source and is not unprotected until it reaches
its destination. However, in order for this scheme to work, routing
information must remain unprotected so that the intermediate gateways
Cerf, et al. Expires November 2001 [Page 37]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
are able to determine how to forward the data without having the
opportunity to read or change the data.
One problem with the end-to-end paradigm is that it can only work
across a network where there are end-to-end protocols (e.g., TCP).
There has to be a protocol below the data that provides the ability
to route. One example of this is the Internet Engineering Task
Force's (IETF) Internet Protocol Security Protocol (IPSEC). The
IPSEC Encapsulating Security Payload (ESP) protocol provides an end-
to-end security service between the IP and TCP protocol layers.
However, in the IPN, as has already been discussed earlier, IP and
TCP are not necessarily carried end-to-end protocols. Rather, those
protocols can/will be terminated in a local internet (e.g. a ground
segment on earth or another celestial body) and not carried end-to-
end through the IPN. The data will be carried end-to-end via the
bundle protocol that would in turn, be transported by other IPN
protocols (e.g., LTP, TCP) using the pony express model.
Since the IPN must be structured on a store-and-forward basis, and
since users may not trust the IPN gateways, solutions such as IPSEC
cannot be employed. In order to provide an end-to-end like security
solution, security mechanisms can only be applied to the data and not
any other protocol layer(s) below the bundle protocol. This is the
way the Secure Sockets Layer (SSL) (now being specified within the
IETF as the Transport Layer Security (TLS) protocol) and secure email
techniques such as S/MIME and OpenPGP work. In essence, security
services are only applied to the data portion of a packet, leaving
all of the packet's protocol headers in the open and available for
use by intermediate systems. For example, to transmit the string
"hello world" across the Internet would entail encapsulating the
string into a TCP header, an IP header, and a media access (MAC)
layer header. To provide end-to-end security services, only the
payload ("hello world") would have security services applied to it
(e.g., it might be encrypted to provide privacy/confidentiality).
However the TCP, IP, and MAC headers would all remain without
security services applied to them. In comparison, when using IPSEC
ESP, the TCP header would have security services applied to it to
protect it as well as the payload data. And if it were operating in
"tunnel" mode, ESP would encapsulate the entire payload - the TCP and
IP headers - inside of an IP packet.
Given that data will be transported across the IPN in bundles using
an email-like paradigm, borrowing technology from email security is a
solution for the IPN.
5.2. Secure Email Technology
Since secure electronic mail operates in a non-interactive mode, and
since both communicating parties have not necessarily communicated
with one another previously, a technique different than used by IPSEC
was developed. One way in which email could be securely wrapped
Cerf, et al. Expires November 2001 [Page 38]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
(e.g., encrypted and/or digitally signed) is via through the use of
public key cryptography. Using this technology, an email sender
would use the public key of the intended recipient to encrypt the
payload (i.e., the message) and the recipient would use its private
key to decrypt the payload. Likewise, the sender could digitally
sign the message (in order to authenticate the message) using its
private key and the recipient would verify the message's authenticity
using the sender's public key. However, there are two problems with
this scheme for the IPN.
The first problem is that public key cryptography is quite
consumptive of processor power. Therefore, the common practice is to
use symmetric key (i.e., shared secret) algorithms for large amounts
of data. With changes in technology, this problem may disappear, at
least for earth-based systems. However, it is not clear that space-
based assets will catch up as quickly. The second problem is that
public key cryptography is consumptive of bandwidth. A public key
exchange technology that allows two communicating entities to derive
a shared symmetric key by virtue of knowing each other's public keys
and exchanging some random information is known as a Diffie-Hellman
exchange. However, the exchange of information must be performed in
a near-real-time environment so that the shared key can be used to
encrypt transmissions. This is not practical in the IPN since there
are no real-time exchanges of data on an end-to-end basis.
However, the secure email community has realized that they also
operate under a pony-express model. The secure email community has
also realized that secure email may be sent to entities with which
one has not previously communicated. Therefore, there needed to be a
base-level expectation of a common, minimal set of security services
that both sides could use.
The general technique used by the secure email community is that the
sending entity decides what security mechanism(s) to employ (e.g.,
encryption for confidentiality, digital signature for authentication
and integrity, or both). If the data to be sent via email is to be
encrypted, the sender generates a random key that is used to encrypt
the data and the data is encrypted. The sender then either has in
its possession the public key of the receiver, or queries a public
key server (e.g., a public key infrastructure or PKI) to obtain the
receiver's public key, which would be contained in a digital
certificate. At the expense of additional bandwidth, the sender can
transmit its digital certificate in the email message, which the
receiver can verify as genuine based on the well-known certificate
authority's signature on the certificate. The key used to encrypt
the data is encrypted using the public key of the receiver and the
message is sent. The receiver uses its private key to first decrypt
the key used to encrypt the message and then uses the decrypted key
to decrypt the data.
When a digital signature is used, the sender calculates a hash of the
message using a digest algorithm such as MD5 or the Secure Hash
Cerf, et al. Expires November 2001 [Page 39]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Algorithm (SHA-1). The resulting hash is then encrypted using the
sender's private key. The encrypted hash is sent along with the
message data. The receiver uses its copy of the sender's public key
to authenticate the fact that the message was sent by the sender.
5.3. Application of Secure Email Technology to the IPN
Given that the IPN and electronic mail share the same operational
paradigm, the IPN's notion of "bundle-space" is directly analogous to
a secure email MIME body part. As was previously explained, in
secure email the contents of the message are encrypted using a
symmetric key. The actual message contents are "containerized"
(which can be analogized to the contents being "bundled") into a MIME
body part. Additional MIME body parts are also attached which would
contain the encrypted symmetric key and additional information needed
for decryption.
Therefore, the same security techniques can be applied to the IPN's
notion of "bundle-space." It can be seen how bundle payload data
carried through the IPN is equivalent to an email message.
Therefore, the bundle payload data, like the email message, can have
security services applied to it before being sent as a protected
entity via bundling across the IPN.
In essence, the really difficult part of deploying an IPN security
solution based on secure email is the problem of deploying a public
key infrastructure (PKI) in the IPN. Deploying a PKI on Earth is
equally difficult as can be seen by the continuing problems faced in
the various PKI pilot programs currently underway.
On Earth, the problem is how do competing PKIs cross certify public
keys and who acts as the root certification authority? (PKI is based
on a tree hierarchy where a root certificate authority's public key
is well known in order to certify public key certificates). On
Earth, when a user does not have the necessary public keys, the user
can go off to a public key certificate server to obtain the needed
certificates which contain digitally signed (i.e., certified and
authenticated) public keys. In the case of email, the sender can
attach its public key certificate as a MIME body part. The receiver
then validates the digital certificate and uses it to obtain the
information contained in the message. Eventually, a cache of digital
certificates is formed containing the information needed to securely
communicate among a cadre of entities without having to resort to the
use of a key/certificate server.
The IPN, however, will not have the luxury of being able to query on-
line PKI certificate servers (at least not in the same sense as is
being contemplated on Earth). The problem is very much analogous to
the one with the Domain Name System (DNS). When a user sends a
secured "bundle" to an IPN entity on another celestial body, we would
not want the receiver to have to first query a PKI for a public key
certificate on Earth to obtain the receiver's public key. A local
Cerf, et al. Expires November 2001 [Page 40]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
PKI might exist. However there is the issue of the dynamic nature of
such a server and the high likelihood that it would quickly drop out
of synch. It would seem reasonable that the public key certificates
of the various space-based entities would not change often, but this
could not be said about the Earth-based entities.
An earth-based sender may be able to query a certificate authority to
obtain a certificate for an IPN entity not located on earth.
However, an IPN entity in space sending a bundle would not be able to
perform such a query. Therefore, it would appear that a set of pre-
placed, cached certificates containing the public keys of those IPN
entities that are expected to be communicated with would make sense.
In addition, an IPN sender should always include its public key
certificate as part of the bundled transmission across the IPN as is
done using secure email. This would "cost" us in terms of bandwidth
(a typical X.509 public key certificate is about 1K bytes in size but
can be larger when certificate chains are included). However, by
including the certificate, we can be assured that the receiver will
not have to use any additional bandwidth, or incur any additional
delays in making use of the transmitted data.
5.4. Protecting IPN Data and the IPN Backbone Infrastructure
The IPN will have few and precious resources. Therefore, not only
the user data flowing across the IPN will require security services -
the backbone infrastructure itself will require similar services. In
order for the IPN infrastructure to be self-protecting, it must be
built using the paradigm of mutual suspicion. Mutual suspicion
requires each entity of the IPN to not assume that another IPN entity
with which it is communicating is a "friendly" entity. That is, it
should not assume that the IPN entity is who it claims to be without
a verified means of identification. Based on a verified identity,
access controls can be performed to allow or disallow communications
between entities.
Infrastructure information such as routing updates, node management
information, distribution of digital certificates, and certificate
revocation lists need to be protected from unauthorized modification
and potentially from unauthorized disclosure. The IPN nodes would
use the same mechanisms as are used to provide protection for user
data - namely the security services available to a bundle aware
application (e.g., a "bundle-agent").
A bundle aware application would encrypt and/or digitally sign the
IPN infrastructure payload to be transmitted to another IPN node.
The signed and/or encrypted payload would then be presented to a
bundle-agent that would prepare the payload data to be carried across
the IPN in a bundle. Potentially, the bundle-agent might also sign
and/or encrypt for hop-by-hop security protection, which would allow
each bundle-agent receiving the bundle to authenticate the identity
of the transmitting bundle-agent. This mechanism provides the
ability to ensure that no other entity can masquerade as a rogue
Cerf, et al. Expires November 2001 [Page 41]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
bundle-agent.
The bundle-agent residing at a ground-based IPN gateway should check
the signature of the bundle payload to perform an access control
check to limit access to the IPN only to those who are authorized to
use it. The access control check can be accomplished via an access
control list.
Finally, the receiving application would make the final security
checks before accepting the data payload from the final receiving
bundle-agent. Should all the final checks succeed, the receiving
application would then be free to consume the data.
6. Building a Stable Backbone for the IPN
Just as the performance and capability of the terrestrial Internet
are largely determined by the capacity and stability of its backbone
links, so will the performance and capability of the Interplanetary
Internet depend in large part on the capacity and stability of the
interplanetary backbone.
By "backbone" we mean a set of high-capacity, high-availability links
between points of access to high-activity subnets. In the
terrestrial Internet, backbone links are typically between the high-
activity subnets for cities such as Houston and Chicago. In the
Interplanetary Internet the backbone links will be between the high-
activity "subnets" for planets such as Earth and Mars.
But the IPN backbone will differ from familiar terrestrial backbones
in several important ways.
* The media by which information is transmitted obviously differ.
Terrestrial backbone links used to be copper wire and are now
optical fiber. In the Interplanetary Internet, all information
will be via radiation _ either RF or optical _ through empty
space.
* The nature of the connectivity between backbone points of presence
(POPs) will be fundamentally different. Terrestrial Internet
connectivity is structural and relatively static: nodes are
physically attached to fiber. Interplanetary Internet
connectivity will be operational, directed, and highly dynamic:
radiation must be directed at the right moment toward nodes that
are prepared to detect it, and the transmitting and receiving
entities will be in continuous movement relative to one another.
* The costs of deploying, repairing, and upgrading infrastructure
will be much higher for the interplanetary backbone than for
terrestrial backbones.
* The costs of configuring, operating, and managing the
interplanetary backbone will likely be somewhat higher than for
terrestrial backbones. One factor in this is the scarcity and
high cost of electrical power at IPN nodes deployed off Earth.
* Perhaps most important of all, the distances between nodes of the
Cerf, et al. Expires November 2001 [Page 42]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
interplanetary backbone will be vastly greater than the distances
between nodes of terrestrial backbones _ so great that the speed
of light becomes a very significant constraint on backbone
operations.
6.1. Backbone Design Considerations
The differences described above imply two general constraints on the
design of the interplanetary backbone.
1.
Bandwidth is not free, or even cheap. Reliable delivery cost per
bit will be far higher across the interplanetary backbone than
across any terrestrial backbone, so bandwidth must be thoughtfully
allocated. Both retransmission and forward error correction
coding contribute to reliable delivery, but both cost bandwidth;
we must seek a balance between the two that minimizes overall
overhead.
2.
Interactive protocols don't work, at least not well. Negotiation
_ connection establishment, flow control, congestion control _ can
easily take so long as to consume all available transmission time.
Reliable, in-order stream delivery can be so severely delayed by
retransmission latency as to be operationally useless.
These design constraints, in turn, must be accommodated at four
layers of the protocol stack.
At the physical layer, the relevant research is in radiant RF or
optical communication. The physical infrastructure of the
interplanetary backbone consists mainly of antennae, many of them
mounted aboard orbiting or landed spacecraft, rather than cable in
buried or undersea conduit. In the near to medium term, the
principal elements of this infrastructure will be Earth-based
tracking stations, such as NASA's Deep Space Network, and the planned
"Marsnet" of low-Mars orbiters plus a possible areostationary relay
satellite. In the long term these assets might be augmented by
optical communication satellites orbiting Earth and/or other
planetary bodies, and perhaps by additional relay satellites
positioned at the LaGrange points of planetary orbits (possibly using
solar sailing technology for autonomous station-keeping).
Although this infrastructure will not be subject to some of the
problems of terrestrial backbones _ for example, we needn't worry
about backhoes cutting through underground fiber _ there will be
other challenging operational issues. Accuracy in pointing and
transmission scheduling at the backbone antennae will be critical to
assuring the most efficient use of these assets. This means that all
elements of the interplanetary backbone infrastructure must share a
common understanding of (a) one another's orbital dynamics and (b)
the current time. The latter argues for the importance of reliable
space-borne clocks, together with a protocol for frequent correction
of clock drift based on current navigation data.
Cerf, et al. Expires November 2001 [Page 43]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Moreover, there will be times at which connectivity between a given
pair of backbone antennae will be impossible due to the interposition
of large, radiant planetary bodies _ or worse, the sun. These
occultations will be predictable, given good models of orbital
dynamics, but will nonetheless necessarily constrain the manner in
which data flow through the backbone.
At the link layer, the design issues are somewhat less exotic. The
interplanetary backbone will require link protocols that minimize
overhead and that have no inherent expectations of low noise or low
point-to-point transmission latency. CCSDS protocol standards such
as Proximity-1 are plausible candidates. They support robust encoding
to reduce bit error rates, at some cost in bandwidth consumption.
Our current thinking is that no interplanetary backbone functionality
will be required at the network layer. The reason for this is that
we expect the endpoints of transport-layer protocol on the backbone
to be in direct line-of-sight contact as discussed below; since there
will be no intervening nodes to route through, there will be no need
for routing functionality at the network layer. [Selection of
endpoints for point-to-point transport-layer communication does
indeed imply the need for routing and network functionality, but this
requirement is addressed at the Bundling layer above Transport.]
Finally, the general constraints on the design of the interplanetary
backbone mandate constraints on the protocol used at the transport
layer. The TCP transport protocol used in both the backbones and the
subnets of the terrestrial Internet _ and of other planetary subnets
of the IPN _ will not be suitable: connection negotiation and in-
order stream delivery are incompatible with the enormous distances
between nodes of the interplanetary backbone.
As noted earlier, the bundle protocol residing just above the
transport layer is by nature relatively optimistic about transmission
success but it must have transport-layer-like properties, i.e., it
must be able to recover from transmission failure at lower layers.
The capacity for timeout detection and custodian-to-custodian
retransmission has to be built into it.
However, the bundle protocol's structural optimism would result in
poor performance if such a capacity were exercised frequently. That
optimism has to be justified by the general trustworthiness of lower
layers:
* When the bundle protocol runs over TCP in a deployed internet,
TCP's own retransmission regime automatically recovers from errors
in the network and link layers, and only a failure in TCP itself
will trigger retransmission at the bundle layer.
* When the bundle protocol is operating over interplanetary
distances, a similar level of trustworthiness must be provided by
a new interplanetary transport protocol.
Cerf, et al. Expires November 2001 [Page 44]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
This protocol will necessarily be very different in operation from
TCP. For example, connection time within the interplanetary backbone
will be too rare and valuable to squander in waiting for reciprocal
traffic of any kind. Consequently the basic mode of operation in the
interplanetary transport protocol will be asynchronous: data will be
issued continuously throughout each episode of connectivity, with
return traffic _ acknowledgements and coordination messages _ matched
up to the corresponding original data as it arrives. Data will
arrive at the final destination out of order (because retransmitted
data will arrive late); in many cases, even out-of-order delivery to
applications maybe preferable to waiting the minutes, hours, or even
days required for complete, error-free acquisition of data.
Fortunately, out-of-order delivery due to retransmission delay may be
made relatively rare by forward error correction at the link layer.
Only a failure in that error correction need trigger retransmission
at the transport layer.
Accurate timeout detection will be critical to the success of this
retransmission regime. Premature timeouts would result in
unnecessary retransmission, consuming precious bandwidth and
degrading link utilization; late detection of timeouts would delay
both bundle delivery and the recovery of retransmission processing
and storage resources. We believe interplanetary transmission
timeouts can be detected accurately, but only when round-trip times
can be calculated from firm operational data rather than estimated
from past experience: the extreme variability in round-trip time
introduced by intermittent connectivity means that the total round-
trip time for transmission N might be hundreds of times greater than
that for transmission N _ 1. The required operational data could in
theory be provided regardless of the topological relationship between
the Transport-layer endpoints, but we believe support for any model
other than point-to-point, direct line-of-sight transmission between
the endpoints will not scale. [Note that accurate clock
synchronization across the interplanetary backbone is as important
for implementation of timeouts at the transport layer as it is for
resource scheduling at the physical layer.]
Any number of blocks of transport-layer data traffic might
concurrently be in various stages of transmission and retransmission,
so the aggregate storage space allocated to transmission state data
and retransmission buffers might be very large. Moreover, loss of
this information due to (for example) an unplanned power cycle could
abort any number of transmissions _ unlike an unplanned reboot of a
TCP host, which will crash only the messages that are currently being
transmitted on all open streams. For these reasons, it may well be
desirable to retain interplanetary transport protocol data in non-
volatile storage rather than dynamic memory.
In some cases a set of transport protocol agents _ e.g., the set of
relay satellites orbiting a planet _ may be most useful if they can
Cerf, et al. Expires November 2001 [Page 45]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
operate in concert as a single "aggregate" entity. By distributing
the retransmission workload across multiple agents as they
successively acquire connectivity to a common peer agent, this
collective behavior might reduce the total elapsed time required to
effect reliable completion of a large block. If so, we'll
additionally need some sort of application-layer coordination
protocol operating among the members of this aggregate entity.
Flow control and congestion control are possibly the least tractable
interplanetary backbone design issues at the transport layer. TCP's
strategies are based on reciprocal message exchange that the large
delays in interplanetary communication make generally impractical.
Alternative approaches remain to be discovered.
7. Deployed Internets in the IPN
We differentiate between two types of networks in the IPN: the long
haul backbone and deployed internets. The long haul backbone,
described in the previous section, is characterized by long
propagation times, due to the great physical separation among the
communicating elements. Its protocols are designed to operate
efficiently in long-delay, intermittent-connectivity environments.
As discussed in [ref: Why not TCP], as delays increase, the
efficiency of the Internet suite steadily decreases until
applications fail altogether due to embedded time outs.
We designate a network to be a "deployed internet" if it meets the
following criteria:
* It has connectivity to an interplanetary gateway, and through that
gateway can reach the interplanetary backbone or other deployed
networks.
* It has a communication environment that doesn't inherently
preclude the use of (possibly enhanced) internet protocols.
* It uses the Domain Name space as a common means of referencing
objects and systems across deployed internets.
This designation covers a wide range of possible configurations. The
following list provides examples of deployed internets:
* A single lander that hosts an interplanetary gateway and a (real
or virtual) lander-internal network;
* A small number of cooperating robots on a planetary surface, such
as a single lander and a single rover;
* An orbiter, a lander, and a sample-return vehicle communicating
among themselves and via interplanetary gateways hosted in the
orbiter and/or the lander;
* Multiple "cells" of robots on a planet surface communicating
beyond line of sight via low-orbiting satellites that serve as
relays and as interplanetary gateways;
* Similar scenarios as above, but with one or more planet-stationary
satellites serving as interplanetary gateways;
Cerf, et al. Expires November 2001 [Page 46]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
* Spacecraft-onboard networks that communicate with remote endpoints
via interplanetary gateways;
* The earth's Internet is the initial instantiation of a "deployed
internet."
7.1. Applications of deployed internets in the IPN
It is appropriate to consider the types of activities that will
exercise the deployed internets. Clearly, we do not yet know all of
the types of applications that will be used in deployed internets,
but we do have a basic idea of some of the types of applications that
will be used in the relatively near term. In developing this notion
of applications within deployed internets, we want to eventually
develop the ability to characterize their use of the IPN in terms of
arrival rates, data volume, latency and reliability requirements,
number and location of producers, number and location of consumers
(within the IPN), etc. At this moment, we have only highly
qualitative notions of these kinds of data. Over time, we will
develop a more quantitative representation of this traffic as an
adjunct for testing. The model that we develop will probably never
accurately reflect the actual use of the IPN, because the use of the
IPN will be affected by its availability: users will adapt their
usage patterns as their understanding of the network's capabilities
develops. We don't ever expect to hit this moving target, but we
hope to come close enough to discover most of the usage-related
issues that might exist.
One of the primary uses of a deployed internet is to return science
data from the point where it is collected to the point where it will
be processed. The processing point could be on a remote internet
(say, on earth) or at some point in the local deployed internet.
Depending on the resource availability at the gathering site versus
the availability at the processing point, the transfer could be
largely unprocessed data, shipped as a formatted stream or as a file.
Alternatively, the data could be processed to reduce its transmission
size, which would likely result in a file transfer operation. Sizes,
frequency, precise reliability requirements, and so forth remain to
be determined. However, in general, this data is not particularly
time-sensitive (although the equipment may be quite sensitive to the
amount of time that it takes to complete a transfer due to power
considerations, which will be discussed later).
Another primary application, the reporting of health and status
telemetry, is typically accomplished via repetitive, unreliable
transmission in today's spacecraft. There is underway a slow
evolution toward data-summarization on board spacecraft. But there
is, understandably, some resistance to filtering or summarizing data
at the spacecraft, since in the event of a spacecraft catastrophe,
data not previously transmitted is not available for post-mortem
analysis. An important aspect of current telemetry systems is its
delivery characteristics, which are either "stream-oriented" or
periodically delivered. Mission operators perceive that "heartbeat"-
Cerf, et al. Expires November 2001 [Page 47]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
style information that is ancillary to a periodic or "continuous"
stream of telemetry is an important characteristic that will not
displace event-driven telemetry for the foreseeable future.
A third type of application is the command and control of in-situ
elements. Command and control refers to the closed-loop control of
remote systems. The endpoints of the control loop could be separated
by interplanetary space, as in the case of terrestrial commanding of
a rover. Alternatively, the endpoints could be in reasonably close
proximity, but resident on physically separate platforms connected by
wireless media. This is the case when a lander controls a rover.
These types of applications must be designed to accommodate the
delays intrinsic in the network, but the network can permit more
responsive control operations by providing qualities of service that
mitigate those delays. With command and control applications, data
volumes tend to be fairly low, and the commands tend to be followed
by responses (although in some instances, command responses are
inferred from returned telemetry, described above).
The final type of application we consider is telescience and the
development of a "virtual presence." This type of application is
intended to synthesize great volumes of information about the local
environment in order to allow earth-resident scientists, in-situ
controlling robots, or eventually in-situ astronauts to interact with
high-fidelity models of a portion of a remote environment. To an
extent, this technology has already been used to assist in planning
the Mars Pathfinder/Sojourner excursions (ref
http://www.piercorp.com/Projects/mars/mars2.htm). However, the
technology for fully exploiting this type of adjunct to exploration
is still in development. Terrestrial research into telepresence and
tele-immersion (ref:
http://www.advanced.org/tele-immersion/publications.html) may provide
insight into the workload that this type of application might
represent for the IPN.
7.2. Characteristics of remote deployed internets in the IPN
Although we consider the Earth's Internet to be the first instance of
a deployed internet, the environmental characteristics affecting
internets deployed in remote locations may differ significantly from
those affecting Earth's Internet. In examining these
characteristics, we must differentiate between two different types of
environments in the Earth's Internet: the traditional, "wired"
environments; and the emerging mobile, ad hoc wireless environments.
First and foremost among the environmental characteristics of note is
that of power availability. In the "wired" Internet on Earth, power
is cheap and plentiful. Power availability is more of an issue in
terrestrial mobile, ad hoc networks (MANETs), because the mobile
nodes operate using portable power sources, typically batteries.
However, in terrestrial MANETs, the mobile nodes eventually have
access to the same cheap and abundant sources of power that the
Cerf, et al. Expires November 2001 [Page 48]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
"wired" nodes use. This is not the case for remote deployed
internets (RDIs) in the IPN. For the foreseeable future, the primary
source of power for RDIs in the IPN will be the sun. Solar power
conversion is currently relatively inefficient, but even if dramatic
improvements in conversion efficiency occur, the fact remains that as
one moves away from the sun, the available power diminishes. In the
orbit of Mars, the average solar intensity is 590 W/m2, less than
half of the 1370 W/m2 in Earth orbit [ref:
http://powerweb.lerc.nasa.gov/pv/marspower.html ]. On the surface of
a planet, the solar intensity is by seasonal variations, by dust
build-up on solar panels, and by dust erosion of the solar panels
over time. As a result, power availability is and will be of
overriding importance to all aspects of communication in RDIs, and
will dictate a need for efficiency at all protocol layers.
The signal-to-noise ratios (SNRs) experienced in the terrestrial
wired Internet are currently very high, making loss due to data
corruption a rare experience. In terrestrial MANETs, SNRs are lower,
due to both power availability and to node density. In remote
deployed internets, SNRs will be very low, due primarily to power
availability.
In the terrestrial wired Internet, the bulk of the routing
infrastructure is fixed in terms of its location. Conversely, in
terrestrial MANETs, the infrastructure is deployed on an as-needed
basis, and may be mobile. In RDIs, this infrastructure is also
deployable and mobile, but will have a large component of satellite-
based resources.
If we examine the transmission media in use, the terrestrial wired
Internet uses primarily copper cable and optical fiber. Terrestrial
MANETs use free-space propagation in the radio frequency (RF) range
or the infrared range of the electromagnetic spectrum. In RDIs, we
anticipate that free-space RF will be the primary communication
medium, even for many permanent, immobile installations as they
develop, due to the cost of landing, deploying, and maintaining cable
or fiber.
The cost characteristics of deployment, operations, repair of RDIs is
not currently well understood. However, deployment cost of anything
that must be landed on the surface of another planetary body is quite
high. Correspondingly, repair or replacement costs of components of
the RDIs are also high. We do not yet have a clear understanding of
how the operations costs of the RDIs will compare to the operating
costs of terrestrial MANETs or of the terrestrial wired Internet.
7.3. Effects of environmental characteristics on protocols for the
IPN RDIs
In considering how to deal with the anticipated operating environment
for RDIs, we are encouraged that a significant amount of relevant
research is currently under way to support protocols for terrestrial
Cerf, et al. Expires November 2001 [Page 49]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
MANETs. This section briefly reviews some of the areas in which work
is required, and is organized by protocol layer.
At the physical layer, spectrum management is an issue that requires
coordination. There is currently no "Federal Communications
Commission" or International Telecommunications Union that regulates
use of radio-frequency spectrum outside of earth. However,
coordination of this sort is needed sooner rather than later, due to
the attractiveness of certain frequency bands as a result of their
free-space propagation loss and refractive characteristics [ref:
private communication].
The cost of landing equipment on remote planetary surfaces motivates
designers to keep as much communication infrastructure in orbit as
possible. Low-orbiting satellites will be used extensively to
support surface-to-surface communication and RDI-to-RDI
communication. Antenna design to support wideband communication
between surface-based mobile nodes and these low-orbiting satellites
is an area where research might yield great benefit. Although we
have heard of some research in these areas to support military on-
the-move applications, we have seen nothing in detail and feel that
it is an area that is ripe for additional study.
At the link layer, management of low SNRs is of significant
importance, with many areas of relevant research. While many
different coding schemes are available, it is not yet clear whether
one of these is best for all RDI use or whether the QOS requirements
of different types of traffic will dictate the use of different
coding schemes on a per-packet basis. Many codes are currently being
considered, including convolutional coding, concatenated codes (such
as Reed Solomon codes), and the emerging Turbo codes. Each of these
has different properties related to delay, residual error rate, and
link acquisition characteristics.
Resource reservation schemes at the media access level may be of
significant importance in supporting closed-loop control operations.
Some of these schemes have the benefits of providing bounded delays
and of avoiding interference, which is important in power-constrained
environments.
Link-layer status detection and signaling (to upper layers) is
important in resource-constrained environments. We believe that the
link layers must detect and signal link availability, link capacity
and congestion status, and current error conditions. As the
terrestrial cellular telephone industry becomes internet-enabled,
some of these issues are beginning to be addressed in single-hop
wireless environments. There is also ongoing research of interest in
MANETs and in DARPA-sponsored research [ref.
ftp://ftp.rooftop.com/pub/apis/link_api.pdf ].
At the network layer, there is a need for routing protocols to
support both fast- and slow-moving mobile nodes. The routing
Cerf, et al. Expires November 2001 [Page 50]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
protocols must be able to constitute and maintain networks comprised
of a combination of fixed and mobile nodes. Significant research in
this area is currently being performed, and is being coordinated by
the IETF's MANET working group [ref.
http://www.ietf.org/html.charters/manet-charter.html ].
Another area of relevant network-layer research concerns "vertical
handoffs," which allow nodes to adapt to changes in available links
or the resources available via particular links. This work [ref.
http://daedalus.cs.Berkeley.edu/publications/monet97.ps.gz ] is
focused on "last-hop" use. Its potential for use within wireless
routers requires further research.
As link layers enhance their capabilities to monitor and report their
status internally to the network layer, network layer control
protocols and algorithms must be enhanced to appropriately propagate
this information to affected parties within the RDI. This topic is
an active research area, with some applicable work having been done
within the SCPS project [ref: http://www.scps.org ].
Just as resource reservation schemes within the link layer may be
important mechanisms for providing bounded link-layer delays,
resource allocation schemes at the network layer may also be
important for providing such bounds on end-to-end communication
within the RDI. However, resource allocation and resource
reservation within mobile wireless networks is still a subject of
ongoing research. Should wireless networks use the integrated
services model of resource allocation? Or should they use the
differentiated services model? It is relatively clear that
guarantees of network capacity are not feasible in a highly dynamic
environment. However, the specification of operating ranges may be
more tractable, and may provide useful services
* To applications (by giving them feedback on available network
capacity),
* To the transport layer (by providing dynamic rate control
information), and
* To the network layer (by ensuring that resources are not over
allocated in the non-bottleneck portions of the network).
The applicability to MANETs of the current integrated services versus
differentiated services models is the subject of ongoing research
[ref: http://comet.ctr.columbia.edu/insignia/,
http://www.cs.ucla.edu/~terzis/mobile.html].
The networks that will be deployed remotely are truly without any
fixed infrastructure. Therefore, the elements of this network will
need to establish and maintain the set of services necessary to
bootstrap the network. Self-configuration has relevance here in the
areas of addresses allocation and management, name-to-address
binding, and dynamic hierarchical organization. There is ongoing
research in all of these areas that is of potential utility, ref:
http://www.ietf.org/html.charters/zeroconf-charter.html
Cerf, et al. Expires November 2001 [Page 51]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
http://www.ietf.org/html.charters/dhc-charter.html
http://www.ietf.org/html.charters/svrloc-charter.html
http://www.ietf.org/html.charters/dnsind-charter.html
http://gershwin.utdallas.edu/publications].
At the transport layer, there is a need for development of new
protocols or extensions to existing protocols that are capable of
participating in power efficient and mobility-aware communication
schemes [ref:
http://www.isi.edu/workshop/public_html/wmcw97/nsf4.html ].
These enhanced transport protocols must be able to adapt to changing
network conditions. This applies in particular to adaptation of the
protocol's behavior in order to meet application Quality of Service
requirements. Additionally, support is required for explicit
signaling of and responses to changing network conditions such as
link outages, congestion, and corruption trends.
For the foreseeable future, some links in the RDIs will exhibit
significant data rate asymmetry (on the order of 100s: 1). The
effective asymmetries may be even higher, since the low-capacity link
may not necessarily be dedicated to supporting the movement of data
across the high capacity link. Accordingly, accommodation of
significant data rate asymmetries will be required while still
maintaining a high degree of power efficiency in the presence of
errors.
At the application layer, service location in mobile ad hoc networks
is an issue that is closely related to self-configuration at the
network layer, and is of current research interest, ref:
http://www.ietf.org/html.charters/zeroconf-charter.html,
http://www.ietf.org/html.charters/svrloc-charter.html ).
This is important both in initial network startup, and also in the
(potentially frequent) case that RDIs with low connectivity become
partitioned, either expectedly or unexpectedly.
Efficient, autonomous network management and control in RDIs is an
area of interest. There is some ongoing research in the area, ref:
http://www.ietf.org/html.charters/disman-charter.html,
ftp://ftp.ece.orst.edu:/pub/users/singh/papers/anmp.ps.gz
that is of potential interest, but this is an area in which further
research is required.
Finally, monitoring the health and status of mobile nodes (not
necessarily just the networking components) is of significant
importance in RDIs. The cost to land new elements on a planetary
surface is extremely high. Therefore, we are motivated to perform
preemptive maintenance and to repair, rather than replace, failed
parts. The ability to perform these activities autonomously is an
area where a significant amount of research is required (and sounds
really fun).
Cerf, et al. Expires November 2001 [Page 52]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
7.4. Summary
The effects of power limitations in RDIs are significant and will be
present in at least some portions of the IPN for the foreseeable
future. These effects strongly suggest the likelihood of some
divergence from the standard internet suite of protocols that is in
current use on Earth. Whether these changes become the standard
internet suite on Earth, as a result of the significant increase in
the use of mobile, wireless internet nodes, remains to be determined.
It is also likely that some of the RDIs that are deployed will use a
model of organization that is substantially different than that
employed in the Internet. It may be that the IPN treats these
networks as single, distributed nodes, but further investigation is
required.
8. Working Conclusions
With the increasing pace of space exploration, Earth will distribute
large numbers of robotic vehicles, landers, and possibly even humans,
to asteroids and other planets in the coming decades. Possible
future missions include lander/rover/orbiter sets, sample return
missions, aircraft communicating with orbiters, and outposts of
humans or computers remotely operating rovers. All of these missions
involve clusters of entities in relatively close proximity
communicating with each other in challenging environments. These
clusters, in turn, will be in intermittent contact with one another,
possibly across interplanetary space. This dual-mode communications
environment: relatively cheap round-trips and more constant
connectivity when communicating with "local" elements coupled with
the very long-delay and intermittent connectivity with non-local
elements has led us to the architecture just described, with the
following working conclusions.
We need to use a "Pony-Express" model and bundles for interplanetary
communication
For communicating between IPN domains such as Earth and Mars, the
standard Internet protocols fail, both at the application and
transport layers, mainly due to the large delays. Intermittent
connectivity and bit error rate are also significant issues, but
delay cannot be mitigated by any current means. Communicating over
these distances requires a fundamentally different model that does
not assume that round-trip-times are cheap. To combat this we
propose a "pony-express" model, where senders submit bundles
describing entire transactions to the network, which then uses
custody transfers to move bundles from node to node. Under this
model the sender has little or no expectation of real-time delivery
of the bundle, which may be stored at intermediate nodes for
arbitrary periods of time. In time, the communications links
connecting the various deployed internets will grow to form a stable
interplanetary backbone network.
Cerf, et al. Expires November 2001 [Page 53]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
We can't support most legacy applications, even with application-layer
proxies
SMTP (e-mail) is perhaps the only application that could possibly be
tuned to work over interplanetary distances, but even SMTP has
embedded timers that would have to be altered significantly before it
would work.
Name tuples consisting of a routing part and an administrative part are
the means of reference
To assist in interplanetary routing, we have introduced the notion of
an IPN name tuple. These tuples, each consisting of a topology-
related routing part and an administratively controlled
administrative part are the analog of IP addresses in the terrestrial
Internet. Just as with IP addresses, bundles carry these name tuples
(source and destination) end-to-end. The routing part of an IPN name
tuple serves as an identifier of the destination internet and is
consulted to find the next bundle-hop destination whenever the bundle
is NOT in its destination IPN region. The administrative part of an
IPN name tuple serves the same function as DNS names in the
terrestrial Internet, allowing symbolic names to be used in place of
network-layer addresses. For the IPN region that includes Earth, and
possibly others, we envision using actual DNS names as the
administrative parts of the tuples.
This two-part naming approach has a number of advantages. First, it
explicitly allows routing information to be encoded in the IPN name
tuple (in the routing part) without interfering with the
administrative part. Second, since the administrative parts of name
tuples are only resolved in the destination IPN region,
administrative parts of IPN names do not have to be exchanged between
IPN regions for the purpose of routing. Third, while all IPN nodes
must be able to interpret the routing part of a name tuple, different
name-to-address binding mechanisms for the administrative part can be
used in the different regions. Finally, decoupling the
administrative parts of the various IPN regions from one another
allows autonomy of the administrative naming in different regions.
IPN Nodes will be used as "impedance-matchers" between the (relatively)
rich environment of local communications and the long-delay
interplanetary environment
We refer to the nodes that perform the store-and-forward operations
on bundles simply as interplanetary nodes, and the bundle-agents on
these nodes are responsible for routing bundles through the IPN.
We need flexible and bandwidth-efficient security (particularly
authentication and access control)
For the foreseeable future, the Interplanetary Internet will be an
expensive resource, and an irresistible lure to hackers. For these
Cerf, et al. Expires November 2001 [Page 54]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
reasons, access to the IPN will need to be strictly controlled and
IPN gateways and resources will need to authenticate commands sent to
them. In many cases, data privacy will be required by scientists to
ensure that they get first access to the data from their instruments,
and integrity will of course be required for command sequences and
some telemetry return. At the same time, the goal of the IPN is to
allow scientists, researchers, and on occasion the general public,
easy access to information from outside of Earth's domain. These
competing goals will require a flexible security scheme.
In addition to being flexible, the security mechanisms used over the
deep-space links will also need to be bit efficient. Standard
Diffie-Hellman exchanges cannot be used to generate traffic keys, as
they require the exchange of several rounds of random information in
near-real time. Something similar to secure email may be a better
approach, where the sender applies the appropriate security to the
payload of a bundle and then attaches its public key certificate to
the bundle. This approach, while costly in terms of bandwidth, will
allow the receiver to immediately authenticate/verify/decode the
contents of received bundles.
The long-haul transport protocol used to carry bundles in interplanetary
space will be very different from TCP
The transport layer that moves these bundles across interplanetary
space will necessarily be very different from TCP, the predominant
transport protocol in the terrestrial Internet. Since bandwidth is a
precious commodity in the IPN, the LTP must be "connectionless" in
that it cannot wait a round-trip to establish a connection before
beginning to send data. It is quite possible that the LTP will
provide partial data delivery where data with "holes" or errors is
delivered to the application out of order instead of in order, as
with TCP. A major challenge of the LTP is how to perform flow and
congestion control without timely feedback.
Deployed internets may or may not use internet protocols.
For the "local" communications, between nodes on a spacecraft LAN,
elements on the surface of a planet, or between elements on the
surface and in orbit, for example, round-trip times are comparable to
those in the terrestrial Internet. In these cases, it is feasible
and indeed very reasonable to use protocols like the standard
Internet Protocols (TCP/IP). Using an internet model, and perhaps
even versions of the standard internet protocols themselves, will
allow deployed devices to easily communicate with one another and to
share a common communications infrastructure. A common, shared
infrastructure will free each mission from having to design, build,
test, and operate its own custom communications system, and will pay
off in terms of development time, development cost, mass,
interoperability (reliability), and efficiency. Thus we envision the
deployments of fragments that are similar to Earth's Internet to
remote locations of interest. Current advances such as Mobile IP and
Cerf, et al. Expires November 2001 [Page 55]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
SCPS that extend the internet model to include mobile nodes and
stressed environments will be useful in these environments.
At the same time, it may be advantageous to use protocols entirely
foreign to the internet suite in an "RDI." For example, it might be
preferable in some instances to deploy a self-organizing sensor
network in which only data sets, not individual nodes, are
addressable. We believe that the naming method under consideration,
where each entity name is divided into an IPN domain part and a
domain-specific part, would support these types of RDIs. The
mechanics of how to interface such a network with the rest of the IPN
and whether/how such a network could be integrated with an RDI using
internet protocols are topics of current discussion.
Terrestrial advances in Mobile Ad Hoc networking are necessary but
possibly not sufficient for RDIs
There is a large and growing body of ongoing work in the IETF,
universities, and government and private agencies to develop
protocols for mobile ad hoc networks. While much of this work will
be directly applicable to the remotely deployed internets, RDIs will
remain a breed apart from terrestrial wireless networks. The main
differences are in power and node density. Terrestrial MANETs, while
they may be power-starved, still have the prospect of recharging at a
wall socket; elements of an RDI have no such expectation. Secondly,
the density of elements in an RDI may be low compared to a typical
terrestrial MANET (although it is too soon to know that a "typical"
terrestrial MANET will look like). The node density may be so low in
fact that real-time communications between members of the RDI may not
be possible. This will be the case if many landed elements
communicate with one another via a small number of low-altitude
orbiters, for example.
9. Security Considerations
Security is an integral concern of the Interplanetary Internet.
Section 5 of this document is devoted to examining the security
requirements in the IPN, some potential approaches for securing data
in the IPN, and some approaches for securing the backbone
infrastructure of the IPN.
Cerf, et al. Expires November 2001 [Page 56]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
10. Authors' Addresses
Dr. Vinton G. Cerf
MCI WorldCom
22001 Loudoun County Parkway
Building F2, Room 4115, ATTN: Vint Cerf
Ashburn, VA 20147
Telephone +1 (703) 886-1690
FAX +1 (703) 886-0047
Email vcerf@mci.net
Scott C. Burleigh
Jet Propulsion Laboratory
4800 Oak Grove Drive
M/S: 179-206
Pasadena, CA 91109-8099
Telephone +1 (818) 393-3353
FAX +1 (818) 354-1075
Email Scott.Burleigh@jpl.nasa.gov
Adrian J. Hooke
Jet Propulsion Laboratory
4800 Oak Grove Drive
M/S: 303-400
Pasadena, CA 91109-8099
Telephone +1 (818) 354-3063
FAX +1 (818) 393-3575
Email Adrian.Hooke@jpl.nasa.gov
Leigh Torgerson
Jet Propulsion Laboratory
4800 Oak Grove Drive
M/S: T1710-
Pasadena, CA 91109-8099
Telephone +1 (818) 393-0695
FAX +1 (818) 354-9068
Email Leigh.Torgerson@jpl.nasa.gov
Robert C. Durst
The MITRE Corporation
1820 Dolley Madison Blvd.
M/S W650
McLean, VA 22102
Telephone +1 (703) 883-7535
FAX +1 (703) 883-7142
Email durst@mitre.org
Cerf, et al. Expires November 2001 [Page 57]
Internet Draft draft-irtf-ipnrg-arch-00.txt May 2001
Dr. Keith L. Scott
The MITRE Corporation
1820 Dolley Madison Blvd.
M/S W650
McLean, VA 22102
Telephone +1 (703) 883-6547
FAX +1 (703) 883-7142
Email kscott@mitre.org
Eric J. Travis
Global Science and Technology, Inc.
6411 Ivy Lane
Suite 300
Greenbelt, MD 20770
Telephone +1 (301) 474-9696
FAX +1 (301) 474-5970
Email travis@gst.com
Howard S. Weiss
SPARTA, Inc.
9861 Broken Land Parkway
Columbia, MD 21046
Telephone +1 (410) 381-9400 x201
FAX +1 (410) 381-5559
Email hsw@columbia.sparta.com
Cerf, et al. Expires November 2001 [Page 58]
