NFSv4 Working Group J. Lentini
Internet-Draft C. Everhart
Intended status: Standards Track NetApp
Expires: January 11, 2011 D. Ellard
Raytheon BBN Technologies
R. Tewari
M. Naik
IBM Almaden
July 10, 2010
Administration Protocol for Federated Filesystems
draft-ietf-nfsv4-federated-fs-admin-05
Abstract
This document describes the administration protocol for a federated
file system that enables file access and namespace traversal across
collections of independently administered fileservers. The protocol
specifies a set of interfaces by which fileservers with different
administrators can form a fileserver federation that provides a
namespace composed of the filesystems physically hosted on and
exported by the constituent fileservers.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 11, 2011.
Copyright Notice
Lentini, et al. Expires January 11, 2011 [Page 1]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
Lentini, et al. Expires January 11, 2011 [Page 2]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Error Values . . . . . . . . . . . . . . . . . . . . . . . . . 9
4. Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.1. FedFsNsdbName Equality . . . . . . . . . . . . . . . . . . 12
5. Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.1. FEDFS_CREATE_JUNCTION . . . . . . . . . . . . . . . . . . 14
5.2. FEDFS_DELETE_JUNCTION . . . . . . . . . . . . . . . . . . 15
5.3. FEDFS_LOOKUP_JUNCTION . . . . . . . . . . . . . . . . . . 16
5.4. FEDFS_CREATE_REPLICATION . . . . . . . . . . . . . . . . . 18
5.5. FEDFS_DELETE_REPLICATION . . . . . . . . . . . . . . . . . 19
5.6. FEDFS_LOOKUP_REPLICATION . . . . . . . . . . . . . . . . . 20
5.7. FEDFS_SET_NSDB_PARAMS . . . . . . . . . . . . . . . . . . 20
5.8. FEDFS_GET_NSDB_PARAMS . . . . . . . . . . . . . . . . . . 21
5.9. FEDFS_GET_LIMITED_NSDB_PARAMS . . . . . . . . . . . . . . 21
6. Internationalization . . . . . . . . . . . . . . . . . . . . . 22
7. Security Considerations . . . . . . . . . . . . . . . . . . . 22
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
9. Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25
10.1. Normative References . . . . . . . . . . . . . . . . . . . 25
10.2. Informative References . . . . . . . . . . . . . . . . . . 26
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . . 26
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 27
Lentini, et al. Expires January 11, 2011 [Page 3]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
1. Introduction
A federated filesystem enables file access and namespace traversal in
a uniform, secure and consistent manner across multiple independent
fileservers within an enterprise (and possibly across multiple
enterprises) with reasonably good performance.
Traditionally, building a namespace that spans multiple fileservers
has been difficult for two reasons. First, the fileservers that
export pieces of the namespace are often not in the same
administrative domain. Second, there is no standard mechanism for
the fileservers to cooperatively present the namespace. Fileservers
may provide proprietary management tools and in some cases an
administrator may be able to use the proprietary tools to build a
shared namespace out of the exported filesystems. Relying on vendor-
proprietary tools does not work in larger enterprises or when
collaborating across enterprises because it is likely that the system
will contain fileservers running different software, each with their
own protocols, with no common protocol to manage the namespace or
exchange namespace information.
The requirements for federated namespaces are described in [RFC5716].
The filesystem federation protocol described in [FEDFS-NSDB] allows
fileservers from different vendors and/or with different
administrators to cooperatively build a namespace.
This document describes the protocol used by administrators to
configure the fileservers and construct the namespace.
2. Protocol
The RPC protocol used by the administration operations is ONC RPC
[RFC5531]. The data structures used for the parameters and return
values of these procedures are expressed in this document in XDR
[RFC4506].
The XDR definitions below are formatted to allow the reader to easily
extract them from the document. The reader can use the following
shell script to extract the definitions:
<CODE BEGINS>
#!/bin/sh
grep '^ *///' | sed 's?^ */// ??' | sed 's?^ *///$??'
Lentini, et al. Expires January 11, 2011 [Page 4]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
<CODE ENDS>
If the above script is stored in a file called "extract.sh", and this
document is in a file called "spec.txt", then the reader can do:
<CODE BEGINS>
sh extract.sh < spec.txt > admin1.xdr
<CODE ENDS>
The effect of the script is to remove leading white space from each
line, plus a sentinel sequence of "///".
The protocol definition in XDR notation is shown below. We begin by
defining basic constants and structures used by the protocol. We
then present the procedures defined by the protocol.
<CODE BEGINS>
/// enum FedFsStatus {
/// FEDFS_OK = 0,
/// FEDFS_ERR_ACCESS = 1,
/// FEDFS_ERR_BADCHAR = 2,
/// FEDFS_ERR_BADNAME = 3,
/// FEDFS_ERR_NAMETOOLONG = 4,
/// FEDFS_ERR_LOOP = 5,
/// FEDFS_ERR_BADXDR = 6,
/// FEDFS_ERR_EXIST = 7,
/// FEDFS_ERR_INVAL = 8,
/// FEDFS_ERR_IO = 9,
/// FEDFS_ERR_NOSPC = 10,
/// FEDFS_ERR_NOTDIR = 11,
/// FEDFS_ERR_NOTEMPTY = 12,
/// FEDFS_ERR_NOTJUNCT = 13,
/// FEDFS_ERR_NOTLOCAL = 14,
/// FEDFS_ERR_PERM = 15,
/// FEDFS_ERR_ROFS = 16,
/// FEDFS_ERR_SVRFAULT = 17,
/// FEDFS_ERR_NSDB_ROUTE = 18,
/// FEDFS_ERR_NSDB_DOWN = 19,
/// FEDFS_ERR_NSDB_CONN = 20,
/// FEDFS_ERR_NSDB_AUTH = 21,
/// FEDFS_ERR_NSDB_LDAP = 22,
/// FEDFS_ERR_NSDB_LDAP_VAL = 23,
Lentini, et al. Expires January 11, 2011 [Page 5]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
/// FEDFS_ERR_NSDB_NOFSN = 24,
/// FEDFS_ERR_NSDB_NOFSL = 25,
/// FEDFS_ERR_NSDB_RESPONSE = 26,
/// FEDFS_ERR_NSDB_FAULT = 27,
/// FEDFS_ERR_NSDB_PARAMS = 28,
/// FEDFS_ERR_PATH_TYPE_UNSUPP = 29
/// };
///
/// typedef opaque utf8string<>;
/// typedef utf8string utf8str_cs;
/// typedef utf8string utf8str_cis;
///
/// typedef opaque FedFsUuid[16];
///
/// struct FedFsNsdbName {
/// unsigned int port;
/// utf8str_cis hostname;
/// };
///
/// typedef utf8str_cs FedFsPathComponent;
/// typedef FedFsPathComponent FedFsPathName<>;
///
/// struct FedFsFsn {
/// FedFsUuid fsnUuid;
/// FedFsNsdbName nsdbName;
/// };
///
/// enum FedFsFslType {
/// FEDFS_NFS_FSL = 0
/// /* other types TBD */
/// };
///
/// struct FedFsNfsFsl {
/// FedFsUuid fslUuid;
/// unsigned int port;
/// utf8str_cis hostname;
/// FedFsPathName path;
/// };
///
/// union FedFsFsl switch(FedFsFslType type) {
/// case FEDFS_NFS_FSL:
/// FedFsNfsFsl nfsFsl;
/// };
///
/// enum FedFsPathType {
/// FEDFS_PATH_SYS = 0,
/// FEDFS_PATH_NFS = 1
/// /* other types TBD */
Lentini, et al. Expires January 11, 2011 [Page 6]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
/// };
///
/// union FedFsPath switch(FedFsPathType type) {
/// case FEDFS_PATH_SYS: /* administrative path */
/// FedFsPathName admin_path;
/// case FEDFS_PATH_NFS: /* NFS namespace path */
/// FedFsPathName nfs_path;
/// };
///
/// struct FedFsCreateJunctionArgs {
/// FedFsPath path;
/// FedFsFsn fsn;
/// };
///
/// enum FedFsResolveType {
/// FEDFS_RESOLVE_NONE = 0,
/// FEDFS_RESOLVE_CACHE = 1,
/// FEDFS_RESOLVE_NSDB = 2
/// };
///
/// struct FedFsLookupFsnArgs {
/// FedFsPath path;
/// FedFsResolveType resolve;
/// };
///
/// struct FedFsLookupFsnResOk {
/// FedFsFsn fsn;
/// FedFsFsl fsl<>;
/// };
///
/// union FedFsLookupFsnRes switch (FedFsStatus status) {
/// case FEDFS_OK:
/// FedFsLookupFsnResOk resok;
/// case FEDFS_ERR_NSDB_LDAP_VAL:
/// unsigned int ldapResultCode;
/// default:
/// void;
/// };
///
/// enum FedFsConnectionSec {
/// FEDFS_SEC_NONE = 0,
/// FEDFS_SEC_TLS = 1 /* StartTLS mechanism; RFC4513, Section 3 */
/// /* other mechanisms TBD */
/// };
///
/// union FedFsNsdbParams switch (FedFsConnectionSec secType) {
/// case FEDFS_SEC_TLS:
/// opaque secData<>;
Lentini, et al. Expires January 11, 2011 [Page 7]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
/// default:
/// void;
/// };
///
/// struct FedFsSetNsdbParamsArgs {
/// FedFsNsdbName nsdbName;
/// FedFsNsdbParams params;
/// };
///
/// union FedFsGetNsdbParamsRes switch (FedFsStatus status) {
/// case FEDFS_OK:
/// FedFsNsdbParams params;
/// default:
/// void;
/// };
///
/// union FedFsGetLimitedNsdbParamsRes switch (FedFsStatus status) {
/// case FEDFS_OK:
/// FedFsConnectionSec secType;
/// default:
/// void;
/// };
///
/// program FEDFS_PROG {
/// version FEDFS_V1 {
/// void FEDFS_NULL(void) = 0;
/// FedFsStatus FEDFS_CREATE_JUNCTION(
/// FedFsCreateJunctionArgs) = 1;
/// FedFsStatus FEDFS_DELETE_JUNCTION(
/// FedFsPathName) = 2;
/// FedFsLookupFsnRes FEDFS_LOOKUP_JUNCTION(
/// FedFsLookupFsnArgs) = 3;
/// FedFsStatus FEDFS_CREATE_REPLICATION(
/// FedFsCreateJunctionArgs) = 7;
/// FedFsStatus FEDFS_DELETE_REPLICATION(
/// FedFsPathName) = 8;
/// FedFsLookupFsnRes FEDFS_LOOKUP_REPLICATION(
/// FedFsLookupFsnArgs) = 9;
/// FedFsStatus FEDFS_SET_NSDB_PARAMS(
/// FedFsSetNsdbParamsArgs) = 4;
/// FedFsGetNsdbParamsRes FEDFS_GET_NSDB_PARAMS(
/// FedFsNsdbName) = 5;
/// FedFsGetLimitedNsdbParamsRes FEDFS_GET_LIMITED_NSDB_PARAMS(
/// FedFsNsdbName) = 6;
/// } = 1;
/// } = 100418;
<CODE ENDS>
Lentini, et al. Expires January 11, 2011 [Page 8]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
3. Error Values
The results of successful operations will consist of a status of
FEDFS_OK. The results of unsuccessful operations will begin with a
status, other than FEDFS_OK, that indicates the reason why the
operation failed.
Many of the error status names and meanings (and the prose for their
descriptions) are taken from the specification for NFSv4 [RFC3530].
Note, however, that the literal values for the status codes are
different.
Note that the status of an unsuccessful operation will generally only
indicate the first error encountered during the attempt to execute
the operation.
FEDFS_OK: No errors were encountered. The operation was a success.
FEDFS_ERR_ACCESS: Permission denied. The caller does not have the
correct permission to perform the requested operation.
FEDFS_ERR_BADCHAR: A UTF-8 string contains a character which is not
supported by the server in the context in which it being used.
FEDFS_ERR_BADNAME: A name string in a request consisted of valid
UTF-8 characters supported by the server, but the name is not
supported by the server as a valid name for the current operation.
FEDFS_ERR_NAMETOOLONG: Returned when the pathname in an operation
exceeds the server's implementation limit.
FEDFS_ERR_LOOP: Returned when too many symbolic links were
encountered in resolving pathname.
FEDFS_ERR_BADXDR: The server encountered an XDR decoding error while
processing an operation.
FEDFS_ERR_EXIST: The junction specified already exists.
FEDFS_ERR_INVAL: Invalid argument for an operation.
FEDFS_ERR_IO: A hard error occurred while processing the requested
operation.
FEDFS_ERR_NOSPC: The requested operation would have caused the
server's filesystem to exceed some limit (for example, if there is
a fixed number of junctions per fileset or per server).
Lentini, et al. Expires January 11, 2011 [Page 9]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
FEDFS_ERR_NOTDIR: The caller specified a non-directory in an
operation that requires a directory.
FEDFS_ERR_NOTEMPTY: The caller specified a directory that is not
empty as the operand of an operation that requires an empty
directory.
FEDFS_ERR_NOTJUNCT: The caller specified a path that does not end in
a junction as the operand for an operation that requires the last
component of the path to be a junction.
FEDFS_ERR_NOTLOCAL: The caller specified a path that contains a
junction in any position other than the last component.
FEDFS_ERR_PERM: The operation was not allowed because the caller is
either not a privileged user or not the owner of an object that
would be modified by the operation.
FEDFS_ERR_ROFS: A modifying operation was attempted on a read-only
filesystem.
FEDFS_ERR_SVRFAULT: An unanticipated non-protocol error occurred on
the server.
FEDFS_ERR_NSDB_ROUTE: The fileserver was unable to find a route to
the NSDB.
FEDFS_ERR_NSDB_DOWN: The fileserver determined that the NSDB was
down.
FEDFS_ERR_NSDB_CONN: The fileserver was unable to establish a
connection with the NSDB.
FEDFS_ERR_NSDB_AUTH: The fileserver was unable to authenticate and
establish a secure connection with the NSDB.
FEDFS_ERR_NSDB_LDAP: An LDAP error occurred on the connection
between the fileserver and NSDB.
FEDFS_ERR_NSDB_LDAP_VAL: Indicates the same error as
FEDFS_ERR_NSDB_LDAP, and allows the LDAP protocol error value to
be returned back to the client.
FEDFS_ERR_NSDB_NOFSN: The fileserver was unable to locate the given
FSN in the appropriate NSDB.
Lentini, et al. Expires January 11, 2011 [Page 10]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
FEDFS_ERR_NSDB_NOFSL: The fileserver was unable to locate any FSLs
for the given FSN in the appropriate NSDB.
FEDFS_ERR_NSDB_RESPONSE: The fileserver received a malformed
response from the NSDB.
FEDFS_ERR_NSDB_FAULT: An unanticipated error related to the NSDB
occurred.
FEDFS_ERR_NSDB_PARAMS: The fileserver does not have any connection
parameters on record for the specified NSDB.
FEDFS_ERR_PATH_TYPE_UNSUPP: The fileserver does not support the
specified FedFsPathType value.
4. Data Types
The basic data types defined above MUST be formatted as follows:
FedFsUuid: A universally unique identifier (UUID) as described in
[RFC4122] as a version 1 UUID. The UUID should be formatted in
network byte order.
FedFsNsdbName: A (hostname, port) pair.
The hostname is a variable length UTF-8 string that represents an
NSDB's network location in DNS name notation. The DNS name MUST
be represented using a fully qualified domain name. A system
(i.e. fileserver or administrative host) SHOULD resolve the fully
qualified domain name to a network address using the system's
standard resolution mechanisms.
The port is the NSDB transport port. The port value MUST be in
the range 0 (zero) to 65535. A value of 0 (zero) indicates that
the standard LDAP port number, 389, SHOULD be assumed.
FSNs are immutable and invariant. The attributes of an FSN,
including the fedfsNsdbName, are expected to remain constant.
Therefore, a FedFsNsdbName SHOULD NOT contain a network address,
such as an IPv4 or IPv6 address, as this would indefinitely assign
the network address.
FedFsPathComponent: A case sensitive UTF-8 string containing a
filesystem path component.
Lentini, et al. Expires January 11, 2011 [Page 11]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
FedFsPathName: A variable length array of FedFsPathComponent values
representing a filesystem path. The path's first component is
stored at the first position of the array, the second component is
stored at the second position of the array, and so on.
FedFsPath: A pathname container. The format and semantics of the
pathname are defined by the FedFsPathType value. type specific
description of a pathname.
A FEDFS_PATH_SYS is an implementation dependent administrative
pathname. For example, it could be a local file system path.
A FEDFS_PATH_NFS is a pathname in the NFSv4 server's single-server
namespace.
FedFsNsdbParams: A set of parameters for connecting to an NSDB.
Conceptually the fileserver contains a data structure that maps an
NSDB name (DNS name and port value) to these LDAP connection
parameters.
The secType field indicates the security mechanism that MUST be
used to protect all connections to the NSDB with the connection
parameters.
A value of FEDFS_SEC_NONE indicates that no security mechanism is
necessary. In this case, the secData array will have 0 (zero)
length.
A value of FEDFS_SEC_TLS indicates that the StartTLS security
mechanism [RFC4513] MUST be used to protect all connections to the
NSDB. In this case, the secData array will contain an X.509v3
certificate in binary DER format [RFC5280]. The certificate
SHOULD be used by the fileserver to authenticate the identity of
the NSDB. In particular, this certificate SHOULD be used to
validate the NSDB's TLS certificate list chain (see 7.4.2 of
[RFC5246]). The certificate could be that of a certificate
authority or a self-signed certificate.
4.1. FedFsNsdbName Equality
Two FedFsNsdbNames are considered equal if both their DNS name and
port values are the same. As described above, the standard LDAP port
number, 389, SHOULD be assumed if a port number of 0 (zero) is
specified. Therefore, the FedFsNsdbName "(nsdb.example.com, 0)" is
considered equal to "(nsdb.example.com, 389)" but not equal to
"(nsdb.example.com, 1066)" since the port number is different or
"(nsdb.foo.example.com, 389)" since the DNS name is different.
Lentini, et al. Expires January 11, 2011 [Page 12]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
5. Procedures
Fileservers that participate as "internal" nodes in the federated
namespace MUST provide these procedures:
FEDFS_NULL The null RPC, which is included, by convention, in every
ONC RPC protocol.
FEDFS_CREATE_JUNCTION Create a new junction from some location on
the server (defined as a pathname) to an FSN.
FEDFS_DELETE_JUNCTION Delete an existing junction from some location
on the server (defined as a pathname).
FEDFS_LOOKUP_JUNCTION Query the server to discover the current value
of the junction (if any) at a given path in the server namespace.
FEDFS_CREATE_REPLICATION Set an FSN representing the replication
information for the fileset containing the pathname.
FEDFS_DELETE_REPLICATION Remove the replication information for the
fileset containing the pathname.
FEDFS_LOOKUP_REPLICATION Query the server to discover the current
replication information (if any) at the given path.
FEDFS_SET_NSDB_PARAMS Set the connection parameters for the
specified NSDB.
FEDFS_GET_NSDB_PARAMS Get the connection parameters for the
specified NSDB.
FEDFS_GET_LIMITED_NSDB_PARAMS Get a limited subset of the connection
parameters for the specified NSDB.
The procedures above are described in more detail in the following
sections.
Fileservers that participate as "leaf" nodes in the namespace (i.e.,
fileservers that host filesets that are the target of junctions, but
that do not contain any junctions) are not required to implement any
of these operations.
Note that operations that modify the state of a replicated fileset
MUST result in the update of all of the replicas in a consistent
manner. Ideally all of the replicas SHOULD be updated before any
operation returns. If one or more of the replicas are unavailable,
the operation MAY succeed, but the changes MUST be applied before the
Lentini, et al. Expires January 11, 2011 [Page 13]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
unavailable replicas are brought back online. We assume that
replicas are updated via some protocol that permits state changes to
be reflected consistently across the set of replicas in such a manner
that the replicas will converge to a consistent state within a
bounded number of successful message exchanges between the servers
hosting the replicas.
5.1. FEDFS_CREATE_JUNCTION
This operation creates a junction from a server-relative path to a
(potentially) remote fileset named by the given FSN.
The junction directory on the server is identified by a pathname in
the form of an array of one or more UTF-8 path component strings. It
is not required that this path be accessible in any other manner
(e.g., to a client). This path does not appear in the federated
namespace, except by coincidence; there is no requirement that the
global namespace parallel the server namespace, nor is it required
that this path be relative to the server pseudo-root. It does not
need to be a path that is accessible via NFS (although the junction
will be of limited utility if the directory specified by the path is
not also accessible via NFS).
If the fileset is read-only, then this operation SHOULD indicate this
with a status of FEDFS_ERR_ROFS.
If the path contains an invalid UTF-8 character, then status
FEDFS_ERR_BADCHAR must be returned.
The path is REQUIRED to exist and be completely local to the server.
It MUST NOT contain a junction. If the last component of the path is
a junction (i.e., this operation is attempting to create a junction
where one already exists), then this operation MUST return the error
FEDFS_ERR_EXISTS (even if the requested junction is identical to the
current junction). If any other component of the path is a junction,
then this operation MUST fail with status FEDFS_ERR_NOTLOCAL. The
path may contain a symbolic link (if supported by the local server),
but the traversal of the path must remain within the server-local
namespace.
If any component of the path does not exist, then the operation fails
with status FEDFS_ERR_INVAL.
The server MAY enforce the local permissions on the path, including
the final component. If the path cannot be traversed because of
insufficient permissions, or the final component is an unexecutable
or unwritable directory, then the operation MAY fail with status
FEDFS_ERR_ACCESS.
Lentini, et al. Expires January 11, 2011 [Page 14]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
The operation SHOULD fail with status FEDFS_ERR_NSDB_PARAMS if the
fileserver does not have any connection parameters on record for the
specified NSDB.
The association between the path and the FSN MUST be durable before
the operation may return successfully. If the operation return codes
indicates success, then the caller may assume that the junction was
successfully created and is immediately accessible.
If successful, subsequent references via NFSv4 [RFC3530] or NFSv4.1
[RFC5661] clients to the directory that has been replaced by the
junction will result in a referral to a current location of the
target fileset [FEDFS-NSDB].
Note that the effective permissions of the directory that is
converted, by this operation, into a junction are the permissions of
the root directory of the target fileset. The original permissions
of the directory (and any other attributes it might have) are
subsumed by the junction.
Note that this operation does not create a fileset at the location
targeted by the junction. If the target fileset does not exist, the
junction will still be created. An NFS client will discover the
missing fileset when it traverses the junction.
5.2. FEDFS_DELETE_JUNCTION
This operation removes a junction specified by a server-relative
path.
As with FEDFS_CREATE_JUNCTION, the junction on the server is
identified by a pathname in the form of an array of one or more UTF-8
path component strings. It is not required that this path be
accessible in any other manner (e.g., to a client). This path does
not appear in the federated namespace, except by coincidence; there
is no requirement that the global namespace reflect the server
namespace, nor is it required that this path be relative to the
server pseudo-root. It does not need to be a path that is accessible
via NFS.
If the fileset is read-only, then this operation SHOULD indicate this
with a status of FEDFS_ERR_ROFS.
If the path contains an invalid UTF-8 character, then status
FEDFS_ERR_BADCHAR must be returned.
The path used to delete a junction might not be the same path that
was used to create the junction. If the namespace on the server has
Lentini, et al. Expires January 11, 2011 [Page 15]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
changed, then the junction may now appear at a different path than
where it was created. If there is more than one valid path to the
junction, any of them may be used.
The path is REQUIRED to exist and be completely local to the server.
It MUST NOT contain a junction, except as the final component, which
MUST be a junction. If any other component of the path is a
junction, then this operation MUST fail with status
FEDFS_ERR_NOTLOCAL. If the last component of the path is not a
junction then this operation MUST return status FEDFS_ERR_INVAL. The
path may contain a symbolic link (if supported by the local server),
but the traversal of the path must remain within the server-local
namespace.
The server MAY enforce the local permissions on the path, including
the final component. If the path cannot be traversed because of
insufficient permissions, or the parent directory of the junction is
an unexecutable or unwritable directory, then the operation MAY fail
with status FEDFS_ERR_ACCESS.
The removal of the association between the path and the FSN MUST be
durable before the operation may return successfully. If the
operation return codes indicates success, then the caller may assume
that the junction was successfully destroyed.
The effective permissions and other attributes of the directory that
is restored by this operation SHOULD be identical to their value
prior to the creation of the junction.
After removal of the junction, the fileserver MAY check if any of its
existing junctions reference the NSDB specified in the removed
junction's FSN. If the NSDB is not referenced, the fileserver MAY
delete the connection parameters of the unreferenced NSDB.
5.3. FEDFS_LOOKUP_JUNCTION
This operation queries a server to determine whether a given path
ends in a junction, and if so, the FSN to which the junction refers
and the filerserver's ability to resolve the junction.
Ordinary NFSv4 operations do not provide any general mechanism to
determine whether an object is a junction -- there is no encoding
specified by the NFSv4 protocol that can represent this information.
As with FEDFS_CREATE_JUNCTION, the pathname must be in the form of an
array of one or more UTF-8 path component strings. It is not
required that this path be accessible in any other manner (e.g., to a
client). This path does not appear in the federated namespace,
Lentini, et al. Expires January 11, 2011 [Page 16]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
except by coincidence; there is no requirement that the global
namespace reflect the server namespace, nor is it required that this
path be relative to the server pseudo-root. It does not need to be a
path that is accessible via NFS.
If the path contains an invalid UTF-8 character, then status
FEDFS_ERR_BADCHAR must be returned.
The path used to lookup a junction might not be the same path that
was used to create the junction. If the namespace on the server has
changed, then a junction may now appear at a different path than
where it was created. If there is more than one valid path to the
junction, any of them may be used.
The path is REQUIRED to exist and be completely local to the server.
It MUST NOT contain a junction, except as the final component. If
any other component of the path is a junction, then this operation
MUST fail with status FEDFS_ERR_NOTLOCAL. If the last component of
the path is not a junction then this operation MUST return the status
FEDFS_ERR_NOTJUNCT. The path may contain a symbolic link (if
supported by the local server), but the traversal of the path must
remain within the server-local namespace.
The server MAY enforce the local permissions on the path, including
the final component. If the path cannot be traversed because of
insufficient permissions, or the parent directory of the junction is
an unexecutable or unwritable directory, then the operation MAY fail
with status FEDFS_ERR_ACCESS.
If the junction exists, the resolve parameter allows for testing the
fileserver's ability to resolve the junction. If the junction does
not exist, the fileserver will ignore the resolve parameter.
If the junction exists and the resolve parameter is set to
FEDFS_RESOLVE_NONE, the fileserver MUST NOT attempt to resolve the
FSN. This will allow the administrator to obtain the junction's FSN
even if the resolution would fail. Therefore on success, the result
of a FEDFS_RESOLVE_NONE call will return a 0 (zero) length fsl list
in the FedFsLookupFsnResOk structure.
If the junction exists and the resolve parameter is set to
FEDFS_RESOLVE_CACHE, the fileserver MUST attempt to resolve the FSN
using its FSL cache, if one exists. The fileserver MUST NOT resolve
the FSN by contacting the appropriate NSDB. If the fileserver does
not have a cache or its cache does not have a mapping for the FSN in
question, the result of the operation MUST be FEDFS_OK with the
FedFsResolveRes's resolve value set to FEDFS_RESOLVE_NONE.
Lentini, et al. Expires January 11, 2011 [Page 17]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
If the junction exists and the resolve parameter is set to
FEDFS_RESOLVE_NSDB, the fileserver MUST attempt to resolve the FSN by
contacting the appropriate NSDB. The FSN MUST NOT be resolved using
cached information. The resolution MAY fail with
FEDFS_ERR_NSDB_ROUTE, FEDFS_ERR_NSDB_DOWN, FEDFS_ERR_NSDB_CONN,
FEDFS_ERR_NSDB_AUTH, FEDFS_ERR_NSDB_LDAP, FEDFS_ERR_NSDB_LDAP_VAL,
FEDFS_ERR_NSDB_NOFSN, FEDFS_ERR_NSDB_NOFSL, FEDFS_ERR_NSDB_RESPONSE,
or FEDFS_ERR_NSDB_FAULT depending on the nature of the failure.
In the case of a LDAP failure, the fileserver MAY return either
FEDFS_ERR_NSDB_LDAP or FEDFS_ERR_NSDB_LDAP_VAL. FEDFS_ERR_NSDB_LDAP
indicates that an LDAP protocol error occurred during the resolution.
FEDFS_ERR_NSDB_LDAP_VAL also indicates that an LDAP protocol error
occurred during the resolution and allows the LDAP protocol error
value to be returned in the FedFsLookupFsnRes's ldapResultCode field
(see the resultCode values in Section 4.1.9 of [RFC4511]).
If the fileserver has a cache of FSL records, the process of
resolving an FSN using an NSDB SHOULD result in the cache being
updated. New FSLs for the given FSN SHOULD be added to the cache and
deleted FSLs SHOULD be removed from the cache. This behavior is
desirable because it allows an administrator to proactively request
that the fileserver refresh its FSL cache. For example, the
administrator might like to refresh the fileserver's cache when
changes are made to an FSN's FSLs.
If the junction is resolved, the fileserver will indicate the type of
resolution that was performed using the FedFsResolveRes's resolve
value and include a list of UUIDs for the FSN's FSLs in the
FedFsResolveRes's fslUuid array.
5.4. FEDFS_CREATE_REPLICATION
This operation indicates the replication information to be returned
for a particular fileset. An NFSv4.x client may request fs_locations
or fs_locations_info at any time to detect other copies of this
fileset, and this operation supports this by supplying the FSN the
fileserver should use to respond. This FSN should be associated with
the entire fileset in which the path resides, and should be used to
satisfy fs_locations or fs_locations_info attribute requests whenever
no junction is being accessed; if a junction is being accessed, the
FSN specified by FEDFS_CREATE_JUNCTION will take precedence.
This operation differs from FEDFS_CREATE_JUNCTION in that it controls
a fileset-wide attribute not associated with a junction.
The server SHOULD permit this operation even on read-only filesets,
but MAY return FEDFS_ERR_ROFS if this is not possible.
Lentini, et al. Expires January 11, 2011 [Page 18]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
If the path contains an invalid UTF-8 character, then status
FEDFS_ERR_BADCHAR must be returned.
The path is REQUIRED to exist and be completely local to the server.
It MUST NOT contain a junction. If any component of the path is a
junction, then this operation MUST fail with status
FEDFS_ERR_NOTLOCAL. The path may contain a symbolic link (if
supported by the local server), but the traversal of the path must
remain within the server-local namespace.
The server MAY enforce the local permissions on the path, including
the final component. If the path cannot be traversed because of
insufficient permissions, or the final component is an unexecutable
or unwritable directory, then the operation MAY fail with status
FEDFS_ERR_ACCESS.
The operation SHOULD fail with status FEDFS_ERR_NSDB_PARAMS if the
fileserver does not have any connection parameters on record for the
specified NSDB.
5.5. FEDFS_DELETE_REPLICATION
This operation removes any replication information from the fileset
in which the path resides, such that NFSv4.x client requests for
fs_locations or fs_locations_info in the absence of a junction will
not be satisfied.
This operation differs from FEDFS_DELETE_JUNCTION in that it controls
a fileset-wide attribute not associated with a junction.
The server SHOULD permit this operation even on read-only filesets,
but MAY return FEDFS_ERR_ROFS if this is not possible.
If the path contains an invalid UTF-8 character, then status
FEDFS_ERR_BADCHAR must be returned.
The path is REQUIRED to exist and be completely local to the server.
It MUST NOT contain a junction. If any component of the path is a
junction, then this operation MUST fail with status
FEDFS_ERR_NOTLOCAL.
The server MAY enforce the local permissions on the path, including
the final component. If the path cannot be traversed because of
insufficient permissions, or the parent directory of the junction
unexecutable or unwritable directory, then the operation MAY fail
with status FEDFS_ERR_ACCESS.
Lentini, et al. Expires January 11, 2011 [Page 19]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
5.6. FEDFS_LOOKUP_REPLICATION
This operation queries a server to determine whether a fileset
containing the given path has replication information associated with
it, and if so, the FSN for that replication information.
This operation differs from FEDFS_LOOKUP_JUNCTION in that it inquires
about a fileset-wide attribute not associated with a junction.
If the path contains an invalid UTF-8 character, then status
FEDFS_ERR_BADCHAR must be returned.
The path is REQUIRED to exist and be completely local to the server.
It MUST NOT contain a junction. If any component of the path is a
junction, then this operation MUST fail with status
FEDFS_ERR_NOTLOCAL.
The server MAY enforce the local permissions on the path, including
the final component. If the path cannot be traversed because of
insufficient permissions, or the parent directory of the junction is
an unexecutable or unwritable directory, then the operation MAY fail
with status FEDFS_ERR_ACCESS.
Interpretation of the 'resolve' parameter shall be the same as
specified in Section 5.3 for the FEDFS_LOOKUP_JUNCTION operation.
5.7. FEDFS_SET_NSDB_PARAMS
This operations allows the administrator to set the connection
parameters for a given NSDB.
If a record for the given NSDB does not exist, a new record is
created with the specified connection parameters.
If a record for the given NSDB does exist, the existing connection
parameters are replaced with the specified connection parameters.
An NSDB is specified using a FedFsNsdbName. The rules in Section 4.1
define when two FedFsNsdbNames are considered equal
The given NSDB need not be referenced by any junctions on the
fileserver. This situation will occur when connection parameters for
a new NSDB are installed.
The format of the connection parameters is described above.
On success, this operation returns FEDFS_OK. When the operation
returns, the new connection parameters SHOULD be used for all
Lentini, et al. Expires January 11, 2011 [Page 20]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
subsequent LDAP connections to the given NSDB. Existing connections
MAY be terminated and re-established using the new connection
parameters. The connection parameters SHOULD be durable across
fileserver reboots.
On failure, an error value indicating the type of error is returned.
The operation MAY return FEDFS_ERR_ACCESS if the operation's
associated user does not have sufficient permissions to create/modify
NSDB connection parameters.
5.8. FEDFS_GET_NSDB_PARAMS
This operations allows the administrator to retrieve connection
parameters, if they exist, for the given NSDB.
An NSDB is specified using a FedFsNsdbName. The rules in Section 4.1
define when two FedFsNsdbNames are considered equal
A set of connection parameters is considered a match if their
associated NSDB is equal (as defined above) to the operation's NSDB
argument. Therefore, there is at most one set of connection
parameters that can match the query described by this operation.
The format of the connection parameters is described above.
On success, this operation returns FEDFS_OK and the connection
parameters on record for the given NSDB.
On failure, an error value indicating the type of error is returned.
This operation MAY return FEDFS_ERR_NSDB_PARAMS to indicate that
there are no connection parameters on record for the given NSDB. The
operation MAY return FEDFS_ERR_ACCESS if the operation's associated
user does not have sufficient permissions to view NSDB connection
parameters.
5.9. FEDFS_GET_LIMITED_NSDB_PARAMS
This operations allows the administrator to retrieve a limited subset
of information on the connection parameters, if they exist, for the
given NSDB.
An NSDB is specified using a FedFsNsdbName. The rules in Section 4.1
define when two FedFsNsdbNames are considered equal
A set of connection parameters is considered a match if their
associated NSDB is equal (as defined above) to the operation's NSDB
argument. Therefore, there is at most one set of connection
parameters that can match the query described by this operation.
Lentini, et al. Expires January 11, 2011 [Page 21]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
This operation returns a limited subset of the connection parameters.
Only the FedFsConnectionSec mechanism that is used to protect
communication between the fileserver and NSDB is returned.
Viewing the limited subset of NSDB connection parameters returned by
FEDFS_GET_LIMITED_NSDB_PARAMS MAY be a less privileged operation than
viewing the entire set of NSDB connection parameters returned by
FEDFS_GET_NSDB_PARAMS. For example, the full contents of an NSDB's
connection parameters could contain sensitive information for some
security mechanisms. FEDFS_GET_LIMITED_NSDB_PARAMS allows the
fileserver to communicate a subset of the connection parameters (the
security mechanism) to users with sufficient permissions without
revealing more sensitive information.
On success, this operation returns FEDFS_OK and the
FedFsConnectionSec value on record for the given NSDB.
On failure, an error value indicating the type of error is returned.
This operation MAY return FEDFS_ERR_NSDB_PARAMS to indicate that
there are no connection parameters on record for the given NSDB. The
operation MAY return FEDFS_ERR_ACCESS if the operation's associated
user does not have sufficient permissions to view the subset of NSDB
connection parameters returned by this procedure.
6. Internationalization
TBD. All UTF-8 strings and DNS names will conform to the rules
defined by rfc3530bis.
7. Security Considerations
The ONC RPC protocol supports authentication, integrity and privacy
via the RPCSEC_GSS framework [RFC2203]. Fileservers which support
the FedFS administration protocol described above MUST support
RPCSEC_GSS.
8. IANA Considerations
A range of ONC RPC program numbers were assigned for use by FedFS
using the procedure described in Section 7.3 "Program Number
Assignment" of [RFC5531]. The FedFS range is:
Lentini, et al. Expires January 11, 2011 [Page 22]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
IETF NFSv4 Working Group - FedFS 100418 - 100421
This document describes version 1 of the ONC RPC program 100418.
9. Glossary
Administrator: user with the necessary authority to initiate
administrative tasks on one or more servers.
Admin Entity: A server or agent that administers a collection of
fileservers and persistently stores the namespace information.
Client: Any client that accesses the fileserver data using a
supported filesystem access protocol.
Federation: A set of server collections and singleton servers that
use a common set of interfaces and protocols in order to provide
to their clients a federated namespace accessible through a
filesystem access protocol.
Fileserver: A server exporting a filesystem via a network filesystem
access protocol.
Fileset: The abstraction of a set of files and the directory tree
that contains them. A fileset is the fundamental unit of data
management in the federation.
Note that all files within a fileset are descendants of one
directory, and that filesets do not span filesystems.
Filesystem: A self-contained unit of export for a fileserver, and
the mechanism used to implement filesets. The fileset does not
need to be rooted at the root of the filesystem, nor at the export
point for the filesystem.
A single filesystem MAY implement more than one fileset, if the
client protocol and the fileserver permit this.
Filesystem Access Protocol: A network filesystem access protocol
such as NFSv2 [RFC1094], NFSv3 [RFC1813], NFSv4 [RFC3530], or CIFS
(Common Internet File System) [MS-SMB] [MS-SMB2] [MS-CIFS].
FSL (Fileset Location): The location of the implementation of a
fileset at a particular moment in time. An FSL MUST be something
that can be translated into a protocol-specific description of a
resource that a client can access directly, such as an fs_location
(for NFSv4), or share name (for CIFS). Note that not all FSLs
Lentini, et al. Expires January 11, 2011 [Page 23]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
need to be explicitly exported as long as they are contained
within an exported path on the fileserver.
FSN (Fileset Name): A platform-independent and globally unique name
for a fileset. Two FSLs that implement replicas of the same
fileset MUST have the same FSN, and if a fileset is migrated from
one location to another, the FSN of that fileset MUST remain the
same.
Junction: A filesystem object used to link a directory name in the
current fileset with an object within another fileset. The
server-side "link" from a leaf node in one fileset to the root of
another fileset.
Namespace: A filename/directory tree that a sufficiently authorized
client can observe.
NSDB (Namespace Database) Service: A service that maps FSNs to FSLs.
The NSDB may also be used to store other information, such as
annotations for these mappings and their components.
NSDB Node: The name or location of a server that implements part of
the NSDB service and is responsible for keeping track of the FSLs
(and related info) that implement a given partition of the FSNs.
Referral: A server response to a client access that directs the
client to evaluate the current object as a reference to an object
at a different location (specified by an FSL) in another fileset,
and possibly hosted on another fileserver. The client re-attempts
the access to the object at the new location.
Replica: A replica is a redundant implementation of a fileset. Each
replica shares the same FSN, but has a different FSL.
Replicas may be used to increase availability or performance.
Updates to replicas of the same fileset MUST appear to occur in
the same order, and therefore each replica is self-consistent at
any moment.
We do not assume that updates to each replica occur
simultaneously. If a replica is offline or unreachable, the other
replicas may be updated.
Server Collection: A set of fileservers administered as a unit. A
server collection may be administered with vendor-specific
software.
The namespace provided by a server collection could be part of the
Lentini, et al. Expires January 11, 2011 [Page 24]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
federated namespace.
Singleton Server: A server collection containing only one server; a
stand-alone fileserver.
10. References
10.1. Normative References
[RFC1094] Nowicki, B., "NFS: Network File System Protocol
specification", RFC 1094, March 1989.
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
Version 3 Protocol Specification", RFC 1813, June 1995.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2203] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol
Specification", RFC 2203, September 1997.
[RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally
Unique IDentifier (UUID) URN Namespace", RFC 4122,
July 2005.
[RFC4506] Eisler, M., "XDR: External Data Representation Standard",
STD 67, RFC 4506, May 2006.
[RFC4511] Sermersheim, J., "Lightweight Directory Access Protocol
(LDAP): The Protocol", RFC 4511, June 2006.
[RFC4513] Harrison, R., "Lightweight Directory Access Protocol
(LDAP): Authentication Methods and Security Mechanisms",
RFC 4513, June 2006.
[RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security
(TLS) Protocol Version 1.2", RFC 5246, August 2008.
[RFC5280] Cooper, D., Santesson, S., Farrell, S., Boeyen, S.,
Housley, R., and W. Polk, "Internet X.509 Public Key
Infrastructure Certificate and Certificate Revocation List
(CRL) Profile", RFC 5280, May 2008.
[RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol
Specification Version 2", RFC 5531, May 2009.
Lentini, et al. Expires January 11, 2011 [Page 25]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
10.2. Informative References
[FEDFS-NSDB]
Lentini, J., Everhart, C., Ellard, D., Tewari, R., and M.
Naik, "NSDB Protocol for Federated Filesystems",
draft-ietf-nfsv4-federated-fs-protocol (Work In Progress),
2010.
[MS-CIFS] Microsoft Corporation, "Common Internet File System (CIFS)
Protocol Specification", MS-CIFS 2.0, November 2009.
[MS-SMB] Microsoft Corporation, "Server Message Block (SMB)
Protocol Specification", MS-SMB 17.0, November 2009.
[MS-SMB2] Microsoft Corporation, "Server Message Block (SMB) Version
2 Protocol Specification", MS-SMB2 19.0, November 2009.
[RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "Network File System
(NFS) version 4 Protocol", RFC 3530, April 2003.
[RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File
System (NFS) Version 4 Minor Version 1 Protocol",
RFC 5661, January 2010.
[RFC5662] Shepler, S., Eisler, M., and D. Noveck, "Network File
System (NFS) Version 4 Minor Version 1 External Data
Representation Standard (XDR) Description", RFC 5662,
January 2010.
[RFC5716] Lentini, J., Everhart, C., Ellard, D., Tewari, R., and M.
Naik, "Requirements for Federated File Systems", RFC 5716,
January 2010.
Appendix A. Acknowledgments
We would like to thank Robert Thurlow of Sun Microsystems for helping
to author this document, including drafting the replication procedure
text.
We would also like to thank Paul Lemahieu of EMC and Mario Wurzl of
EMC for helping to author this document.
We would also like to thank Trond Myklebust for suggesting
improvements to the FSL pathname format, Chuck Lever for suggesting
improvements to the XDR type definitions and error codes, and Nicolas
Williams for his suggestions.
Lentini, et al. Expires January 11, 2011 [Page 26]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
The extract.sh shell script and formatting conventions were first
described by the authors of the NFSv4.1 XDR specification [RFC5662].
Authors' Addresses
James Lentini
NetApp
1601 Trapelo Rd, Suite 16
Waltham, MA 02451
US
Phone: +1 781-768-5359
Email: jlentini@netapp.com
Craig Everhart
NetApp
7301 Kit Creek Rd
Research Triangle Park, NC 27709
US
Phone: +1 919-476-5320
Email: everhart@netapp.com
Daniel Ellard
Raytheon BBN Technologies
10 Moulton Street
Cambridge, MA 02138
US
Phone: +1 617-873-8000
Email: dellard@bbn.com
Renu Tewari
IBM Almaden
650 Harry Rd
San Jose, CA 95120
US
Email: tewarir@us.ibm.com
Lentini, et al. Expires January 11, 2011 [Page 27]
Internet-Draft Admin Protocol for Federated Filesystems July 2010
Manoj Naik
IBM Almaden
650 Harry Rd
San Jose, CA 95120
US
Email: manoj@almaden.ibm.com
Lentini, et al. Expires January 11, 2011 [Page 28]