INTERNET-DRAFT                                           David Noveck
Expires: January 2006                         Network Appliance, Inc.
                                                    Rodney C. Burnett
                                                            IBM, Inc.

                                                            July 2005


              Implementation Guide for Referrals in NFSv4
                   draft-ietf-nfsv4-referrals-00.txt


Status of this Memo

     By submitting this Internet-Draft, each author represents
     that any applicable patent or other IPR claims of which he
     or she is aware have been or will be disclosed, and any of
     which he or she becomes aware will be disclosed, in
     accordance with Section 6 of BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF), its areas, and its working groups.  Note that
     other groups may also distribute working documents as Internet-
     Drafts.

     Internet-Drafts are draft documents valid for a maximum of six
     months and may be updated, replaced, or obsoleted by other
     documents at any time.  It is inappropriate to use Internet-Drafts
     as reference material or to cite them other than as "work in
     progress."

     The list of current Internet-Drafts can be accessed at
         http://www.ietf.org/ietf/1id-abstracts.txt The list of
     Internet-Draft Shadow Directories can be accessed at
         http://www.ietf.org/shadow.html.

Copyright Notice

     Copyright (C) The Internet Society (2005).  All Rights Reserved.













Noveck, Burnett           Expires January 2006                  [Page 1]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


Abstract

     RFC3530 describes a migration feature using the NFS4ERR_MOVED error
     code and the fs_locations attribute.  The description focuses on
     the case of migration (i.e. relocation) of a file system already
     known to the client.  The simpler limiting case in a which a file
     system not previously known to the client was located elsewhere,
     which we here call a referral, was not clearly described.  Because
     of this and also because of some inconsistencies and ambiguities in
     the text of RFC3530, there has been confusion about how the client
     and server should interact in performing a referral.  This document
     provides a guide to the implementation of referrals, and in so
     doing, addresses the relevant problems in RFC3530.

Table Of Contents

     1.   Introduction . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.   Referrals as a Limiting Case of Migrations . . . . . . .   3
     1.2.   Pure Referrals . . . . . . . . . . . . . . . . . . . . .   3
     2.   Issues as to When NFS4ERR_MOVED May/Should be Returned . .   4
     2.1.   PUTFH Returning NFS4ERR_MOVED  . . . . . . . . . . . . .   5
     2.2.   GETFH Returning NFS4ERR_MOVED  . . . . . . . . . . . . .   6
     2.3.   GETATTR Returning NFS4ERR_MOVED  . . . . . . . . . . . .   6
     2.4.   Ops Not Allowed to Return NFS4ERR_MOVED  . . . . . . . .   7
     3.   Attribute Issues . . . . . . . . . . . . . . . . . . . . .   7
     3.1.   Number of fs_locations . . . . . . . . . . . . . . . . .   7
     3.2.   General Issues of Incomplete Attribute Sets  . . . . . .   8
     3.3.   Attributes Needed for Root of an Absent Fs . . . . . . .  11
     3.4.   Attributes Valid for Root of an Absent Fs  . . . . . . .  13
     3.5.   Attributes Not Valid for Root of an Absent Fs  . . . . .  14
     4.   Referral Examples  . . . . . . . . . . . . . . . . . . . .  15
     4.1.   Referral Example (LOOKUP)  . . . . . . . . . . . . . . .  15
     4.2.   Referral Example (READDIR) . . . . . . . . . . . . . . .  20
     5.   Additional Client-side Considerations  . . . . . . . . . .  22
          Acknowledgements . . . . . . . . . . . . . . . . . . . . .  23
          Normative References . . . . . . . . . . . . . . . . . . .  23
          Authors' Addresses . . . . . . . . . . . . . . . . . . . .  23
          Full Copyright Statement . . . . . . . . . . . . . . . . .  23


1.  Introduction

     RFC3530 describes a migration feature using the NFS4ERR_MOVED error
     code and the fs_locations attribute.  The description focuses on
     the case of migration of a file system already known to the client,
     while the alternate case in a which a file system not previously
     known to the client was located elsewhere, which we here call a
     referral, was not clearly described.



Noveck, Burnett           Expires January 2006                  [Page 2]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     Analysis of how this case would be dealt with has shown that the
     protocol in RFC3530 is capable of dealing properly with this sub-
     case.  However, there are difficulties for implementers in
     determining how precisely this is to be done, because the text of
     RFC3530 does not address this case.  In addition, there are a
     number of ambiguities and inconsistencies in the text, that
     increase the difficulties for implementers and that raise the
     probability that clients and servers may not properly interact.

     This document is an attempt to deal with these spec issues and to
     provide a clear guide to follow in developing inter-operable
     referral implementations.

     This document may deal with some aspects of issues related to the
     migration case where the data actually relocated from one server to
     another while under client access.  For example, clarification of
     apparent contradictions in the spec may relate to both referrals
     and the general migration case.  However, the focus of this
     document is on the case of referrals and other aspects of migration
     will only be dealt with as necessary to accomplish that main
     purpose.

1.1.  Referrals as a Limiting Case of Migrations

     If a server implements file system migration, individual clients,
     not being synchronized with the migration event, will encounter
     different situations from the client point of view.  That is, even
     though some clients will have received filehandles within the
     migrated filesystem, others may at that time be first referencing
     the root of the migrated filesystem and need to be redirected to
     the new fs location, just as those clients that have already
     accessed the filesystem (when it was present) need to be
     redirected.

     Thus referrals are a limiting sub-case of migration and when a
     migration event occurs some clients will see an ordinary migration
     in the case in which access (by that client) to the filesystem
     being moved has already occurred, while others will see a referral
     when they first attempt to access the absent filesystem.

1.2.  Pure Referrals

     Given that clients supporting migration need to be prepared for a
     migration event, a server can establish a situation, which we refer
     to as a pure referral, in which all clients see a referral.  Since
     no client can enter the absent filesystem, the presumptive
     migration of the absent filesystem, assumes a purely conventional
     character, in that for each client it appears to have happened



Noveck, Burnett           Expires January 2006                  [Page 3]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     before all access by that client.  This allows the server to
     establish a situation in which referral to an absent filesystem may
     occur for filesystems that were never present on the server.

     This functionality may be used to allow construction of multi-
     server namespaces and may serve as a building block for the
     construction of wider global namespaces.

2.  Issues as to When NFS4ERR_MOVED May/Should be Returned

     RFC3530 contains the following statement in section 6.2: "The
     NFS4ERR_MOVED error is returned for all operations except PUTFH and
     GETATTR.", which is contradicted by the error lists in the detailed
     operation descriptions.  PUTFH and GETATTR, while the statement
     above suggests  otherwise, have NFS4ERR_MOVED listed as error
     codes, whereas there are six other ops (PUTROOTFH, PUTPUBFH, RENEW,
     SETCLIENTID, SETCLIENTID_CONFIRM, RELEASE_OWNER) which are not
     allowed to return NFS4ERR_MOVED despite the "all operations except"
     clause given above.

     The individual ops will be discussed below but in order to
     understand how these contradictions arose and how they are to be
     dealt with, it is important to be clear on what point the current
     filesystem (i.e. the filesystem associated with the current
     filehandle) is to be tested for absence in deciding whether to
     return NFS4ERR_MOVED.  For operations which change the current
     filehandle, (i.e. PUTFH, PUTROOTFH, PUTPUBFH, RESTOREFH, LOOKUP,
     LOOKUPP), this distinction is critical.  The spec is, however, not
     very clear on this issue so we need to consider the alternatives.

     Let's first consider a check on the current filehandle after the
     operation.  In part, that seems dubious because it returns an error
     for an operation where the circumstances that give rise to the
     error (i.e. changing the filehandle) depend on the successful
     execution of the operation giving rise to the error.  However, the
     main reason such an interpretation is insupportable is that it
     would make some situations, wherein fs_locations is needed,
     impossible to deal with.  For example, LOOKUP followed by
     GETATTR(fs_locations) would not be possible if the LOOKUP returned
     NFS4ERR_MOVED when the current filehandle at the start of the
     LOOKUP was in a present filesystem while the current filehandle at
     the end of the LOOKUP was at the root of an absent filesystem.  An
     NFS4ERR_MOVED returned by LOOKUP would not allow execution of the
     GETATTR, any exception for GETATTR of fs_locations notwithstanding.

     On the other hand, making the check at the start of each operation
     (except some GETATTR's) allows fs_locations to be determined
     whenever necessary and is the best way to resolve the issue.



Noveck, Burnett           Expires January 2006                  [Page 4]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     Except where RFC3530 has made this impossible by not listing
     NFS4ERR_MOVED as an error code for certain ops, each op should
     check at the beginning of the operation for a current filehandle
     within an absent fs.  If so, NFS4ERR_MOVED is returned.  A partial
     exception is GETATTR for which the return of NFS4ERR_MOVED (or not)
     is dependent on the set of attributes requested.  This is discussed
     further below.

2.1.  PUTFH Returning NFS4ERR_MOVED

     As noted above, section 6.2 indicates that PUTFH will not return
     NFS4ERR_MOVED while the detailed description for PUTFH (section
     14.2.20), lists NFS4ERR_MOVED as a valid return.  How are we to
     resolve the contradiction?

     It would seem that the exception for PUTFH in section 6.2, is
     unnecessary and derives from a confusion about the logic of
     NFS4ERR_MOVED.  Clearly the sequence of PUTFH(fh within absent
     filesystem) followed by GETATTR(fs_locations) must be allowed but
     it requires no specific exception for PUTFH to do so, since only
     GETATTR, and not PUTFH, has a filehandle within an absent
     filesystem as the current handle at the start of the operation.
     Thus no exception for PUTFH is required, as is recognized by
     section 14.2.20.

     PUTFH will never return NFS4ERR_MOVED because the filehandle it is
     establishing belongs to an absent filesystem.  This is because the
     current filehandle/filesystem is checked for absence at the start
     of each op and the new filehandle for PUTFH has not been
     established at the start of the PUTFH.  This allows the sequence
     PUTFH-GETATTR(fs_locations) to proceed without any specific
     exception for PUTFH or any analysis of the op stream that
     encompasses multiple ops (e.g. looking ahead to the GETATTR).

     NFS4ERR_MOVED can happen as a result of a PUTFH, but only in rather
     odd circumstances such as the following:

     o    PUTFH (fh within absent filesystem)

     o    PUTFH (fh within present filesystem)

     In this case, NFS4ERR_MOVED would be returned on the second PUTFH
     since the current filehandle at the start of that op is within an
     absent filesystem.  It may seem odd that the PUTFH which is
     establishing a current filehandle within a present filesystem
     should get an error while the one that established a current
     filehandle within an absent filesystem does not but this behavior
     is a consequence of a uniform set of rules for NFS4ERR_MOVED, the



Noveck, Burnett           Expires January 2006                  [Page 5]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     basis of which is that the current filehandle is tested at the
     start of an operation.

2.2.  GETFH Returning NFS4ERR_MOVED

     While RFC 3530 does not make any exception for GETFH when the
     current filehandle is within an absent filesystem, the fact that
     GETFH is such a passive, purely interrogative operation, may lead
     readers to wrongly suppose that an NFSERR_MOVED error will not
     arise in this situation.

     In fact, GETFH will result in NFS4ERR_MOVED being returned if the
     current file handle is for an absent filesystem.  This is
     particularly helpful in dealing with pure referral situations since
     a filehandle that a client does not ever see can pose no
     difficulties.  For example, it is irrelevant whether the target
     filesystem has persistent or volatile filehandles.  An inherently
     unobserved filehandle poses no expiration difficulties regardless
     of whether the target filesystem (on the target server) has
     persistent or volatile filehandles.  Servers should never return
     filehandles within absent filesystems for pure referral cases.

2.3.  GETATTR Returning NFS4ERR_MOVED

     As noted above, Section 6.2 indicates that NFS4ERR_MOVED is not
     returned for a GETATTR operation, but NFS4ERR_MOVED is listed as an
     error that can be returned by GETATTR (in section 14.2.7).  Since
     the purpose of the exception for GETATTR is to allow fs_locations
     to be fetched, the best resolution of this contradiction is for
     NFS4ERR_MOVED to be returned by GETATTR's that ask for some
     unavailable attribute and that do not interrogate the fs_locations
     attribute.  This maintains the exception which allows GETATTR to be
     used to get fs_locations information by establishing the rule that
     GETATTR's which interrogate fs_locations (with or without
     additional attributes) will not return NFS4ERR_MOVED.  Neither will
     GETATTR's that only request attributes that are available.  These
     considerations are further discussed in section 3.

     Sometimes clients need a quickly checked indication of whether a
     filesystem is absent or not.  In most cases, because fs_locations
     is not requested, this indication will be the NFS4ERR_MOVED being
     returned by the GETATTR.  When fs_locations is requested, a
     convenient way to provide this indication is to request the
     filehandle attribute.  Since this attribute will not be provided
     for referrals (see section 3.4.2), examining the mask of returned
     attributes for the presence of the filehandle attribute is quick
     way to determine whether the directory in question is part of an
     absent filesystem.



Noveck, Burnett           Expires January 2006                  [Page 6]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


2.4.  Ops Not Allowed to Return NFS4ERR_MOVED

     As noted above, despite the fact that section 6.2 suggests
     otherwise, a number of ops, all of which do not require a current
     filehandle, are not listed as returning NFS4ERR_MOVED.  These ops
     are PUTROOTFH, PUTPUBFH, RENEW, SETCLIENTID, and RELEASE_LOCKOWNER.
     Note also that DELEGPURGE, another op which does not require a
     current filehandle, is listed as able to return NFS4ERR_MOVED.

     Although this contradiction could be resolved by changing the op
     description to match 6.2, it does not seem that there is any strong
     reason to do so.  The protocol will function adequately, if these
     are treated as exceptions, and NFS4ERR_MOVED is returned by the
     next op allowed to do so, as long as the current filehandle at the
     start of that op is one within an absent fs.  Note that PUTROOTFH
     and PUTPUBFH could cause a pending absent fs indication not to
     result in an error code, since they change the current filehandle
     to be within a new, presumably present, filesystem.  However, in
     this case the client would not have issued any operations prior to
     the PUTROOTFH or PUTPUBFH that actually operated on the absent fs.

3.  Attribute Issues

     RFC3530 anticipates that clients will request additional attributes
     beyond fs_locations (see section 6.2 of that document) and allows
     the server to return fs_locations only.  The return of other
     attributes, implicit in the statement "server may return
     fs_locations only" is not clearly dealt with.  While some of these,
     such as fsid, are important for all forms of migration, this
     omission is most critical in the case of referrals.  There are a
     number of attributes besides fs_locations that need to be provided
     at the root of an absent filesystem for server and client to
     properly collaborate to perform a referral.

     There are also a large number of other attributes which the server
     should not provide for absent filesystem since providing them is
     the province of the referral's target server.  Providing possibly
     conflicting values on the server that is the source of the referral
     can only contribute to confusion.

3.1.  Number of fs_locations

     It should be noted that with referrals, it is valid for an
     fs_locations response to describe multiple locations if the target
     of the referral is a replicated filesystem. This allows the client
     to find one of the replicas when one or more of the supplied
     locations are unavailable. Once the client establishes contact with




Noveck, Burnett           Expires January 2006                  [Page 7]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     a target server, it will then obtain fs_locations as appropriate
     from the server where the referral target data resides.

     Although it is valid for fs_locations to specify more than one
     location in the case of referral, it must specify at least one.
     This is in contrast to the case in which fs_locations is fetched
     for a filesystem which is present in order to determine replica
     locations in the case of failure.  In that case, it is valid for a
     server to return an empty fs_locations attribute, since the client
     already has a valid location for the server and is only requesting
     additional locations and when there aren't any, an empty
     fs_locations gives him exactly the information being sought.  When
     handling the returned attribute, the client should consider the
     replying server as the implied last entry in the list when it is
     not present in the returned fs_locations.

     In the referral case, the client is getting NFS4ERR_MOVED and has
     no current valid location for the filesystem and thus at least one
     location is required.  In other cases, although a server may
     include his own location for the filesystem in the fs_locations
     attribute, he is not required to do so.

     When the server validly returns an fs_locations which has a null
     array of locations, it may choose to return an empty (zero
     component) fs_root since the only purpose of fs_root is to aid in
     the interpretation of the associated locations in the fs_locations
     attribute.

3.2.  General Issues of Incomplete Attribute Sets

     Migration or referral events naturally create situations in which
     all of the attributes normally supported on a server are not
     obtainable.  RFC3530 is in places ambivalent and/or apparently
     self-contradictory on such issues.

     The first problem concerns the statement in the third paragraph of
     section 6.2: "If the client requests more attributes than just
     fs_locations, the server may return fs_locations only.  This is to
     be expected since the server has migrated the filesystem and may
     not have a method of obtaining additional attribute data."

     While the above seems quite reasonable, it is seemingly
     contradicted by the following text from section 14.2.7 in the
     second paragraph of the DESCRIPTION for GETATTR: "The server must
     return a value for each attribute that the client requests if the
     attribute is supported by the server.  If the server does not
     support an attribute or cannot approximate a useful value then it
     must not return the attribute value and must not set the attribute



Noveck, Burnett           Expires January 2006                  [Page 8]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     bit in the result bitmap.  The server must return an error if it
     supports an attribute but cannot obtain its value.  In that case no
     attribute values will be returned."

     The above is a helpful restriction in that it allows clients to
     simplify their attribute interpretation code.  It allows them to
     assume that all of the attributes they request are present, often
     making it possible to get successive attributes at fixed offsets
     within the data stream.  However, it seems to contradict what is
     said in section 6.2, where it is clearly anticipated, at least when
     fs_locations is requested, that fewer (often many fewer) attributes
     will be available than are requested.  It could be argued that you
     could harmonize these two by being creative with the interpretation
     of the phrase "if the attribute is supported by the server".  One
     could argue that many attributes are not supported by the server
     for an absent fs even though the text talking about attributes
     "supported by a server" seems to indicate that this is not allowed
     to be different for different fs's (which is troublesome in itself
     as one server might have some filesystems that do support ACLs and
     some that don't support then, for example).

     Note however that the following paragraph in the description says,
     "All servers must support the mandatory attributes as specified in
     the section 'File Attributes'".  That's reasonable enough in
     general, but for an absent fs it is not reasonable and so section
     14.2.7 and section 6.2 are contradictory.

     The text in section 14.2.7 would create great obstacles in the
     implementation of migration given section 6.2.  In order to get the
     additional attributes mentioned, the client would have to guess at
     the set to be provided and, if that guess were not valid, get an
     error, presumably NFS4ERR_MOVED, returned.  For the purposes of
     this document, we will treat the specific statement regarding
     fs_locations in section 6.2 as controlling.  The contradictory text
     in section 14.2.7 will be treated as an overgeneralization to which
     an exception, for the case of migration and referrals, must be
     understood.

     When fs_locations is requested, then a subset of the requested
     attributes may be provided by the server without generating the
     error NFS4ERR_MOVED.  The subset returned must always include
     fs_locations.  In addition, if all of the attributes requested can
     be provided, then NFS4ERR_MOVED will also not be returned but in
     this case, a subset of the requested attributes may not be
     provided.  The following sections give guidance on the attributes
     the server should provide for filehandles within an absent
     filesystem.




Noveck, Burnett           Expires January 2006                  [Page 9]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     A related issue concerns attributes in a READDIR.  There has been
     discussion, not yet fully resolved, on the working group list about
     whether all the attributes specified in the attribute mask for
     READDIR must be provided for all readdir entries when those
     attributes are supported by the associated filesystems.  It can be
     argued that that would be implied if one combined the words of
     section 14.2.7 with text that (even though somewhat more loosely)
     implies that the attributes in READDIR are to be provided via
     GETATTR.  It has also been argued that the role of READDIR with
     attributes as a replacement for v3 READDIRPLUS makes it more
     appropriate to treat the attribute mask parameters as a hint with
     the server allowed to omit any attributes requested (possibly with
     some exceptions) whenever inconvenient.  In the discussion, there
     have also been occasional suggestions that an attribute's status
     (as mandatory or recommended) should have role in deciding whether
     servers are obliged to provide them when requested for READDIR.

     Regardless of how this debate is to be resolved in the general
     case, it is clear in the case of an absent filesystem, if there is
     a general obligation to provide all requested attributes, that an
     exception must be made for directory entries that represent the
     root of an absent fs (a referral).  For these entries, the full set
     of requested attributes may simply not be available, as they
     likewise would not be available for GETATTR.

     Further, we will assume in our examples, that for the few
     attributes necessary to perform the referral (see below for
     details), the server will always provide these when requested as it
     should be easy for the server to do.

     So in this document, we will assume that regardless of the general
     resolution of this issue, in the case of an absent filesystem some
     flexibility must be provided, even if it is determined that making
     the attribute mask for READDIR a hint in general is not justified.

     So we will assume that for READDIR:

     o    When fs_locations is among the attributes requested, the
          server may provide a subset of the other requested attributes
          together with fs_locations for roots of absent fs's, without
          causing any error for the READDIR as a whole.  If rdattr_error
          is also requested and there are attributes which are not
          available, then rdattr_error will receive the value
          NFS4ERR_MOVED.

     o    When fs_locations is not requested, but all of the attributes
          requested can be provided, then they will be provided and no




Noveck, Burnett           Expires January 2006                 [Page 10]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


          NFS4ERR_MOVED will be generated. An example would be READDIR's
          that request mounted_on_fileid either with or without fsid.

     o    When fs_locations is not requested, but rdattr_error is and
          some attributes requested are not available because of the
          absence of the filesystem, the server will return
          NFS4ERR_MOVED for the rdattr_error attribute and, in addition,
          the requested attributes that are valid for the root of an
          absent filesystem.

     o    When neither fs_locations nor rdattr_error is requested and
          there is a directory within an absent fs within the directory
          being read, if some unavailable attributes are requested, no
          data will be returned and the READDIR will get an
          NFS4ERR_MOVED error.  (The examples will suppose this but
          clients will not depend on this behavior and clients should
          work with a more liberal interpretation if this is decided on.

3.3.  Attributes Needed for Root of an Absent Fs

     The following options need to be provided for referrals.  While it
     is true that some of these are not mandatory in NFSv4, and the spec
     treats providing others as optional in the case of an absent
     filesystem, they do need to be provided to do a referral as
     envisioned herein.

3.3.1.  fsid

     The fsid attribute allows clients to recognize when fs boundaries
     have been crossed.  This applies also when one crosses into an
     absent filesystem and servers should provide this information when
     fs_locations is being requested for a filehandle within an absent
     filesystem, most typically at the root of that absent filesystem.

     To avoid misunderstanding, it should be noted that the fsid
     provided in this case is solely so that the fs boundaries can be
     properly noted and that the fsid returned will not necessarily be
     valid after resolution of the referral or migration event.  The
     logic of fsid handling for NFSv4 is that fsid's are only unique
     within a per-server context.  This would seem to be a strong
     indication that they need not be persistent when file systems are
     moved from server to server, although RFC 3530 does not
     specifically address the matter.

     A key reason for always providing an fsid can be seen by
     considering client behavior.  When a client gets the error
     NFS4ERR_MOVED, it has a number of key steps to get through.  It
     must determine if it has previously accessed the filesystem on the



Noveck, Burnett           Expires January 2006                 [Page 11]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     server that the moved error involves. If so, then it represents a
     data migration event in the strict sense, rather than a referral.
     The client must recover state and properly use rootpath data from
     the old and new server to graft the new server's namespace into the
     client's local space. If a client gets an NFS4ERR_MOVED error and
     it cannot get an fsid, then it must use fs_locations, fs_root, and
     rootpath data to locally determine if it has been accessing the
     filesystem or if the error applies to an fh in a different.
     (absent) filesystem.  If it uses fs_root and there have been
     renames along the path to the fh, then the client will not be able
     to distinguish a migration from a referral. The best solution here
     is for the client to always get an fsid. All a server has to do is
     generate an fsid value that cannot represent actual exported data
     at the server. We want to have a model where referrals can still
     function even if renames or other events change the path to the
     referral point.  Always providing fsid's improves the usefulness of
     the NFS namespace since referral and migration events can continue
     to be handled in the face of namespace management. This also helps
     to reduce client complexity.

3.3.2.  mounted_on_fileid

     The mounted_on_fileid attribute is of particular importance to many
     clients, in that they need this information to form a proper
     response to a readdir() call.  When a readdir() call is done within
     UNIX, the d_ino field of each of the entries needs to have a unique
     value normally derived from the NFSv4 fileid attribute.  It is in
     the case in which a file system boundary is crossed that using the
     fileid attribute for this purpose, particularly when crossing into
     an absent fs, will pose problems.  Note first that the fileid
     attribute, since it is within a new fs and thus a new fileid space,
     need not be unique within the directory.  Also, since the fs, at
     its new location, may arrange things differently, the fileid
     decided on at the directing server may be overridden at the target
     server, making it of little value.

     Neither of these problems arise in the case of mounted_on_fileid
     since that fileid is in the context of the mounted-on fs and unique
     within it.  Servers need to support and always return valid data
     for mounted-on-fileid, even for the case of an absent filesystem.
     The returned data represents a fileid for the entry in the
     directory that represents the absent fs. It's not the fileid of the
     root of the absent fs itself. Per the attribute handling rules
     presented in section 3.1, the server may return mounted-on-fileid
     as a subset of the requested attributes. The presence of
     fs_locations and rdattr_error attributes in a READDIR request
     determines if and how the server sets rdattr_error in the partial
     return case.



Noveck, Burnett           Expires January 2006                 [Page 12]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     Clients are strongly encouraged to always include mounted-on-fileid
     and rdattr_error in READDIR requests. This increases the potential
     that the client will receive information needed to satisfy the
     request and avoid further READDIR requests with a more restricted
     set of attributes. As a result, client performance is improved and
     network traffic is reduced.

3.3.3.  rdattr_error attribute

     This attribute needs to be provided for READDIR's or else a READDIR
     of a directory that contains the root of absent fs's cannot be done
     effectively.  In order to avoid NFS4ERR_MOVED on the directory as a
     whole, providing this error via the rdattr_error attribute allows
     attributes for the entry in question, which can be provided to the
     client, as well allowing the READDIR to provide information about
     other entries within the directory.


3.4.  Attributes Valid for Root of an Absent Fs

     Beyond the essential attributes discussed above, there are several
     other attributes that a server could decide to return for an absent
     fs. These should pose little potential for inconsistencies or
     client side problems. They are mentioned below, but clients should
     not expect servers to return them.


3.4.1.  type attribute

     For the root of absent fs, providing the value NF4DIR poses no
     difficulties as the target will provide the same value at the root
     of the target filesystem.

     Such a value isn't really needed as clients can assume that when
     there is a change of fsid value, the root of new filesystem will be
     a directory, as it must be.

3.4.2.  change attribute

     When a server implements pure referrals for a large number of
     filesystems, it may be convenient for clients to cache the
     destination fs_locations and interrogate the change attribute to
     see if there has been any change in the locations attribute to
     require a refetch.  Therefore, if the server does return a value
     for the change attribute when fetched for an absent filesystem, the
     server must update the change attribute when the target of the
     referral, i.e. the fs_locations value, changes.




Noveck, Burnett           Expires January 2006                 [Page 13]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     If the server does provide this information, the client must
     understand that there is no necessary continuity between the change
     values on referring server and on the target, as there may not be
     in the general migration case.  Clients are best advised to refetch
     such values on the target server and assume that it is possible
     that a change to the object may have occurred before the fetch of
     the change attribute on the target, and that this fact means that
     data and attribute caches on he client are best flushed.

3.5.  Attributes Not Valid for Root of an Absent Fs

     Attributes not listed in the sections above should not be provided
     in the case of the root of an absent filesystem (the referral
     case).  The general principle is that such information is
     appropriately provided at the destination specified by the
     fs_locations attribute and that specifying a value at the point of
     referral will either be incorrect or superfluous.

     We present a few examples of this principle below for particular
     attributes but the same sort of logic applies to all of the other
     attributes, to some degree or other.

3.5.1.  supp_attr attribute

     Since the referring server has no basis to determine what
     attributes are supported on the target, it is best not to provide
     this attribute on the referring server.

     It could be argued a value limited to attributes actually provided
     by the referring server could be provided, with the clear
     understanding that the value on the target will be different.
     However, there is very little value in doing that since there is
     only a single accessible object on the referring server for each
     fs, and the supported attributes are simply those that the server
     returns together with fs_locations when requested.

3.5.2.  filehandle attribute

     Just as a filehandle for the absent fs cannot be provided to the
     client via GETFH, it should similarly not be provided as the
     filehandle attribute and for the same reasons.

     A filehandle provided for an absent filesystem poses the difficulty
     of possible expiration or remapping.  This is an issue when
     migration happens after the file system has been accessed, but in
     the pure referral case, this issue can and should be avoided by
     never allowing the client to see such a filehandle at all.




Noveck, Burnett           Expires January 2006                 [Page 14]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


3.5.3.  fh_expire_type attribute

     In the pure referral case, the issue of filehandle expiration type
     is rendered moot by the lack of visibility of filehandles within
     the absent filesystem.  Providing a value for this attribute would
     limit the value on the target server to no purpose.

     By not providing a value, the referring server allows the target
     server flexibility to choose the value without fear of confusing
     the client.

     Note that the same logic applies to such attributes as
     link_support, symlink_support, case_insensitive, case_preserving,
     chown_restricted, and homogeneous.  The referring server should
     support any choice of such values on the target and the client
     should have no interest in guesses on the part of the server.

3.5.4.  fileid attribute

     Specifying a value for the root of the absent filesystem would
     serve no purpose as the value at the target would have to be
     definitive and any value at the referring server might conflict
     with a value at the target server.

4.  Referral Examples

     The details of how referrals proceed are implicit in the
     specification of migration in RFC 3530.  However, because the
     details of handling of this case are so different from those in the
     cases discussed therein, examples tailored to the referral
     situation are needed to clarify matters and allow correct and
     consistent implementations.

     The examples given in the sections below are somewhat artificial in
     that an actual client will not typically do a multi-component
     lookup, but will have cached information regarding the upper levels
     of the name hierarchy.  However, these example are chosen to make
     the required behavior clear and easy to put within the scope of a
     small number of requests, without getting unduly into details of
     how specific clients might choose to cache things.

4.1.  Referral Example (LOOKUP)


     Let us suppose that the following COMPOUND is issued in an
     environment in which /src/linux/2.7/latest is absent from the
     target server.  This may be for a number of reasons.  It may be the
     case that the file system has moved, or, it may be the case that



Noveck, Burnett           Expires January 2006                 [Page 15]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     the target server is functioning mainly, or solely, to refer
     clients to the servers on which various file systems are located.

     o    PUTROOTFH

     o    LOOKUP "src"

     o    LOOKUP "linux"

     o    LOOKUP "2.7"

     o    LOOKUP "latest"

     o    GETFH

     o    GETATTR fsid,fileid,size,ctime

     Under the given circumstances, the following will be the result.

     o    PUTROOTFH  --> NFS_OK

          Current fh is root of pseudo-fs.

     o    LOOKUP "src" --> NFS_OK

          Current fh is for /src and is within pseudo-fs.

     o    LOOKUP "linux" --> NFS_OK

          Current fh is for /src/linux and is within pseudo-fs.

     o    LOOKUP "2.7" --> NFS_OK

          Current fh is for /src/linux/2.7 and is within pseudo-fs.

     o    LOOKUP "latest" --> NFS_OK

          Current fh is for /src/linux/2.7/latest and is within a new,
          absent fs, but ...

          The client will never see the value of that fh.

     o    GETFH --> NFS4ERR_MOVED

          Fails because current fh is in an absent fs at the start of
          the operation and the spec makes no exception for GETFH.





Noveck, Burnett           Expires January 2006                 [Page 16]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     o    GETATTR fsid,fileid,size,ctime

          Not executed because the failure of the GETFH stops processing
          of the COMPOUND.

     Given the failure of the GETFH, the client has the job of
     determining the root of the absent file system and where to find
     that file system, i.e. the server and path relative to that
     server's root fh.  Note here that in this example, the client did
     not obtain filehandles and attribute information (e.g. fsid) for
     the intermediate directories, so that he would not be sure where
     the absent file system starts.  It could be the case, for example,
     that /src/linux/2.7 is the root of the moved filesystem and that
     the reason that the lookup of "latest" succeeded is that the
     filesystem was not absent on that op but was moved between the last
     LOOKUP and the GETFH (since COMPOUND is not atomic).  Even if we
     had the fsid's for all of the intermediate directories, we could
     have no way of knowing that /src/linux/2.7/latest was the root of a
     new fs, since we don't yet have its fsid.

     In order to get the necessary information, let us re-issue the
     chain of lookup's with GETFH's and GETATTR's to at least get the
     fsid's so we can be sure where the appropriate fs boundaries are.
     The client could choose to get fs_locations at the same time but in
     most cases the client will have a good guess as to where fs
     boundaries are (because of where NFS4ERR_MOVED was gotten and where
     not) making fetching of fs_location info unnecessary.

     o    PUTROOTFH  --> NFS_OK

          Current fh is root of pseudo-fs.

     o    GETATTR(fsid)  --> NFS_OK

          Just for completeness.  Normally, clients will know the fsid
          of the pseudo-fs as soon as they establish communication with
          a server.

     o    LOOKUP "src" --> NFS_OK

     o    GETATTR(fsid)  --> NFS_OK

          Get current fsid to see where fs boundaries are.  The fsid
          will be that for the pseudo-fs in this example, so no
          boundary.

     o    GETFH --> NFS_OK




Noveck, Burnett           Expires January 2006                 [Page 17]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


          Current fh is for /src and is within pseudo-fs.

     o    LOOKUP "linux" --> NFS_OK

          Current fh is for /src/linux and is within pseudo-fs.

     o    GETATTR(fsid)  --> NFS_OK

          Get current fsid to see where fs boundaries are.  The fsid
          will be that for the pseudo-fs in this example, so no
          boundary.

     o    GETFH --> NFS_OK

          Current fh is for /src/linux and is within pseudo-fs.

     o    LOOKUP "2.7" --> NFS_OK

          Current fh is for /src/linux/2.7 and is within pseudo-fs.

     o    GETATTR(fsid)  --> NFS_OK

          Get current fsid to see where fs boundaries are.  The fsid
          will be that for the pseudo-fs in this example, so no
          boundary.

     o    GETFH --> NFS_OK

          Current fh is for /src/linux/2.7 and is within pseudo-fs.

     o    LOOKUP "latest" --> NFS_OK

          Current fh is for /src/linux/2.7/latest and is within a new,
          absent fs, but ...

          The client will never see the value of that fh

     o    GETATTR(fsid, fs_locations)  --> NFS_OK

          We are getting the fsid to know where the fs boundaries are.
          While RFC 3530 does not oblige the server to give us anything
          but fs_locations, we are assuming that the server is following
          the recommendations herein and providing it.  Note that the
          fsid we are given will not necessarily be preserved at the new
          location.  That fsid might be different and in fact the fsid
          we have for this fs might a valid fsid of a different fs on
          that new server.




Noveck, Burnett           Expires January 2006                 [Page 18]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


          In this particular case, we are pretty sure anyway that what
          has moved is /src/linux/2.7/latest rather than /src/linux/2.7
          since we have the fsid of the latter and it is that of the
          pseudo-fs, which presumably cannot move.  However, in other
          examples, we might not have this kind of information to rely
          on (e.g. /src/linux/2.7 might be a non-pseudo filesystem
          separate from /src/linux/2.7/latest), so we need to have
          another reliable source information on the boundary of the fs
          which is moved.  If, for example, the filesystem "/src/linux"
          had moved we would have a case of migration rather than
          referral and once the boundaries of the migrated filesystem
          was clear we could fetch fs_locations.

          We are fetching fs_location because the fact that we got an
          NFS4ERR_MOVED at this point means that it most likely that
          this is a referral and we need the destination.  Even if it is
          the case that "/src/linux/2.7" is a filesystem which has
          migrated, we will still need the location information for that
          file system.

     o    GETFH --> NFS4ERR_MOVED

          Fails because current fh is in an absent fs at the start of
          the operation and the spec makes no exception for GETFH.  Note
          that this has the happy consequence that we don't have to
          worry about the volatility or lack thereof of the fh.  If the
          root of the fs on the new location is a persistent fh, then we
          can assume that this fh, which we never saw is a persistent
          fh, which, if we could see it, would exactly match the new fh.
          At least, there is no evidence to disprove that.  On the other
          hand, if we find a volatile root at the new location, then the
          filehandle which we never saw must have been volatile or at
          least nobody can prove otherwise.

     Given the above, the client knows where the root of the absent file
     system is, by noting where the change of fsid occurred.  The
     fs_locations attribute also gives the client the actual location of
     the absent file system, so that the referral can proceed.  The
     server gives the client the bare minimum of information about the
     absent file system so that there will be very little scope for
     problems of conflict between information sent by the referring
     server and information of the file system's home.  No filehandles
     and very few attributes are present on the referring server and the
     client can treat those it receives as basically transient
     information with the function of enabling the referral.






Noveck, Burnett           Expires January 2006                 [Page 19]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


4.2.  Referral Example (READDIR)

     Another context in which a client may encounter referrals is when
     it does a READDIR on directory in which some of the sub-directories
     are the roots of absent file systems.

     Suppose such a directory is read as follows:

     o    PUTROOTFH

     o    LOOKUP "src"

     o    LOOKUP "linux"

     o    LOOKUP "2.7"

     o    READDIR (fsid, size, ctime, mounted_on_fileid)

     In this case, because rdattr_error is not requested, fs_locations
     is not requested, and some of attributes cannot be provided the
     result will be an NFS4ERR_MOVED error on the READDIR, with the
     detailed results as follows:

     o    PUTROOTFH  --> NFS_OK

          Current fh is root of pseudo-fs.

     o    LOOKUP "src" --> NFS_OK

          Current fh is for /src and is within pseudo-fs.

     o    LOOKUP "linux" --> NFS_OK

          Current fh is for /src/linux and is within pseudo-fs.

     o    LOOKUP "2.7" --> NFS_OK

          Current fh is for /src/linux/2.7 and is within pseudo-fs.

     o    READDIR (fsid, size, ctime, mounted_on_fileid) -->
          NFS4ERR_MOVED

          Note that the same error would have been returned if
          /src/linux/2.7 had migrated, when in fact it is because the
          directory contains the root of an absent fs.

     So now suppose that we reissue with rdattr_error:




Noveck, Burnett           Expires January 2006                 [Page 20]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     o    PUTROOTFH

     o    LOOKUP "src"

     o    LOOKUP "linux"

     o    LOOKUP "2.7"

     o    READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid)

     The results will be:

     o    PUTROOTFH  --> NFS_OK

          Current fh is root of pseudo-fs.

     o    LOOKUP "src" --> NFS_OK

          Current fh is for /src and is within pseudo-fs.

     o    LOOKUP "linux" --> NFS_OK

          Current fh is for /src/linux and is within pseudo-fs.

     o    LOOKUP "2.7" --> NFS_OK

          Current fh is for /src/linux/2.7 and is within pseudo-fs.

     o    READDIR (rdattr_error, fsid, size, ctime, mounted_on_fileid)
          --> NFS_OK

          The attributes for "latest" will only contain rdattr_error
          with the value will be NFS4ERR_MOVED, together with an fsid
          value and an a value for mounted_on_fileid.

     So suppose we do another READDIR to get fs_locations info, although
     we could have used a GETATTR directly, as in the previous section.

     o    PUTROOTFH

     o    LOOKUP "src"

     o    LOOKUP "linux"

     o    LOOKUP "2.7"

     o    READDIR (rdattr_error, fs_locations, mounted_on_fileid, fsid,
          size, ctime)



Noveck, Burnett           Expires January 2006                 [Page 21]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     The results would be:

     o    PUTROOTFH  --> NFS_OK

          Current fh is root of pseudo-fs.

     o    LOOKUP "src" --> NFS_OK

          Current fh is for /src and is within pseudo-fs.

     o    LOOKUP "linux" --> NFS_OK

          Current fh is for /src/linux and is within pseudo-fs.

     o    LOOKUP "2.7" --> NFS_OK

          Current fh is for /src/linux/2.7 and is within pseudo-fs.

     o    READDIR (rdattr_error, fs_locations, mounted_on_fileid, fsid,
          size, ctime) --> NFS_OK

          The attributes for "latest" will only contain

          +    rdattr_error (value: NFS4ERR_MOVED)

          +    fs_locations (value: target:/path/on/target)

          +    mounted_on_fileid (value: unique fileid within referring
               fs)

          +    fsid (value: unique value within referring server)

          The attribute entry for "latest" will not contain size or
          ctime.

5.  Additional Client-side Considerations

     When clients make use of servers that implement referrals and
     migration, care should be taken so that a user who mounts a given
     filesystem that includes a referral or a relocated filesystem
     continue to see a coherent picture of that user-side filesystem
     despite the fact that it contains a number of server-side
     filesystems which may be on different servers.

     One important issue is upward navigation from the root of a server-
     side filesystem to its parent (specified as ".." in UNIX).  The
     client needs to determine when it hits an fsid root going up the
     filetree.  When at such a point, and needs to ascend to the parent,



Noveck, Burnett           Expires January 2006                 [Page 22]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


     it must do so locally instead of sending a LOOKUPP call to the
     server.  The LOOKUPP would normally return the ancestor of the
     target filesystem on the target server, which may not be part of
     the space that the client mounted.

     Another issue concerns refresh of referral locations.  When
     referrals are used extensively, they may change as server
     configurations change.  It is expected that clients will cache
     information related to traversing referrals so that future client
     side requests are resolved locally without server communication.
     This is usually rooted in client-side name lookup caching. Clients
     should periodically purge this data for referral points in order to
     detect changes in location information.  When the change attribute
     changes for directories that hold referral entries or for the
     referral entries themselves, clients should consider any associated
     cached referral information to be out of date.

6.  Acknowledgements

     The authors wish to thank Neil Brown and Ted Anderson for their
     helpful comments on various drafts of the material presented here.

7.  Normative References

     [RFC3530]
          S. Shepler, et. al., "NFS Version 4 Protocol", Standards Track
          RFC

Authors' Addresses

     David Noveck
     Network Appliance, Inc.
     375 Totten Pond Road
     Waltham, MA 02451 USA

     Phone: +1 781 768 5347
     EMail: dnoveck@netapp.com

     Rodney C. Burnett
     IBM, Inc.
     13001 Trailwood Rd
     Austin, TX 78727 USA

     Phone: +1 512 838 8498
     EMail: cburnett@us.ibm.com






Noveck, Burnett           Expires January 2006                 [Page 23]


Internet-Draft Implementation Guide for Referrals in NFSv4     July 2005


Full Copyright Statement

     Copyright (C) The Internet Society (2005).  This document is
     subject to the rights, licenses and restrictions contained in BCP
     78, and except as set forth therein, the authors retain all their
     rights.

     This document and the information contained herein are provided on
     an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
     REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
     THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
     EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
     THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
     ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
     PARTICULAR PURPOSE.

Intellectual Property

     The IETF takes no position regarding the validity or scope of any
     Intellectual Property Rights or other rights that might be claimed
     to pertain to the implementation or use of the technology described
     in this document or the extent to which any license under such
     rights might or might not be available; nor does it represent that
     it has made any independent effort to identify any such rights.
     Information on the procedures with respect to rights in RFC
     documents can be found in BCP 78 and BCP 79.

     Copies of IPR disclosures made to the IETF Secretariat and any
     assurances of licenses to be made available, or the result of an
     attempt made to obtain a general license or permission for the use
     of such proprietary rights by implementers or users of this
     specification can be obtained from the IETF on-line IPR repository
     at http://www.ietf.org/ipr.

     The IETF invites any interested party to bring to its attention any
     copyrights, patents or patent applications, or other proprietary
     rights that may cover technology that may be required to implement
     this standard.  Please address the information to the IETF at ietf-
     ipr@ietf.org.


Acknowledgement

     Funding for the RFC Editor function is currently provided by the
     Internet Society.






Noveck, Burnett           Expires January 2006                 [Page 24]