IETF T. Keiser
Internet-Draft S. Jenkins
Intended status: BCP Sine Nomine
Expires: December 17, 2010 June 15, 2010
AFSVol Tag-Length-Value Remote Procedure Call Extensions
draft-tkeiser-afs3-volser-tlv-02
Abstract
AFS-3 is a distributed file system based upon prototypes developed at
Carnegie Mellon University during the 1980s. AFS-3 heavily leverages
Remote Procedure Calls (RPCs) as the foundation for its distributed
architecture. In 2003, new RPCs were introduced into AFS-3 that
provide for capability introspection between file servers and cache
managers. This memo introduces equivalent functionality to the
volume server RPC interface, thus making the volume management
interface more extensible.
Furthermore, this memo extends the volume management interface to
support getting and setting of AFS volume attributes via an
extensible Tag-Length-Value (TLV) encoding, which is based upon XDR
discriminated unions. TLV-based get and set RPCs are specified,
along with a tag enumeration RPC. The TLV encoding side-steps the
typical XDR union decode problem, whereby failure to decode a union
leg causes the entire RPC payload decode to fail, by mandating an XDR
opaque default leg for the union, along with a standard mechanism for
encoding new leg types inside the XDR opaque blob.
Finally, tags are allocated for existing volume and transaction
metadata, and implementation-private tags are allocated for metadata
related to the OpenAFS Demand Attach File Server and RxOSD protocol.
Internet Draft Comments
Comments regarding this draft are solicited. Please include the
AFS-3 protocol standardization mailing list
(afs3-standardization@openafs.org) as a recipient of any comments.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Keiser & Jenkins Expires December 17, 2010 [Page 1]
Internet-Draft AFSVol TLV RPCs June 2010
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 17, 2010.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Keiser & Jenkins Expires December 17, 2010 [Page 2]
Internet-Draft AFSVol TLV RPCs June 2010
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1. Motivations . . . . . . . . . . . . . . . . . . . . . . . 6
1.2. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 6
2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 7
3. AFSVol Capability Introspection Interface . . . . . . . . . . 7
3.1. Capability Bit Interpretation . . . . . . . . . . . . . . 8
3.2. Capability Bit Allocations . . . . . . . . . . . . . . . . 8
3.3. Capabilities Cache Coherence . . . . . . . . . . . . . . . 8
4. TLV Interface . . . . . . . . . . . . . . . . . . . . . . . . 9
4.1. Encoding . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.1.1. Data Value Types . . . . . . . . . . . . . . . . . . . 11
4.1.2. TLV Flags . . . . . . . . . . . . . . . . . . . . . . 12
4.2. Qualifiers . . . . . . . . . . . . . . . . . . . . . . . . 13
5. AFSVol TLV Interface . . . . . . . . . . . . . . . . . . . . . 13
5.1. Tag Introspection . . . . . . . . . . . . . . . . . . . . 13
5.1.1. Tag Namespace Cache Coherence . . . . . . . . . . . . 14
5.2. TLV Get . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.3. TLV Streaming Get . . . . . . . . . . . . . . . . . . . . 15
5.3.1. Split call stream encoding . . . . . . . . . . . . . . 16
5.4. TLV Set . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.4.1. Call preprocessing . . . . . . . . . . . . . . . . . . 17
5.4.1.1. Verify tag is supported . . . . . . . . . . . . . 17
5.4.1.2. Verify tag is writeable . . . . . . . . . . . . . 18
5.4.1.3. Verify value encoding is supported . . . . . . . . 18
5.4.1.4. Verify value can be decoded . . . . . . . . . . . 18
5.4.1.5. Verify qualifier is supported . . . . . . . . . . 18
5.4.2. Call processing . . . . . . . . . . . . . . . . . . . 18
6. Mapping of existing metadata onto TLV namespace . . . . . . . 19
6.1. volintXInfo . . . . . . . . . . . . . . . . . . . . . . . 19
6.2. transDebugInfo . . . . . . . . . . . . . . . . . . . . . . 22
6.3. Additional de facto-standardized fields . . . . . . . . . 24
6.4. Day-of-week usage statistics . . . . . . . . . . . . . . . 26
6.4.1. Qualifiers . . . . . . . . . . . . . . . . . . . . . . 26
6.4.1.1. NULL qualifier . . . . . . . . . . . . . . . . . . 26
6.4.1.2. UINT64 qualifier . . . . . . . . . . . . . . . . . 27
6.4.2. Calendar day correlation . . . . . . . . . . . . . . . 27
7. Extended volume state exportation . . . . . . . . . . . . . . 27
7.1. Volume state explanations . . . . . . . . . . . . . . . . 28
7.2. Mapped process types . . . . . . . . . . . . . . . . . . . 30
7.3. TLV tuples . . . . . . . . . . . . . . . . . . . . . . . . 31
8. AFS-3 Object Storage Extensions Policy Attributes . . . . . . 31
9. Backward Compatibility . . . . . . . . . . . . . . . . . . . . 32
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 32
11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33
12. AFS Assign Numbers Registrar Considerations . . . . . . . . . 33
Keiser & Jenkins Expires December 17, 2010 [Page 3]
Internet-Draft AFSVol TLV RPCs June 2010
12.1. Namespace allocations . . . . . . . . . . . . . . . . . . 33
12.1.1. AFSVol Capabilities . . . . . . . . . . . . . . . . . 33
12.1.2. AFSVol TLV Payloads . . . . . . . . . . . . . . . . . 34
12.1.3. AFSVol TLV Tags . . . . . . . . . . . . . . . . . . . 35
12.1.4. AFSVol TLV Flags . . . . . . . . . . . . . . . . . . . 36
12.1.5. AFSVol DoW Stats Flags . . . . . . . . . . . . . . . . 37
12.1.6. AFSVol Vol State Expls . . . . . . . . . . . . . . . . 37
12.1.7. AFSVol Program Types . . . . . . . . . . . . . . . . . 38
12.2. Assigned numbers allocations . . . . . . . . . . . . . . . 39
12.2.1. VICED Capability bits . . . . . . . . . . . . . . . . 39
12.2.2. AFSVol Capabilities . . . . . . . . . . . . . . . . . 39
12.2.3. AFSVol TLV Payloads . . . . . . . . . . . . . . . . . 39
12.2.4. AFSVol TLV Tags . . . . . . . . . . . . . . . . . . . 40
12.2.5. AFSVol TLV Flags . . . . . . . . . . . . . . . . . . . 42
12.2.6. AFSVol DoW Stats Flags . . . . . . . . . . . . . . . . 42
12.2.7. VOLS Error Table . . . . . . . . . . . . . . . . . . . 43
12.2.8. AFSVol Vol State Expls . . . . . . . . . . . . . . . . 43
12.2.9. AFSVol Program Types . . . . . . . . . . . . . . . . . 44
13. Security Considerations . . . . . . . . . . . . . . . . . . . 44
14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 45
14.1. Normative References . . . . . . . . . . . . . . . . . . . 45
14.2. Informative References . . . . . . . . . . . . . . . . . . 45
Appendix A. XDR Definition for FS-CM Capabilities Mechanism . . . 46
Appendix B. Sample XDR Definition for AFSVol Capabilities
Mechanism . . . . . . . . . . . . . . . . . . . . . . 46
Appendix C. Sample XDR Definition for AFSVol TLV Mechanism . . . 46
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 51
Keiser & Jenkins Expires December 17, 2010 [Page 4]
Internet-Draft AFSVol TLV RPCs June 2010
1. Introduction
AFS-3 [AFS1] [AFS2] is a distributed file system that has its origins
in the VICE project [CMU-ITC-84-020] [VICE1] at the Carnegie Mellon
University Information Technology Center [CMU-ITC-83-025], a joint
venture between CMU and IBM. VICE later became AFS when CMU moved
development to a new commercial venture called Transarc Corporation,
which later became IBM Pittsburgh Labs. AFS-3 is a suite of un-
standardized network protocols based on a remote procedure call (RPC)
suite known as Rx. While de jure standards for AFS-3 fail to exist,
the various AFS-3 implementations have agreed upon certain de facto
standards, largely helped by the existence of an open source fork
called OpenAFS that has served the role of reference implementation.
In addition to using OpenAFS as a reference, IBM wrote and donated
developer documentation that contains somewhat outdated
specifications for the Rx protocol and all AFS-3 remote procedure
calls, as well as a detailed description of the AFS-3 system
architecture.
The AFS-3 architecture consists of many administrative domains called
"cells" [CMU-ITC-88-070] which are glued together to form a globally
distributed file system. Each cell consists of: client nodes, which
run cache manager daemons; file servers, which run file server
daemons and volume server daemons; and database server nodes, which
can run volume location database servers, protection database
servers, backup database servers, or several other obscure and/or
deprecated database services.
This memo focuses on the volume server component of AFS-3. The
volume server provides an RPC interface for managing AFS volumes.
Volumes are the unit of storage administration in AFS-3. Each volume
contains a subtree of the file system, along with special directory
entries called mount points, which are used to link volumes together
into a (potentially cyclic) directed graph. Mount points can cross
cell boundaries, thus permitting construction of a cross-
organizational, globally distributed, location-transparent file
system. The file system is location-transparent because mount points
contain volume names and cell names (which are resolved to locations
by contacting the appropriate cell's volume location database),
rather than encoding the data's physical location directly in the
pointer.
This memo extends the AFS-3 volume server RPC interface with:
1. an RPC in support of server capability introspection, and
2. a suite of new RPCs that provide extensible volume metadata get
and set operations.
Keiser & Jenkins Expires December 17, 2010 [Page 5]
Internet-Draft AFSVol TLV RPCs June 2010
1.1. Motivations
The current AFSVol volume metadata introspection routines use hard-
coded XDR [RFC4506] structure definitions. This significantly limits
protocol extensibility because new remote procedure calls and
structure definitions must be defined during each protocol revision.
To some degree, this has been due to the lack of protocol standards
documents: certain sites co-opted unused protocol fields for private
uses, thus eliminating the ability for the standards process to
reclaim these fields without breaking existing deployments. Hence,
each time new functionality needs to be added, a new RPC, and
typically a new XDR data structure, need to be defined. This is a
rather expensive process both in terms of standardization and
implementation. Frequently, this leads to a desire to postpone
protocol feature enhancements until many changes can be aggregated
into a major protocol upgrade.
This memo introduces a new tag-length-value (TLV) encoding mechanism
based upon XDR discriminated unions. This TLV encoding is utilized
for getting and setting AFS-3 volume metadata. The key advantage of
this design is that new TLV tuples can be allocated without defining
a new RPC. Furthermore, because TLV tuples allocated after this
draft are enocoded inside an XDR opaque blob, Rx endpoints will never
fail to decode the XDR call or reply payload; they may only fail to
decode the contents of the opaque. This means that XDR decode error
handling can happen at the application layer instead of deep within
Rx internals.
1.2. Goals
This memo aims to standardize a new TLV encoding mechanism for volume
metadata. In addition, this memo will standardize the TLV encoding
of volume metadata which is currently available via several AFSVol
XDR structures, as well as specify the encoding of several new pieces
of AFS-3 volume metadata that are not currently available via the
AFSVol interface. For example, metadata specific to the OpenAFS
Demand Attach File Server will be made available via the AFSVol
service, whereas in the past it was only available locally on the
file server machine via a proprietary interprocess communication
mechanism.
1.3. Abbreviations
AFS - Historically, AFS stood for the Andrew File System; AFS no
longer stands for anything
Keiser & Jenkins Expires December 17, 2010 [Page 6]
Internet-Draft AFSVol TLV RPCs June 2010
AFSINT - AFS-3 File Server / Cache Manager RPC Interface
AFSVol - AFS-3 Volume Server RPC Interface
CM - AFS-3 Cache Manager
DAFS - OpenAFS Demand Attach File Server
FS - AFS-3 File Server
RPC - Remote Procedure Call
RX - AFS-3 Remote Procedure Call Mechanism
RXAFS - AFS-3 File Server Rx RPC Interface
RXAFSCB- AFS-3 Cache Manager Rx RPC Interface
TLV - Tag-Length-Value encoding
TTL - Time to Live for cached data
VOLSER - AFS-3 Volume Server
XDR - eXternal Data Representation
2. Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
3. AFSVol Capability Introspection Interface
This memo introduces a capabilities namespace, and GetCapabilities
interface to the AFSVol service. The AFSVol GetCapabilities
interface will be be identical to the previously-defined AFSINT
interface, and its Rx interface specification will be:
proc GetCapabilities(
OUT Capabilities * capabilities
) = XXX;
Figure 1
The "Capabilities" type is defined by the existing AFSINT interface,
Keiser & Jenkins Expires December 17, 2010 [Page 7]
Internet-Draft AFSVol TLV RPCs June 2010
which is included here for reference:
const AFSCAPABILITIESMAX = 196;
typedef afs_int32 Capabilities<AFSCAPABILITIESMAX>;
Figure 2
3.1. Capability Bit Interpretation
The capabilities bit vector is used by an AFSVol server to advertise
which advanced protocol features it supports. Because the
GetCapabilities RPC OUT parameter is an XDR variable-length array,
servers MAY return a smaller bit vector than the full 196 elements.
Should a server return an array of length less than 196, all array
elements beyond those returned SHALL be interpreted as zero-filled.
3.2. Capability Bit Allocations
Three new capability bit allocations will need to be processed by the
AFS Assigned Numbers Registrar:
VICED_CAPABILITY_DAFS Announce that this file server supports the
OpenAFS Demand Attach File Server version 1 semantics
AFSVOL_CAPABILITY_DAFS Announce that this volume server supports the
OpenAFS Demand Attach File Server version 1 semantics
AFSVOL_CAPABILITY_TLV Announce that this volume server supports the
Tag-Length-Value RPC
3.3. Capabilities Cache Coherence
One important distinction between this capability introspection
interface and the ones utilized by AFSINT is: AFSINT is a stateful
circuit -- file servers can reset the cached state across themselves
and clients via the RXAFSCB_InitCallBackState,
RXAFSCB_InitCallBackState2, and RXAFSCB_InitCallBackState3 RPCs.
Because AFSVol is a stateless (with the exception of rxkad security
state) client/server protocol, there is no means of maintaining
AFSVol capabilities cache coherence. It is RECOMMENDED that clients
receiving RPC error codes, or critical tags which they cannot decode,
perform a new AFSVolGetCapabilities invocation to ensure that
capabilities cache incoherence is detected.
Clearly, the above technique is open to races; AFSVol clients SHOULD
try to limit race probability by minimizing the time window between
GetCapabilities calls, and invocation of capabilities-dependent RPCs
Keiser & Jenkins Expires December 17, 2010 [Page 8]
Internet-Draft AFSVol TLV RPCs June 2010
(such as the TLV suite defined in Section 4). All AFSVol clients
MUST flush cached capabilities data at most two hours after
retrieving them via AFSVolGetCapabilities. Additionally, if the
implementation permits querying the epoch field of Rx RPC responses,
the client MAY wish to use this as a means of detecting volume server
restarts, and thus as means of detecting when to invalidate cached
volume server capabilities. However, an AFSVol client MUST NOT use
the epoch field as a means to circumvent the two hour AFSVol
capabilities TTL, as AFSVol servers are not required to keep the
capability vector static throughout their operation.
4. TLV Interface
A new suite of RPCs will be standardized to get/set tag-length-value
tuples, and to enumerate supported tags. The tag namespace will be
controlled by the AFS Assigned Numbers Registrar as an assigned
numbers namespace.
4.1. Encoding
The TLV data will be encoded using the following XDR specification:
/* registrar-controlled tag namespace */
enum AFSVol_TLV_tag {
...
};
const AFSVOL_TLV_TAG_MAX = 1024; /* upper-bound on number of
* TLV tuples per RPC */
const AFSVOL_TLV_OPAQUE_MAX = 262144; /* upper-bound on size of
* value payload */
const AFSVOL_TLV_UINT64_MAX = 32768; /* upper-bound on length of
uint64 vector payload */
enum AFSVol_TLV_type {
AFSVOL_TLV_TYPE_NULL = 0,
AFSVOL_TLV_TYPE_TRUE = 1,
AFSVOL_TLV_TYPE_FALSE = 2,
AFSVOL_TLV_TYPE_UINT64 = 3,
AFSVOL_TLV_TYPE_UINT64_VEC = 4,
AFSVOL_TLV_TYPE_UUID = 5,
AFSVOL_TLV_TYPE_STRING = 6,
AFSVOL_TLV_TYPE_VOL_DOW_USE = 7,
AFSVOL_TLV_TYPE_OPAQUE = 8
};
union AFSVol_TLV_value switch(AFSVol_TLV_type type) {
Keiser & Jenkins Expires December 17, 2010 [Page 9]
Internet-Draft AFSVol TLV RPCs June 2010
case AFSVOL_TLV_TYPE_NULL:
case AFSVOL_TLV_TYPE_TRUE:
case AFSVOL_TLV_TYPE_FALSE:
void;
case AFSVOL_TLV_TYPE_UINT64:
afs_uint64 u_64;
case AFSVOL_TLV_TYPE_UINT64_VEC:
afs_uint64 u_64_vec<AFSVOL_TLV_UINT64_MAX>;
case AFSVOL_TLV_TYPE_UUID:
afsUUID u_uuid;
case AFSVOL_TLV_TYPE_STRING:
string u_string<AFSVOL_TLV_OPAQUE_MAX>;
case AFSVOL_TLV_TYPE_VOL_DOW_USE:
/* type defined later in this memo */
AFSVol_stat_use_per_dow u_vol_dow_use;
case AFSVOL_TLV_TYPE_OPAQUE:
default:
opaque u_opaque<AFSVOL_TLV_OPAQUE_MAX>;
};
const AFSVOL_TLV_FLAG_UNSUPPORTED = 0x1;
const AFSVOL_TLV_FLAG_READ_ERROR = 0x2;
const AFSVOL_TLV_FLAG_CRITICAL = 0x4;
const AFSVOL_TLV_FLAG_QUALIFIER_NO_MATCH = 0x8;
struct AFSVol_TLV {
afs_uint32 tlv_tag;
afs_uint32 tlv_flags;
AFSVol_TLV_value tlv_value;
};
TLV XDR specification
Figure 3
In order to solve the XDR discriminated union decoding problem, all
future AFSVol_TLV_type allocations will map to opaque. All
implementations MUST support all arms in the AFSVol_TLV_value XDR
union, as defined above.
When possible, future protocol augmentations requiring the definition
of new data types should request allocation of a new standards-track
Keiser & Jenkins Expires December 17, 2010 [Page 10]
Internet-Draft AFSVol TLV RPCs June 2010
payload type code. Allocation of a type code should coincide with
standardization of the payload encoding associated with the type code
allocation. However, in limited circumstances where:
1. it is known a priori that there will never be any encoding
ambiguity, and
2. the cost of type code allocation and encoding standardization are
deemed too high
use of the type code AFSVOL_TLV_TYPE_OPAQUE may be an acceptable
alternative.
4.1.1. Data Value Types
The core of the TLV definition above is the XDR discriminated union.
The following discriminators are initially defined in this memo:
AFSVOL_TLV_TYPE_NULL = 0
This shall map to type XDR void in the AFSVol_TLV_value union.
AFSVOL_TLV_TYPE_TRUE = 1
This shall map to type XDR void in the AFSVol_TLV_value union.
It is used to communicate the boolean value true.
AFSVOL_TLV_TYPE_FALSE = 2
This shall map to type XDR void in the AFSVol_TLV_value union.
It is used to communicate the boolean value false.
AFSVOL_TLV_TYPE_UINT64 = 3
This shall map to type afs_uint64 in the AFSVol_TLV_value union.
The semantics of this field are defined by the tag.
AFSVOL_TLV_TYPE_UINT64_VEC = 4
This shall map to an XDR variable length vector of up to 32768
afs_uint64 values. The semantics of this field are defined by
the tag.
AFSVOL_TLV_TYPE_UUID = 5
This shall map to an afsUUID type.
Keiser & Jenkins Expires December 17, 2010 [Page 11]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TYPE_STRING = 6
This shall map to an XDR string of maxmimum length 262144. The
semantics of this field are defined by the tag.
AFSVOL_TLV_TYPE_VOL_DOW_USE = 7
This shall map to an type AFSVol_stat_use_per_dow, as defined in
Section 6.4.
AFSVOL_TLV_TYPE_OPAQUE = 8
This shall map to an XDR opaque byte array of maximum length
262144. The semantics and encoding of this field are defined by
the tag.
(default)
All other tag values SHALL map to an XDR opaque byte array, as
above. However, the key difference between
AFSVOL_TLV_TYPE_OPAQUE and the default leg is how implementations
determine which decoding algorithm to use on the embedded value.
Unlike AFSVOL_TLV_TYPE_OPAQUE, where the algorithm is determined
by the tag, here the algorithm is chosen based upon the
discriminator stored in the AFSVol_TLV_value union.
4.1.2. TLV Flags
The AFSVol_TLV structure contains a 32-bit flags field for
communication of various ancillary boolean values. This memo defines
and allocates the following flag bits:
AFSVOL_TLV_FLAG_UNSUPPORTED = 0x1
When this flag is asserted, it tells the RPC caller that this tag
is not supported by this server.
AFSVOL_TLV_FLAG_READ_ERROR = 0x2
When this flag is asserted, it tells the RPC caller that the
server was unable to read a value for this tag, despite the tag
being supported by the server.
AFSVOL_TLV_FLAG_CRITICAL = 0x4
When this flag is asserted, it informs the peer that failure to
decode the payload associated with this tag is a fatal error that
should result in aborting this RPC call.
Keiser & Jenkins Expires December 17, 2010 [Page 12]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_FLAG_QUALIFIER_NO_MATCH = 0x8
When this flag is asserted, it informs the caller that the
qualifier passed in did not match any record.
4.2. Qualifiers
In some cases the value associated with a tag will be large,
structured data. A qualifier is a tag-specific parameter which
allows a caller to address a subset of the value stored in a tag.
For TLV get interfaces, specifying a qualifer can reduce the amount
of data sent over the wire. For TLV set interfaces, specifying a
qualifier permits a client to modify a subset of a structured value
without endangering cache coherence. Qualifiers are marshalled over
the wire as type AFSVol_TLV_value. Unless otherwise noted, it should
be assumed that a tag only supports the null qualifier (XDR union
discriminator set to AFSVOL_TLV_TYPE_NULL). The null qualifier
always references the entire value for a given tag.
5. AFSVol TLV Interface
5.1. Tag Introspection
The Rx procedure specification for the tag support RPC will be as
follows:
proc GetVolumeTLVTags(
IN AFSVol_TLV_tag offset,
OUT AFSVol_TLV_tag * tags<AFSVOL_TLV_TAG_MAX>
) = XXX;
Figure 4
The call parameters are defined as follows:
offset The offset IN parameter specifies the numeric offset of the
first tag to return. A value of zero indicates that the client
wants to start the enumeration at the beginning of the tag list.
tags The tags OUT parameter contains a sorted list of supported
tags, beginning with the first supported tag greater than or
equal to the offset IN parameter.
Keiser & Jenkins Expires December 17, 2010 [Page 13]
Internet-Draft AFSVol TLV RPCs June 2010
5.1.1. Tag Namespace Cache Coherence
Because the AFSVol interface is stateless, cache coherence cannot be
maintained via the normal AFS mechanism. Thus, AFSVol clients MUST
treat enumerated tags as ephemeral with a TTL of two hours.
As described in Section 3.3, a client MAY use the Rx epoch returned
by the AFSVol server as an indication that the cache should be
invalidated prior to the two hour TTL, but MUST NOT use this as an
optimization to extend cache lifetime beyond the two hour TTL, as the
server may change its supported tag enumeration at runtime.
5.2. TLV Get
The Rx procedure specification for the TLV get interface will be as
follows:
struct AFSVol_TLV_query {
AFSVol_TLV_tag tq_tag;
AFSVol_TLV_value tq_qualifier;
};
proc GetOneVolumeTLV(
IN afs_uint32 partId,
IN afs_uint64 volId,
IN AFSVol_TLV_query queries<AFSVOL_TLV_TAG_MAX>,
OUT AFSVol_TLV * tuples<AFSVOL_TLV_TAG_MAX>
) = XXX;
Figure 5
The call parameters are defined as follows:
partId The partId IN parameter specifies the disk partition on which
the volume is located.
volId The volId IN parameter specifies the volume for which TLV
tuples are being requested.
queries The queries IN parameter specifies an optional list of tags
for which TLV tuples are desired. If this parameter is zero-
length, then the server will return up to AFSVOL_TLV_TAG_MAX TLV
tuples. If an unknown tag identifier is passed in the tags
parameter, then the server will return a tuple with the
AFSVOL_TLV_FLAG_UNSUPPORTED bit asserted in AFSVol_TLV.tlv_flags,
and the tlv type set to AFSVOL_TLV_TYPE_NULL. Similarly, if the
server is unable to retrieve the value for a supported tag, then
a tuple will be returned with AFSVOL_TLV_FLAG_READ_ERROR set in
Keiser & Jenkins Expires December 17, 2010 [Page 14]
Internet-Draft AFSVol TLV RPCs June 2010
the AFSVol_TLV.tlv_flags field, and the tlv type set to
AFSVOL_TLV_TYPE_NULL. The AFSVol_TLV_query.tq_qualifier field
contains optional tag-specific qualifiers which would allow the
implementation to return a subset of the data for a specific tag.
When a non-NULL qualifier is passed, and the qualifier fails to
match any record, then the flag bit
AFSVOL_TLV_FLAG_QUALIFIER_NO_MATCH will be set in
AFSVol_TLV.tlv_flags field, and the tlv type set to
AFSVOL_TLV_TYPE_NULL.
tuples The tuples OUT parameter contains up to AFSVOL_TLV_TAG_MAX
TLV tuples for this volume.
5.3. TLV Streaming Get
This call is similar to the call described in the previous section,
with the exception that TLV tuples will be returned for multiple
volumes at once using an Rx split call interface. The Rx procedure
specification is as follows:
const AFSVOL_BULK_GETVOLUME_MAX = 1024;
proc GetVolumesTLV(
IN afs_uint32 partIds<AFSVOL_BULK_GETVOLUME_MAX>,
IN afs_uint64 volIds<AFSVOL_BULK_GETVOLUME_MAX>,
IN AFSVol_TLV_query queries<AFSVOL_TLV_TAG_MAX>
) split = XXX;
Figure 6
The call parameters are defined as follows:
partIds The partIds IN parameter specifies as list of vice
partitions. If this list is zero-length, then TLV information is
requested for all volumes on all vice partitions. If this list
is non-zero length, then TLV information is requested only for
volumes on specific vice partitions.
volIds The volIds IN parameter specifies a list of volume IDs. If
this list is zero-length, then TLV information is requested for
all volumes on the vice partitions specified in partIds.
If the volIds array is non-zero length, then its length MUST
match the length of the partIds array. In this case, each
matching index in the partIds and volIds arrays together form a
tuple which uniquely addresses a volume on a given vice
partition.
Keiser & Jenkins Expires December 17, 2010 [Page 15]
Internet-Draft AFSVol TLV RPCs June 2010
queries The queries IN parameter specifies an optional list of tags
for which TLV tuples are desired. If this parameter is zero-
length, then the server will return up to AFSVOL_TLV_TAG_MAX TLV
tuples. If an unknown tag identifier is passed in the tags
parameter, then the server will return a tuple with the
AFSVOL_TLV_FLAG_UNSUPPORTED bit asserted in AFSVol_TLV.tlv_flags,
and the tlv type set to AFSVOL_TLV_TYPE_NULL. Similarly, if the
server is unable to retrieve the value for a supported tag, then
a tuple will be returned with AFSVOL_TLV_FLAG_READ_ERROR set in
the AFSVol_TLV.tlv_flags field, and the tlv type set to
AFSVOL_TLV_TYPE_NULL. The AFSVol_TLV_query.tq_qualifier field
contains optional tag-specific qualifiers which would allow the
implementation to return a subset of the data for a specific tag.
When a non-NULL qualifier is passed, and the qualifier fails to
match any record, then the flag bit
AFSVOL_TLV_FLAG_QUALIFIER_NO_MATCH will be set in
AFSVol_TLV.tlv_flags field, and the tlv type set to
AFSVOL_TLV_TYPE_NULL.
5.3.1. Split call stream encoding
The contents of the split call stream shall be an xdrrec stream
containing a finite sequence of XDR-encoded AFSVol_TLV structures,
each of which shall be marked as a separate record (typically by
calling xdrrec_endofrecord). End of sequence will be annotated by a
dummy tuple containing the special tag type AFSVOL_TLV_TAG_EOS.
5.4. TLV Set
The Rx procedure specification for the TLV set interface will be as
follows:
struct AFSVol_TLV_store {
AFSVol_TLV ts_tuple;
AFSVol_TLV_value ts_qualifier;
};
proc SetVolumeTLV(
IN afs_int32 trans,
IN AFSVol_TLV_store tuples<AFSVOL_TLV_TAG_MAX>,
OUT afs_int32 * results<AFSVOL_TLV_TAG_MAX>
) = XXX;
Figure 7
The call parameters are defined as follows:
Keiser & Jenkins Expires December 17, 2010 [Page 16]
Internet-Draft AFSVol TLV RPCs June 2010
trans
The trans IN parameter specifies the transaction ID returned by a
previous invocation of AFSVolTransCreate.
tuples
The tuples IN parameter contains the list of TLV tuples to be set
by the server.
results
The results OUT parameter contains a list of error codes, one per
tuple. These error codes provide specific information regarding
the success/failure of each TLV set operation. Valid error codes
include:
* VOLSERTAGUNSUPPORTED
* VOLSERTAGREADONLY
* VOLSERTAGWRITEFAILED
* VOLSERTAGDECODEFAILED
* VOLSERTAGUNSUPPORTEDENCODING
* VOLSERTLVQUALIFIERUNSUPPORTEDENCODING
* VOLSERTLVQUALIFIERDECODEFAILED
* VOLSERTLVQUALIFIERINVALID
* VOLSERFAILEDOP
5.4.1. Call preprocessing
The SetVolumeTLV begins by scanning all elements within the tuples
array. If any elements have the AFSVOL_TLV_FLAG_CRITICAL bit
asserted in tuples[i].ts_tuple.ts_flags, then preprocessing of the
tuple must occur. For each tuple with the critical bit set, several
preprocessing validation steps will be taken.
5.4.1.1. Verify tag is supported
The tag stored in tuples[i].ts_tuple.tlv_tag is checked to ensure
that the server supports it. In the event that the tag is not
supported, then the corresponding array index in the results array
Keiser & Jenkins Expires December 17, 2010 [Page 17]
Internet-Draft AFSVol TLV RPCs June 2010
will be set to VOLSERTAGUNSUPPORTED, and the RPC call abort at the
conclusion of critical tuple preprocessing with error code
VOLSERFAILEDOP.
5.4.1.2. Verify tag is writeable
The tag stored in tuples[i].ts_tuple.tlv_flag is checked to ensure
that it is a writeable property. In the event that the tag is read-
only, then the corresponding array index in the results array will be
set to VOLSERTAGREADONLY, and the RPC call will abort at the
conclusion of critical tuple preprocessing with error code
VOLSERFAILEDOP.
5.4.1.3. Verify value encoding is supported
The XDR union discriminator in tuples[i].ts_tuple.tlv_value is
checked to make sure that it is a supported type. If the
discriminator is not a supported type, then the corresponding array
index in the results array will be set to
VOLSERTAGUNSUPPORTEDENCODING, and the RPC call will abort at the
conclusion of critical tuple preprocessing with error code
VOLSERFAILEDOP.
5.4.1.4. Verify value can be decoded
The value stored in tuples[i].ts_tuple.tlv_value is checked to make
sure that it can be decoded. If the wire-encoded data cannot be
decoded, then the corresponding array index in the results array will
be set to VOLSERTAGDECODEFAILED, and the RPC call will abort at the
conclusion of critical tuple preprocessing with error code
VOLSERFAILEDOP.
5.4.1.5. Verify qualifier is supported
Qualifiers are specific to a given tag. If for any reason the tag-
specific validation logic determines that the qualifier is invalid,
it may set the corresponding array index in the results array to one
of VOLSERTLVQUALIFIERUNSUPPORTEDENCODING,
VOLSERTLVQUALIFIERDECODEFAILED, or VOLSERTLVQUALIFIERINVALID. As
with the other validation steps, if a critical tuple fails qualifier
validation, then the RPC call will abort at the conclusion of
critical tuple preprocessing with error code VOLSERFAILEDOP.
5.4.2. Call processing
Once the necessary validation steps have been performed, the call
will perform the set operations for each tuple. Errors encountered
during the processing of each tuple will be recorded in the
Keiser & Jenkins Expires December 17, 2010 [Page 18]
Internet-Draft AFSVol TLV RPCs June 2010
appropriate array index of the results array. At the conclusion the
RPC will either return 0 if all set operations succeeded, or
VOLSERFAILEDOP if any failed.
6. Mapping of existing metadata onto TLV namespace
Existing metadata available from several interfaces will also be
exported as TLV tuples. This is being done not only for
completeness, but also to prevent data races between
AFSVolGetOneVolumeTLV, and the various legacy introspection
interfaces.
6.1. volintXInfo
All metadata exported via the volintXInfo XDR structure will now be
exported as TLV tuples. Unless otherwise specified, the values
associated with each tag shall be identical to that returned for the
associated field in volintXInfo by the AFSVolXListOneVolume
interface. The following tuples will be allocated to export existing
members of volintXInfo:
AFSVOL_TLV_TAG_VOL_NAME
This is the TLV analogue of volintXInfo.name. This tuple MUST
have a payload of type AFSVOL_TLV_TYPE_STRING. The u_string
payload field MUST contain a null-terminated string.
AFSVOL_TLV_TAG_VOL_STATUS
This is the TLV analogue of volintXInfo.status. This tuple MUST
have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_IN_USE
This is the TLV analogue of volintXInfo.inUse. This tuple will
contain a boolean value, and therefore MUST have a payload type
of either: AFSVOL_TLV_TYPE_TRUE, or AFSVOL_TLV_TYPE_FALSE.
AFSVOL_TLV_TAG_VOL_ID
This is the TLV analogue of volintXInfo.volid. This tuple MUST
have a payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TYPE
This is the TLV analogue of volintXInfo.type. This tuple MUST
have a payload of type AFSVOL_TLV_TYPE_UINT64.
Keiser & Jenkins Expires December 17, 2010 [Page 19]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_CLONE_ID
This is the TLV analogue of volintXInfo.cloneID. This tuple MUST
have a payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_BACKUP_ID
This is the TLV analogue of volintXInfo.backupID. This tuple
MUST have a payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_PARENT_ID
This is the TLV analogue of volintXInfo.parentID. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_COPY_DATE
This is the TLV analogue of volintXInfo.copyDate. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This timestamp
shall be encoded using the rules specified in the forthcoming
afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_CREATE_DATE
This is the TLV analogue of volintXInfo.creationDate. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This timestamp
shall be encoded using the rules specified in the forthcoming
afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_ACCESS_DATE
This is the TLV analogue of volintXInfo.accessDate. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This timestamp
shall be encoded using the rules specified in the forthcoming
afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_UPDATE_DATE
This is the TLV analogue of volintXInfo.updateDate. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This timestamp
shall be encoded using the rules specified in the forthcoming
afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_BACKUP_DATE
This is the TLV analogue of volintXInfo.backupDate. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This timestamp
shall be encoded using the rules specified in the forthcoming
Keiser & Jenkins Expires December 17, 2010 [Page 20]
Internet-Draft AFSVol TLV RPCs June 2010
afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_SIZE
This is the TLV analogue of volintXInfo.size. This tuple MUST
have payload of type AFSVOL_TLV_TYPE_UINT64. The uint64 value
describes the size of the volume, in units of 1KiB blocks.
AFSVOL_TLV_TAG_VOL_FILE_COUNT
This is the TLV analogue of volintXInfo.filecount. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_QUOTA_BLOCKS
This is the TLV analogue of volintXInfo.maxquota. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This uint64
value specifies the maximum volume size, in units of 1KiB blocks.
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY
This is the TLV analogue of volintXInfo.dayUse. This tuple MUST
have payload of type AFSVOL_TLV_TYPE_UINT64. This field tracks
volume accesses by AFS-3 clients over the course of this calendar
day, since midnight local time of the file server.
Operational monitoring applications which need to correlate the
start time for the counter against a date SHOULD simultaneously
query the value of tag AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY_DATE.
For further discussion of the cache coherence implications,
please see Section 6.4.2.
It should be noted that the definition of an "access" is
implementation-private, and thus comparison of access rates
across AFS-3 implementations is not possible.
AFSVOL_TLV_TAG_VOL_STAT_USE_PER_DOW
This is the TLV exportation of the daily usage statistics for the
past week. This tuple may have two different payload types,
depending upon whether or not a qualifier is delivered. The
payload and qualifier types will be discussed in Section 6.4.
It should be noted that the definition of an "access" is
implementation-private, and thus comparison of access rates
across AFS-3 implementations is not possible.
Keiser & Jenkins Expires December 17, 2010 [Page 21]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_STAT_READS
This is the TLV analogue of volintXInfo.stat_reads. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64_VEC. This
vector SHALL be of length 4.
AFSVOL_TLV_TAG_VOL_STAT_WRITES
This is the TLV analogue of volintXInfo.stat_reads. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64_VEC. This
vector SHALL be of length 4.
AFSVOL_TLV_TAG_VOL_STAT_FILE_SAME_AUTHOR
This is the TLV analogue of volintXInfo.stat_fileSameAuthor.
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64_VEC.
This vector SHALL be of length 6.
AFSVOL_TLV_TAG_VOL_STAT_FILE_DIFFERENT_AUTHOR
This is the TLV analogue of volintXInfo.stat_fileDiffAuthor.
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64_VEC.
This vector SHALL be of length 6.
AFSVOL_TLV_TAG_VOL_STAT_DIR_SAME_AUTHOR
This is the TLV analogue of volintXInfo.stat_dirSameAuthor. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64_VEC. This
vector SHALL be of length 6.
AFSVOL_TLV_TAG_VOL_STAT_DIR_DIFFERENT_AUTHOR
This is the TLV analogue of volintXInfo.stat_dirDiffAuthor. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64_VEC. This
vector SHALL be of length 6.
6.2. transDebugInfo
All metadata exported via the transDebugInfo XDR structure will now
be exported as TLV tuples. Unless otherwise specified, the values
associated with each tag shall be identical to that returned for the
associated field in transDebugInfo by the AFSVolMonitor interface.
The following tuples will be allocated to export existing members of
transDebugInfo:
Keiser & Jenkins Expires December 17, 2010 [Page 22]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_TRANS_ID
This is the TLV analogue of transDebugInfo.tid. This tuple MUST
have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_TIME
This is the TLV analogue of transDebugInfo.time. This tuple MUST
have payload of type AFSVOL_TLV_TYPE_UINT64. This timestamp
shall be encoded using the rules specified in the forthcoming
afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_TRANS_CREATE_TIME
This is the TLV analogue of transDebugInfo.creationTime. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This
timestamp shall be encoded using the rules specified in the
forthcoming afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_TRANS_RETURN_CODE
This is the TLV analogue of transDebugInfo.returnCode. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_ATTACH_MODE
This is the TLV analogue of transDebugInfo.iflags. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_STATUS
This is the TLV analogue of transDebugInfo.vflags This tuple MUST
have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_FLAGS
This is the TLV analogue of transDebugInfo.tflags. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_LAST_PROC_NAME
This is the TLV analogue of transDebugInfo.lastProcName. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_STRING. The
u_string payload field MUST contain a null-terminated string.
Keiser & Jenkins Expires December 17, 2010 [Page 23]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_TRANS_CALL_VALID
This is the TLV analogue of transDebugInfo.callValid. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_READ_NEXT
This is the TLV analogue of transDebugInfo.readNext. This tuple
MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_XMIT_NEXT
This is the TLV analogue of transDebugInfo.transmitNext. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_TRANS_LAST_RECV_TIME
This is the TLV analogue of transDebugInfo.lastReceiveTime. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This
timestamp shall be encoded using the rules specified in the
forthcoming afs3 RPC refresh document.
AFSVOL_TLV_TAG_VOL_TRANS_LAST_SEND_TIME
This is the TLV analogue of transDebugInfo.lastSendTime. This
tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64. This
timestamp shall be encoded using the rules specified in the
forthcoming afs3 RPC refresh document.
6.3. Additional de facto-standardized fields
Certain fields from the IBM AFS and OpenAFS file server's
VolumeDiskData header are generally useful. In particular, several
fields exported via the AFSVolGetFlags and AFSVolSetFlags RPCs should
be exported via the TLV interface. The full list of supported TLV
tuples are:
AFSVOL_TLV_TAG_VOL_IN_SERVICE
This tuple will contain a boolean value, and therefore MUST have
a payload type of either: AFSVOL_TLV_TYPE_TRUE, or
AFSVOL_TLV_TYPE_FALSE. When this bit is not asserted, the volume
is administratively prohibited from coming online.
AFSVOL_TLV_TAG_VOL_BLESSED
This tuple will contain a boolean value, and therefore MUST have
a payload type of either: AFSVOL_TLV_TYPE_TRUE, or
Keiser & Jenkins Expires December 17, 2010 [Page 24]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TYPE_FALSE. When this bit is not asserted, the volume
is administratively prohibited from coming online.
AFSVOL_TLV_TAG_VOL_RESTORED_FROM_ID
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
When this field is non-zero, it contains the volume ID contained
in the dump from which it was restored.
AFSVOL_TLV_TAG_VOL_DESTROYED
This tuple will contain a boolean value, and therefore MUST have
a payload type of either: AFSVOL_TLV_TYPE_TRUE, or
AFSVOL_TLV_TYPE_FALSE. When this bit is asserted, this volume is
flagged for deletion.
AFSVOL_TLV_TAG_VOL_NEEDS_SALVAGE
This tuple will contain a boolean value, and therefore MUST have
a payload type of either: AFSVOL_TLV_TYPE_TRUE, or
AFSVOL_TLV_TYPE_FALSE. When this bit is asserted, this volume
requires a salvage.
AFSVOL_TLV_TAG_VOL_OFFLINE_MESSAGE
This tuple MUST have payload of type AFSVOL_TLV_TYPE_STRING. The
u_string payload field MUST contain a null-terminated string.
This field stores an administrative message to indicate why the
volume is offline.
AFSVOL_TLV_TAG_VOL_EXPIRATION_DATE
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
This timestamp shall be encoded using the rules specified in the
forthcoming afs3 RPC refresh document. To the best knowledge of
the authors, this field is not standardized by any
implementation.
AFSVOL_TLV_TAG_VOL_QUOTA_RESERVATION
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
This field, otherwise known as minquota, specifies the amount of
storage (in units of 1024 octets) that are reserved on the
underlying storage for use by this volume.
Keiser & Jenkins Expires December 17, 2010 [Page 25]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY_DATE
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64.
This field, otherwise known as dayUseDate, specifies the
timestamp when AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY was reset to
zero, and the previous value rolled over to index 0 of
AFSVOL_TLV_TAG_VOL_STAT_USE_PER_DOW. This timestamp shall be
encoded using the rules specified in the forthcoming afs3 RPC
refresh document.
6.4. Day-of-week usage statistics
The day-of-week usage statistics accessed via tag
AFSVOL_TLV_TAG_VOL_STAT_USE_PER_DOW provide access to historic data
for the 7 days prior to the current access counter available via tag
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY. Depending on the desired mode of
statistics collection, two qualifier types are supported by this tag.
6.4.1. Qualifiers
6.4.1.1. NULL qualifier
When the qualifier is of type AFSVOL_TLV_TYPE_NULL, then a custom
payload of type AFSVOL_TLV_TYPE_VOL_DOW_USE will be used to deliver
day-of-week usage data for the past week. This type is defined as
follows:
stuct AFSVol_stat_use_per_dow {
afs_uint64 stat_dow[7];
afs_uint32 stat_flags;
};
Figure 8
Seven bits in the stat_flags field are used to assert data validity
for each day of week. These bits are present to help monitoring
applications distinguish between days for which no data was collected
(e.g. due to the volume being less than eight days old) and days when
there were exactly zero accesses. These bits are defined as follows:
Keiser & Jenkins Expires December 17, 2010 [Page 26]
Internet-Draft AFSVol TLV RPCs June 2010
Flag Description
----- -----------
AFSVOL_VOL_STAT_DOW0_VALID stat_dow[0] is valid
AFSVOL_VOL_STAT_DOW1_VALID stat_dow[1] is valid
AFSVOL_VOL_STAT_DOW2_VALID stat_dow[2] is valid
AFSVOL_VOL_STAT_DOW3_VALID stat_dow[3] is valid
AFSVOL_VOL_STAT_DOW4_VALID stat_dow[4] is valid
AFSVOL_VOL_STAT_DOW5_VALID stat_dow[5] is valid
AFSVOL_VOL_STAT_DOW6_VALID stat_dow[6] is valid
AFSVOL_VOL_STAT_DOW_FUZZY server incapable of guaranteeing validity
Day-of-week statistics flags
Server implementations which are incapable of distinguishing between
days when there was no usage, and for which there is no data SHOULD
make a best-effort to populate the 7 per-day bits, and MUST assert
the 0x80 stat_flags bit.
6.4.1.2. UINT64 qualifier
When the qualifier is of type AFSVOL_TLV_TYPE_UINT64, then a payload
of type AFSVOL_TLV_TYPE_UINT64 will be used to deliver day-of-week
usage data for the day of week specified in the uint64 qualifier.
Valid qualifiers are in the range 0 to 6, where 0 means the day prior
to the current day, and 6 means 7 days prior to the current day.
6.4.2. Calendar day correlation
Clients who need to poll AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY or
AFSVOL_TLV_TAG_VOL_STAT_USE_PER_DOW, and need to correlate this
statistical data with specific calendar days SHOULD simultaneously
query for the value stored at tag
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY_DATE. By querying these tags in
the same RPC invocation, the caller will be able correlate the usage
statistics with calendar days in a cache coherent manner. Querying
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY_DATE in a separate RPC invocation
is not guarnteed to yield correct results, as there is no way to
guarantee the value didn't change between the two RPC invocations.
7. Extended volume state exportation
In addition to exporting the existing volser state, DAFS state
metadata will also be exported via the TLV interface. Specifically,
an extended volume state field, and a raw DAFS state debugging tag,
will be exported.
Keiser & Jenkins Expires December 17, 2010 [Page 27]
Internet-Draft AFSVol TLV RPCs June 2010
7.1. Volume state explanations
Given that volume state information is useful across all server
implementations, a collection of generic state explanations shall be
standardized. These standardized enumeration values shall be
published via a special volume state explanation tag. The following
states are initially defined in the namespace:
AFSVOL_VOL_STATE_EXPL_NONE
No further explanation is deemed necessary.
AFSVOL_VOL_STATE_EXPL_UNKNOWN
This volume is in its current state for unknown reasons.
AFSVOL_VOL_STATE_EXPL_OUT_OF_SERVICE
This volume is administratively out of service. For example, the
IBM AFS and OpenAFS implementations both permit an administrator
to force a volume offline by mutating the blessed or inService
disk header bits.
AFSVOL_VOL_STATE_EXPL_DELETED
This volume no longer exists on-disk. This record merely serves
as a pointer to tell clients that the volume has been permanently
deleted, or moved to a new location.
AFSVOL_VOL_STATE_EXPL_READY
This volume is ready to service requests. If the primary volume
state is offline, this means the volume is ready to be brought
online as soon as a remote procedure call needs to access this
volume.
AFSVOL_VOL_STATE_EXPL_ATTACHING
This volume is busy attaching. Assuming the process completes
successfully, the volume will be brought online.
AFSVOL_VOL_STATE_EXPL_DETACHING
This volume is busy detaching.
Keiser & Jenkins Expires December 17, 2010 [Page 28]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_VOL_STATE_EXPL_BUSY
This volume is busy performing some ancillary operation which
requires exclusive access.
AFSVOL_VOL_STATE_EXPL_IO_BUSY
This volume is busy performing an I/O operation which requires
exclusive access.
AFSVOL_VOL_STATE_EXPL_SALVAGING
This volume is currently being salvaged in the background.
AFSVOL_VOL_STATE_EXPL_SALVAGE_NEEDED
This volume is offline, and will require a salvage before it can
be brought online.
AFSVOL_VOL_STATE_EXPL_ERROR
This volume has been forced offline due to a non-recoverable
error. Manual intervention by an administrator will be necessary
to bring this volume back to an operable state.
AFSVOL_VOL_STATE_EXPL_VOLUME_OPERATION
This volume is currently offline because a volume transaction
requires exclusive access.
enum AFSVol_vol_state_expl {
AFSVOL_VOL_STATE_EXPL_NONE = 0,
AFSVOL_VOL_STATE_EXPL_UNKNOWN = 1,
AFSVOL_VOL_STATE_EXPL_OUT_OF_SERVICE = 2,
AFSVOL_VOL_STATE_EXPL_DELETED = 3,
AFSVOL_VOL_STATE_EXPL_READY = 4,
AFSVOL_VOL_STATE_EXPL_ATTACHING = 5,
AFSVOL_VOL_STATE_EXPL_DETACHING = 6,
AFSVOL_VOL_STATE_EXPL_BUSY = 7,
AFSVOL_VOL_STATE_EXPL_IO_BUSY = 8,
AFSVOL_VOL_STATE_EXPL_SALVAGING = 9,
AFSVOL_VOL_STATE_EXPL_SALVAGE_NEEDED = 10,
AFSVOL_VOL_STATE_EXPL_ERROR = 11,
AFSVOL_VOL_STATE_EXPL_VOLUME_OPERATION = 12
};
XDR definition of Volume State Enumeration
Keiser & Jenkins Expires December 17, 2010 [Page 29]
Internet-Draft AFSVol TLV RPCs June 2010
7.2. Mapped process types
It is useful to be able to track volume ownership by process type.
In order to do this, a new program type namespace must be defined.
The following types are initially defined in the program type
namespace:
AFSVOL_PROGRAM_TYPE_NONE
This value refers to the absence of a process.
AFSVOL_PROGRAM_TYPE_FILE_SERVER
An afs file server process (Rx service ID 1).
AFSVOL_PROGRAM_TYPE_VOLUME_SERVER
An afs volume server process (Rx service ID 4).
AFSVOL_PROGRAM_TYPE_SALVAGER
An afs stand-alone salvager process.
AFSVOL_PROGRAM_TYPE_SALVAGE_SERVER
An OpenAFS DAFS salvage server process.
AFSVOL_PROGRAM_TYPE_VOLUME_UTILITY
Any ancillary stand-alone volume utility process.
AFSVOL_PROGRAM_TYPE_UNKNOWN
This value refers to an unknown process type.
enum AFSVol_program_type {
AFSVOL_PROGRAM_TYPE_NONE = 0,
AFSVOL_PROGRAM_TYPE_FILE_SERVER = 1,
AFSVOL_PROGRAM_TYPE_VOLUME_SERVER = 2,
AFSVOL_PROGRAM_TYPE_SALVAGER = 3,
AFSVOL_PROGRAM_TYPE_SALVAGE_SERVER = 4,
AFSVOL_PROGRAM_TYPE_VOLUME_UTILITY = 5,
AFSVOL_PROGRAM_TYPE_UNKNOWN = 6
};
XDR definition of Program Type Enumeration
Keiser & Jenkins Expires December 17, 2010 [Page 30]
Internet-Draft AFSVol TLV RPCs June 2010
7.3. TLV tuples
Volume state will be exported via five new TLV tuples:
AFSVOL_TLV_TAG_VOL_STATE_ONLINE
This tuple MUST have payload of either type AFSVOL_TLV_TYPE_TRUE,
or AFSVOL_TLV_TYPE_FALSE. This value SHALL tell the caller
whether or not the volume is fully online.
AFSVOL_TLV_TAG_VOL_STATE_AVAILABLE
This tuple MUST have payload of either type AFSVOL_TLV_TYPE_TRUE,
or AFSVOL_TLV_TYPE_FALSE. This tuple shall tell the caller
whether or not the volume is available. This SHOULD be asserted
either when the volume is fully online, or when the volume can be
brought online on-demand within a reasonable length of time
following receipt of an RPC call to Rx service id 1 requesting
access to the volume.
AFSVOL_TLV_TAG_VOL_STATE_EXPL
This tuple MUST have payload of type AFSVOL_TLV_TYPE_UINT64. The
u_64 payload shall contain a volume state explanation enumeration
value, as defined in Section 7.1.
AFSVOL_TLV_TAG_VOL_STATE_DAFS_RAW
For servers exporting capability AFSVOL_CAPABILITY_DAFS, this
payload MUST be of type AFSVOL_TLV_TYPE_OPAQUE. Encoding of raw
state is unspecified and implementation-private.
AFSVOL_TLV_TAG_VOL_STATE_OWNING_PROCESS
This tag should only be advertised as available on server
implementations which support tracking volume ownership by
process type. When available, this payload MUST be of type
AFSVOL_TLV_TYPE_UINT64. The u_64 payload shall contain a program
type enumeration value, as defined in Section 7.2.
8. AFS-3 Object Storage Extensions Policy Attributes
RxOSD requires two TLV tuples to encode new quota types:
Keiser & Jenkins Expires December 17, 2010 [Page 31]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_QUOTA_BLOCKS_STORED_LOCALLY
The value in this tuple defines the maximum allowable storage, in
units of blocks, that may be stored on the local file server
partition. When storage is required beyond this limit, some data
must be migrated to object storage devices (OSDs). This tuple
MUST have a payload of type AFSVOL_TLV_TYPE_UINT64.
AFSVOL_TLV_TAG_VOL_QUOTA_FILES
The value in this tuple defines the maximum allowable file count
for this volume. This tuple MUST have a payload of type
AFSVOL_TLV_TYPE_UINT64.
9. Backward Compatibility
AFSVol services providing extended Tag-Length-Value RPCs MUST provide
backwards compatible interfaces to both legacy clients and servers.
Additionally, interoperability between TLV versions must also be
specified if they do not comply with the following requirements:
1. AFSVol TLV servers replying to legacy AFSVol clients MUST provide
the identical response to an AFSVol server.
2. AFSVol TLV clients communicating with AFSVol servers MUST fall
back to using non-TLV AFSVol RPCs.
3. AFSVol TLV clients to AFSVol TLV servers:
A. Where capabilities match or the server can provide
capabilities including those which the client requests, the
server MUST reply with exactly the capabilities requested.
B. Where the client requests capabilities that the server does
not provide it MUST either return an 'unknown tag' error
code, or (OPTIONAL) fall back to an non-TLV AFSVol response.
10. Acknowledgements
We would like to thank all of the participants at the 2009 Edinburgh
AFS hackathon for their input into the design of this TLV mechanism.
Alistair Ferguson has provided much useful feedback, especially with
regard to backwards compatibility and discriminated union type
identifier namespace allocations. Andrew Deason and Michael Meffie
have provided considerable input with regard to the discriminated
union XDR decoding problem, AFS registrar and namespace allocation
Keiser & Jenkins Expires December 17, 2010 [Page 32]
Internet-Draft AFSVol TLV RPCs June 2010
concerns, what metadata should be exported in the initial revision,
the notion of data qualifiers, as well as commentary about how they
envision this extension being used to support future protocol
extensions. Derrick Brashear has provided helpful feedback with
regard to restructuring the volume state reporting tags. Thanks to
Christof Hanke and Hartmut Reuter for collaborating to make this memo
compatible with their RxOSD protocol enhancments, and, furthermore,
for providing helpful feedback regarding the language in this draft.
Finally, special thanks to Jeffrey Hutzelman for providing
considerable help with restructuring this memo to improve readability
and limit its scope to something tractable.
11. IANA Considerations
This memo includes no request to IANA.
12. AFS Assign Numbers Registrar Considerations
The AFS Assigned Numbers Registrar will need to consider several
assigned numbers requests.
12.1. Namespace allocations
First and foremost, this memo requests that the AFS Registrar assume
control over several new registries:
1. AFSVol Capability bit namespace
2. AFSVol TLV payload type namespace
3. AFSVol TLV tag namespace
4. AFSVol TLV flag namespace
5. AFSVol TLV Day-of-Week Stats flag namespace
6. AFSVol Mapped Volume State namespace
7. AFSVol Program Type namespace
12.1.1. AFSVol Capabilities
This memo requests the allocation of a new registry with the formal
name "AFSVol Capabilities". This registry will be used to track
allocations of AFSVol capability bits. The capability bit namespace
contains 6272 bits, subdivided into 196 32-bit buckets. Allocation
Keiser & Jenkins Expires December 17, 2010 [Page 33]
Internet-Draft AFSVol TLV RPCs June 2010
requests for this namespace MUST be in the form of an RFC.
Furthermore, final approval for allocations SHALL be made by a
Designated Expert [RFC5226] to be nominated by the AFS-3 Working
Group. Should the AFS-3 Working Group be unable to assign a
Designated Expert, the AFS Assigned Numbers Registrar will be free to
appoint one or more Designated Experts to aid the registrar in the
process of vetting requests for this namespace. All allocation
requests for this registry MUST include the following information:
o capability name
o RFC section reference to definition of how this capability bit
alters AFSVol protocol semantics
In addition, an allocation request MAY include the following optional
elements:
o capability description
o desired capability bucket number and bit position
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
12.1.2. AFSVol TLV Payloads
This memo requests the allocation of a new registry with the formal
name "AFSVol TLV Payloads". This registry will be used to track
allocations of enumeration values in the AFSVol_TLV_type XDR enum,
and the mapping of these values onto their respective XDR type
definitions. This is a 32-bit unsigned namespace. Allocations can
fall into one of a few categories:
Range Description
----- -----------
0 to 0xfeffffff - AFS-STDS Early Assignment
0xf0000000 - Private Assignment
to 0xfffeffff
0xffff0000 - reserved
to 0xffffffff
Subdivision into allocation policy regions
In the table above, "AFS-STDS Early Assignment" refers to the
allocation policy described in [AFS3-STDS-CHARTER]; "Private
Keiser & Jenkins Expires December 17, 2010 [Page 34]
Internet-Draft AFSVol TLV RPCs June 2010
Assignment", and "Reserved" are as-described in [RFC5226].
Allocation requests for the "AFS-STDS Early Assignment" region MUST
contain the following information:
o type name
o RFC section reference to definition of data encoding associated
with this type enumeration value
In addition, an "AFS-STDS Early Assignment" allocation request MAY
include the following optional elements:
o type description
o desired value in AFSVol_TLV_type enumeration
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
12.1.3. AFSVol TLV Tags
This memo requests the allocation of a new registry with the formal
name "AFSVol TLV Tags". This registry will be used to track
allocations of enumeration values in the AFSVol_TLV_tag XDR enum, and
the mapping of these values onto legal tags and qualifiers. This is
a 32-bit unsigned namespace. Allocations can fall into one of a few
categories:
Range Description
----- -----------
0 to 0xfeffffff - AFS-STDS Early Assignment
0xf0000000 - Private Assignment
to 0xfffeffff
0xffff0000 - reserved
to 0xffffffff
Subdivision into allocation policy regions
In the table above, "AFS-STDS Early Assignment" refers to the
allocation policy described in [AFS3-STDS-CHARTER]; "Private
Assignment", and "Reserved" are as-described in [RFC5226].
Allocation requests for the "AFS-STDS Early Assignment" region MUST
contain the following information:
Keiser & Jenkins Expires December 17, 2010 [Page 35]
Internet-Draft AFSVol TLV RPCs June 2010
o tag name
o RFC section reference to definition of tag semantics
In addition, an "AFS-STDS Early Assignment" allocation request MAY
include the following optional elements:
o tag description
o desired value in AFSVol_TLV_tag enumeration
o RFC section reference to definition of qualifier semantics for
this tag
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
12.1.4. AFSVol TLV Flags
This memo requests the allocation of a new registry with the formal
name "AFSVol TLV Flags". This registry will be used to track
allocations of flag bits in the AFSVol_TLV.tlv_flags field. This is
a 32-bit flag namespace. All flag bit allocations shall fall under
the "AFS-STDS Early Assignment" allocation policy, as described in
[AFS3-STDS-CHARTER]. Flag bit allocation requests MUST contain the
following information:
o flag name
o RFC section reference to definition of flag semantics
In addition, an allocation request MAY include the following optional
elements:
o flag description
o desired flag bit value
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
Keiser & Jenkins Expires December 17, 2010 [Page 36]
Internet-Draft AFSVol TLV RPCs June 2010
12.1.5. AFSVol DoW Stats Flags
This memo requests the allocation of a new registry with the formal
name "AFSVol DoW Stats Flags". This registry will be used to track
allocations of flag bits in the AFSVol_stat_use_per_dow.stat_flags
field. This is a 32-bit flag namespace. All flag bit allocations
shall fall under the "AFS-STDS Early Assignment" allocation policy,
as described in [AFS3-STDS-CHARTER]. Flag bit allocation requests
MUST contain the following information:
o flag name
o RFC section reference to definition of flag semantics
In addition, an allocation request MAY include the following optional
elements:
o flag description
o desired flag bit value
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
12.1.6. AFSVol Vol State Expls
This memo requests the allocation of a new registry with the formal
name "AFSVol Vol State Expls". This registry will be used to track
allocations of enumeration values in the AFSVol_vol_state_expl enum
(see Section 7.1). This is a 32-bit unsigned namespace. Allocations
can fall into one of a few categories:
Range Description
----- -----------
0 to 0xfeffffff - AFS-STDS Early Assignment
0xf0000000 - Private Assignment
to 0xffffffff
Subdivision into allocation policy regions
In the table above, "AFS-STDS Early Assignment" refers to the
allocation policy described in [AFS3-STDS-CHARTER]; "Private
Assignment" is as-described in [RFC5226].
Allocation requests for the "AFS-STDS Early Assignment" region MUST
Keiser & Jenkins Expires December 17, 2010 [Page 37]
Internet-Draft AFSVol TLV RPCs June 2010
contain the following information:
o state name
o RFC section reference to definition of this volume state
enumeration value
In addition, an "AFS-STDS Early Assignment" allocation request MAY
include the following optional elements:
o state description
o desired value in AFSVol_vol_state_expl enumeration
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
12.1.7. AFSVol Program Types
This memo requests the allocation of a new registry with the formal
name "AFSVol Program Types". This registry will be used to track
allocations of enumeration values in the AFSVol_program_type enum
(see Section 7.2). This is a 32-bit unsigned namespace. Allocations
can fall into one of a few categories:
Range Description
----- -----------
0 to 0xfeffffff - AFS-STDS Early Assignment
0xf0000000 - Private Assignment
to 0xffffffff
Subdivision into allocation policy regions
In the table above, "AFS-STDS Early Assignment" refers to the
allocation policy described in [AFS3-STDS-CHARTER]; "Private
Assignment" is as-described in [RFC5226].
Allocation requests for the "AFS-STDS Early Assignment" region MUST
contain the following information:
o program name
o RFC section reference to definition of this program type
enumeration value
Keiser & Jenkins Expires December 17, 2010 [Page 38]
Internet-Draft AFSVol TLV RPCs June 2010
In addition, an "AFS-STDS Early Assignment" allocation request MAY
include the following optional elements:
o program description
o desired value in AFSVol_program_type enumeration
o RFC section reference to discussion regarding backwards
compatibility
o RFC section reference to relevant security considerations
12.2. Assigned numbers allocations
In addition to requesting the allocation of new registries, this memo
also requests several new allocations within existing assigned
numbers registries.
12.2.1. VICED Capability bits
One new capability bit is requested:
o VICED_CAPABILITY_DAFS (see Section 3.2)
12.2.2. AFSVol Capabilities
The following initial allocations are requested in the newly-created
registry "AFSVol Capabilites":
o AFSVOL_CAPABILITY_DAFS = 0x1 (see Section 3.2)
o AFSVOL_CAPABILITY_TLV = 0x2 (see Section 3.2)
12.2.3. AFSVol TLV Payloads
The following initial allocations are requested in the newly-created
registry "AFSVol TLV Payloads":
o AFSVOL_TLV_TYPE_NULL = 0 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_TRUE = 1 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_FALSE = 2 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_UINT64 = 3 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_UINT64_VEC = 4 (see Section 4.1.1)
Keiser & Jenkins Expires December 17, 2010 [Page 39]
Internet-Draft AFSVol TLV RPCs June 2010
o AFSVOL_TLV_TYPE_UUID = 5 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_STRING = 6 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_VOL_DOW_USE = 7 (see Section 4.1.1)
o AFSVOL_TLV_TYPE_OPAQUE = 8 (see Section 4.1.1)
12.2.4. AFSVol TLV Tags
The following initial allocations are requested in the newly-created
registry "AFSVol TLV Tags":
o AFSVOL_TLV_TAG_EOS = 0 (see Section 5.3.1)
o AFSVOL_TLV_TAG_VOL_NAME = 1 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STATUS = 2 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_IN_USE = 3 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_ID = 4 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_TYPE = 5 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_CLONE_ID = 6 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_BACKUP_ID = 7 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_PARENT_ID = 8 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_COPY_DATE = 9 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_CREATE_DATE = 10 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_ACCESS_DATE = 11 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_UPDATE_DATE = 12 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_BACKUP_DATE = 13 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_SIZE = 14 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_FILE_COUNT = 15 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_QUOTA_BLOCKS = 16 (see Section 6.1)
Keiser & Jenkins Expires December 17, 2010 [Page 40]
Internet-Draft AFSVol TLV RPCs June 2010
o AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY = 17 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_USE_PER_DOW = 18 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_READS = 19 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_WRITES = 20 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_FILE_SAME_AUTHOR = 21 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_FILE_DIFFERENT_AUTHOR = 22 (see
Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_DIR_SAME_AUTHOR = 23 (see Section 6.1)
o AFSVOL_TLV_TAG_VOL_STAT_DIR_DIFFERENT_AUTHOR = 24 (see
Section 6.1)
o AFSVOL_TLV_TAG_VOL_TRANS_ID = 25 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_TIME = 26 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_CREATE_TIME = 27 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_RETURN_CODE = 28 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_ATTACH_MODE = 29 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_STATUS = 30 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_FLAGS = 31 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_LAST_PROC_NAME = 32 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_CALL_VALID = 33 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_READ_NEXT = 34 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_XMIT_NEXT = 35 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_LAST_RECV_TIME = 36 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_TRANS_LAST_SEND_TIME = 37 (see Section 6.2)
o AFSVOL_TLV_TAG_VOL_IN_SERVICE = 38 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_BLESSED = 39 (see Section 6.3)
Keiser & Jenkins Expires December 17, 2010 [Page 41]
Internet-Draft AFSVol TLV RPCs June 2010
o AFSVOL_TLV_TAG_VOL_RESTORED_FROM_ID = 40 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_DESTROYED = 41 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_NEEDS_SALVAGE = 42 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_OFFLINE_MESSAGE = 43 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_EXPIRATION_DATE = 44 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_QUOTA_RESERVATION = 45 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY_DATE = 46 (see Section 6.3)
o AFSVOL_TLV_TAG_VOL_STATE_ONLINE = 47 (see Section 7)
o AFSVOL_TLV_TAG_VOL_STATE_AVAILABLE = 48 (see Section 7)
o AFSVOL_TLV_TAG_VOL_STATE_EXPL = 49 (see Section 7)
o AFSVOL_TLV_TAG_VOL_STATE_DAFS_RAW = 50 (see Section 7)
o AFSVOL_TLV_TAG_VOL_STATE_OWNING_PROCESS = 51 (see Section 7)
o AFSVOL_TLV_TAG_VOL_QUOTA_BLOCKS_STORED_LOCALLY = 52 (see
Section 8)
o AFSVOL_TLV_TAG_VOL_QUOTA_FILES = 53 (see Section 8)
12.2.5. AFSVol TLV Flags
The following initial allocations are requested within the newly-
created registry "AFSVol TLV Flags":
o AFSVOL_TLV_FLAG_UNSUPPORTED = 0x1 (see Section 4.1.2)
o AFSVOL_TLV_FLAG_READ_ERROR = 0x2 (see Section 4.1.2)
o AFSVOL_TLV_FLAG_CRITICAL = 0x4 (see Section 4.1.2)
o AFSVOL_TLV_FLAG_QUALIFIER_NO_MATCH = 0x8 (see Section 4.1.2)
12.2.6. AFSVol DoW Stats Flags
The following initial allocations are requested within the newly-
created registry "AFSVol DoW Stats Flags":
Keiser & Jenkins Expires December 17, 2010 [Page 42]
Internet-Draft AFSVol TLV RPCs June 2010
o AFSVOL_VOL_STAT_DOW0_VALID = 0x1 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW1_VALID = 0x2 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW2_VALID = 0x4 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW3_VALID = 0x8 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW4_VALID = 0x10 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW5_VALID = 0x20 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW6_VALID = 0x40 (see Section 6.4)
o AFSVOL_VOL_STAT_DOW_FUZZY = 0x80 (see Section 6.4)
12.2.7. VOLS Error Table
Within the VOLS error table (offset 1492325120), several new codes
need to be allocated:
o VOLSERTAGUNSUPPORTED
o VOLSERTAGREADONLY
o VOLSERTAGWRITEFAILED
o VOLSERTAGDECODEFAILED
o VOLSERTAGUNSUPPORTEDENCODING
o VOLSERTLVQUALIFIERUNSUPPORTEDENCODING
o VOLSERTLVQUALIFIERDECODEFAILED
o VOLSERTLVQUALIFIERINVALID
12.2.8. AFSVol Vol State Expls
The following initial allocations are requested within the newly-
created registry "AFSVol Vol State Expls":
o AFSVOL_VOL_STATE_EXPL_NONE = 0 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_UNKNOWN = 1 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_OUT_OF_SERVICE = 2 (see Section 7.1)
Keiser & Jenkins Expires December 17, 2010 [Page 43]
Internet-Draft AFSVol TLV RPCs June 2010
o AFSVOL_VOL_STATE_EXPL_DELETED = 3 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_READY = 4 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_ATTACHING = 5 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_DETACHING = 6 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_BUSY = 7 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_IO_BUSY = 8 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_SALVAGING = 9 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_SALVAGE_NEEDED = 10 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_ERROR = 11 (see Section 7.1)
o AFSVOL_VOL_STATE_EXPL_VOLUME_OPERATION = 12 (see Section 7.1)
12.2.9. AFSVol Program Types
Within the new AFS program type namespace, the following allocations
are requested:
o AFSVOL_PROGRAM_TYPE_NONE = 0 (see Section 7.2)
o AFSVOL_PROGRAM_TYPE_FILE_SERVER = 1 (see Section 7.2)
o AFSVOL_PROGRAM_TYPE_VOLUME_SERVER = 2 (see Section 7.2)
o AFSVOL_PROGRAM_TYPE_SALVAGER = 3 (see Section 7.2)
o AFSVOL_PROGRAM_TYPE_SALVAGE_SERVER = 4 (see Section 7.2)
o AFSVOL_PROGRAM_TYPE_VOLUME_UTILITY = 5 (see Section 7.2)
o AFSVOL_PROGRAM_TYPE_UNKNOWN = 6 (see Section 7.2)
13. Security Considerations
Security and authorization issues are tag-specific. The legacy
AFSVol RPCs permitted rxnull connections to perform the four
ListVolume RPCs, and AFSVolMonitor. Arguably, it is time to re-
evaluate this decision, and restrict access to certain tags, as they
do permit potentially sensitive volume or operational metadata to
leak onto public networks.
Keiser & Jenkins Expires December 17, 2010 [Page 44]
Internet-Draft AFSVol TLV RPCs June 2010
14. References
14.1. Normative References
[AFS3-STDS-CHARTER]
Wilkinson, S., "Options for AFS Standardisation",
September 2008, <http://
michigan-openafs-lists.central.org/archives/
afs3-standardization/2008-September/000244.html>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 5226,
May 2008.
14.2. Informative References
[AFS1] Howard, J., "An Overview of the Andrew File System"",
Proc. 1988 USENIX Winter Tech. Conf. pp. 23-26,
February 1988.
[AFS2] Howard, J., Kazar, M., Menees, S., Nichols, D.,
Satyanarayanan, M., Sidebotham, R., and M. West, "Scale
and Performance in a Distributed File System", ACM Trans.
Comp. Sys. Vol. 6, No. 1, pp. 51-81, February 1988.
[AFS3-FSCM]
Zayas, E., "AFS-3 Programmer's Reference: File Server/
Cache Manager Interface", Transarc Corp. Tech. Rep. FS-00-
D162, August 1991.
[AFS3-VVL]
Zayas, E., "AFS-3 Programmer's Reference: Volume Server/
Volume Location Server Interface", Transarc Corp. Tech.
Rep. FS-00-D165, August 1991.
[CMU-ITC-83-025]
Morris, J., Van Houweling, D., and K. Slack, "The
Information Technology Center", CMU ITC Tech. Rep. CMU-
ITC-83-025, 1983.
[CMU-ITC-84-020]
West, M., "VICE File System Services", CMU ITC Tech.
Rep. CMU-ITC-84-020, August 1984.
[CMU-ITC-88-070]
Keiser & Jenkins Expires December 17, 2010 [Page 45]
Internet-Draft AFSVol TLV RPCs June 2010
Zayas, E. and C. Everhart, "Design and Specification of
the Cellular Andrew Environment", CMU ITC Tech. Rep. CMU-
ITC-88-070, August 1988.
[RFC4506] Eisler, M., "XDR: External Data Representation Standard",
STD 67, RFC 4506, May 2006.
[VICE1] Satyanarayanan, M., Howard, J., Nichols, D., Sidebotham,
R., Spector, A., and M. West, "The ITC Distributed File
System: Principles and Design", Proc. 10th ACM Symp.
Operating Sys. Princ. Vol. 19, No. 5, December 1985.
Appendix A. XDR Definition for FS-CM Capabilities Mechanism
const AFSCAPABILITIESMAX = 196;
typedef afs_uint32 Capabilities<AFSCAPABILITIESMAX>;
/* Viced Capability Flags */
const VICED_CAPABILITY_ERRORTRANS = 0x0001;
const VICED_CAPABILITY_64BITFILES = 0x0002;
const VICED_CAPABILITY_WRITELOCKACL = 0x0004;
const VICED_CAPABILITY_SANEACLS = 0x0008;
/* Cache Manager Capability Flags */
const CLIENT_CAPABILITY_ERRORTRANS = 0x0001;
Appendix B. Sample XDR Definition for AFSVol Capabilities Mechanism
const AFSVOLCAPABILITIESMAX = 196;
typedef afs_uint32 AFSVolCapabilities<AFSVOLCAPABILITIESMAX>;
/* Viced Capability Flags */
const AFSVOL_CAPABILITY_DAFS = 0x0001;
const AFSVOL_CAPABILITY_TLV = 0x0002;
GetCapabilities (
OUT AFSVolCapabilities * caps
) = XXX;
Appendix C. Sample XDR Definition for AFSVol TLV Mechanism
const AFSVOL_TLV_TAG_MAX = 1024; /* upper-bound on number of
* TLV tuples per RPC */
Keiser & Jenkins Expires December 17, 2010 [Page 46]
Internet-Draft AFSVol TLV RPCs June 2010
const AFSVOL_TLV_OPAQUE_MAX = 262144; /* upper-bound on size of
* value payload */
const AFSVOL_TLV_UINT64_MAX = 32768; /* upper-bound on length of
uint64 vector payload */
const AFSVOL_BULK_GETVOLUME_MAX = 1024; /* upper-bound on
* (partition, volume)
* tuples per RPC */
const AFSVOL_TLV_FLAG_UNSUPPORTED = 0x1;
const AFSVOL_TLV_FLAG_READ_ERROR = 0x2;
const AFSVOL_TLV_FLAG_CRITICAL = 0x4;
const AFSVOL_TLV_FLAG_QUALIFIER_NO_MATCH = 0x8;
enum AFSVol_TLV_type {
AFSVOL_TLV_TYPE_NULL = 0,
AFSVOL_TLV_TYPE_TRUE = 1,
AFSVOL_TLV_TYPE_FALSE = 2,
AFSVOL_TLV_TYPE_UINT64 = 3,
AFSVOL_TLV_TYPE_UINT64_VEC = 4,
AFSVOL_TLV_TYPE_UUID = 5,
AFSVOL_TLV_TYPE_STRING = 6,
AFSVOL_TLV_TYPE_VOL_DOW_USE = 7,
AFSVOL_TLV_TYPE_OPAQUE = 8
};
const AFSVOL_VOL_STAT_DOW0_VALID = 0x1;
const AFSVOL_VOL_STAT_DOW1_VALID = 0x2;
const AFSVOL_VOL_STAT_DOW2_VALID = 0x4;
const AFSVOL_VOL_STAT_DOW3_VALID = 0x8;
const AFSVOL_VOL_STAT_DOW4_VALID = 0x10;
const AFSVOL_VOL_STAT_DOW5_VALID = 0x20;
const AFSVOL_VOL_STAT_DOW6_VALID = 0x40;
const AFSVOL_VOL_STAT_DOW_FUZZY = 0x80;
stuct AFSVol_stat_use_per_dow {
afs_uint64 stat_dow[7];
afs_uint32 stat_flags;
};
enum AFSVol_vol_state_expl {
AFSVOL_VOL_STATE_EXPL_NONE = 0,
AFSVOL_VOL_STATE_EXPL_UNKNOWN = 1,
AFSVOL_VOL_STATE_EXPL_OUT_OF_SERVICE = 2,
AFSVOL_VOL_STATE_EXPL_DELETED = 3,
AFSVOL_VOL_STATE_EXPL_READY = 4,
AFSVOL_VOL_STATE_EXPL_ATTACHING = 5,
AFSVOL_VOL_STATE_EXPL_DETACHING = 6,
Keiser & Jenkins Expires December 17, 2010 [Page 47]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_VOL_STATE_EXPL_BUSY = 7,
AFSVOL_VOL_STATE_EXPL_IO_BUSY = 8,
AFSVOL_VOL_STATE_EXPL_SALVAGING = 9,
AFSVOL_VOL_STATE_EXPL_SALVAGE_NEEDED = 10,
AFSVOL_VOL_STATE_EXPL_ERROR = 11,
AFSVOL_VOL_STATE_EXPL_VOLUME_OPERATION = 12
};
enum AFSVol_program_type {
AFSVOL_PROGRAM_TYPE_NONE = 0,
AFSVOL_PROGRAM_TYPE_FILE_SERVER = 1,
AFSVOL_PROGRAM_TYPE_VOLUME_SERVER = 2,
AFSVOL_PROGRAM_TYPE_SALVAGER = 3,
AFSVOL_PROGRAM_TYPE_SALVAGE_SERVER = 4,
AFSVOL_PROGRAM_TYPE_VOLUME_UTILITY = 5,
AFSVOL_PROGRAM_TYPE_UNKNOWN = 6
};
union AFSVol_TLV_value switch(AFSVol_TLV_type type) {
case AFSVOL_TLV_TYPE_NULL:
case AFSVOL_TLV_TYPE_TRUE:
case AFSVOL_TLV_TYPE_FALSE:
void;
case AFSVOL_TLV_TYPE_UINT64:
afs_uint64 u_64;
case AFSVOL_TLV_TYPE_UINT64_VEC:
afs_uint64 u_64_vec<AFSVOL_TLV_UINT64_MAX>;
case AFSVOL_TLV_TYPE_UUID:
afsUUID u_uuid;
case AFSVOL_TLV_TYPE_STRING:
string u_string<AFSVOL_TLV_OPAQUE_MAX>;
case AFSVOL_TLV_TYPE_VOL_DOW_USE:
AFSVol_stat_use_per_dow u_vol_dow_use;
case AFSVOL_TLV_TYPE_OPAQUE:
default:
opaque u_opaque<AFSVOL_TLV_OPAQUE_MAX>;
};
/* registrar-controlled tag namespace */
enum AFSVol_TLV_tag {
AFSVOL_TLV_TAG_EOS = 0,
AFSVOL_TLV_TAG_VOL_NAME = 1,
Keiser & Jenkins Expires December 17, 2010 [Page 48]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_STATUS = 2,
AFSVOL_TLV_TAG_VOL_IN_USE = 3,
AFSVOL_TLV_TAG_VOL_ID = 4,
AFSVOL_TLV_TAG_VOL_TYPE = 5,
AFSVOL_TLV_TAG_VOL_CLONE_ID = 6,
AFSVOL_TLV_TAG_VOL_BACKUP_ID = 7,
AFSVOL_TLV_TAG_VOL_PARENT_ID = 8,
AFSVOL_TLV_TAG_VOL_COPY_DATE = 9,
AFSVOL_TLV_TAG_VOL_CREATE_DATE = 10,
AFSVOL_TLV_TAG_VOL_ACCESS_DATE = 11,
AFSVOL_TLV_TAG_VOL_UPDATE_DATE = 12,
AFSVOL_TLV_TAG_VOL_BACKUP_DATE = 13,
AFSVOL_TLV_TAG_VOL_SIZE = 14,
AFSVOL_TLV_TAG_VOL_FILE_COUNT = 15,
AFSVOL_TLV_TAG_VOL_QUOTA_BLOCKS = 16,
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY = 17,
AFSVOL_TLV_TAG_VOL_STAT_USE_PER_DOW = 18,
AFSVOL_TLV_TAG_VOL_STAT_READS = 19,
AFSVOL_TLV_TAG_VOL_STAT_WRITES = 20,
AFSVOL_TLV_TAG_VOL_STAT_FILE_SAME_AUTHOR = 21,
AFSVOL_TLV_TAG_VOL_STAT_FILE_DIFFERENT_AUTHOR = 22,
AFSVOL_TLV_TAG_VOL_STAT_DIR_SAME_AUTHOR = 23,
AFSVOL_TLV_TAG_VOL_STAT_DIR_DIFFERENT_AUTHOR = 24,
AFSVOL_TLV_TAG_VOL_TRANS_ID = 25,
AFSVOL_TLV_TAG_VOL_TRANS_TIME = 26,
AFSVOL_TLV_TAG_VOL_TRANS_CREATE_TIME = 27,
AFSVOL_TLV_TAG_VOL_TRANS_RETURN_CODE = 28,
AFSVOL_TLV_TAG_VOL_TRANS_ATTACH_MODE = 29,
AFSVOL_TLV_TAG_VOL_TRANS_STATUS = 30,
AFSVOL_TLV_TAG_VOL_TRANS_FLAGS = 31,
AFSVOL_TLV_TAG_VOL_TRANS_LAST_PROC_NAME = 32,
AFSVOL_TLV_TAG_VOL_TRANS_CALL_VALID = 33,
AFSVOL_TLV_TAG_VOL_TRANS_READ_NEXT = 34,
AFSVOL_TLV_TAG_VOL_TRANS_XMIT_NEXT = 35,
AFSVOL_TLV_TAG_VOL_TRANS_LAST_RECV_TIME = 36,
AFSVOL_TLV_TAG_VOL_TRANS_LAST_SEND_TIME = 37,
AFSVOL_TLV_TAG_VOL_IN_SERVICE = 38,
AFSVOL_TLV_TAG_VOL_BLESSED = 39,
AFSVOL_TLV_TAG_VOL_RESTORED_FROM_ID = 40,
AFSVOL_TLV_TAG_VOL_DESTROYED = 41,
AFSVOL_TLV_TAG_VOL_NEEDS_SALVAGE = 42,
AFSVOL_TLV_TAG_VOL_OFFLINE_MESSAGE = 43,
AFSVOL_TLV_TAG_VOL_EXPIRATION_DATE = 44,
AFSVOL_TLV_TAG_VOL_QUOTA_RESERVATION = 45,
AFSVOL_TLV_TAG_VOL_STAT_USE_TODAY_DATE = 46,
AFSVOL_TLV_TAG_VOL_STATE_ONLINE = 47,
AFSVOL_TLV_TAG_VOL_STATE_AVAILABLE = 48,
AFSVOL_TLV_TAG_VOL_STATE_EXPL = 49,
Keiser & Jenkins Expires December 17, 2010 [Page 49]
Internet-Draft AFSVol TLV RPCs June 2010
AFSVOL_TLV_TAG_VOL_STATE_DAFS_RAW = 50,
AFSVOL_TLV_TAG_VOL_STATE_OWNING_PROCESS = 51,
AFSVOL_TLV_TAG_VOL_QUOTA_BLOCKS_STORED_LOCALLY = 52,
AFSVOL_TLV_TAG_VOL_QUOTA_FILES = 53
};
struct AFSVol_TLV {
afs_uint32 tlv_tag;
afs_uint32 tlv_flags;
AFSVol_TLV_value tlv_value;
};
struct AFSVol_TLV_query {
AFSVol_TLV_tag tq_tag;
AFSVol_TLV_value tq_qualifier;
};
struct AFSVol_TLV_store {
AFSVol_TLV ts_tuple;
AFSVol_TLV_value ts_qualifier;
};
proc GetVolumeTLVTags(
IN AFSVol_TLV_tag offset,
OUT AFSVol_TLV_tag * tags<AFSVOL_TLV_TAG_MAX>
) = XXX;
proc GetOneVolumeTLV(
IN afs_uint32 partId,
IN afs_uint64 volId,
IN AFSVol_TLV_query queries<AFSVOL_TLV_TAG_MAX>,
OUT AFSVol_TLV * tuples<AFSVOL_TLV_TAG_MAX>
) = XXX;
proc GetVolumesTLV(
IN afs_uint32 partIds<AFSVOL_BULK_GETVOLUME_MAX>,
IN afs_uint64 volIds<AFSVOL_BULK_GETVOLUME_MAX>,
IN AFSVol_TLV_query queries<AFSVOL_TLV_TAG_MAX>
) split = XXX;
proc SetVolumeTLV(
IN afs_int32 trans,
IN AFSVol_TLV_store tuples<AFSVOL_TLV_TAG_MAX>,
OUT afs_int32 * results<AFSVOL_TLV_TAG_MAX>
) = XXX;
Keiser & Jenkins Expires December 17, 2010 [Page 50]
Internet-Draft AFSVol TLV RPCs June 2010
Figure 9
Authors' Addresses
Thomas Keiser
Sine Nomine Associates
43596 Blacksmith Square
Ashburn, VA 20147
USA
Phone: +1 703 723 6673
Email: tkeiser@sinenomine.net
Steven Jenkins
Sine Nomine Associates
43596 Blacksmith Square
Ashburn, VA 20147
USA
Phone: +1 703 723 6673
Email: steven.jenkins@gmail.com
Keiser & Jenkins Expires December 17, 2010 [Page 51]