Adding an Uncacheable File Data Attribute to NFSv4.2
draft-ietf-nfsv4-uncacheable-files-03
| Document | Type | Active Internet-Draft (nfsv4 WG) | |
|---|---|---|---|
| Author | Thomas Haynes | ||
| Last updated | 2026-01-16 (Latest revision 2026-01-13) | ||
| Replaces | draft-ietf-nfsv4-uncacheable | ||
| RFC stream | Internet Engineering Task Force (IETF) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Additional resources | Mailing list discussion | ||
| Stream | WG state | In WG Last Call | |
| Document shepherd | (None) | ||
| IESG | IESG state | I-D Exists | |
| Consensus boilerplate | Unknown | ||
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-ietf-nfsv4-uncacheable-files-03
Network File System Version 4 T. Haynes
Internet-Draft Hammerspace
Intended status: Standards Track 13 January 2026
Expires: 17 July 2026
Adding an Uncacheable File Data Attribute to NFSv4.2
draft-ietf-nfsv4-uncacheable-files-03
Abstract
Network File System version 4.2 (NFSv4.2) clients commonly perform
client-side caching of file data in order to improve performance. On
some systems, applications may influence client data caching
behavior, but there is no standardized mechanism for a server or
administrator to indicate that particular file data should not be
cached by clients for reasons of performance or correctness. This
document introduces a new file data caching attribute for NFSv4.2.
Files marked with this attribute are intended to be accessed with
client-side caching of file data suppressed, in order to support
workloads that require predictable data visibility. This document
extends NFSv4.2 (see RFC7862).
Note to Readers
Discussion of this draft takes place on the NFSv4 working group
mailing list (nfsv4@ietf.org), which is archived at
https://mailarchive.ietf.org/arch/search/?email_list=nfsv4. Source
code and issues list for this draft can be found at
https://github.com/ietf-wg-nfsv4/uncacheable-files.
Working Group information can be found at https://github.com/ietf-wg-
nfsv4.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
Haynes Expires 17 July 2026 [Page 1]
Internet-Draft Uncacheable File January 2026
This Internet-Draft will expire on 17 July 2026.
Copyright Notice
Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Requirements Language . . . . . . . . . . . . . . . . . . 4
2. Client-Side Caching of File Data . . . . . . . . . . . . . . 4
2.1. Non-Goals . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2. Uncacheable File Data . . . . . . . . . . . . . . . . . . 5
3. Setting the Uncacheable File Data Attribute . . . . . . . . . 6
4. Implementation Status . . . . . . . . . . . . . . . . . . . . 7
5. XDR for Uncacheable Attribute . . . . . . . . . . . . . . . . 7
6. Extraction of XDR . . . . . . . . . . . . . . . . . . . . . . 8
7. Security Considerations . . . . . . . . . . . . . . . . . . . 8
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 8
9.1. Normative References . . . . . . . . . . . . . . . . . . 8
9.2. Informative References . . . . . . . . . . . . . . . . . 9
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 10
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10
1. Introduction
Clients of remote filesystems commonly perform client-side caching of
file dat in order to improve performance. Such caching may include
retaining data read from the server to satisfy subsequent READ
requests, as well as retaining data written by applications in order
to delay or combine WRITE requests before transmitting them to the
server. While these techniques are effective for many workloads,
they may be unsuitable for workloads that require predictable data
visibility or involve concurrent modification of shared files by
multiple clients.
Haynes Expires 17 July 2026 [Page 2]
Internet-Draft Uncacheable File January 2026
In some cases, Network File System version 4.2 (NFSv4.2) (see
[RFC7862]) mechanisms such as file delegations can reduce the impact
of concurrent access. However, delegations are not always available
or effective, particularly for workloads with frequent concurrent
writers or rapidly changing access patterns.
There have been prior efforts to bypass file data caching in order to
address these issues. In High-Performance Computing (HPC) workloads,
file data caching is often bypassed to improve predictability and to
avoid read-modify-write hazards when multiple clients write disjoint
byte ranges of the same file.
Applications on some systems can request bypass of the client data
cache by opening files with the O_DIRECT flag (see [OPEN-O_DIRECT]).
However, this approach has limitations, including the requirement
that each application be explicitly modified and the lack of a
standardized mechanism for communicating this intent between servers
and clients.
This document introduces the uncacheable file data attribute to
NFSv4.2. This OPTIONAL attribute allows a server to indicate that
client-side caching of file data for a particular file is unsuitable.
When both the client and the server support this attribute, the
client is advised suppress client-side caching of file data for that
file, in accordance with the semantics defined in this document.
The uncacheable file data attribute is read-write, applies on a per-
file basis, and has a data type of boolean.
Support for the uncacheable file data attribute is specific to the
exported filesystem and may differ between filesystems served by the
same server. A client can determine whether the attribute is
supported for a given file by issuing a GETATTR request and examining
the returned attribute list.
The uncacheable file data attribute applies only to regular files
(NF4REG). Attempts to query or set this attribute on objects of
other types MUST result in an error of NFS4ERR_INVAL. Since the
uncacheable file data attribute applies only to regular files,
attempts to apply it to other object types represent an invalid use
of the attribute.
Using the process described in [RFC8178], the revisions in this
document extend NFSv4.2 [RFC7862]. They are built on top of the
external data representation (XDR) [RFC4506] generated from
[RFC7863].
Haynes Expires 17 July 2026 [Page 3]
Internet-Draft Uncacheable File January 2026
1.1. Definitions
client-side caching of file data The retention of file data by a
client in a local data cache, commonly referred to as the page
cache, for the purpose of satisfying subsequent READ requests or
delaying transmission of WRITE data to the server.
write-behind caching A form of file data caching in which WRITE data
is retained by the client and transmission of the data to the
server is delayed in order to combine multiple WRITE operations or
improve efficiency.
direct I/O An access mode in which file data is transferred between
application buffers and the underlying storage without populating
or consulting the client's file data cache. Direct I/O suppresses
both read caching and write-behind caching of file data.
write hole A write hole is an instance of data corruption that
arises when multiple clients modify disjoint byte ranges within
the same encoded data block without having a consistent view of
the existing contents. This can result in stale data overwriting
newer updates, particularly in environments that use erasure
encoding or striped storage. (Adapted from
[I-D.haynes-nfsv4-flexfiles-v2].)
This document assumes familiarity with the NFSv4 protocol operations,
error codes, object types, and attributes as defined in [RFC8881].
1.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2. Client-Side Caching of File Data
The uncacheable file data attribute advises the client to bypass its
page cache for a file in certain troublesome cases. These include
forms of client-side caching of file data such as write-behind
caching, in which multiple pending WRITEs are combined and
transmitted to the server at a later time for efficiency. The
uncacheable file data attribute inhibits such behavior with an effect
similar to that of using the O_DIRECT flag with the open call
([OPEN-O_DIRECT]).
Haynes Expires 17 July 2026 [Page 4]
Internet-Draft Uncacheable File January 2026
The intent of this attribute is to allow a server or administrator to
indicate that client-side caching of file data for a particular file
is unsuitable. The server is often in a better position than
individual clients to determine sharing patterns, access behavior, or
correctness requirements associated with a file. By exposing this
information via an attribute, the server can advise clients to
suppress file data caching in a consistent manner.
One important use case for this attribute arises in connection with
High-Performance Computing (HPC) workloads. These workloads often
involve large data transfers and concurrent access by multiple
clients. In such environments, client-side caching of file data can
introduce unpredictable latency or correctness hazards when data is
buffered and flushed at a later time.
Another aspect of such workloads is the need to support concurrent
writers to shared files. When application data spans a data block in
a client cache, delayed transmission of WRITE data can result in
clients modifying stale data and overwriting updates written by
others. Prompt transmission of WRITE data enables the prompt
detection of write holes and reduces the risk of data corruption.
2.1. Non-Goals
This attribute does not require clients to provide strict coherency,
does not replace existing NFS cache consistency mechanisms, and does
not mandate any specific client implementation strategy. It provides
advisory guidance intended to reduce latency and correctness risks in
selected workloads.
2.2. Uncacheable File Data
When a file object is marked as uncacheable file data, the attribute
advises the client that client-side caching of file data for the file
is unsuitable. In particular, the client is advised to transmit
modifications to the file promptly rather than retaining them in a
local data cache. Note that a client that does not query this
attribute cannot be expected to observe the behavior described in
this section.
For uncacheable file data, the client is advised not to retain file
data in its local data cache for the purpose of satisfying subsequent
READ requests or delaying transmission of WRITE data. In such cases,
READ operations bypass the client data cache, and WRITE data is not
retained for read-after-write satisfaction or for the purpose of
combining multiple WRITE requests.
Haynes Expires 17 July 2026 [Page 5]
Internet-Draft Uncacheable File January 2026
Caching of unstably written data used to reissue WRITEs lost because
of server failure prior to COMMIT is not affected by the advice
provided by the uncacheable file data attribute. This is because the
server is made aware of the WRITE operation without the sort of
delays introduced by write-behind caching.
Suppressing read caching in addition to suppressing write-behind
caching reduces the risk of stale-data overwrite in multi-writer
workloads. If a client retains cached READ data while other clients
concurrently modify disjoint byte ranges of the same file, the client
may perform a read-modify-write operation using stale data and
overwrite updates written by others. This risk exists even when
WRITE operations are transmitted promptly.
Disabling READ caching allows clients to observe the most recent data
prior to modification and reduces read-modify-write hazards for
shared files. This behavior is consistent with direct I/O semantics
such as those provided by the O_DIRECT flag in Linux and the
directio/forcedirectio mechanisms in Solaris.
If the fattr4_uncacheable_file_data attribute is not set when a file
is opened and is changed while the file is open, the client is not
expected to retroactively alter its caching behavior. A client may
choose to flush cached data and apply the advice to subsequent I/O,
but such behavior is not required until the file is closed and
reopened.
The presence of the uncacheable file data attribute does not
invalidate file delegations. A server that wishes to ensure prompt
client I/O may choose not to issue write delegations for files marked
as uncacheable, but clients are not required to suppress delegations
solely due to the presence of this attribute.
3. Setting the Uncacheable File Data Attribute
The uncacheable file data attribute provides a mechanism by which
applications that do not support O_DIRECT can request DIRECT-I/O-like
semantics for file access. In particular, the attribute allows a
server to advise clients that client-side caching of file data for a
file is unsuitable, including both read caching and write-behind
caching.
Haynes Expires 17 July 2026 [Page 6]
Internet-Draft Uncacheable File January 2026
Suppressing read caching is necessary in addition to suppressing
write-behind caching to avoid read-modify-write hazards in multi-
writer workloads. If clients retain cached READ data while other
clients concurrently modify disjoint byte ranges of the same file,
stale cached data may be merged with new WRITE data and overwrite
updates written by others. This risk exists even when WRITE data is
transmitted promptly and is not addressed by suppressing write-behind
caching alone.
One possible deployment model is for a server or administrator to
configure a mount (see [MOUNT]) option such that newly created files
under a given export are marked as uncacheable file data. In such a
configuration, the NFSv4.2 client could use SETATTR to set the
fattr4_uncacheable_file_data attribute at file creation time.
This approach is conceptually similar int intent to the Solaris
forcedirectio mount option (see [SOLARIS-FORCEDIRECTIO]), but differs
in scope and visibility in that it allows DIRECT-I/O-like behavior to
be applied without requiring changes to individual applications.
However, unlike the Solaris option, the NFSv4.2 attribute is visible
to all clients accessing the file and is intended to convey server-
side knowledge or policy in a distributed environment.
4. Implementation Status
There is a prototype Hammerspace server which implements the
uncacheable file data attribute and a prototype Linux client which
treats the uncacheable file data attribute as an indication to use
O_DIRECT. For the prototype, all files created under the mount point
have the fattr4_uncacheable_file_data set to be true.
Experience with the prototype indicates that the uncacheable file
data attribute can provide many of the practical benefits of O_DIRECT
without requiring application modification. For applications that
issue well-formed I/O requests, this approach has been observed to
improve performance in many cases, while also reducing memory
pressure and CPU utilization in the NFS client.
5. XDR for Uncacheable Attribute
///
/// typedef bool fattr4_uncacheable_file_data;
///
/// const FATTR4_UNCACHEABLE_FILE_DATA = 87;
///
Haynes Expires 17 July 2026 [Page 7]
Internet-Draft Uncacheable File January 2026
6. Extraction of XDR
This document contains the external data representation (XDR)
[RFC4506] description of the uncacheable file attribute. The XDR
description is presented in a manner that facilitates easy extraction
into a ready-to-compile format. To extract the machine-readable XDR
description, use the following shell script:
<CODE BEGINS>
#!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'
<CODE ENDS>
For example, if the script is named 'extract.sh' and this document is
named 'spec.txt', execute the following command:
<CODE BEGINS>
sh extract.sh < spec.txt > uncacheable_prot.x
<CODE ENDS>
This script removes leading blank spaces and the sentinel sequence
'///' from each line. XDR descriptions with the sentinel sequence
are embedded throughout the document.
Note that the XDR code contained in this document depends on types
from the NFSv4.2 nfs4_prot.x file (generated from [RFC7863]). This
includes both nfs types that end with a 4, such as offset4, length4,
etc., as well as more generic types such as uint32_t and uint64_t.
While the XDR can be appended to that from [RFC7863], the code
snippets should be placed in their appropriate sections within the
existing XDR.
7. Security Considerations
The attribute does not introduce new authorization mechanisms or
alter existing access control semantics; existing NFSv4.2 security
mechanisms continue to apply.
8. IANA Considerations
This document has no IANA actions.
9. References
9.1. Normative References
Haynes Expires 17 July 2026 [Page 8]
Internet-Draft Uncacheable File January 2026
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[RFC4506] Eisler, M., Ed., "XDR: External Data Representation
Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May
2006, <https://www.rfc-editor.org/rfc/rfc4506>.
[RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor
Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862,
November 2016, <https://www.rfc-editor.org/rfc/rfc7862>.
[RFC7863] Haynes, T., "Network File System (NFS) Version 4 Minor
Version 2 External Data Representation Standard (XDR)
Description", RFC 7863, DOI 10.17487/RFC7863, November
2016, <https://www.rfc-editor.org/rfc/rfc7863>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
[RFC8178] Noveck, D., "Rules for NFSv4 Extensions and Minor
Versions", RFC 8178, DOI 10.17487/RFC8178, July 2017,
<https://www.rfc-editor.org/rfc/rfc8178>.
[RFC8881] Noveck, D., Ed. and C. Lever, "Network File System (NFS)
Version 4 Minor Version 1 Protocol", RFC 8881,
DOI 10.17487/RFC8881, August 2020,
<https://www.rfc-editor.org/rfc/rfc8881>.
9.2. Informative References
[I-D.haynes-nfsv4-flexfiles-v2]
Haynes, T., "Parallel NFS (pNFS) Flexible File Layout
Version 2", Work in Progress, Internet-Draft, draft-
haynes-nfsv4-flexfiles-v2-02, 20 October 2025,
<https://datatracker.ietf.org/doc/html/draft-haynes-nfsv4-
flexfiles-v2-02>.
[MOUNT] Linux man-pages project, "mount(2) - mount filesystem",
Linux Programmer's Manual, 2024,
<https://man7.org/linux/man-pages/man2/mount.2.html>.
[OPEN-O_DIRECT]
Linux man-pages project, "open(2) - Linux system call for
opening files (O_DIRECT)", 2024,
<https://man7.org/linux/man-pages/man2/open.2.html>.
Haynes Expires 17 July 2026 [Page 9]
Internet-Draft Uncacheable File January 2026
[SOLARIS-FORCEDIRECTIO]
Oracle Solaris Documentation, "mount -o forcedirectio -
Solaris forcedirectio mount option",
Solaris Administration Guide, 2023,
<https://docs.oracle.com/en/operating-systems/solaris/
oracle-solaris/11.4/manage-nfs/mount-options-for-nfs-file-
systems.html>.
Acknowledgments
Trond Myklebust, Mike Snitzer, Jon Flynn, Keith Mannthey, and Thomas
Haynes all worked on the prototype at Hammerspace.
Rick Macklem, Chuck Lever, and Dave Noveck reviewed the document.
Chris Inacio, Brian Pawlowski, and Gorry Fairhurst helped guide this
process.
Author's Address
Thomas Haynes
Hammerspace
Email: loghyr@gmail.com
Haynes Expires 17 July 2026 [Page 10]