Caching subgroup of HTTP working group Jeffrey Mogul/DECWRL
Internet-Draft 15 February 1996
Expires: 20 August 1996
A design for caching in HTTP 1.1
Preliminary Draft
draft-mogul-http-caching-00.txt
STATUS OF THIS MEMO
THIS IS A PRELIMINARY DRAFT. IT DOES NOT AT THE MOMENT
REPRESENT THE CONSENSUS OF THE CACHING SUBGROUP OF THE HTTP
WORKING GROUP, OR OF THE HTTP WORKING GROUP.
This document is an Internet-Draft. Internet-Drafts are
working documents of the Internet Engineering Task Force
(IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of
six months and may be updated, replaced, or obsoleted by
other documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other
than as "work in progress".
To learn the current status of any Internet-Draft, please
check the "1id-abstracts.txt" listing contained in the
Internet-Drafts Shadow Directories on ftp.is.co.za
(Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific
Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US
West Coast).
WARNING: The specification in this document is subject to
change, and will certainly change. It is inappropriate AND
STUPID to implement to the proposed specification in this
document. In particular, anyone who implements to this
specification and then complains when it changes will be
properly viewed as an idiot, and any such complaints shall
be ignored. YOU HAVE BEEN WARNED.
Distribution of this document is unlimited. Please send
comments to the HTTP working group at
<http-wg@cuckoo.hpl.hp.com>. Discussions of the working
group are archived at
<URL:http://www.ics.uci.edu/pub/ietf/http/>. General
discussions about HTTP and the applications which use HTTP
should take place on the <www-talk@w3.org> mailing list.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 1]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
ABSTRACT
I propose a caching design for HTTP 1.1, including the
definition of terms, the description of a model, the
specification of related HTTP headers, and the algorithms
to be used.
The proposed design uses opaque cache validators and
explicit expiration values to allow the server to control
the tradeoff between cache performance and staleness of the
data presented to users. The server may choose to ensure
that a user never unwittingly sees stale data, or to
minimize network traffic, or to compromise between these
two extremes. The proposed design also allows the server
to control whether a client sees stale data after another
client performs an update.
This draft is not intended to become a standards-track
document. The ultimate output of the HTTP subgroup will be
a set of changes to the HTTP/1.1 draft specification [1].
In the interests of making rapid forward progress and in
capturing some of the motivation and rationale of our
design(s), it seems useful to produce an Internet-draft
document in the interim. Note that this draft in its
entirety does not currently represent the consensus of the
subgroup, although several sections do represent an
apparent consensus.
TABLE OF CONTENTS
1 Introduction 3
2 Philosophy: make everything work, optimize for common cases 4
3 Caching model 5
3.1 Overview 5
3.2 Definitions 5
3.3 Goals 8
3.4 Cache keys 9
3.5 Cache validation and the ``immutable cache'' concept 9
3.6 Opaque validators 11
3.7 Cache operation when receiving validator from client 13
3.8 Cache operation when receiving errors or incomplete 14
responses
3.9 Bounding freshness 14
3.9.1 Disambiguating expiration values 16
3.9.2 Disambiguating multiple responses 17
3.10 Explicit Cache-control 18
3.11 Warnings 20
3.12 Compatibility with earlier versions of HTTP 21
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 2]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
3.12.1 Compatibility between HTTP/1.1 clients and older 21
servers
3.12.2 Compatibility between HTTP/1.1 servers and older 21
clients
3.12.3 Compatibility between HTTP/1.1 and older proxies 21
3.13 Update consistency 21
3.14 Volume validation 23
3.15 Side effects of GET and HEAD 25
3.16 Write-through mandatory 26
4 HTTP headers related to caching 26
4.1 Cache-validation 26
4.2 Expiration values 27
4.3 Age values 28
4.4 Cache-control 28
4.5 Warnings 29
4.6 Volume validation 29
5 Rules summarized by method 30
6 Cache replacement algorithms 30
7 Specific issues 30
7.1 Content negotiation 30
7.2 Authentication 30
7.3 State 30
7.4 Caching and methods with side effects 31
7.5 Network partitions 31
7.6 Caching of negative responses 31
7.7 History Lists 31
7.8 Hit Metering 31
7.9 Bypassing in caching hierarchies 32
7.10 Explicit warnings regarding user-specified overrides 33
8 Security Considerations 33
8.1 Caching of responses to authorized requests 33
8.2 Location: headers and spoofing 34
9 Acknowledgements 36
10 References 36
11 Author's address 36
1 Introduction
The World Wide Web is a distributed system, and so its performance
can be improved by the use of caches. These caches are typically
placed at proxies and in the clients themselves.
The use of caches introduces two problems: how do we ensure that the
caches return the ``correct'' values, and how do we obtain the best
possible performance. In general, there may be a tradeoff between
how ``correct'' the returned values are, and how effectively the
caches perform.
At issue in the specification of HTTP, therefore, is the design of a
caching mechanism that produces the most appropriate possible
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 3]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
tradeoff between correctness and performance. In this document, I
propose a design that uses opaque cache validators and explicit
expiration values to allow the server to control the tradeoff between
cache performance and staleness of the data presented to users. The
server may choose to ensure that a user never unwittingly sees stale
data, or to minimize network traffic, or to compromise between these
two extremes.
One can get into lengthy debates about what ``correct'' means with
respect to HTTP caching. To sidestep such pointless debates, I will
avoid using the word ``correctness'' and use the specific term
``semantic transparency.'' A cache is completely semantically
transparent if clients cannot tell that it exists, except for the
speed at which responses are delivered.
Part of the semantic transparency problem is that when multiple
copies of a value exist, and one copy is updated, a client using
another copy may not see the update. This can lead to
inconsistencies. The proposed design also allows the server to
control whether a client sees stale data after another client
performs an update.
Server-based control is also important because HTTP may be used for a
wide variety of ``applications.'' The design of a Web application
(for example, a stock-trading system) may be peculiar to the server,
while Web browsers are generic to all Web applications. Because the
precise behavior of an application cannot be known to the implementor
of a browser, but can be controlled by the implementor of a server,
servers need to have the option of direct control over the caching
mechanism. Because the world is not perfect, we also need to give
users and browsers some control over caching, but this is at best a
contingency plan.
2 Philosophy: make everything work, optimize for common cases
Applications in which HTTP is used span a wide space of interaction
styles. For some of those applications, the origin server needs to
impose strict controls on when and where values are cached, or else
the application simply fails to work properly. We can refer to these
as the ``corner cases.'' In (perhaps) most other cases, on the other
hand, caching does not interfere with the application semantics. We
can call this the ``common case.''
Caching in HTTP should provide the best possible performance in the
common case, but the HTTP protocol must entirely support the
semantics of the corner cases, and in particular an origin server
must be able to defeat caching in such a way that any attempt to
override this decision cannot be made without an explicit
understanding that in doing so the proxy or client is going to suffer
from incorrect behavior. In other words, if the origin server says
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 4]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
``do not cache'' and you decide to cache anyway, you have to do the
equivalent of signing a waiver form.
We explicitly reject an approach in which the protocol is designed to
maximize performance for the common case by making the corner cases
fail to work correctly.
3 Caching model
In this section, I present a model for HTTP caching but do not
provide detailed specifications for headers and algorithms.
3.1 Overview
The caching model has these major design elements:
1. The basic protocol provides a way to enforce strict
semantic transparency; if servers, caches, and clients
follow these rules, then users will never see something
other than what the server intends.
2. The protocol provides ways for the servers, caches, and
clients to loosen the strictness rules. In effect, the
server grants permission for a cache to behave loosely.
The amount of looseness is explicitly bounded by the
participants, and user agents are always aware if a
response may be the result of a loosening of the rules.
In this proposal, loosening is done by bounding the amount
of time that an invalid cached response can be used, but
other techniques might be developed.
3. A cache can check if a cached value is valid by checking
with a server, and only reloading the value if the cached
version may be invalid. This is done using ``conditional''
HTTP methods, whose semantics depends on the validity
check.
4. The protocol includes an explicit Cache-control mechanism
that allows servers or clients to directly control aspects
of caching.
5. The protocol includes a warning mechanism to transmit
indications of possibly non-transparent caching.
3.2 Definitions
Here are some useful terms that need specific definition.
First, we have found it useful to distinguish between various places
where responses, or pointers to responses, might be stored, and
between idealized and practical instances of each:
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 5]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
ideal cache exists to model the origin server's responses to
requests. Semantically transparent to the user for
all practical purposes, but should improve
performance according to one or more metrics.
ideal history buffer
records what the user has seen during past
interactions, and provides a means for the user to
revisit those views exactly as they were seen.
ideal location log
records what locations the user has visited during
past interactions; provides a means for the user to
revisit those locations to discover their current
contents.
practical cache exists to model the origin server's responses to
requests. Semantically transparent to the user,
except that the origin server, user, or cache
administrator may choose to compromise transparency
in the service of performance or availability.
Should improve performance according to one or more
metrics.
practical history buffer
records what the user has seen during some subset of
past interactions, and provides a means for the user
to revisit those views exactly as they were seen, or
as best as can be approximated given finite buffer
space.
practical location log
records a subset of the locations the user has
visited during past interactions; provides a means
for the user to revisit those locations to discover
their current contents.
Note that we are not aware of any existing implementations that make
such clear distinctions, but they seem useful for the purposes of
specifying protocol behavior.
The rest of this document specifies the behavior of caches, not
history buffers or location logs.
We leave the behavior of a history buffer up to the browser
implementor. If a browser confuses the distinctions made
between caches and history buffers, then any resulting
erroneous behavior is not the fault of the servers or proxies.
A browser implementor may choose to use the browser's cache to
approximate the history buffer, in which case it is possible
that a user will see something other than the ``historically
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 6]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
accurate view'' of a page, but that is between the user and the
browser vendor; it is not something that the HTTP protocol
specification can define.
This includes issues such as ``does `subset' mean terminal
subsequence?'' and does `` `approximated' mean one can retrieve
a new copy of the page if the old view didn't fit into the
buffer?''
Similarly, we do not specify the implementation of a location
log, although we recommend that it include a terminal
subsequence of the locations visited.
XXX Shel Kaphan may add something more specific here.
conditional method
An HTTP method whose semantics depends on the value
of one or more request headers.
cached object The unit of caching, which consists of an entity
body, entity headers, perhaps other response headers,
perhaps some related request headers, and perhaps the
request entity body.
cache key The portion of a cached object that the caching
algorithms use to determine if the cached object
matches a request.
document expiration time
The time, if any, at which an entity should no longer
be seen by a user.
valid A cached entity is valid, with respect to a given
request at a given time, if it is exactly what the
origin server would return in response to that
request at that time.
invalid A cached entity is invalid, with respect to a given
request at a given time, if it is not exactly what
the origin server would return in response to that
request at that time.
unknown validity
A cached entity is of unknown validity if the cache
does not know whether it is valid or invalid.
firsthand A response is firsthand if it comes directly and
without unnecessary delay from the origin server,
perhaps via one or more proxies. A response is also
firsthand if its validity has just been checked
directly with the origin server.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 7]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
expiration time The time at which the origin server intends that an
entity should no longer be returned by a cache unless
the cache first checks with the origin server to see
if the cached object is valid. All cached objects
have an expiration time, but this might be
``Forever'' (infinitely in the future), ``already''
(in the past), or ``undefined.''
fresh A cached entity is fresh if it has not reached its
expiration time.
stale A cached entity is stale if it has reached its
expiration time. Stale entities are not necessarily
useless.
inbound towards the origin server; the request chain flows
inbound.
outbound towards the ultimate client system; the response
chain flows outbound.
3.3 Goals
This caching design values semantic transparency over performance.
That is, performance considerations are not allowed to overrule
transparency considerations in the design itself. In actual
operation of HTTP, it is possible for servers, proxies, or clients to
explicitly choose performance over transparency, but the protocol
does not force this choice on them.
Some people have argued that a ``correct'' caching design must
optimize performance as well as any semantic correctness. For
the purposes of this document, I will explicitly OMIT
performance from the definition of ``correctness.''
It is understood that a GOOD caching design includes good
performance as well as some level of semantic transparency.
This does NOT make performance part of the definition of
semantic transparency.
This depends on a clear definition of what it is to be ``semantically
transparent.'' I will start with this definition:
A response returned by a cache in response to a request is a
semantically transparent result if and only if it includes
exactly the same entity headers and entity body that would be
returned by the origin server to the same client making the
same request at the same time.
A cached response that includes slightly different general headers
(see section 4.3 of [1]) may still be transparent.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 8]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Another way to look at transparency is that:
A user should never have to take explicit action to see exactly
the version of an entity that the server wants the user to see.
Semantic transparency means that the only user-visible effect of a
cache is a probable improvement in performance. We do not insist on
semantic transparency in all cases, but we insist that if the user
and server both want semantic transparency, then the caches MUST
provide it.
Users should not have to use ``Reload'' buttons to simply assure
themselves that they are seeing valid copies. One will still have to
use ``Reload'' to refresh the page currently on your screen, but only
to indicate to the user agent that you have finished reading it and
are willing to see a new copy, if any.
I now state three goals:
- The protocol must allow semantic transparency, in which the
user is guaranteed to see a valid copy of an object unless
some network failure prevents it.
- The protocol must allow users, servers, and proxies to
choose performance over transparency if they want to, but
must not force them to do so.
- When the protocol delivers a response that may be
non-transparent, the client must made be aware of this.
Because the world is already full of of HTTP implementations, we also
must require that a HTTP/1.1 implementation must interoperate with
pre-1.1 implementations, without being any less transparent than a
pre-1.1 implementation would be, and without being significantly
slower.
3.4 Cache keys
The first step in using a cache is to decide if the requested entity
is present in the cache. This is done by searching the set of cached
objects for one with a cache key that matches the request headers.
For now, we ignore the problem of how the cache matches requests
against its cached objects. This is covered in section 7.1. In
order to understand the other aspects of the model, one can pretend
that the cache key is simply the URL of a cached object.
3.5 Cache validation and the ``immutable cache'' concept
The primary mechanism for ensuring semantic transparency is cache
validation. When a downstream client makes a request via a cache,
the cache must not return a cached entity unless it is fresh, or the
cache has checked with the inbound server that its cached object is
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 9]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
valid. A cache may always return a firsthand (uncached) response, of
course. When multiple caches are involved, the validation check may
be answered by a outbound cache, but in most cases will be answered
by the origin server.
To restate this: a cache may not return a stale entry without first
checking its validity, unless the user or cache administrator has
explicitly allowed the use of stale responses.
This differs from the language in section 10.19 of the draft HTTP/1.1
specification [1], which says ``Applications must not cache this
entity beyond the [Expires:] date given.'' That section should be
reworded to say ``Applications must not return this entity from their
cache after the date given, without first validating it with the
origin server.'' This changes the spec to make it clear when a cache
must use a conditional method before responding to a request.
In HTTP, a client may say to a server ``tell me if my cached copy is
valid, and if it is not, please return a valid copy.'' If a cached
object is found to be invalid, the cache normally will receive a new
copy of the object from the server, return that firsthand (and
presumptively valid) copy to the client, and may enter the new copy
into the cache. Such requests are normally done using the GET method
(although this mechanism might apply to other methods), and if so,
this is referred to as a ``conditional GET.''
Note: the current HTTP/1.1 draft says that there is no such
thing as a ``conditional HEAD'' request. Do we want/need to
retain this restriction? Or is there some possible value to a
conditional HEAD?
This mechanism, all by itself, yields semantically transparent
caching in all cases, although it may not yield optimal performance.
Note that it says nothing at all about removing objects from the
cache.
One could in theory implement a cache as an immutable time-ordered
log of cache objects, where once an object is entered into the cache,
it stays there forever. Because this ``immutable cache'' is a
time-ordered log, only the most recent entry with a given cache key
will match a request. While the log will sometimes contain stale,
invalid entries that are not hidden behind newer entries with the
same key, this is not a problem for transparency because the cache
will never return such entries to a client.
The performance of an infinite immutable cache would not actually be
that bad, if we ignore the cost of doing a key lookup in an infinite
database. Clearly, the cache-hit ratio would be optimal, since all
misses would be compulsory or coherency misses, not capacity misses.
Consider a cache that discarded invalid cache entries. In this case,
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 10]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
the server would return the same number of object copies, but it
would do fewer validity checks and so might see slightly less load.
However, in most cases an invalid entry would immediately be hidden
behind a newer copy of the entity, and so these would seldom be found
on a cache lookup. The only case that seems to matter is when a
subsequent request arrives at the cache before the retrieval is
complete; performance considerations suggest that the cache should
not make parallel requests to the server for the same URI.
Note that this design does not involve any explicit flushing of cache
entries; in fact, the ``immutable cache'' model is incompatible with
flushing (that is, removing one or more entries). Cache-flushing has
been proposed to deal with updates, removals, and other changes, but
it is in fact both unnecessary and insufficient. This will be
discussed in section 3.13.
Some people have suggested the adoption of a mechanism by which a
server can explicitly revoke a permission it had previously given to
cache something loosely. This might be done, for example, using a
callback mechanism. This is a promising approach but requires more
consideration than is feasible for HTTP/1.1. However, section 3.14
presents a technique that should make revocation less necessary.
3.6 Opaque validators
Section 3.5 states ``the cache must not return a cached entity unless
it is either fresh or the cache has checked with the inbound server
that its cached object is valid.'' How does the cache perform this
check?
In the existing protocol (HTTP/1.0 and earlier), the cache can use
the If-modified-since: header to instruct the server to send a full
response only if the entity has not changed since the cache entry was
made. We must retain this mechanism in HTTP/1.1 to provide
compatibility with existing implementations, but If-modified-since
has at least three major faults:
1. This header uses the HTTP-date format to express a time
value. HTTP-date has a resolution of 1 second, and so is
not sufficient to detect if a document is changing more
frequently than that. For example, if client retrieves a
document with a modification time of 9:08:07.01, and then
the document is modified again at 9:08:07.04, the
If-modified-since: mechanism will fail to detect the
change.
2. This forces the server to maintain modification times for
all cachable entities. While a traditional file-system
based HTTP server can do this fairly easily, if the server
stores entities in a relational database, or constructs
them in some other manner, it may be inconvenient for the
server to store modification-date values with every
entity.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 11]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
3. On some popular operating systems, including UNIX,
renaming a file does not change its modification date.
This can confuse the if-modified-since logic. For
example, if a URL initially points to file A with a
modification time of 9:07, and then (through server-side
renaming operations) the same URL later points to an older
file B with a modification time of 9:06, a client doing a
``GET if-modified-since'' with a comparison date of 9:07
will not get the new value of the resource.
A proposal was made in [1] to allow the client to do the
cache-validity check using a general predicate on all header fields
(the ``logic bag'') approach. While this avoids some of the problems
with If-modified-since, it still does not give the server full
flexibility, since the choice of how to determine validity is
essentially left up to the client. This also makes the
implementation of logic bags mandatory, and such an implementation
could be complex or slow. While logic bags may be useful for other
purposes, they seem inappropriate for checking cache validity.
Instead, we can use an ``opaque validator'' mechanism to give the
server full flexibility, while making the implementation as simple as
possible. In this approach, the server returns (using a new response
header) a ``cache validator'' value with each entity. A client MUST
NOT do anything with this value except to store it with a cached
object, and to present it to the server as part of a cache-validation
check. A cache may compare two validators in certain circumstances,
but MUST NOT do anything else except for storing or presenting it.
When the cache wants to check the validity of a cached object, it
presents the cache-validator value to the server (using a new request
header), along with the object's URI. The server then decides
whether the cached object associated with that cache-validator value
is still valid, and informs the cache (for example, by returning "304
Not Modified", or by returning the new value).
When a cache receives a request, checks the validity of the cached
object, and the server indicates that the object is still valid, the
cache may then treat its cached response as a firsthand response, but
only for the purpose of responding to the current request.
An origin server can use almost any algorithm it chooses to decide if
the validator indicates that the cached object is still valid. It
need not use simple equality comparison; for example, the server may
decide that although the cached entity is not exactly the same as the
current copy, it is close enough for any practical purpose. However,
validators are also used by intermediate caches to decide whether to
forward a conditional request to the origin server, and caches can
only perform equality comparisons (because validators are opaque).
This puts a strict requirement on the validator: if two values for
the entity associated with a given resource are semantically
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 12]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
different, then the associated validators MUST be different. In most
cases, therefore, simple equality is the best comparison algorithm.
The server could use any number of methods to construct
cache-validator values. For example, it could simply use the
modification date. However, since the value is opaque, it need not
use the HTTP-date format; it could encode a 64-bit (seconds,
nanoseconds) representation as 16 hex digits. Or it could use a
fingerprinting algorithm, such as MD5. Or it could use the address
of a LISP cell, concatenated with a garbage-collection generation
number. And it could use different algorithms for different kinds of
object.
A cache or client MUST NOT assume that two cached entities with
different URIs but identical cache validators are identical. For
example, the server may be using modification times in its validator,
and two different documents may have identical modification dates.
An HTTP/1.1 server SHOULD return a cache-validator value with every
response.
If a server does not return a cache-validator with a response, but
does return a Last-Modified: date, a client or cache SHOULD validate
a cached object using the If-Modified-Since: header, as is done in
HTTP/1.0. However, HTTP/1.1 clients MUST NOT use
``If-Modified-Since:'' if a cache-validator value was provided by the
server. The provision of a cache-validator value is the server's
only way of stating that the Last-Modified date is not a useful cache
validator (but is being supplied for other reasons).
XXX what if neither a validator or Last-Modified date?
3.7 Cache operation when receiving validator from client
When a client makes a request to a cache, and includes a
cache-validator value (that is, the request method is conditional on
the cache-validator value), if the cache does not have a fresh cached
object that matches the request, it simply forwards the request to
the origin server (or the next inbound cache).
If the request does match a fresh cached object, then the cache MUST
check the offered cache-validator value against the one stored with
the cached object. If the cache-validator value matches, then the
cache can return ``304 Not Modified''. Otherwise, it MUST return a
full response (either from its cache or from the server).
The cache checks the cache-validator values by comparing them for
equality, with the proviso that a null value can never compare equal
to any value.
If a request does not include a cache-validator, then it is not a
conditional request and the cache MUST return a full response.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 13]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
3.8 Cache operation when receiving errors or incomplete responses
A cache that receives an incomplete response (for example, with few
bytes of data than specified in a Content-length: header) MUST not
cache the response or any part of it.
XXX We need to come up with rules for whether to cache
responses with statuses other than
- 200 OK
- 206 Partial content
- 300 Multiple Choices
- 301 Moved Permanently
- 304 Not Modified
- 410 Gone
which seem safe to cache, and
- 302 Moved Temporarily
which are cachable if the server explicitly allows it.
3.9 Bounding freshness
The server bounds the looseness of caching for an entity by
specifying how long it remains fresh. A server that wants to enforce
complete strictness can set this duration to zero; in other cases, a
server may allow an entry to be fresh for a definite period, or
forever.
Section 3.5 states ``the cache must not return a cached entity unless
it is either fresh or the cache has checked with the inbound server
that its cached object is valid.'' How does the cache decide what is
fresh and what is stale?
Recall that a cached entity is fresh until it reaches its expiration
time, and is stale thereafter. The expiration time is provided by
the server in the response headers of the initial response that loads
the entity into the cache. It may also be updated as the result of a
validation check that does not otherwise change the entity (e.g., a
conditional GET that returns "304 Not Modified").
In order to avoid problems caused by incorrectly synchronized clocks,
the protocol provides a means for caches to convey the amount of time
(``age'') they have held a response. We compute the total allowed
duration of a response's freshness by subtracting the origin server's
Date: value from its Expires: value, and then use the response's age
to decide if it is still fresh.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 14]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Each cache keeps track of how long it has stored a response without
its being refreshed (which is done by obtaining a new expiration time
from the origin server). We call this the ``age'' of the cached
response. When a cached response is sent to a client, the response
headers transmit the response's current age using a (new) ``Age:''
header. If a response has been stored by multiple caches along the
path, each one adds its own Age: value. We allow a cache to coalesce
multiple Age: headers into one such header, by performing the
obvious addition.
It is possible that a response may be held for some time by an
HTTP/1.0 cache, which would not include the Age: header. This could
lead to improper comparisons between ages and expiration values.
Therefore, we adopt the rule that the initial value for a cached
response's age is not zero, but rather
max(0, ((now - date_value) - age_sum))
where ``now'' is the cache's current clock, ``date_value'' is the
value of the origin server's Date: header, and ``age_sum'' is the sum
of all of the Age: header values already carried by the response.
This means that each HTTP/1.1 cache is responsible for correcting for
any ``missing'' time that the response spent in an HTTP/1.0 cache,
since the last time that response left an HTTP/1.1 system.
- This correction mechanism adds extra age if the cache's
clock is running ahead of the origin server's clock, and
fails to correct for missing age if the cache's clock is
running behind. However, it is the best that can be done
in the presence of HTTP/1.0 caches.
A similar algorithm allows the Age: values to be used to partially
correct Expiration: dates in the presence of clock skew.
Because of network-imposed delays, some significant interval may pass
from the time that an origin server generates an expiration value,
and the time it is received at the next outbound cache or client. If
uncorrected, this delay could result in improperly low ages.
We can correct for this by noting that the request that resulted in
the returned Age: value must have taken place prior to the value's
generation. Thus, when an Age: value is received, it MUST be
interpreted relative to the time the request was sent, not the time
that the response was received. This algorithm results in
conservative behavior no matter how much delay is experienced.
To decide whether a response is fresh, a cache performs this
algorithm:
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 15]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
/*
* age_sum
* is the sum of all of the Age: header values received
* by the cache with this response.
* date_value
* is the value of the origin server's Date: header
* request_time
* is the (local) time when the cache made the request
* that resulted in this cached response
* response_time
* is the (local) time when the cache received the
* cached response
* now
* is the current (local) time
* expires_value
* is the expiration value sent by the origin server
* request_max_age
* is the Cache-control: max-age value, if any, in the
* request
*/
apparent_age = response_time - date_value;
age_compensation = max(0, ((apparent_age) - age_sum));
local_age = (now - request_time) + age_compensation;
total_age = local_age + age_sum;
freshness_duration = max(0, expires_value - date_value);
if ((total_age < freshness_duration)
&& (total_age < request_max_age)) {
/* response is still fresh */
}
else {
/* response is stale */
}
In section 4.2, we discuss how to transmit the expiration value,
including an encoding for ``Forever'' (infinitely in the future) and
one for ``undefined'' (the server explicitly grants the cache
permission to use any reasonable heuristic).
In section 4.3, we discuss how to transmit the Age: value.
An HTTP/1.1 server SHOULD provide an Expires: value with every
cachable entity, but if it does not, a cache uses a heuristic to
decide how long to treat the response as fresh. See section 4.2 for
constraints on this heuristic.
3.9.1 Disambiguating expiration values
Because expiration values are assigned optimistically, it is possible
that two caches may contain different fresh values for the same
resource.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 16]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
If a client doing a retrieval receives a non-firsthand response for a
resource that was already fresh in its own cache, and the Date:
header in its existing cache entry is newer than the Date: on the new
response, then the client MAY ignore the response. If so, it MAY
retry the request with a ``Cache-control: revalidate'' directive (see
section 3.10), to force a check with the origin server.
XXX Why would a client make a request if it still has a fresh
copy in its own cache, and not make that request demand
revalidation? Perhaps because it is honoring a fresh-min
directive?
If a cache that is pooling cached responses from other caches sees
two fresh responses for the same resource with different validators,
it SHOULD use the one with the newer Date: header.
3.9.2 Disambiguating multiple responses
Because a client may be receiving responses via multiple paths, so
that some responses flow through one set of caches and other respones
flow through a different set of caches, a client may receive
responses in an order different from that in which the origin server
generated them. We would like the client to use the most recently
generated response, even if older responses are still apparently
fresh.
Neither the opaque validator nor the expiration value can impose an
ordering on responses, since it is possible that a later response
intentionally carries an earlier expiration time. However, the
HTTP/1.1 specification requires the transmission of Date: headers on
every response, and the Date: values are ordered to a granularity of
one second.
If a client does a [XXX conditional?] request for a resource that it
already has in its cache, and the response it receives contains a
Date: value that appears to be older than the one it already has in
its cache, then the client SHOULD repeat the request unconditionally,
and include
Cache-control: revalidate
to force any intermediate caches to validate their copies directly
with the origin server, or
Cache-control: reload
to force any intermediate caches to obtain a new copy from the origin
server. This prevents certain paradoxes arising from the use of
multiple caches.
If the Date: values are equal, then the client may use either
response (or may, if it is being extremely prudent, request a new
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 17]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
response). Servers MUST not depend on clients being able to choose
deterministically between responses generated during the same second,
if their expiration times overlap.
3.10 Explicit Cache-control
It is useful to provide an explicit mechanism to control certain
aspects of the caching mechanism.
The HTTP/1.1 draft includes a new Cache-Control: header, which can
carry four kinds of directive: ``cachable'', ``max-age'',
``private'', and ``no-cache''. This can be sent in either direction
(inbound or outbound), although the ``cachable'' and ``private''
directives are not allowed in requests.
We replace the proposed ``cachable'' directive with the more explicit
``public'' directive, used on responses to indicate that a response
to a request which included an Authorization header field may be
returned from a cache entry. This overrides restrictions stated
elsewhere in the specification on cache behavior for such requests,
allowing a server that uses authentication for purposes other than
limiting access to encourage caching. See section 8.1 for more
details.
XXX Action item: is ``public'' allowed on responses to any
method, or do we need to restrict it?
The ``no-cache'' directive on responses is useful, since it is a more
polite way to defeat caching than simply returning a bogus
cache-validator value and a past expiration time. With
``Cache-Control: no-cache'', the cache knows not to waste storage
space on the entity.
The ``no-cache'' directive on requests is also useful, because it
allows a user agent (or other client) to override a cache's belief
that a cached object is fresh. It is likely that some servers will
assign expiration times that turn out to be optimistic, or that by
some accident an incorrect cache-validator may be stored with the
cached object, and clients need a way to force reloading. However,
it seems unnecessarily economical to use the same directive for two
purposes, and might be less confusing to use the name ``reload''
here.
It also seems useful to provide a ``revalidate'' directive for
requests, to force a cache to do a conditional GET on a cached
object, but not necessarily to reload it. This would be useful if
the expiration time is wrong, but the cache-validator is right. An
over-optimistic expiration seems far more likely to happen than an
incorrect cache-validator.
The ``private'' directive, which is like a ``no-cache'' response
except that it only applies to shared caches, is clearly useful. See
section 8.1 for more details.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 18]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
The original proposal for ``no-cache'' and ``private'' allows them to
specify particular fields of the response, but it is not clear that
this is necessary or useful. It means that a subsequent request to
the cache cannot use the entire cached entity. Does this mean that
the cache should return only a partial entity (which seems relatively
useless)? Does it mean that the cache should retrieve the entire
entity from the origin server (in which case, why cache any of it)?
Or does it suggest that the cache ought to be able to request a
partial entity from the origin server to fill in the missing pieces
(but the protocol does not currently include a means to do so)?
XXX re-think this in the context of extensibility
The ``max-age'' directive on responses effectively duplicates the
expiration time, and (as currently specified) does not provide as
much protection against timing delay, so it is unnecessary.
The original purpose of the ``max-age'' directive on requests is to
tell a cache to revalidate the value if it has been in the cache more
than a specified number of seconds. This is basically a way for the
client, rather than the server, to change the looseness bounds on a
cached entity. We can use the same age-computation algorithm here as
is used with expiration values (see section 3.9).
In addition to ``max-age'', we can use two new directives for
requests to allow the client to set looseness bounds. The
``fresh-min'' directive specifies the minimum number of seconds of
remaining freshness necessary. For example, ``fresh-min 10'' means
that the client only wants to see a cached response if it has at
least 10 seconds of freshness left (measured by the cache). This
allows the client to increase (make more strict) the looseness
bounds.
Similarly, the ``stale-max'' directive specifies a maximum number of
seconds past the expiration time that the client is willing to
accept. For example, ``stale-max 5'' means that the client is
willing to accept a cached response that has already been stale for 5
seconds. This allows the client to decrease (make less strict) the
looseness bounds. If both fresh-min and stale-max are given in a
request, the stale-max is ignored.
It may be appropriate, in some circumstances, for a user to specify
that the response to a request should not be cached. (This might be
because the user believes that an intervening cache may be insecure,
but has no choice but to use the cache.) We include the
``dont-cache-response'' directive for this purpose; caches SHOULD
obey it but the user MUST never rely on this. We do not specify what
should happen if the cache already contains the response.
XXX anything else?
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 19]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
3.11 Warnings
Because one of our goals is that possibly non-transparent responses
should not be represented as transparent ones, we introduce the
notion of a ``warning''. A warning is a header that may be added to
a response, either by the origin server or by a cache, indicating
that the response might in some way be suspect. This is not done
using the Status-line, because such responses are not really
failures; they are simply somewhat suspicious.
Warnings are advisory; they may be used by a user agent to control
the presentation of a response, but do not otherwise affect the
handling of responses.
Warnings are always cachable, both because they are advisory and
because they never weaken the transparency of a response. This means
that warnings can be passed to HTTP/1.0 caches without danger; such
caches will simply pass the warning along as a response header.
Warnings are assigned numbers between 0 and 99. This specification
defines the code numbers and meanings of each warning, but a server
may send the warning message text in any appropriate natural language
(perhaps based on the client's Accept: headers). Multiple warning
messages may be attached to a response (either by the origin server
or by a cache), including multiple warnings with the same code
number. This allows a server to, for example, provide the same
warning with texts in both English and Basque.
The currently defined warnings are
90 Not firsthand
MUST be sent by a cache with any response that is
taken from a cache entry, except if the cache has
validated the cached object with the origin server as
a result of the current request. Note that if the
cache attempts to validate and receives a ``90 Not
firsthand'', then the validation did not come from
the origin server.
91 Response is stale
MUST be included whenever the returned response is
stale. A cache may add this warning to any response,
but may never remove it until the response is known
to be fresh.
92 Revalidation failed
MUST be included if a cache returns a stale response
because an attempt to revalidate the response failed,
due to an inability to reach the server. A cache may
add this warning to any response, but may never
remove it until the response is known to be fresh.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 20]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
98 Caching may violate law
SHOULD be sent with any response if the server is
aware of any law that specifically prohibits the
caching (including storage) of the response. This
may include copyright laws, confidential-records
laws, etc. Such responses SHOULD also be marked with
an expiration value in the past, to prevent an
unwitting cache from returning the value in a cached
response.
99 Miscellaneous warning
The warning text may include arbitrary information to
be presented to a human user, or logged. A system
receiving this warning MUST NOT take any automated
action.
XXX anything else?
3.12 Compatibility with earlier versions of HTTP
3.12.1 Compatibility between HTTP/1.1 clients and older servers
XXX when must an HTTP/1.1 client validate (using Get I-M-S) a
response that comes from an HTTP/1.0 origin server?
3.12.2 Compatibility between HTTP/1.1 servers and older clients
An HTTP/1.1 server SHOULD send a Last-Modified: date whenever
possible. This allows older clients and proxies to use the
If-Modified-Since: header, and may provide useful information even
for HTTP/1.1 clients.
An HTTP/1.1 server SHOULD send a cache-validator value (perhaps null)
in every potentially cachable response. Older clients will simply
ignore this, but the response may be cached by a proxy that speaks
HTTP/1.1, and the cached value may later be used by an HTTP/1.1
client.
3.12.3 Compatibility between HTTP/1.1 and older proxies
Since a proxy is always acting as either a client or a server, the
rules for compatibility between clients and servers should suffice to
provide compatibility between caches and servers (cache acting as
client), and between caches and clients (cache acting as server).
XXX is this true?
3.13 Update consistency
Various operations, either by an HTTP client or local to the origin
server, can change a resource so as to make cached copies invalid.
This leads to inconsistencies, in that different clients may then see
different versions of the resource. We would like to provide as much
``update consistency'' as possible.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 21]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
The draft HTTP/1.1 specification [1] includes several statements of
the form:
If the request passes through a cache and the Request-URI
identifies a currently cached entity, that entity must be
removed from the cache.
There is no harm in taking such action, and it can eliminate certain
inconsistencies that cannot be prevented by other means, but it is
not sufficient to prevent inconsistency.
For example, suppose that an entity is cached by two different
caching proxies A and B, and client X of proxy A performs a DELETE
method via that proxy. Proxy A will remove the entry from its cache,
but proxy B will know nothing about the interaction, and so will
continue to cache the deleted entity. This means that a different
client Y, using proxy B, will have a view that is inconsistent with
client X.
Even worse, if the two proxies are pooling their entries, proxy A
might end up re-acquiring the (bogus) cached entry from proxy B. For
this reason, we should change the wording quoted above from ``must be
removed from the cache'' to ``must be marked invalid in the cache'',
at least for those caches that participate in pooling.
Regardless of the wording here, any requirements that caches must
delete or invalidate certain entries when doing certain operations
are at best ``prudent,'' and at worst ``ineffective.'' There is no
way for an origin server to force other caches to remove or
invalidate entries; in fact, there is no way for an origin server
even to discover the set of caches that might be holding an entry.
We must recognize this, and look for other solutions to the update
consistency problem.
One simple solution would be for origin servers to mark potentially
updatable resources as uncachable (using a past expiration value, or
Cache-control: no-cache). If this is done conservatively, it solves
the update consistency problem, but it also eliminates the
performance benefits of caching for client requests between updates.
Another solution would be for the server to refuse to perform update
methods (such as PUT or DELETE) as long as any existing cache entries
might be fresh. This requires the server to record the latest
expiration value it has issued for a resource, and is similar to the
Leases mechanism of Gray and Cheriton [2]. This would solve the
update consistency problem without preventing caching, but it could
lead to confused users, whose update operations would fail at
unpredictable times. (This includes local users or administrators of
the server, as well as HTTP clients.) One might take the approach
that updates that conflict with existing expiration leases are
accepted but queued for later application; this avoids unpredictable
failures, but leads to mysterious effects.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 22]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Either of these solutions can be implemented unilaterally by the
origin server, with the caching mechanisms described elsewhere in
this document.
Another approach, inspired by some work done at CMU XXX find
appropriate reference XXX, uses the concept of a ``volume''
containing a number of individual resources, and allows the server to
invalidate an entire volume at once. A cache finds out about a
volume invalidation whenever it contacts the origin server regarding
any member of the volume; this improves the chances that a cache will
notice that a resource has been updated before it provides an
inconsistent cache response to a client. However, it cannot
eliminate inconsistency. We give a specific proposal for volume
validation in section 3.14.
Some combination of these approaches (uncachable responses, leases,
and volume validation) may allow servers to reduce the update
consistency problem to manageable levels, while still allowing
reasonably effective caching and predictable responses to users doing
updates. However, it is easy to construct scenarios in which it is
simply impossible to optimize on all three metrics at once. In order
to solve this problem, we would have to introduce some sort of
``callback'' mechanism, which is not feasible for HTTP/1.1.
3.14 Volume validation
We propose a simple and optional ``volume validation'' mechanism for
the HTTP protocol, to reduce the chances that caches are inconsistent
with origin servers or with each other.
If a server wants to use volume validation, it assigns each resource
to one of a number of sets, called ``volumes.'' Volumes do not
necessarily map onto storage-hierarchy concepts such as disks; they
might be based on file type, for example.
Members of a volume ought to have similar lifetimes. The server
might assign all of its resources to the same volume, or it might use
a number of volumes to distinguish between (for example), probably
immutable resources, things that change slowly (say, once a week),
things that change often (say, once an hour), and things that are
very dynamic (changing at intervals of seconds or minutes). If a
group of resources is typically changed together, then that group
also forms a natural volume.
When the server returns a response to a cache, it includes the usual
cache control info (as specified elsewhere in this draft), and it
also returns these three new header values:
Volume-ID: <opaque string>
Volume-version: <opaque-string>
Volume-expires: HTTP-date (in RFC-822 form)
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 23]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
XXX we probably have to correct the volume-expiration value
using the same Age: headers that we deduct from expiration
values.
The cache, in addition to keeping the individual resources, also
keeps a cache of this per-volume information. Each of the individual
resource cache entries includes a pointer to the associated volume
info (which is managed as a cache, and therefore might not always be
present). Also, the Volume-version: value is stored with each
individual resource entry.
Note that the per-volume information must also include some sort of
unique ID for the server, such as its IP address or host name.
Each time the server receives a response from a server, it can update
the per-volume information from that response. This allows the
server to keep increasing the expiration value for a volume.
However, if any resource in the volume is modified, then the server
must change the Volume-version: value to one it has never used
before. (This could be done using a timestamp or sequence number;
note that the clients and caches MUST NOT attempt to impose an
ordering on these values.)
When the cache receives a client request, it would normally check the
expiration information stored with the relevant individual-resource
entry to decide if it has to reload the response from the server.
However, if the resource is associated with a volume that the cache
knows about, then it can do this:
if (volume-version stored with resource matches current
volume version stored in per-volume entry) then
if (volume-expiration time has not yet been reached)
or if (resource expiration time not yet reached)
then it's OK to return the cached response
else
must do conditional GET from server
else
if (resource expiration time not yet reached)
then it's OK to return the cached response
else
must do conditional GET from server
In other words, the per-volume information is used to extend the
expiration time for what might be a large set of cached resources (if
the version value remains unchanged) or to invalidate a large set of
cached resources (if the version value changes).
Of course, this would only pay off when a proxy caches a sufficiently
large number of resources from the same server (and from the same
volume). But it lets the server assign relatively short expiration
times to the individual resources (which makes update inconsistency
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 24]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
less likely), and still prevent a busy proxy from bombarding it with
conditional requests for resources that haven't been modified.
It is entirely optional for the server or the cache to implement
this, and the implementation at the cache side of things seems to
pretty simple. On the server side, implementation complexity will
depend on how the server detects whether a member of a volume is
modified. A small amount of support from the underlying file system
or database might be quite useful, but in many cases a fairly simple
scheme would work. For example, if all of the items in a product
catalog were assigned to the same volume, the server administrator
could simply change the volume-version value whenever the catalog is
updated.
If a cache receives a response that changes the Volume-ID of a
resource currently in the cache (whether fresh or not), the cache
MUST set the volume-expiration value for the old volume to be zero,
forcing the cache to check with the origin server on any subsequent
request for a member of the old volume. A change in the Volume-ID
for a cached resource implies that the volume itself has changed in a
significant way.
XXX should we insist that all of the cached responses from the
old volume be revalidated? In case they are no longer in this
volume? In general, does this design adequately deal with the
problem of moving resources between volumes?
XXX should we provide some mechanism for a cache to advise an
origin server that it is willing to obey the volume validation
protocol (using Connection: to avoid confusion from intervening
caches)? This would allow the origin server to allow looser
caching without as much risk of inconsistency.
3.15 Side effects of GET and HEAD
Section 14.2 (``Safe Methods'') of the draft HTTP/1.1
specification [1] implies that GET and HEAD methods should not have
any side effects that would prevent caching the results of these
methods, unless the origin server explicitly prohibits caching. We
make this explicit: unless the server explicitly disables caching,
GET and HEAD methods SHOULD NOT have side effects.
XXX alternate wording: unless the server explicitly disables
caching, GET and HEAD methods SHOULD NOT have side effects that
would lead to erroneous behavior if their responses are taken
from a cache. They may still have side effects, but a cache is
not required to consider such side effects in its caching
decisions. Caches are always expected to observe an origin
server's explicit restrictions on caching.
Apparently, some applications have used GETs and HEADs with query
URLs (those containing a ``?'' in the rel_path part) to perform
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 25]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
operations with significant side effects. Therefore, caches MUST NOT
treat responses to such URLs as fresh unless the server provides an
explicit expiration time.
This specifically means that responses from HTTP/1.0 servers for such
URIs should not be taken from a cache, since an HTTP/1.0 server
provides no way to indicate that two query responses issued during
the same second are in fact different.
3.16 Write-through mandatory
All methods that may be expected to cause modifications to the origin
server's resources MUST be written through to the origin server.
This currently includes all methods except for GET and HEAD. A cache
MUST NOT reply to such a request from a client before having
transmitted the request to the inbound server, and having received a
corresponding response from the inbound server.
The alternative (known as ``write-back'' or ``copy-back'' caching) is
not allowed in HTTP/1.1, due to the difficulty of providing
consistent updates and the problems arising from server, cache, or
network failure prior to write-back.
4 HTTP headers related to caching
XXX this section is still quite sketchy
4.1 Cache-validation
Validators are quoted strings of printing ASCII characters:
opaque-value = quoted-string
The empty string (``""'') is explicitly allowed, and is defined as
the null value, for which all equality comparisons MUST fail. In
particular, two null strings do not compare as equal.
XXX do we need to specify a maximum length for these strings?
Servers return cache-validator values using
Cache-validator: opaque-value
Clients use
If-invalid: opaque-value
for conditional retrievals.
Clients use
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 26]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Modify-if-valid: opaque-value
for conditional updates.
4.2 Expiration values
XXX this section includes certain provisional parameters in
[brackets].
Expiration values are absolute dates in HTTP-date format. Although
previous HTTP specifications have allowed several formats, including
one with only two digits to represent the date, we need to be able to
unambiguously represent expiration times in the future. Therefore,
origin servers MUST NOT use the RFC-850 form, and SHOULD use the
RFC-822 form (as already stated in the HTTP/1.1 specification).
An HTTP/1.1 server MUST always send a Date: header if it sends an
Expires: header. The two headers should use the same date form.
Origin servers MUST NOT use the RFC-850 form, and SHOULD use the
RFC-822 form.
Although servers MUST NOT send dates in RFC-850 form, clients and
caches MUST be accept these if they are at all sensible. HTTP/1.1
clients and caches SHOULD assume that an RFC-850 date which appears
to be more than [50]XXX years in the future is in fact in the past
(this helps solve the "year 2000" problem).
To mark a response as ``already expired,'' an origin server should
use an Expires: date that is equal to the Date: header value.
Clients and caches MUST assume that a response whose Expires: and
Date: fields are representations of the same value is immediately
stale. Also, any response whose Expires: value is older than its
Date: value is immediately stale.
To mark a response as ``never expires,'' an origin server should use
Expires: date approximately [one]XXX year from the time the response
is generated. HTTP/1.1 servers should not send Expires: dates more
than [one]XXX year in the future.
XXX Proposed rules for requirements on parsing and representation of
dates:
HTTP/1.1 clients (and caches) SHOULD be able to parse HTTP-date
values at least one year in the future (from the time that a
response is received). HTTP/1.1 clients and caches should
assume that an RFC-850 date which appears to be more than
[50]XXX years in the future is in fact in the past (this helps
solve the "year 2000" problem). All HTTP/1.1 implementations
MUST be capable of correctly parsing any HTTP-date value that
is no less than [25]XXX years in the future from the date that
the implementation is sold or otherwise distributed. An
HTTP/1.1 implementation may internally represent a parsed
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 27]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Expires: date as earlier than the proper value, but MUST NOT
internally represent a parsed Expires: date as later than the
proper value.
If the origin server includes an Expires: header but not a Date:
header, a cache or client should compare the expiration value to its
current local clock to decide if a cached copy is fresh or stale
If the origin server does not include an Expires: value, and does not
otherwise prohibit caching, then a cache may use a heuristic to
determine how long the response may be treated as fresh.
Caches should not treat such ``undefined'' expiration values as
``forever.'' A service administrator who fails to assign an
Expires: value should be aware that some caches may treat this
as practically ``forever''.
This MUST never be longer than [7]XXX days.
XXX Action item: We need a heuristic and a maximum value. The
heuristic should deduct any Age: value already associated with
the response. Also, we probably need to say that the heuristic
must return zero for methods other than GET or HEAD.
4.3 Age values
Caches transmit age values using:
Age = "Age" ":" delta-seconds
Age values are non-negative decimal integers, representing time in
seconds. If a cache receives a value larger than the largest
positive integer it can represent, it should replace with the largest
representable positive integer. Caches SHOULD use a representation
with at least 31 bits of range.
Caches MUST NOT delete Age: headers, although they may coalesce them
into a single value. They may also combine multiple Age: headers
into one, with the values separated by commas.
4.4 Cache-control
The Cache-control: header can include one or more of the following
directives on requests:
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 28]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
reload
revalidate
"fresh-min" "=" delta-seconds
"stale-max" "=" delta-seconds
dont-cache-response
The Cache-control: header can include one or more of the following
directives on responses:
no-cache
private
public
4.5 Warnings
Warning headers are sent with responses using:
"Warning" ":" 2DIGIT SP *TEXT
The currently defined Status-code/Reason-Phrase pairs allowed in
warnings include:
90 Not firsthand
91 Response is stale
92 Revalidation failed
98 Caching may violate law
99 Miscellaneous warning
4.6 Volume validation
Servers send volume validation information in responses using these
headers:
Volume-ID: <opaque string>
Volume-version: <opaque-string>
Volume-expires: HTTP-date (in RFC-822 format)
Volume-ID strings and Volume-version strings are opaque to clients
and caches. The Volume-expiration values follow the same rules as do
expiration values.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 29]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Servers are not required to send these headers; clients and caches
are not required to implement them. However, if a cache does
implement any of them, it MUST follow the specification for all of
them.
5 Rules summarized by method
This section includes caching-related rules for each HTTP/1.1 method.
TBS
6 Cache replacement algorithms
TBS
7 Specific issues
XXX this section is likely to be reorganized
7.1 Content negotiation
TBS
7.2 Authentication
TBS
7.3 State
XXX From Shel Kaphan, may need editing and references to other
documents:
Briefly, cookies can't be cached by public (shared) caches, but since
public documents may make up part of a "stateful dialog", and in
particular the first document in a stateful dialog may be (for
example) a public and cachable home page, servers that wish to
receive the client's cookie on each request, or to issue a new cookie
on requests for a document must set the document up to require
validation on each request. (e.g., by having it be "pre-expired").
Otherwise, the cache control headers for responses control what a
proxy has to do. If a document is fresh in a cache, a request
containing a cookie does not have to be forwarded to the origin
server, since (by definition) if the document is servable from a
cache, there aren't important side effects at the origin relating to
requests for that document, and so, no changes to the cookie.
One important issue bearing on caching is that for conditional
requests that go through to the origin server, for which the origin
server responds with 304 and also with a set-cookie header, caches
must splice the set-cookie sent by the origin server into their own
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 30]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
response. This is, for example, how it can work to have a home page
that is in a cache, but stale, so that the only traffic to the origin
server is to validate the home page, receiving a 304 and potentially
a new cookie.
7.4 Caching and methods with side effects
TBS
7.5 Network partitions
TBS
7.6 Caching of negative responses
TBS
7.7 History Lists
TBS
XXX should be treated as separate from caches
XXX browser could alert the user to stale page, without automatically
refreshing.
7.8 Hit Metering
Some customers insist that the hit counts reported at servers include
the hits that happen only at caches. In the absence of a mechanism
to do this, some servers disable caching simply to obtain hit counts,
which is not good for overall performance.
It would not be possible to make hit-metering mandatory for caches,
simply because there is no obvious way to enforce an anti-cheating
policy. Also, hit metering may run afoul of privacy concerns, which
in some jurisdictions have the force of law. But it should be
possible to provide a mechanism for cooperative caches and servers to
use.
Customers have asked for several kinds of hit-metering, with
increasing levels of complexity and cost:
1. simple hit counts: the cache tells the server how many
hits have been made on a particular cached value.
2. access-log transfers: the cache sends to the server
complete access logs (in some standardized format) so that
the server can tell which clients have made what requests
and when.
3. client navigation traces: the cache sends to the server
enough information to determine how the user got from one
page to another.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 31]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
It is not entirely clear whether (3) is really different from (2), or
whether the access logs contain enough information to satisfy the
people who want (3).
XXX Netscape has internal specifications for (1) and (2?), and will
write them up and circulate them to the subgroup.
XXX Jim Gettys adds: Rohit Khare did point out that people want to
know very much not only that a document has been hit, but that it was
cancelled before delivery was complete. This is how webmasters track
down GIFS or boring pages. Adding this much to the simple hit count
reporting seems like the point in the design space we should shoot
for.
7.9 Bypassing in caching hierarchies
Many HTTP caching configurations involve hierarchies of caches, often
designed to reduce bandwidth requirements rather than improving
latency. However, if a cache at a low level in the hierarchy is sure
that the cache(s) above it do not contain a cache entry to match a
given request, that low-level cache can transmit the request directly
to the origin server. This improves retrieval latency without
increasing total bandwidth requirements (it even eliminates some
packet transmissions) and is entirely appropriate for resources whose
values are explicitly not cached.
We call this technique ``request bypassing.'' Note that although the
bypassing decision might be done by the ultimate client, in many
cases the use of firewalls or unsophisticated clients means that the
decision must be made by an intermediate-level cache.
In order to do request bypassing in the most efficient possible way,
the caches must be able to determine from the request whether the
response is likely to be cachable. (It is important to err on the
side of assuming cachability, since the assuming converse could
seriously reduce the effectiveness of the higher-level caches.)
The current HTTP/1.1 draft specification does not include a foolproof
mechanism to mark requests in this way. While we generally do not
allow caching of responses to GET requests for URLs with a ``?'' in
the rel_path part (see section 3.15), we also allow the origin server
to mark responses to such queries as cachable. Therefore, any
bypassing done using this heuristic runs the risk of giving up
perfectly good opportunities to cache some resources.
XXX we have discussed various approaches for marking requests,
all of which apparently require some kind of change to HTML to
allow the origin server to pass the marks to the ultimate
client. Some people suggest using special methods that are
explicitly always cachable (``POST_WITH_NO_SIDE_EFFECTS'', or
more concisely ``POSTC'') or never cachable (``GET_QUERY'', or
more concisely ``GETQ''). Others have suggested adding tags to
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 32]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
HTML that would cause subsequent requests to carry some special
sort of header. Neither solution has resulted in a consensus.
An origin server would be able to use POSTC only with HTTP/1.1
clients and proxies, and so would have to return different HTML
forms depending on the protocol version in the request header.
This would also imply using the proposed Vary: header with
some token that indicates ``varies based on request HTTP
version,'' since we don't want a cache returning one of these
HTML responses to an HTTP/1.0 client.
7.10 Explicit warnings regarding user-specified overrides
Many browsers make it possible for users to override the basic
caching mechanisms. For example, the browser may allow the user to
specify that cached entities (even explictly stale ones) are never
validated. Or the browser might habitually add ``Cache-control:
stale-max=3600'' or ``Cache-control: reload'' to every request. We
recognize that there may be situations which require such overrides,
although browsers SHOULD NOT default to any behavior contrary to the
HTTP/1.1 specification. That is, the user should have to explicitly
request either non-transparent behavior, or behavior that results in
abnormally ineffective caching.
If the user has overridden the basic caching mechanisms, the browser
should explicitly warn the user whenever this results in the display
of information that might not meet the server's transparency
requirements (in particular, if the displayed resource is known to be
stale). Since the protocol normally allows the browser to determine
if responses are stale or not, this indication need only be displayed
when this actually happens. The indication need not be a dialog box;
it could be an icon (for example, a picture of a rotting fish) or
some other visual indicator.
If the user has overridden the caching mechanisms in a way that would
abnormally reduce the effectiveness of caches, the browser should
continually display a warning indication (for example, a picture of
currency in flames) so that the user does not inadvertently consume
excess resources or suffer from excessive latency.
8 Security Considerations
8.1 Caching of responses to authorized requests
The draft HTTP/1.1 specification says (section 10.6):
Responses to requests containing an Authorization field are not
cachable.
We suggest restating this as
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 33]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Responses to requests containing an Authorization field MUST
not be returned from a shared cache to any user other than the
user that made the initial request. If the cache cannot
securely determine the identity of the user, then it must act
as if the identity does not match.
A ``shared cache'' is one that accepts requests from more than
one user. This definition normally applies to all caches
except those integrated with single-user browsers (and not
accessible except from their containing browsers). Such
non-shared caches may return cached responses to authorized
requests.
Protection against replay attacks seems to be a function of how the
cache ``securely'' determines user identies. That is, if the
mechanism does not protect against replay attacks, then it is by
definition not secure.
8.2 Location: headers and spoofing
TBS
XXX Contribution from Shel Kaphan, may need editing and shortening:
The location header in 2xx responses identifies a URI at which the
client could request another copy of the resource being included as
the body of the response. We discussed two issues in connection with
this: "invalidation" (i.e. "making stale") of other objects in a
cache due to the receipt of a response containing a Location header,
and spoofing.
I'm going to use the word "invalidation" to mean "making stale",
since it sounds better.
If a cache were to use the Location URI to cause the response entity
to be cached in such a way that a subsequent GET could retrieve the
entity from the cache using that URI, then a response containing a
Location URI not under the same authority as the request-URI could
accidentally or intentionally confuse a cache, and cause subsequent
requests to receive erroneous data in response. If such replacement
is allowed, there seems to be no obvious way to keep this spoofing
from happening. (Any ideas?)
Paul Leach suggested that it might be "necessary but not sufficient"
to at least require the Location URI to share the same hostname as
the request URI. It is not sufficient because there may be multiple
authorities operating on the same host. It is also not necessary
because a given authority may span multiple hosts, and a given host
may have multiple names. A legitimate Location URI might not match
the request URI's hostname, whereas an illegimate one might sometimes
match the request hostname. Paul seemed to have some other
objections related to worrying about this spoofing problem that I
didn't understand.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 34]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
Jeff Mogul suggested that invalidation of objects in the cache whose
request-URI matches the location URI is not sufficient to solve the
cache coherency problem. This is of course true -- if there are
multiple caches containing copies of some object, invalidating
objects in one cache has no effect on other caches. Therefore, Jeff
suggested that if it is critical for an up-to-date object to be
served for every request, that its expiration should be set so that
it is validated on each request. I agree with that.
However, I believe there is a class of resources for which it is
desirable but not critical for an up-to-date version to be seen, and
for these objects it seems reasonable that a single cache should try
to do its best (forgive the anthropomorphizing) to provide the
up-to-date version. Perhaps this can be viewed as an optional
optimization not required by the protocol, since it is always
possible that a series of requests may be served by different caches
anyway, and so the protocol doesn't guarantee that a cachable
("fresh") object would ever be invalidated.
A context in which this invalidation could be very useful is content
negotiation. If a Location header in a negotiated response ever
matches an explicit request URI for some object already in a cache,
but the negotiated response contains a different version that has
come into existence before the expiration date of the previously
cached version, a subsequent request on that URI in this cache will
retrieve the previous version. This seems rather annoying, and can't
be fixed by replacement using the Location URI because of the
spoofing problem mentioned above. The best that can be done is to
invalidate other objects in the cache that were keyed by a
request-URI matching this location-URI, so that future requests will
require validation. Since this is useful for the content negotiation
case, the same mechanism may also be used to help with other uses of
the Location header -- for instance a response to a POST may be a new
version of a GETtable resource that may already have been cached.
Again, as Jeff points out, if this is critical, that GETtable
resource should be set to require validation on each request, but it
might be just desirable and not critical.
Another minor point: in the content negotiation example above, if
the explicit GET occurred after the negotiated GET, then invalidation
would still be desired, but this time the request-URI of the GET
(assuming no Location header was returned) would be used to
invalidate not only other occurrences of the request-URI in the
cache, but also entries for which that URI was returned as a Location
URI. So, for this purpose the Location URI would serve as a
secondary key to the cache.
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 35]
Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02
9 Acknowledgements
A lot of the content and presentation of this document is due to
suggestions and comments from individuals including: Shel Kaphan,
Paul Leach, Koen Holtman, David Morris, Larry Masinter, and Roy
Fielding.
10 References
1. Roy T. Fielding, Henrik Frystyk Nielsen, and Tim Berners-Lee.
Hypertext Transfer Protocol -- HTTP/1.1. Internet-Draft
draft-ietf-http-v11-spec-00.txt, HTTP Working Group, November, 1995.
2. Cary G. Gray and David R. Cheriton. Leases: An Efficient
Fault-Tolerant Mechanism for Distributed File Cache Consistency.
Proc. 12th Symp. on Operating Systems Principles, Litchfield Park,
AZ, December, 1989, pp. 202-210.
11 Author's address
Jeffrey C. Mogul
Western Research Laboratory
Digital Equipment Corporation
250 University Avenue
Palo Alto, California, 94305, U.S.A.
Email: mogul@wrl.dec.com
Phone: 1 415 617 3304 (email preferred)
DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 36]