Network Working Group J. Mogul, DECWRL
Internet-Draft 12 September 1997
Expires: 26 March 1998
Generation of the Age header field in HTTP/1.1
draft-mogul-http-age-00.txt
STATUS OF THIS MEMO
This document is an Internet-Draft. Internet-Drafts are
working documents of the Internet Engineering Task Force
(IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of
six months and may be updated, replaced, or obsoleted by
other documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other
than as "work in progress."
To learn the current status of any Internet-Draft, please
check the "1id-abstracts.txt" listing contained in the
Internet-Drafts Shadow Directories on ftp.is.co.za
(Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific
Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US
West Coast).
Distribution of this document is unlimited. Please send
comments to the HTTP working group at
<http-wg@cuckoo.hpl.hp.com>. Discussions of the working
group are archived at
<URL:http://www.ics.uci.edu/pub/ietf/http/>. General
discussions about HTTP and the applications which use HTTP
should take place on the <www-talk@w3.org> mailing list.
ABSTRACT
The 'Age' response-header field in HTTP/1.1 [RFC 2068] is
intended to provide a lower bound of an estimate of a
response message's age (time since generation), by
explicitly indicating the amount of time that is known to
have passed since the response message was retrieved or
revalidated. There has been considerable controversy over
when the Age header field should be added to a response.
This document explains the issues, rebuts a previous
proposal, and provides a set of proposed changes for the
revision of RFC 2068.
Mogul [Page 1]
Internet-Draft HTTP Age generation 12 September 1997 13:48
TABLE OF CONTENTS
1 Introduction 2
2 Proposed change to RFC2068 3
3 Discussion 3
3.1 Quantifying the error of the Age estimate: Interpretation 4
C
3.2 Quantifying the error of the Age estimate: Interpretation 5
B
3.3 Prevalence of clock skew in the Web 5
3.4 Implications of overestimating the Age value: 6
Interpretation C
3.5 Implications of underestimating the Age value: 7
Interpretation B
3.6 So what's all the fuss about, anyway? 8
4 Acknowledgements 9
5 References 9
6 Author's address 9
1 Introduction
RFC2068 [3] defines, in section 14.6, the Age header field for
HTTP/1.1:
The Age response-header field conveys the sender's estimate of
the amount of time since the response (or its revalidation) was
generated at the origin server. A cached response is "fresh" if
its age does not exceed its freshness lifetime. Age values are
calculated as specified in section 13.2.3.
Age = "Age" ":" age-value
age-value = delta-seconds
Age values are non-negative decimal integers, representing time
in seconds.
If a cache receives a value larger than the largest positive
integer it can represent, or if any of its age calculations
overflows, it MUST transmit an Age header with a value of
2147483648 (2^31). HTTP/1.1 caches MUST send an Age header in
every response. Caches SHOULD use an arithmetic type of at least
31 bits of range.
Roy Fielding has taken issue with the statement ``HTTP/1.1 caches
MUST send an Age header in every response.'' [2]. Fielding correctly
points out that the use of the word ``caches'' in this sentence is
incorrect. He lists two possible rewordings for this sentence:
Mogul [Page 2]
Internet-Draft HTTP Age generation 12 September 1997 13:48
a) An HTTP/1.1 server that includes a cache MUST send an Age
header field in every response.
b) An HTTP/1.1 server that includes a cache MUST include an Age
header field in every response generated from its own cache.
There is, however, a third possible rewording, if one realizes that
the original drafting error was to sloppily substitute the word
``cache'' where ``proxy'' was meant (mea culpa):
c) An HTTP/1.1 proxy MUST send an Age header field in every
response.
Note that an HTTP proxy does not necessarily include a cache. The
other sentences referring to ``caches'' in that paragraph reflect the
same drafting error.
2 Proposed change to RFC2068
For this paragraph in section 14.6 (Age),
If a cache receives a value larger than the largest positive
integer it can represent, or if any of its age calculations
overflows, it MUST transmit an Age header with a value of
2147483648 (2^31). HTTP/1.1 caches MUST send an Age header in
every response. Caches SHOULD use an arithmetic type of at least
31 bits of range.
substitute
If a proxy server receives a value larger than the largest
positive integer it can represent, or if any of its age
calculations overflows, it MUST transmit an Age header with a
value of 2147483648 (2^31). An HTTP/1.1 proxy MUST send an Age
header in every response. Proxies SHOULD use an arithmetic type
of at least 31 bits of range to represent Age values.
3 Discussion
Because of facts of life such as resolution granularity, clock skews,
and the theory of relativity, the Age value can never be exactly
accurate; it is always an estimate. We would like to bound the
inaccuracy of this estimate. Most important, we would like to ensure
that any inaccuracy remaining in the Age estimate not cause avoidable
and significant harm.
Mogul [Page 3]
Internet-Draft HTTP Age generation 12 September 1997 13:48
3.1 Quantifying the error of the Age estimate: Interpretation C
If interpretation (c) is used (i.e., the change proposed above in
section 2), then the value of the Age header field might overestimate
the actual amount of time since the response was generated at the
origin server. One can calculate the size of the estimation error
for a path containing N HTTP/1.1 proxies:
Error_C = Mean_RTT * N * (N + 1)/2
where Mean_RTT is the mean round-trip time (RTT) between neighboring
pairs of proxies. Note that this measures the error in the Age value
at the final recipient client, not the error at the last HTTP/1.1
proxy on the path.
Actually, this analysis assumes that the RTT on each hop is the
Mean_RTT; in fact, the contribution from the hops closest to the
origin server contribute more to the true error value than this
simplified formula, and so it is not necessarily an upper bound on
the error. If one uses
Error_C_bound = Max_RTT * N * (N + 1)/2
where Max_RTT is the largest of the RTTs over all of the hops, then
the Error_C_bound estimate is an upper bound.
In order to quantify this bound, we need parameter values for N and
Max_RTT. It's not clear exactly how many proxy caches are used on an
arbitrary path through the Internet, especially since many of these
paths start within large intranets, but I think it would be realistic
to assume that most such paths have somewhere between N=1 and N=6.
Clearly, the Max_RTT value is harder to quantify, since over
extremely lossy paths, the RTT can be nearly infinite. As a
surrogate for this value, I looked at the total retrieval time for
each of 504736 retrievals in a two-day proxy trace made in December,
1996. (See [5] for more discussion of this trace.) Among these
retrievals, the mean total retrieval time was 1.25 seconds, with a
median of 0.25 seconds and a standard deviation of 13.7 seconds. As
far as I can tell, a large part of the variation is due to servers
that drop SYN packets due to buggy TCP stacks. Given this, and given
that the median is closer to 1/4 second, and given that a full
HTTP/1.0 retrieval takes at least two round-trips (and probably
more), I think it's fair to assume a value of 2 seconds as an
approximate upper bound for the Max_RTT for most paths.
Therefore, doing the math
Error_C_bound[N=1] = 2 seconds
Error_C_bound[N=6] = 42 seconds
and, just for comparison, if one uses the observed median retrieval
Mogul [Page 4]
Internet-Draft HTTP Age generation 12 September 1997 13:48
time (0.25 seconds) and assumes the use of no more than 3 HTTP/1.1
proxies, then Error_C_bound = 1.5 seconds.
3.2 Quantifying the error of the Age estimate: Interpretation B
If interpretation (b) is used, then the value of the Age header field
might underestimate the actual amount of time since the response was
generated at the origin server. The scenario that generates the
greatest underestimate is when the path between the origin server and
the HTTP/1.1 client's cache includes an HTTP/1.0 proxy cache, and the
client has a skewed clock. In particular, the client's clock is set
into the past by some amount.
In this scenario, since an HTTP/1.0 proxy cache will not add the Age
header, there is no indication that the response was delayed by a
cache between the origin server and the first HTTP/1.1 cache. The
only way for that HTTP/1.1 cache to construct its initial Age
estimate is to compare its own clock against the Date value. If the
client's clock is wrong, this error is, in general, impossible to
correct. For example, suppose the client's clock is one hour slow.
If the origin server generates the response at 10:00AM, and then the
response sits in the HTTP/1.0 cache for 65 minutes, when the response
arrives at the HTTP/1.1 client, it will appear to be only 5 minutes
old.
The benefit of using interpretation (c) is that if there is a
non-caching HTTP/1.1 proxy between the HTTP/1.0 cache and the client,
unless its clock is also skewed, it will add a nearly-correct Age
header field to the response. I.e., the client will realize that the
response is at least 65 minutes old, regardless of its clock skew.
So, while the error is impossible to correct "in general", it is
possible to correct in certain specific cases, and this is the value
of interpretation (c).
3.3 Prevalence of clock skew in the Web
Is clock skew a real problem? Unfortunately, I know of no systematic
study of HTTP client clock skews. This is difficult, in part,
because HTTP requests generally do not include a Date header.
However, since I do have access to a trace of the headers flowing
through a proxy whose clock, at the time of the trace, was carefully
synchronized using NTP, I was able to look at the clock-skew
distribution of a large set of HTTP servers. (The trace covers 22034
distinct server IP addresses.) While this is not the same as a
population of HTTP clients, one might actually expect a set of HTTP
servers to have better clock synchronization characteristics than a
set of HTTP clients. After all, many HTTP clients run on personal
computers or workstations, and are managed by non-experts; most Web
servers on the Internet have at least some semblance of
administration (e.g., someone at least had to obtain a DNS name). In
other words, whatever the situation with Web server clocks, one would
expect the situation among clients to be worse.
Mogul [Page 5]
Internet-Draft HTTP Age generation 12 September 1997 13:48
For each response in the trace, I compared the Date header field
value (if any) to the proxy's NTP-synchronized timestamps for the
start of the connection and the end of the connection. If the
server's clock is accurate, the Date value ought to be between those
two timestamps. If the server's clock is slow, the Date value would
be lower than the start-timestamp; if the server's clock is fast, the
Date value would be higher than the end-timestamp.
Because of the 1-second granularity of Date, I treated as "valid" any
values less than 1 second in error. I also treated as "obviously
bogus" any Date where the server's clock appeared to be more than 1
day wrong, since one could assume that such a badly skewed server
clock would be abnormal.
The trace contained 503969 responses with parsable response headers.
Of these, only 286779 actually had Date headers (most of the rest
appear to be PointCast responses). 1087 of these had Date values
that were clearly bogus (by the "1-day-wrong" test). Of the others,
116966 (41%) showed a server with a "slow" clock (by at least one
second), and 83782 (29%) showed a "fast" clock. Only 84944 (30%) had
apparently-synchronized clocks.
What if we set the threshold for an OK clock at +/- 60 seconds
(which, by the earlier analysis, is somewhat larger than the
Error_C_bound for N = 6 and Max_RTT = 2)? In this case, we still
find 79443 (27%) responses indicating "slow" clocks, and 56429
responses (20%) indicating "fast" clocks. In other words, a lot of
the clocks are off by a lot of time.
Using the 1-second threshold, the mean error in the slow clocks is
1287 seconds, with a median error of 113 seconds. For the fast
clocks, the mean error is 1383 seconds, with a median of 97 seconds.
Using the 60-second threshold, the mean error in the slow clocks is
1884 seconds, with a median error of 198 seconds. For the fast
clocks, the mean error is 2039 seconds, with a median of 152 seconds.
(We're removing the small-error samples from these sets, so we're
left with sets biased towards high-error samples.)
In summary, clock skew seems to be prevalent among HTTP servers, and
the skews seem to be fairly large. One might be justified in
guessing that the situation is worse among HTTP clients.
NOTE: I should reanalyze this data, breaking it down by server
address, rather than by response, but that will have to wait
for another draft of this document.
3.4 Implications of overestimating the Age value: Interpretation C
What happens if the Age value is overestimated? If this happens,
some "fresh" responses appear to be "stale", and so unnecessary cache
misses may be generated. Except in the case where the network is
Mogul [Page 6]
Internet-Draft HTTP Age generation 12 September 1997 13:48
partitioned, this is a performance problem, but does not lead to the
delivery of responses with the wrong body or headers. (When the
network is partitioned, caches are allowed to return stale values
with an appropriate Warning, so this is irrelevant to the current
discussion).
Further, only a very small subset of the fresh, cachable responses
will be affected. In particular, this can only affect a response
whose actual remaining freshness lifetime is smaller than the error
in the Age estimate. Presumably, this is only a significant issue
for responses whose initial max-age value is small (since otherwise
it would be a low-probability coincidence that a request arrives at
the cache just before the freshness lifetime expires). (Remember
that the Error_C_bound = 42 for N = 6 and Max_RTT = 2, and is a lot
lower for shorter proxy paths.)
We have no good information on what kinds of lifetimes HTTP/1.1
origin servers will be assigning to their responses (very few current
servers send Expires), but we can make a guess that of the responses
that are likely to be cachable, most will be assigned fairly long
lifetimes. This is because existing caches already assign, by
heuristic, fairly long lifetimes to responses without Expires or
max-age values, and (for the most part) this kind of caching seems to
be tolerated.
Certainly, one would hope that forthcoming HTTP/1.1 server
implementations will make it easier for service authors to specify
small freshness lifetimes. However, it is unlikely that this could
lead to a great improvement in the number of cache hits. Several
trace-based studies [1, 4] have shown that there is an upper limit to
the performance of straightforward HTTP caches, because for many
resources, no two requests result in the same response body.
Therefore, one should not expect the currently-observed cache hit
rates to get a lot better.
3.5 Implications of underestimating the Age value: Interpretation B
What happens if the Age value is underestimated? If this happens,
some "stale" responses appear to be "fresh", and are returned to the
client without any Warning. The client, in this case, naively
obtains the wrong response value.
This is a far more serious error than causing an extra cache miss.
Delivering the wrong response is not always a significant error, but
in some cases it can lead to serious external consequences.
Moreover, the set of cachable responses vulnerable to this error is
much larger than for an overestimation error. The reason is, as we
saw above, clock-skew errors appear to be quite common, especially
slow-clock errors, and the mean error is measured in tens of minutes.
This is likely to be much closer to the magnitude of freshness
lifetimes than the approximately 1-minute overestimation error.
Mogul [Page 7]
Internet-Draft HTTP Age generation 12 September 1997 13:48
The adoption of interpretation (b), in an attempt to avoid
unnecessary cache misses, could have a perverse effect: by
significantly increasing the likelihood of undetected delivery of
stale responses when the freshness lifetime is short, this might
discourage service authors from allowing such responses to be cached
at all. In other words, service authors generating responses with
short cachable lifetimes might be driven to declare these responses
as totally uncachable, to avoid the possibility of unwarranted
caching.
One should also note that a small (but certainly non-empty) fraction
of the bogus Date values in our trace are wrong by days, not hours.
If such a problem afflicts many clients, this could affect the
willingness of service authors to allow caching even of responses
with relatively long lifetimes.
3.6 So what's all the fuss about, anyway?
It might seem that arguing over the issue of caching responses with
relatively short freshness lifetimes is a tempest in a teapot. At
some level, it is: straightforward HTTP caching will probably never
yield the kind of hit rates seen for CPU caches, and this is
especially true for short-lifetime values.
However, there are a number of ways in which the effectiveness of an
HTTP cache could be improved; for example, prefetching seems to be
useful for decreasing latency (although it doesn't help if the path
is already bandwidth-limited). Prefetching might take advantage of
the cachability of responses with relatively short lifetimes, but it
will not be feasible if service authors cannot trust the prefetching
caches to be punctilious about freshness lifetimes.
In other fields of computer science (for example, compilers or
multiprocessor caches), aggressive optimization always depends on a
reliable understanding of the situation at hand. That is, if one
cannot be sure that the transformation done by the optimization
preserves the semantics of the system, one cannot safely do that
optimization (and, consequently, one may be prevented from doing a
lot of other optimizations that are enabled by the first one). Even
when optimizations are done "speculatively", this always involves
being able to check the results for semantic correctness before
committing them.
Attempts to make things faster at the cost of semantic transparency
might have some short-term attractions, especially in a setting where
we have historically been sloppy about semantic correctness. But
this is ultimately a dead-end path; after once giving up on semantic
correctness, it's almost impossible to go back.
Mogul [Page 8]
Internet-Draft HTTP Age generation 12 September 1997 13:48
4 Acknowledgements
I would like to thank Jim Gettys for comments on this draft.
5 References
1. Fred Douglis, Anja Feldmann, Balachander Krishnamurthy, and
Jeffrey Mogul. Rate of Change and Other Metrics: a Live Study of
the World Wide Web. Proc. Symposium on Internet Technologies and
Systems, USENIX, Monterey, CA, December, 1997. To appear.
2. Roy T. Fielding. Age Header Field in HTTP/1.1. Internet-Draft
draft-fielding-http-age-00.txt, Network Working Group, March, 1997.
3. Roy T. Fielding, Jim Gettys, Jeffrey C. Mogul, Henrik Frystyk
Nielsen, and Tim Berners-Lee. Hypertext Transfer Protocol --
HTTP/1.1. RFC 2068, HTTP Working Group, January, 1997.
4. Thomas M. Kroeger, Darrell D. E. Long, and Jeffrey C. Mogul.
Exploring the Bounds of Web Latency Reduction from Caching and
Prefetching. Proc. Symposium on Internet Technologies and Systems,
USENIX, Monterey, CA, December, 1997. To appear.
5. Jeffrey C. Mogul, Fred Douglis, Anja Feldmann, and Balachander
Krishnamurthy. Potential benefits of delta encoding and data
compression for HTTP. Research Report 97/4, DECWRL, July, 1997. URL
http://www.research.digital.com/wrl/techreports/abstracts/97.4.html.
6 Author's address
Jeffrey C. Mogul
Western Research Laboratory
Digital Equipment Corporation
250 University Avenue
Palo Alto, California, 94305, USA
Email: mogul@wrl.dec.com
Mogul [Page 9]