Network Working Group D. Cridland
Internet-Draft C. King
Expires: October 27, 2007 Isode Limited
April 25, 2007
Contexts for IMAP4
draft-cridland-imap-context-01
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on October 27, 2007.
Copyright Notice
Copyright (C) The IETF Trust (2007).
Abstract
The IMAP4rev1 protocol has powerful search facilities as part of the
core protocol, but lacks the ability to create live, updated results
which can be easily handled. This memo provides such an extension,
and shows how it can be used to provide a facility similar to virtual
mailboxes.
Cridland & King Expires October 27, 2007 [Page 1]
Internet-Draft IMAP CONTEXT April 2007
Table of Contents
1. Conventions used in this document . . . . . . . . . . . . . . 3
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Protocol Changes . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2. Context Hint . . . . . . . . . . . . . . . . . . . . . . . 4
3.3. Notifications of changes . . . . . . . . . . . . . . . . . 4
3.3.1. ADDTO Return Data Item . . . . . . . . . . . . . . . . 5
3.3.2. REMOVEFROM Return Data Item . . . . . . . . . . . . . 6
3.4. Partial results . . . . . . . . . . . . . . . . . . . . . 6
3.5. Caching results . . . . . . . . . . . . . . . . . . . . . 7
4. Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . . 8
5. Security Considerations . . . . . . . . . . . . . . . . . . . 9
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 9
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8.1. Normative References . . . . . . . . . . . . . . . . . . . 9
8.2. Informative References . . . . . . . . . . . . . . . . . . 10
Appendix A. Cookbook . . . . . . . . . . . . . . . . . . . . . . 10
A.1. Virtual Mailboxes . . . . . . . . . . . . . . . . . . . . 10
A.2. Trash Mailboxes . . . . . . . . . . . . . . . . . . . . . 10
A.3. Other uses . . . . . . . . . . . . . . . . . . . . . . . . 11
A.4. Resynchronizing Contexts . . . . . . . . . . . . . . . . . 11
Appendix B. Server Implementation Notes . . . . . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12
Intellectual Property and Copyright Statements . . . . . . . . . . 13
Cridland & King Expires October 27, 2007 [Page 2]
Internet-Draft IMAP CONTEXT April 2007
1. Conventions used in this document
In examples, "C:" and "S:" indicate lines sent by the client
messaging user agent and IMAP4rev1 ([IMAP]) server respectively. The
examples show a server which supports [ESEARCH] and [IDLE], neither
extension is required for this specification. The IDLE command is
used to denote an extended period of time during which any response
may be sent to the client.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [KEYWORDS].
Other capitalised words are typically names of IMAP extensions or
commands - these are uppercased for clarity only, and are case-
insensitive.
[[ Editorial comments are like this. XML2RFC working source is held
at http://svn.dave.cridland.net/svn/ietf-drafts/
draft-cridland-imap-contexts.xml ]]
2. Introduction
Although the basic SEARCH command defined in [IMAP], as enhanced by
[ESEARCH], is relatively compact in its representation, this
reduction only saves a certain amount of data, and huge mailboxes can
overwhelm the storage available for results on even relatively high-
end desktop machines.
This memo borrows concepts from [ACAP], providing a windowed view
onto search results, as well as bandwidth and round-trip efficient
updates.
It is intended that the protocol may be easily adapted onto the SORT
command specified in [SORT].
3. Protocol Changes
3.1. Overview
This extension is present in any IMAP4rev1 server which includes the
string "CONTEXT", or any string beginning "CONTEXT=", within its
advertised capabilities.
Such servers also accept three additional return options, and provide
three new result data items, and no new responses. The first search
Cridland & King Expires October 27, 2007 [Page 3]
Internet-Draft IMAP CONTEXT April 2007
return option is CONTEXT, an optional hint that the criteria will be
used repeatedly, and is defined in Section 3.2.
The second is UPDATE, which causes the server to provide efficient
notifications of changes to the results. This is defined in
Section 3.3.
Finally, the PARTIAL return specifier causes the server to return a
subset of the results in set-syntax. This allows for "virtual
scrollbars" and other UI conveniences to be achieved without having
to preload the entire result set, and is described in Section 3.4.
All of the return specifiers have no interaction with either each
other or any return specifiers defined in [ESEARCH].
3.2. Context Hint
The return option CONTEXT SHOULD be used by a client to indicate that
subsequent use of the criteria are likely. Servers MAY ignore this
return option, or use it as a hint to maintain a full result set, or
index.
A client might choose to obtain a count of matching messages prior to
obtaining actual results. Here, the client signifies its intention
to fetch the results themselves:
C: A01 SEARCH RETURN (CONTEXT COUNT) UNDELETED
UNKEYWORD $Junk
S: * ESEARCH (TAG "A01") COUNT 23765
S: A01 OK Search completed.
3.3. Notifications of changes
The search return option UPDATE, if used by a client, causes the
server to issue unsolicited notifications containing updates to the
SEARCH results which would be returned by an unmodified SEARCH.
These results are carried in ADDTO and REMOVEFROM data items in
ESEARCH/ESORT responses.
Both ADDTO and REMOVEFROM data items SHOULD be delivered to clients
in a timely manner, as and when results changes, whether by new
messages arriving in the mailbox, metadata such as flags being
changed, or messages being expunged.
Typically, this would occur at the same time as the FETCH, EXISTS or
EXPUNGE responses carrying the source of the change.
Unlike [ACAP], there is no requirement that a context need be created
Cridland & King Expires October 27, 2007 [Page 4]
Internet-Draft IMAP CONTEXT April 2007
with CONTEXT to use UPDATE, and in addition, the lack of UPDATE with
a CONTEXT does not affect the results caused by later SEARCH commands
- there is no snapshot facility.
There is no interaction between UPDATE and any other return options,
therefore use of RETURN (UPDATE MIN), for example, does not notify
about the minimum UID or sequence number, but notifies instead about
all changes to the set of matching messages.
In particular, this means that a client using UPDATE and PARTIAL on
the same search program MAY receive notifications about messages
which do not interest it currently.
This time, the client will require notifications of updates, and
chooses to obtain a count:
C: B01 UID SEARCH RETURN (UPDATE COUNT) DELETED
KEYWORD $Junk
S: * ESEARCH (TAG "B01") COUNT 74
S: B01 OK Search completed, will notify.
3.3.1. ADDTO Return Data Item
The ADDTO return data item contains, as payload, a list containing
pairs of a position and a set of results to be inserted at the
position. For ESEARCH responses, the position MAY be zero, and MAY
be ignored by clients.
The results are specified as UIDs or message numbers, depending on
how the UPDATE was specified. If the UPDATE was present in a SEARCH
command, the results will be message numbers; in a UID SEARCH
command, they will be UIDs.
C: B02 IDLE
S: + Idle
[...]
S: * 23762 FETCH (FLAGS (\Deleted \Seen))
S: * ESEARCH (TAG "B01") UID ADDTO (0 32768)
C: DONE
S: B02 OK Not Idle.
Note that this example assumes message 23762 with UID 32768
previously had neither \Deleted nor $Junk set. Also note that only
the ADDTO is included, and not the COUNT.
Cridland & King Expires October 27, 2007 [Page 5]
Internet-Draft IMAP CONTEXT April 2007
3.3.2. REMOVEFROM Return Data Item
The REMOVEFROM return data item contains a set of results to be
removed. The results to be removed are referenced by message number
or UID, as appropriate, and need not be in the same order as the
results. Servers SHOULD sort the results in order to use the
sequence-set syntax as efficiently as possible.
There is no requirement on servers to avoid issuing REMOVEFROM return
data at any particular moment, in particular this is distinct from
EXPUNGE responses.
The results are specified as UIDs or message numbers, depending on
how the UPDATE was specified. If the UPDATE was present in a SEARCH
command, the results will be message numbers; in a UID SEARCH
command, they will be UIDs.
Command B03 here is purely an example of a command which prohibits
EXPUNGE messages. The REMOVEFROM could have been sent without any
command in progress.
C: B03 SEARCH RETURN () 1:* ALL
S: * ESEARCH (TAG "B03") ALL 1:49152
S: * ESEARCH (TAG "B01") UID REMOVEFROM 32768
S: B03 OK Search completed.
C: B04 IDLE
S: + Idle
S: * EXPUNGE 23762
[...]
C: DONE
S: B04 OK Not Idle.
3.4. Partial results
The PARTIAL search return option causes the server to provide in an
ESEARCH response the range from the results denoted by the sequence
range given as the mandatory argument. The first result is 1, thus
the first 500 results would be obtained by a return option of
"PARTIAL 1:500", and the second 500 by "PARTIAL 501:1000". This
intentionally mirrors message sequence numbers.
Where a PARTIAL search return option references results which do not
exist, by using a range which starts or ends higher than the COUNT of
results, then the server returns those results which are in the set.
This yields a PARTIAL return data item which has, as payload, the
original range and a potentially missing set of results which may be
shorter than the extent of the range.
Cridland & King Expires October 27, 2007 [Page 6]
Internet-Draft IMAP CONTEXT April 2007
The subset of results are returned in sequence-set syntax, and
servers SHOULD order results from a SEARCH for maximum efficiency.
Clients need not request PARTIAL results in any particular order.
// Recall from A01 that there are 23764 results.
C: A02 UID SEARCH RETURN (PARTIAL 23500:24000) UNDELETED
UNKEYWORD $Junk
C: A03 UID SEARCH RETURN (PARTIAL 1:500) UNDELETED
UNKEYWORD $Junk
C: A04 UID SEARCH RETURN (PARTIAL 24000:24500) UNDELETED
UNKEYWORD $Junk
S: * ESEARCH (TAG "A02") UID PARTIAL (23500:24000 ...)
// 264 results in set syntax elided,
// this spans the end of the results.
S: A02 OK Completed.
S: * ESEARCH (TAG "A03") UID PARTIAL (1:500 ...)
// 500 results in set syntax elided.
S: A03 OK Completed.
S: * ESEARCH (TAG "A04") UID PARTIAL (24000:24500 NIL)
// No results are present, this is beyond the end of the results.
S: A04 OK Completed.
3.5. Caching results
Server implementations MAY cache results from a search or sort,
whether or not hinted to by CONTEXT, in order to make subsequent
searches more efficient, perhaps by recommencing a subsequent PARTIAL
search where a previous search left off. However servers MUST behave
identically whether or not internal caching is taking place,
therefore any such cache is required to be updated as changes to the
mailbox occur. An alternate strategy would be to discard results
when any change occurs to the mailbox.
Cridland & King Expires October 27, 2007 [Page 7]
Internet-Draft IMAP CONTEXT April 2007
4. Formal Syntax
The collected formal syntax. This includes definitions from [IMAP]
and [IMAP-ABNF], and uses ABNF as defined in [ABNF].
capability =/ "CONTEXT" / "CONTEXT=" atom
addto-position = number
;; Number may be 0 for SEARCH result additions.
;; <number> from RFC3501
modifier-context = "CONTEXT"
modifier-partial = "PARTIAL" SP seq-range
;; <seq-range> from [IMAP]
modifier-update = "UPDATE"
search-return-opt =/ modifier-context / modifier-partial /
modifier-update
;; All conform to <search-return-opt>, from [IMAP-ABNF]
ret-data-addto = "ADDTO"
SP "(" addto-position SP sequence-set
*(SP addto-position SP sequence-set)
")"
;; <sequence-set> from [IMAP]
ret-data-partial = "PARTIAL"
SP "(" seq-range SP partial-results ")"
;; <seq-range> is the requested range.
;; <seq-range> from [IMAP]
partial-results = sequence-set / "NIL"
;; <sequence-set> from [IMAP]
;; NIL indicates no results correspond to the requested range.
ret-data-removefrom = "REMOVEFROM" sequence-set
;; <sequence-set> from [IMAP]
search-return-data =/ ret-data-partial / ret-data-addto /
ret-data-removefrom
;; All conform to <search-return-data>, from [IMAP-ABNF]
Cridland & King Expires October 27, 2007 [Page 8]
Internet-Draft IMAP CONTEXT April 2007
5. Security Considerations
It is believed that this specification introduces no serious new
security considerations. However, implementors are advised to refer
to [IMAP].
Creation of contexts, both for UPDATE and PARTIAL, can benefit from
storing potentially large result sets on the server. Implementors
are advised to take care not to provide a method for denial of
service (DoS) attacks based on this; the notes in Appendix B may aid
in implementation decisions. Note that a server avoiding storing the
results will have much increased I/O, which may also be an avenue for
DoS attacks.
6. IANA Considerations
IMAP4 capabilities are registered by publishing a standards track or
IESG approved experimental RFC. The registry is currently located
at:
http://www.iana.org/assignments/imap4-capabilities
This document defines the CONTEXT IMAP capability. IANA is requested
to add it to the registry accordingly.
7. Acknowledgements
Much of the design of this extension can be found in ACAP. Valuable
comments, both in agreement and in dissent, were received from Alexey
Melnikov, Arnt Gulbrandsen, Randall Gellens, Cyrus Daboo, and others,
and many of these comments have had significant influence on the
design or the text. The authors are grateful to all those involved,
including those not mentioned here.
8. References
8.1. Normative References
[ABNF] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 4234, October 2005.
[IMAP] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION
4rev1", RFC 3501, March 2003.
[IMAP-ABNF]
Cridland & King Expires October 27, 2007 [Page 9]
Internet-Draft IMAP CONTEXT April 2007
Melnikov, A. and C. Daboo, "Collected Extensions to IMAP4
ABNF", RFC 4466, April 2006.
[KEYWORDS]
Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
8.2. Informative References
[ACAP] Newman, C. and J. Myers, "ACAP -- Application
Configuration Access Protocol", RFC 2244, November 1997.
[ESEARCH] Melnikov, A. and D. Cridland, "IMAP4 extension to SEARCH
command for controlling what kind of information is
returned", draft-melnikov-imap-search-ret-03 (work in
progress), June 2006.
[IDLE] Leiba, B., "IMAP4 IDLE command", RFC 2177, June 1997.
[SORT] Crispin, M. and K. Murchison, "INTERNET MESSAGE ACCESS
PROTOCOL - SORT AND THREAD EXTENSIONS",
draft-ietf-imapext-sort-18 (work in progress),
November 2006.
Appendix A. Cookbook
A.1. Virtual Mailboxes
It is possible to use the facilities described within this memo to
create a facility largely similar to a virtual mailbox, but handled
on the client side.
Initially, the client SELECTs the real "backing" mailbox. Next, it
can switch to a filtered view at any time by issuing a SEARCH RETURN
(COUNT UPDATE CONTEXT), and using SEARCH RETURN (PARTIAL x:y) as the
user scrolls, feeding the results into a FETCH as required to
populate summary views.
A.2. Trash Mailboxes
Certain contexts are particularly useful for client developers
wishing to present something similar to the common trash mailbox
metaphor in limited bandwidth. The simple criteria of UNDELETED only
matches undeleted messages, and the corresponding DELETED search key
can be used to display a per-mailbox trash-like virtual mailbox.
Cridland & King Expires October 27, 2007 [Page 10]
Internet-Draft IMAP CONTEXT April 2007
A.3. Other uses
It is entirely possible to simultaneously have two or more UPDATE
searches in operation. This can be used to build a grouped message
display in some cases, and also allows for monitoring counts of
messages matching certain complex criteria.
A.4. Resynchronizing Contexts
The creation of a context, and immediate access to it, can all be
accomplished in a single round-trip. Therefore, whilst it is
possible to elide resynchronization if no changes have occurred, it
is simpler in most cases to resynchronize by simply recreating the
context.
Appendix B. Server Implementation Notes
Although a server may cache the results, this is not mandated nor
required. UPDATE processing, for example, can be achieved by
comparison of the old flag state (if any) and the new, and PARTIAL
can be achieved by re-running the search until the suitable window is
required. This is a result of there being no snapshot facility.
For example, on a new message, the server can simply test for matches
against all current UPDATE context search programs, and for any that
match, send the ADDTO return data.
Similarly, for a flag change on an existing message, the server can
check whether the message matched with its old flags, whether it
matches with new flags, and provide ADDTO or REMOVEFROM return data
accordingly if these results differ.
For PARTIAL requests, the server can perform a full search,
discarding results until the lower bound is hit, and stopping the
search when sufficient results have been obtained.
With some additional state, it is possible to restart PARTIAL
searches, thus avoiding performing the initial discard phase.
For the best performance, however, caching the full search results is
needed, which can allow for faster responses at the expense of
memory. One reasonable strategy would be to balance this trade-off
at run-time, discarding search results after a suitable timeout, and
regenerating them as required.
This yields state requirements of storing the search program for any
UPDATE contexts, and optionally storing both search program and
Cridland & King Expires October 27, 2007 [Page 11]
Internet-Draft IMAP CONTEXT April 2007
(updated) results for further contexts as required.
Authors' Addresses
Dave Cridland
Isode Limited
5 Castle Business Village
36, Station Road
Hampton, Middlesex TW12 2BX
GB
Email: dave.cridland@isode.com
Curtis King
Isode Limited
5 Castle Business Village
36, Station Road
Hampton, Middlesex TW12 2BX
GB
Email: cking@mumbo.ca
Cridland & King Expires October 27, 2007 [Page 12]
Internet-Draft IMAP CONTEXT April 2007
Full Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Acknowledgment
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).
Cridland & King Expires October 27, 2007 [Page 13]
