Network Working Group D. Cridland
Internet-Draft C. King
Expires: April 14, 2007 Isode Limited
October 11, 2006
Contexts for IMAP4
draft-cridland-imap-context-00
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on April 14, 2007.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
The IMAP4rev1 protocol has powerful search facilities as part of the
core protocol, and a similarly powerful SORT extension, but lacks the
ability to create live, updated results which can be easily handled.
This memo provides such an extension, and shows how it can be used to
provide a facility similar to virtual mailboxes.
Cridland & King Expires April 14, 2007 [Page 1]
Internet-Draft IMAP CONTEXT October 2006
Table of Contents
1. Conventions used in this document . . . . . . . . . . . . . . 3
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Protocol Changes . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2. Extended SORT . . . . . . . . . . . . . . . . . . . . . . 4
3.3. Context Hint . . . . . . . . . . . . . . . . . . . . . . . 4
3.4. Notifications of changes . . . . . . . . . . . . . . . . . 4
3.4.1. ADDTO Return Data Item . . . . . . . . . . . . . . . . 5
3.4.2. REMOVEFROM Return Data Item . . . . . . . . . . . . . 5
3.5. Partial results . . . . . . . . . . . . . . . . . . . . . 6
3.6. Caching results . . . . . . . . . . . . . . . . . . . . . 6
4. Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . . 7
5. Security Considerations . . . . . . . . . . . . . . . . . . . 7
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8
8.1. Normative References . . . . . . . . . . . . . . . . . . . 8
8.2. Informative References . . . . . . . . . . . . . . . . . . 8
Appendix A. Cookbook . . . . . . . . . . . . . . . . . . . . . . 9
A.1. Virtual Mailboxes . . . . . . . . . . . . . . . . . . . . 9
A.2. Other uses . . . . . . . . . . . . . . . . . . . . . . . . 9
A.3. Resynchronizing Contexts . . . . . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10
Intellectual Property and Copyright Statements . . . . . . . . . . 11
Cridland & King Expires April 14, 2007 [Page 2]
Internet-Draft IMAP CONTEXT October 2006
1. Conventions used in this document
In examples, "C:" and "S:" indicate lines sent by the client
messaging user agent and IMAP4rev1 ([IMAP]) server respectively. The
examples show a server which supports [ESEARCH] and [IDLE], neither
extension is required for this specification. The IDLE command is
used to denote an extended period of time during which any response
may be sent to the client.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [KEYWORDS].
Other capitalised words are typically names of IMAP extensions or
commands - these are uppercased for clarity only, and are case-
insensitive.
[[ Editorial comments are like this. XML2RFC working source is held
at http://svn.dave.cridland.net/svn/ietf-drafts/
draft-cridland-imap-contexts.xml ]]
2. Introduction
Although the basic SEARCH command defined in [IMAP], as enhanced by
[ESEARCH], is relatively compact in its representation, this
reduction only saves a certain amount of data, and huge mailboxes can
overwhelm the storage available for results on even relatively high-
end desktop machines.
For SORT, as defined in [SORT], use of set-syntax saves considerably
less, and so the problem is more acute.
This memo borrows concepts from [ACAP], providing a windowed view
onto search and sort results, as well as low-bandwidth updates.
3. Protocol Changes
3.1. Overview
This extension is present in any IMAP4rev1 server which includes the
string "CONTEXTS", or any string beginning "CONTEXTS=", within its
advertised capabilities.
These servers handle an extended form of the SORT command accepting
return options similar to SEARCH, and an extended sorted results
response ESORT, described in Section 3.2, which mirrors the extended
Cridland & King Expires April 14, 2007 [Page 3]
Internet-Draft IMAP CONTEXT October 2006
SEARCH described in [IMAP-ABNF]. [[ Of course, if this were moved to
SORT, we'd be happy. ]]
Such servers also accept three additional return options, and provide
three new result data items, and no new responses. The first search
return option is CONTEXT, an optional hint that the criteria will be
used repeatedly, and is defined in Section 3.3.
The second is NOTIFY, which causes the server to provide efficient
notifications of changes to the results. This is defined in
Section 3.4.
Finally, the PARTIAL return specifier causes the server to return a
subset of the results in set-syntax. This allows for "virtual
scrollbars" and other UI conveniences to be achieved without having
to preload the entire result set, and is described in Section 3.5.
3.2. Extended SORT
[[ This needs to describe extended SORT in the same terms as RFC4466.
For now, just close your eyes and imagine something largely similar
to ESEARCH and SEARCH return options. ]]
3.3. Context Hint
The return option CONTEXT SHOULD be used by a client to indicate that
subsequent use of the criteria (and sort order, for SORT) are likely.
Servers MAY ignore this return option, or use it as a hint to
maintain a full result set, or index.
A client might choose to obtain a count of matching messages prior to
obtaining actual results. Here, the client signifies its intention
to fetch the results themselves:
C: A01 SEARCH RETURN (CONTEXT COUNT) UNDELETED
UNKEYWORD $Junk
S: * ESEARCH (TAG A01) COUNT 23765
S: A01 OK Search completed.
3.4. Notifications of changes
The search return option NOTIFY, if used by a client, causes the
server to issue unsolicited notifications containing updates to the
SEARCH or SORT results. These results are carried in ADDTO and
REMOVEFROM data items in ESEARCH/ESORT responses.
Both ADDTO and REMOVEFROM data items SHOULD be delivered to clients
in a timely manner, as and when results changes, whether by new
Cridland & King Expires April 14, 2007 [Page 4]
Internet-Draft IMAP CONTEXT October 2006
messages arriving in the mailbox, metadata such as flags being
changed, or messages being expunged.
Typically, this would occur at the same time as the FETCH, EXISTS or
EXPUNGE responses carrying the source of the change.
Unlike [ACAP], there is no requirement that a context need be created
with CONTEXT to use NOTIFY, and in addition, the lack of NOTIFY with
a CONTEXT does not affect the results caused by later SEARCH commands
- there is no snapshot facility.
This time, the client will require notifications of updates, and
chooses to obtain a count:
C: B01 UID SEARCH RETURN (NOTIFY COUNT) DELETED
KEYWORD $Junk
S: * ESEARCH (TAG B01) COUNT 74
S: B01 OK Search completed, will notify.
3.4.1. ADDTO Return Data Item
The ADDTO return data item contains, as payload, a list containing a
position and a set of results to be inserted at the position. For
ESEARCH responses, the position MAY be zero, and MAY be ignored by
clients. For ESORT responses, the position MUST be the position of
the first message referred to in the sequence-set of results,
counting 1 as the first position.
C: B02 IDLE
S: + Idle
[...]
S: * 23762 FETCH (FLAGS (\Deleted \Seen))
S: * ESEARCH (TAG B01) UID ADDTO (0 32768)
C: DONE
S: B02 OK Not Idle.
Note that this example assumes message 23762 with UID 32768
previously had neither \Deleted nor $Junk set. Also note that only
the ADDTO is included, and not the COUNT.
3.4.2. REMOVEFROM Return Data Item
The REMOVEFROM return data item contains a set of results to be
removed. The results to be removed are referenced by message number
or UID, as appropriate, and need not be in the same order as the
results. Servers SHOULD sort the results in order to use the
sequence-set syntax as efficiently as possible.
Cridland & King Expires April 14, 2007 [Page 5]
Internet-Draft IMAP CONTEXT October 2006
There is no requirement on servers to avoid issuing REMOVEFROM return
data at any particular moment, in particular this is distinct from
EXPUNGE responses.
Command B03 here is purely an example of a command which prohibits
EXPUNGE messages. The REMOVEFROM could have been sent without any
command in progress.
C: B03 SEARCH RETURN () 1:* ALL
S: * ESEARCH (TAG B03) ALL 1:49152
S: * ESEARCH (TAG B01) UID REMOVEFROM 32768
S: B03 OK Search completed.
C: B04 IDLE
S: + Idle
S: * EXPUNGE 23762
[...]
C: DONE
S: B04 OK Not Idle.
3.5. Partial results
The PARTIAL search return option causes the server to provide in an
ESEARCH or ESORT response the range from the results denoted by the
sequence range given as the mandatory argument. The first result is
1, thus the first 500 results would be obtained by a return option of
"PARTIAL 1:500", and the second 500 by "PARTIAL 501:1000". This
intentionally mirrors message sequence numbers.
The subset of results are returned in sequence-set syntax, and
servers SHOULD order results from a SEARCH for maximum efficiency.
Clients need not request PARTIAL results in any particular order.
C: A02 UID SEARCH RETURN (PARTIAL 23500:24000) UNDELETED
UNKEYWORD $Junk
S: * ESEARCH (TAG A02) PARTIAL (23500:24000 ...)
// Elided results include 264 results in this sequence.
S: A02 OK Completed.
3.6. Caching results
Server implementations MAY cache results from a search or sort,
whether or not hinted to by CONTEXT, in order to make subsequent
searches more efficient, perhaps by recommencing a subsequent PARTIAL
search where a previous search left off. However servers MUST behave
identically whether or not internal caching is taking place,
therefore any such cache is required to be updated as changes to the
Cridland & King Expires April 14, 2007 [Page 6]
Internet-Draft IMAP CONTEXT October 2006
mailbox occur. An alternate strategy would be to discard results
when any change occurs to the mailbox.
4. Formal Syntax
The collected formal syntax. This includes definitions from [IMAP]
and [IMAP-ABNF], and uses ABNF as defined in [ABNF].
addto-position = number
;; Number can be 0 for SEARCH result additions.
;; number from RFC3501
modifier-context = "CONTEXT"
modifier-partial = "PARTIAL" SP seq-range
;; seq-range from RFC3501
modifier-notify = "NOTIFY"
search-return-opt =/ modifier-context / modifier-partial /
modifier-notify
;; All conform to search-return-opt, from RFC4466
ret-data-addto = "ADDTO"
SP "(" addto-position SP sequence-set ")"
;; sequence-set from RFC3501
ret-data-partial = "PARTIAL"
SP "(" seq-range SP sequence-set ")"
;; sequence-set from RFC3501
ret-data-removefrom = "REMOVEFROM" SP sequence-set
;; sequence-set from RFC3501
search-return-data =/ ret-data-partial / ret-data-addto /
ret-data-removefrom
;; All conform to search-return-data, from RFC4466.
5. Security Considerations
The authors cannot think of any.
6. IANA Considerations
[[ The extension needs registering - TODO: find correct text for this
Cridland & King Expires April 14, 2007 [Page 7]
Internet-Draft IMAP CONTEXT October 2006
bit. ]]
7. Acknowledgements
Much of the design of this extension can be found in ACAP. Valuable
comments, both in agreement and in dissent, were received from Alexey
Melnikov, Arnt Gulbrandsen, Randall Gellens, Cyrus Daboo, and others,
and many of these comments have had significant influence on the
design or the text. The authors are grateful to all those involved,
including those not mentioned here.
8. References
8.1. Normative References
[ABNF] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 4234, October 2005.
[IMAP] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION
4rev1", RFC 3501, March 2003.
[IMAP-ABNF]
Melnikov, A. and C. Daboo, "Collected Extensions to IMAP4
ABNF", RFC 4466, April 2006.
[KEYWORDS]
Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[SORT] Crispin, M. and K. Murchison, "INTERNET MESSAGE ACCESS
PROTOCOL - SORT AND THREAD EXTENSION",
draft-ietf-imapext-sort-17 (work in progress), May 2004.
8.2. Informative References
[ACAP] Newman, C. and J. Myers, "ACAP -- Application
Configuration Access Protocol", RFC 2244, November 1997.
[ESEARCH] Melnikov, A. and D. Cridland, "IMAP4 extension to SEARCH
command for controlling what kind of information is
returned", draft-melnikov-imap-search-ret-03 (work in
progress), June 2006.
[IDLE] Leiba, B., "IMAP4 IDLE command", RFC 2177, June 1997.
Cridland & King Expires April 14, 2007 [Page 8]
Internet-Draft IMAP CONTEXT October 2006
Appendix A. Cookbook
A.1. Virtual Mailboxes
It is possible to use the facilities described within this memo to
create a facility largely similar to a virtual mailbox, but handled
on the client side. Because the messages are still referred to by
their native UIDs, and there is no constraint to fit the IMAP mailbox
interface, this is a simpler method without the potential pitfalls of
stretching metaphors.
Initially, the client SELECTs the real "backing" mailbox. Next, it
can switch to a filtered view at any time by issuing a SEARCH RETURN
(COUNT NOTIFY CONTEXT), and using SEARCH RETURN (PARTIAL x:y) as the
user scrolls, feeding the results into a FETCH to populate summary
views as required.
Similarly, a sorted display can be obtained by using SORT instead of
SEARCH.
A.2. Other uses
It is possible to simultaneously have two or more NOTIFY searches or
sorts in operation. This can be used to build a grouped message
display.
A.3. Resynchronizing Contexts
Although not specifically provided by the protocol as defined, it is
possible for clients to resynchronize a notify unsorted context
without having to refetch the entire result set, if the server
supports CONDSTORE. The following steps are believed to be correct:
Recreate the context as soon as possible, returning the COUNT.
Check the HIGHESTMODSEQ value for changes.
Resynchronize UID mapping, removing any messages from the contexts
which have been expunged.
Find all changed messages (using, perhaps, UID FETCH 1:* FLAGS
(CHANGEDSINCE)).
Either calculate locally, or research across those messages, to
discover the changes to the new context.
Client authors should note that for contexts which contain only a
small percentage of the entire mailbox - or have a very fluid
membership - it will be cheaper in terms of bandwidth to simply
recreate the context from scratch.
Cridland & King Expires April 14, 2007 [Page 9]
Internet-Draft IMAP CONTEXT October 2006
Authors' Addresses
Dave Cridland
Isode Limited
5 Castle Business Village
36, Station Road
Hampton, Middlesex TW12 2BX
GB
Email: dave.cridland@isode.com
Curtis King
Isode Limited
5 Castle Business Village
36, Station Road
Hampton, Middlesex TW12 2BX
GB
Email: cking@mumbo.ca
Cridland & King Expires April 14, 2007 [Page 10]
Internet-Draft IMAP CONTEXT October 2006
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Disclaimer of Validity
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2006). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Cridland & King Expires April 14, 2007 [Page 11]
