Network Working Group                                        D. Cridland
Internet-Draft                                                   C. King
Expires: November 19, 2007                                 Isode Limited
                                                            May 18, 2007

                           Contexts for IMAP4

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on November 19, 2007.

Copyright Notice

   Copyright (C) The IETF Trust (2007).


   The IMAP4rev1 protocol has powerful search facilities as part of the
   core protocol, but lacks the ability to create live, updated results
   which can be easily handled.  This memo provides such an extension,
   and shows how it can be used to provide a facility similar to virtual

Cridland & King         Expires November 19, 2007               [Page 1]

Internet-Draft                IMAP CONTEXT                      May 2007

Table of Contents

   1.  Conventions used in this document  . . . . . . . . . . . . . .  3
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   3.  Protocol Changes . . . . . . . . . . . . . . . . . . . . . . .  3
     3.1.  Overview . . . . . . . . . . . . . . . . . . . . . . . . .  3
     3.2.  Context Hint . . . . . . . . . . . . . . . . . . . . . . .  4
     3.3.  Notifications of changes . . . . . . . . . . . . . . . . .  4
       3.3.1.  Refusing to update contexts  . . . . . . . . . . . . .  5
       3.3.2.  Common Features of ADDTO and REMOVEFROM  . . . . . . .  6
       3.3.3.  ADDTO Return Data Item . . . . . . . . . . . . . . . .  6
       3.3.4.  REMOVEFROM Return Data Item  . . . . . . . . . . . . .  7
       3.3.5.  The FREECONTEXT command  . . . . . . . . . . . . . . .  7
     3.4.  Partial results  . . . . . . . . . . . . . . . . . . . . .  8
     3.5.  Caching results  . . . . . . . . . . . . . . . . . . . . .  9
   4.  Formal Syntax  . . . . . . . . . . . . . . . . . . . . . . . .  9
   5.  Security Considerations  . . . . . . . . . . . . . . . . . . . 11
   6.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 11
   7.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11
   8.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
     8.1.  Normative References . . . . . . . . . . . . . . . . . . . 11
     8.2.  Informative References . . . . . . . . . . . . . . . . . . 12
   Appendix A.  Cookbook  . . . . . . . . . . . . . . . . . . . . . . 12
     A.1.  Virtual Mailboxes  . . . . . . . . . . . . . . . . . . . . 12
     A.2.  Trash Mailboxes  . . . . . . . . . . . . . . . . . . . . . 12
     A.3.  Immediate EXPUNGE notifications  . . . . . . . . . . . . . 13
     A.4.  Other uses . . . . . . . . . . . . . . . . . . . . . . . . 13
     A.5.  Resynchronizing Contexts . . . . . . . . . . . . . . . . . 13
   Appendix B.  Server Implementation Notes . . . . . . . . . . . . . 13
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 14
   Intellectual Property and Copyright Statements . . . . . . . . . . 15

Cridland & King         Expires November 19, 2007               [Page 2]

Internet-Draft                IMAP CONTEXT                      May 2007

1.  Conventions used in this document

   In examples, "C:" and "S:" indicate lines sent by the client
   messaging user agent and IMAP4rev1 ([IMAP]) server respectively.
   Although the examples show a server which supports [ESEARCH], this is
   not a requirement of this specification.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in [KEYWORDS].

   Other capitalised words are typically names of IMAP extensions or
   commands - these are uppercased for clarity only, and are case-

   [[ Editorial comments are like this.  XML2RFC working source is held
   draft-cridland-imap-contexts.xml ]]

2.  Introduction

   Although the basic SEARCH command defined in [IMAP], and as enhanced
   by [ESEARCH], is relatively compact in its representation, this
   reduction saves only a certain amount of data, and huge mailboxes
   might overwhelm the storage available for results on even relatively
   high-end desktop machines.

   This memo borrows concepts from [ACAP], providing a windowed view
   onto search results, as well as bandwidth and round-trip efficient

   It is intended that the protocol may be easily adapted onto the SORT
   command specified in [SORT].

3.  Protocol Changes

3.1.  Overview

   This extension is present in any IMAP4rev1 server which includes the
   string "CONTEXT", or any string beginning "CONTEXT=", within its
   advertised capabilities.

   Such servers also accept three additional return options, and provide
   three new result data items, and no new responses.  The first search
   return option is CONTEXT, an optional hint that the criteria will be
   used repeatedly, and is defined in Section 3.2.

Cridland & King         Expires November 19, 2007               [Page 3]

Internet-Draft                IMAP CONTEXT                      May 2007

   The second is UPDATE, which causes the server to provide efficient
   notifications of changes to the results.  This is defined in
   Section 3.3.

   Finally, the PARTIAL return specifier causes the server to return a
   subset of the results in set-syntax.  This allows for "virtual
   scrollbars" and other UI conveniences to be achieved without having
   to preload the entire result set, and is described in Section 3.4.

   All of the return specifiers have no interaction with either each
   other or any return specifiers defined in [ESEARCH].

3.2.  Context Hint

   The return option CONTEXT SHOULD be used by a client to indicate that
   subsequent use of the criteria are likely.  Servers MAY ignore this
   return option, or use it as a hint to maintain a full result set, or

   A client might choose to obtain a count of matching messages prior to
   obtaining actual results.  Here, the client signifies its intention
   to fetch the results themselves:

          UNKEYWORD $Junk
       S: * ESEARCH (TAG "A01") COUNT 23765
       S: A01 OK Search completed.

3.3.  Notifications of changes

   The search return option UPDATE, if used by a client, causes the
   server to issue unsolicited notifications containing updates to the
   SEARCH results which would be returned by an unmodified SEARCH.
   These results are carried in ADDTO and REMOVEFROM data items in
   ESEARCH/ESORT responses.

   Both ADDTO and REMOVEFROM data items SHOULD be delivered to clients
   in a timely manner, as and when results change, whether by new
   messages arriving in the mailbox, metadata such as flags being
   changed, or messages being expunged.

   Typically, this would occur at the same time as the FETCH, EXISTS or
   EXPUNGE responses carrying the source of the change.

   Updates will cease only when the mailbox is no longer selected, or
   when the FREECONTEXT command is issued by the client, whichever is

Cridland & King         Expires November 19, 2007               [Page 4]

Internet-Draft                IMAP CONTEXT                      May 2007

   Unlike [ACAP], there is no requirement that a context need be created
   with CONTEXT to use UPDATE, and in addition, the lack of UPDATE with
   a CONTEXT does not affect the results caused by later SEARCH commands
   - there is no snapshot facility.

   There is no interaction between UPDATE and any other return options;
   therefore use of RETURN (UPDATE MIN), for example, does not notify
   about the minimum UID or sequence number, but notifies instead about
   all changes to the set of matching messages.

   In particular, this means that a client using UPDATE and PARTIAL on
   the same search program MAY receive notifications about messages
   which do not currently interest it.

   This time, the client will require notifications of updates, and
   chooses to obtain a count:

          KEYWORD $Junk
       S: * ESEARCH (TAG "B01") COUNT 74
       S: B01 OK Search completed, will notify.

3.3.1.  Refusing to update contexts

   In some cases, the server MAY refuse to provide updates, such as if
   an internal limit on the number of update contexts is reached.

   In this case, an untagged NO is generated during processing of the
   command with a response-code of NOUPDATE.  The response-code
   contains, as argument, the tag of the search command for which the
   server is refusing to honour the UPDATE request.

   Other return options specified will still be honoured.

   Servers MUST provide at least one updating context per client, and
   SHOULD provide more - see Appendix B for strategies on reducing the
   impact of additional updating contexts.

   This time, the client will require notifications of updates, and
   chooses to obtain a count:

          KEYWORD $Junk
       S: * ESEARCH (TAG "B01") COUNT 74
       S: * NO [NOUPDATE "B01"] Too many contexts
       S: B02 OK Search completed, will not notify.

Cridland & King         Expires November 19, 2007               [Page 5]

Internet-Draft                IMAP CONTEXT                      May 2007

3.3.2.  Common Features of ADDTO and REMOVEFROM

   The result update set included in the return data item is specified
   as UIDs or message numbers, depending on how the UPDATE was
   specified.  If the UPDATE was present in a SEARCH command, the
   results will be message numbers; in a UID SEARCH command, they will
   be UIDs.

   The client MUST process (and the server MUST generate) ADDTO and
   REMOVEFROM return data items in order, including those within a
   single ESEARCH response.

   As with any response aside from EXPUNGE, ESEARCH responses carrying
   ADDTO and/or REMOVEFROM return data items MAY be sent at any time.
   In particular, servers MAY send such responses when no command is in
   progress, during the processing of any command, or when the client is
   using the IDLE facility described in [IDLE].  Implementors are
   recommended to read [NOTIFY] as a mechanism for clients to signal
   servers that they are willing to process responses at any time, and
   are also recommended to pay close attention to Section 5.3 of [IMAP].

   Client implementors SHOULD NOT reuse tags with any command; in this
   case the tag is used to identify the SEARCH command which created the

3.3.3.  ADDTO Return Data Item

   The ADDTO return data item contains, as payload, a list containing
   pairs of a position and a set of result updates to be inserted at the
   position.  For ESEARCH responses, the position MAY be zero, and MAY
   be ignored by clients.

   If the position is non-zero, the new results in the update are to be
   inserted at the given position; meaning that the new results will
   occupy the indicated start position and all existing results starting
   from that position are shifted to the position after the insertion.
   The client MUST update the position numbers of the shifted results.
   The ADDTO return data item MAY include several new results to be
   inserted - therefore it is imporant to note that the positions
   included in a single ADDTO return data item contain positions before
   the shifting due to other new results take place.

Cridland & King         Expires November 19, 2007               [Page 6]

Internet-Draft                IMAP CONTEXT                      May 2007

       S: * 23762 FETCH (FLAGS (\Deleted \Seen))
       S: * 23763 FETCH (FLAGS ($Junk \Seen))
       S: * ESEARCH (TAG "B01") UID ADDTO (0 32768:32769)

   Note that this example assumes messages 23762 and 23763 with UIDs
   32768 and 32769 respectively previously had neither \Deleted nor
   $Junk set.  Also note that only the ADDTO is included, and not the
   (now changed) COUNT.

3.3.4.  REMOVEFROM Return Data Item

   The REMOVEFROM return data item contains a set of results to be
   removed.  The results to be removed are referenced by message number
   or UID, as appropriate, and need not be in the same order as the
   results, therefore servers are RECOMMENDED to sort the results into
   increasing UID (and sequence number) order, to take full advantage of
   the set-syntax representation.

   Note that although no position information is included, the positions
   of any results after those removed will change.

   Here, a message in the result set is expunged.  The REMOVEFROM here
   is shown to happen without any command in progress, see
   Section 3.3.2.  Note that EXPUNGE responses do not have this

       S: * ESEARCH (TAG "B01") UID REMOVEFROM 32768
       C: B03 NOOP
       S: * EXPUNGE 23762
       S: B03 OK Nothing done.

3.3.5.  The FREECONTEXT command

   When a client no longer wishes to receive updates, it may issue the
   FREECONTEXT command, which will prevent all updates to the contexts
   named in the arguments from being transmitted by the server.  The
   command takes, as arguments, one or more tags of the commands used to
   request updates.

   The server MAY free any resource associated with a context so

       C: B04 FREECONTEXT "B01"
       S: B04 OK No further updates.

Cridland & King         Expires November 19, 2007               [Page 7]

Internet-Draft                IMAP CONTEXT                      May 2007

3.4.  Partial results

   The PARTIAL search return option causes the server to provide in an
   ESEARCH response the range from the results denoted by the sequence
   range given as the mandatory argument.  The first result is 1, thus
   the first 500 results would be obtained by a return option of
   "PARTIAL 1:500", and the second 500 by "PARTIAL 501:1000".  This
   intentionally mirrors message sequence numbers.

   For SEARCH results, the entire result set MUST be ordered in "mailbox
   order", that is, in UID or message sequence number order.

   Where a PARTIAL search return option references results which do not
   exist, by using a range which starts or ends higher than the COUNT of
   results, then the server returns those results which are in the set.
   This yields a PARTIAL return data item which has, as payload, the
   original range and a potentially missing set of results which may be
   shorter than the extent of the range.

   The subset of results are returned in sequence-set syntax, and
   servers SHOULD order results from a SEARCH for maximum efficiency in
   this representation.

   Clients need not request PARTIAL results in any particular order.

       // Recall from A01 that there are 23764 results.
          UNKEYWORD $Junk
          UNKEYWORD $Junk
          UNKEYWORD $Junk
       S: * ESEARCH (TAG "A02") UID PARTIAL (23500:24000 ...)
       // 264 results in set syntax elided,
       // this spans the end of the results.
       S: A02 OK Completed.
       S: * ESEARCH (TAG "A03") UID PARTIAL (1:500 ...)
       // 500 results in set syntax elided.
       S: A03 OK Completed.
       S: * ESEARCH (TAG "A04") UID PARTIAL (24000:24500 NIL)
       // No results are present, this is beyond the end of the results.
       S: A04 OK Completed.

Cridland & King         Expires November 19, 2007               [Page 8]

Internet-Draft                IMAP CONTEXT                      May 2007

3.5.  Caching results

   Server implementations MAY cache results from a search or sort,
   whether or not hinted to by CONTEXT, in order to make subsequent
   searches more efficient, perhaps by recommencing a subsequent PARTIAL
   search where a previous search left off.  However servers MUST behave
   identically whether or not internal caching is taking place,
   therefore any such cache is required to be updated as changes to the
   mailbox occur.  An alternate strategy would be to discard results
   when any change occurs to the mailbox.

4.  Formal Syntax

Cridland & King         Expires November 19, 2007               [Page 9]

Internet-Draft                IMAP CONTEXT                      May 2007

   The collected formal syntax.  This includes definitions from [IMAP]
   and [IMAP-ABNF], and uses ABNF as defined in [ABNF].

   capability          =/ "CONTEXT" / "CONTEXT=" atom

   command-select      =/ "FREECONTEXT" 1*(SP quoted)
       ;; <command-select> from [IMAP]

   addto-position      = number
       ;; Number may be 0 for SEARCH result additions.
       ;; <number> from [IMAP]

   modifier-context    = "CONTEXT"

   modifier-partial    = "PARTIAL" SP seq-range
       ;; <seq-range> from [IMAP]

   modifier-update     = "UPDATE"

   search-return-opt   =/ modifier-context / modifier-partial /
       ;; All conform to <search-return-opt>, from [IMAP-ABNF]

   resp-text-code      =/ "NOUPDATE" SP quoted
       ;; <resp-text-code> from [IMAP]

   ret-data-addto      = "ADDTO"
                          SP "(" addto-position SP sequence-set
                          *(SP addto-position SP sequence-set)
       ;; <sequence-set> from [IMAP]

   ret-data-partial    = "PARTIAL"
                         SP "(" seq-range SP partial-results ")"
       ;; <seq-range> is the requested range.
       ;; <seq-range> from [IMAP]

   partial-results     = sequence-set / "NIL"
       ;; <sequence-set> from [IMAP]
       ;; NIL indicates no results correspond to the requested range.

   ret-data-removefrom = "REMOVEFROM" SP sequence-set
       ;; <sequence-set> from [IMAP]

   search-return-data  =/ ret-data-partial / ret-data-addto /
       ;; All conform to <search-return-data>, from [IMAP-ABNF]

Cridland & King         Expires November 19, 2007              [Page 10]

Internet-Draft                IMAP CONTEXT                      May 2007

5.  Security Considerations

   It is believed that this specification introduces no serious new
   security considerations.  However, implementors are advised to refer
   to [IMAP].

   Creation of contexts, both for UPDATE and PARTIAL, can benefit from
   storing potentially large result sets on the server.  Implementors
   are advised to take care not to provide a method for denial of
   service (DoS) attacks based on this; the notes in Appendix B may aid
   in implementation decisions.  Note that a server avoiding storing the
   results will have much increased I/O, which may also be an avenue for
   DoS attacks.

6.  IANA Considerations

   IMAP4 capabilities are registered by publishing a standards track or
   IESG approved experimental RFC.  The registry is currently located

   This document defines the CONTEXT IMAP capability.  IANA is requested
   to add it to the registry accordingly.

7.  Acknowledgements

   Much of the design of this extension can be found in ACAP.  Valuable
   comments, both in agreement and in dissent, were received from Alexey
   Melnikov, Arnt Gulbrandsen, Cyrus Daboo, Filip Navara, Mark Crispin,
   Randall Gellens, Zoltan Ordogh and others, and many of these comments
   have had significant influence on the design or the text.  The
   authors are grateful to all those involved, including those not
   mentioned here.

8.  References

8.1.  Normative References

   [ABNF]     Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
              Specifications: ABNF", RFC 4234, October 2005.

              4rev1", RFC 3501, March 2003.

Cridland & King         Expires November 19, 2007              [Page 11]

Internet-Draft                IMAP CONTEXT                      May 2007

              Melnikov, A. and C. Daboo, "Collected Extensions to IMAP4
              ABNF", RFC 4466, April 2006.

              Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

8.2.  Informative References

   [ACAP]     Newman, C. and J. Myers, "ACAP -- Application
              Configuration Access Protocol", RFC 2244, November 1997.

   [ESEARCH]  Melnikov, A. and D. Cridland, "IMAP4 Extension to SEARCH
              Command for Controlling What Kind of Information Is
              Returned", RFC 4731, November 2006.

   [IDLE]     Leiba, B., "IMAP4 IDLE command", RFC 2177, June 1997.

   [NOTIFY]   King, C., "The IMAP NOTIFY Extension",
              draft-gulbrandsen-imap-notify-03 (work in progress),
              March 2007.

   [SORT]     Crispin, M. and K. Murchison, "INTERNET MESSAGE ACCESS
              draft-ietf-imapext-sort-18 (work in progress),
              November 2006.

Appendix A.  Cookbook

A.1.  Virtual Mailboxes

   It is possible to use the facilities described within this memo to
   create a facility largely similar to a virtual mailbox, but handled
   on the client side.

   Initially, the client SELECTs the real "backing" mailbox.  Next, it
   can switch to a filtered view at any time by issuing a SEARCH RETURN
   user scrolls, feeding the results into a FETCH as required to
   populate summary views.

A.2.  Trash Mailboxes

   Certain contexts are particularly useful for client developers
   wishing to present something similar to the common trash mailbox
   metaphor in limited bandwidth.  The simple criteria of UNDELETED only

Cridland & King         Expires November 19, 2007              [Page 12]

Internet-Draft                IMAP CONTEXT                      May 2007

   matches undeleted messages, and the corresponding DELETED search key
   can be used to display a per-mailbox trash-like virtual mailbox.

A.3.  Immediate EXPUNGE notifications

   The command "SEARCH RETURN (UPDATE) ALL" can be used to create a
   context which notifies immediately about expunged messages, yet will
   not affect message sequence numbers until the normal EXPUNGE message
   can be sent.  This may be useful for clients wishing to show this
   behaviour without losing the benefit of sequence numbering.

A.4.  Other uses

   It is entirely possible to simultaneously have two or more UPDATE
   contexts in operation.  This can be used to build a grouped message
   display in some cases, and also allows for monitoring counts of
   messages matching certain complex criteria.

A.5.  Resynchronizing Contexts

   The creation of a context, and immediate access to it, can all be
   accomplished in a single round-trip.  Therefore, whilst it is
   possible to elide resynchronization if no changes have occurred, it
   is simpler in most cases to resynchronize by simply recreating the

Appendix B.  Server Implementation Notes

   Although a server may cache the results, this is not mandated nor
   required.  UPDATE processing, for example, can be achieved
   efficiently by comparison of the old flag state (if any) and the new,
   and PARTIAL can be achieved by re-running the search until the
   suitable window is required.  This is a result of there being no
   snapshot facility.

   For example, on a new message, the server can simply test for matches
   against all current UPDATE context search programs, and for any that
   match, send the ADDTO return data.

   Similarly, for a flag change on an existing message, the server can
   check whether the message matched with its old flags, whether it
   matches with new flags, and provide ADDTO or REMOVEFROM return data
   accordingly if these results differ.

   For PARTIAL requests, the server can perform a full search,
   discarding results until the lower bound is hit, and stopping the
   search when sufficient results have been obtained.

Cridland & King         Expires November 19, 2007              [Page 13]

Internet-Draft                IMAP CONTEXT                      May 2007

   With some additional state, it is possible to restart PARTIAL
   searches, thus avoiding performing the initial discard phase.

   For the best performance, however, caching the full search results is
   needed, which can allow for faster responses at the expense of
   memory.  One reasonable strategy would be to balance this trade-off
   at run-time, discarding search results after a suitable timeout, and
   regenerating them as required.

   This yields state requirements of storing the search program for any
   UPDATE contexts, and optionally storing both search program and
   (updated) results for further contexts as required.

Authors' Addresses

   Dave Cridland
   Isode Limited
   5 Castle Business Village
   36, Station Road
   Hampton, Middlesex  TW12 2BX


   Curtis King
   Isode Limited
   5 Castle Business Village
   36, Station Road
   Hampton, Middlesex  TW12 2BX


Cridland & King         Expires November 19, 2007              [Page 14]

Internet-Draft                IMAP CONTEXT                      May 2007

Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at


   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).

Cridland & King         Expires November 19, 2007              [Page 15]