Internet Draft: IMAP Extension for Conditional STORE A. Melnikov Document: draft-melnikov-imap-condstore-05.txt S. Hole Expires: July 2002 ACI WorldWide/MessagingDirect January 2002 IMAP Extension for Conditional STORE operation Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet- Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society 2001-2002. All Rights Reserved. 0.1. Open issues 1). Should conditional STORE be atomic accross message set (i.e. either all messages in message set weren't changed since and conditional STORE succeeds or operation fails for all messages)? This can be difficult to implement for some servers. 2). The document assumes that each flag has a corresponding ANNOTATE entry. This has to be synchronized with ANNOTATE draft. 0.2. Change History Changes from -04 to -05: 1. Added support for SORT extension. 2. Multiple language/spelling fixes by Randall Gellens. Changes from -03 to -04: 1. Added text saying that MODSEQ fetch data items cause server to include MODSEQ data response in all subsuquent unsolicited FETCH responses. 2. Added "authors address" section. Changes from -02 to -03: 1. Changed MODTIME untagged response to MODTIME response code. 2. Added MODTIME response code to the tagged OK response for SEARCH. Updated examples accordingly. 3. Changed rule for sending untagged FETCH response as a result of STORE when .SILENT prefix is used. If .SILENT prefix is used, server doesn't have to send untagged FETCH response, because MODTIME response code already contains modtime. 4. Renamed MODTIME to MODSEQ to make sure there is no confusion between mod-sequence and ACAP modtime. 5. Minor ABNF changes. 6. Minor language corrections. Changes from -01 to -02: 1. Added MODTIME data item to STATUS command. 2. Added OK untagged response to SELECT/EXAMINE. 3. Clarified that MODIFIED response code contains list of UIDs for conditional UID STORE and message set for STORE. 4. Added per-message modtime. 5. Added PERFLAGMODTIME capability. 6. Fixed several bugs in examples. 7. Added more comments to ABNF. Changes from -00 to -01: 1. Refreshed the list of Open Issues. 2. Changed "attr-name" to "entry-name", because modtime applies to entry, not attribute. 3. Added MODTIME untagged response. 4. Cleaned up ABNF. 5. Added "Acknowledgments" section. 6. Fixed some spelling mistakes. Table of Contents <<To be completed later>> 1. Abstract Often, multiple IMAP clients need to coordinate changes to a common IMAP mailbox. Examples include different clients for the same user, and multiple users accessing shared mailboxes. These clients need a mechanism to synchronize state changes for messages within the mailbox. They must be able to guarantee that only one client can change message state (e.g., message flags or annotations) at any time. An example of such an application is use of an IMAP mailbox as a message queue with multiple dequeueing clients. The Conditional Store facility provides a protected update mechanism for message state information that can detect and resolve conflicts between multiple writers. 2. Conventions Used in This Document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [KEYWORDS]. In examples, lines beginning with "S:" are sent by the IMAP server, and lines beginning with "C:" are sent by the client. Line breaks may appear in example commands solely for editorial clarity; when present in the actual message they are represented by "CRLF". Formal syntax is defined using ABNF [ABNF] as modified by [IMAP4]. In examples, "C:" and "S:" indicate lines sent by the client and server respectively. The term metadata or metadata item is used throughout this document. It refers to any system or user defined flag or an annotation [ANNOTATION]. 3. Introduction and Overview The Conditional STORE extension is present in any IMAP4 implementation which returns "CONDSTORE" as one of the supported capabilities in the CAPABILITY command response. Every read-write item of metadata of an IMAP message has an associated unsigned 64-bit value called a modification sequence (mod-sequence). This is an opaque value updated by the server whenever the metadata item is modified. The value is intended to be used only for comparisons within a server. However, the server MUST guarantee that each STORE command (including simultaneous stores to different attributes from different connections) will get a different mod-sequence value. Also, for any two successfull conditional store operations performed in the same session, the mod-sequence of the second operation MUST be greater than the mod-sequence of the first. Note that this rule disallows the use of the system clock as a mod-sequence, because if system time changes (e.g., NTP client adjusting the time), the next generated value might be less than the previous value. Mod-sequence allows a client that supports the CONDSTORE extension to determine if the value of a particular flag has changed since some known moment. Whenever the state of a flag changes (i.e., the flag is added and before it wasn't set, or the flag is removed and before it was set) the value of the modification sequence for that flag MUST be updated. Adding the flag when it is already present or removing when it is not present SHOULD NOT change the mod-sequence. Changes to any flag MUST also update the per-message mod-sequence. Each flag SHOULD have a separate mod-sequence; for example, changes to the '\Draft' flag SHOULD NOT affect the mod-sequence for the '\Deleted' flag. Servers that support per-flag mod-sequences (i.e., that satisfy this SHOULD) MUST also report "PERFLAGMODSEQ" in the CAPABILITY command response. When a message is appended to a mailbox (via the IMAP APPEND command or using an external mechanism) the server assigns the current server modification sequence to every flag or annotation specified in the APPEND command. When an annotation is removed the mod-sequence SHOULD be preserved. This extension makes the following changes to the IMAP4 protocol: a) extends the syntax of the STORE command to allow STORE modifiers b) adds the MODIFIED response code which should be used with a NO response to the STORE command c) adds a new MODSEQ message data item for use with the FETCH command d) adds a new MODSEQ search criterion e) adds a new MODSEQ response code f) adds a new OK untagged response for the SELECT and EXAMINE commands g) adds the HIGHESTMODSEQ status data item to the STATUS command h) adds a new MODSEQ sort criterion The rest of this document describes the protocol changes more rigorously. 4. IMAP Protocol Changes 4.1. OK untagged response for SELECT and EXAMINE This document adds a new OK untagged response for the SELECT and EXAMINE commands. A server supporting the CONDSTORE extension MUST send the following OK untagged response with any successful SELECT or EXAMINE command: OK [HIGHESTMODSEQ <mod-sequence-value>] Where <mod-sequence-value> is the highest mod-sequence value for any metadata item of any message in the mailbox. A disconnected client can use the value of HIGHESTMODSEQ to check if it has to refetch flags from the server. If the value stored in the client's cache is less than the value returned by the server, then some metadata items on the server have changed since the last synchronization, and the client needs to update its cache. The client MAY use SEARCH MODSEQ as described in section 4.4 to find out exactly which metadata items have changed. Example: C: A142 SELECT INBOX S: * 172 EXISTS S: * 1 RECENT S: * OK [UNSEEN 12] Message 12 is first unseen S: * OK [UIDVALIDITY 3857529045] UIDs valid S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited S: * OK [HIGHESTMODSEQ 20010715194045007] S: A142 OK [READ-WRITE] SELECT completed 4.2. STORE and UID STORE Commands Arguments: message set OPTIONAL store modifiers message data item name value for message data item Responses: untagged responses: FETCH OK untagged responses: MODSEQ (See section 4.6) Result: OK - store completed NO - store error: can't store that data BAD - command unknown or arguments invalid This document extends the syntax of the STORE and UID STORE commands (see section 6.4.6 of [IMAP]) to include an optional STORE modifier. The document defines the following modifier: UNCHANGEDSINCE If the mod-sequence of any metadata item specified in the STORE operation for any message in the message set is greater than the specified unchangedsince value, then the command fails. On failure, a MODIFIED response code is returned which includes the message set (for STORE) or set of UIDs (for UID STORE) of all messages that failed the UNCHANGESINCE test. Example: C: a101 STORE 7,5,9 (UNCHANGEDSINCE 20000320162338) +FLAGS.SILENT (\Deleted) S: a101 NO [MODIFIED 7,9] Conditional STORE failed In spite of the failure of the conditional STORE operation for message 7, the server continues to process the conditional STORE in order to find all messages which fail the test. Use of UNCHANGEDSINCE with a modification sequence of 0 always fails if the metadata item exists. Example: C: a102 STORE 12 (UNCHANGEDSINCE 0) +FLAGS.SILENT ($MDNSent) S: a102 NO [MODIFIED 12] Conditional STORE failed If the operation is successful the server MUST update the mod-sequence attribute for every metadata item that was changed. Untagged FETCH responses MUST be sent (unless .SILENT is specified) and MUST include MODSEQ message data items as described in 4.3. Also, servers MUST send a MODSEQ response code to indicate that the client has received all updates to metadata items which have mod-sequence values less than or equal to the indicated mod-sequence value. Example: C: a103 UID STORE 6,4,8 (UNCHANGEDSINCE 200012121230045) +FLAGS.SILENT (\Deleted) S: * 1 FETCH (UID 4 MODSEQ (200012121231000 "/message/flags/system/\\Deleted" 200012121231000)) S: * 2 FETCH (UID 6 MODSEQ (200012101230852 "/message/flags/system/\\Deleted" 200012101230852)) S: * 4 FETCH (UID 8 MODSEQ (200012121130956 "/message/flags/system/\\Deleted" 200012121130956)) S: a103 OK [MODSEQ 4,6,8 200012121231000] Store completed Example: C: a104 STORE * (UNCHANGEDSINCE 200012121230045) +FLAGS.SILENT (\Deleted $Processed) S: * 50 FETCH (MODSEQ (200012111230045 "/message/flags/system/\\Deleted" 200012111230045 "/message/flags/system/$Processed" 200012111230045)) S: a104 OK [MODSEQ 50 200012111230045] Store completed In the latter example the UNCHANGEDSINCE value is checked against the mod-sequences for both flags. Note: If the message is specified multiple times in the message set, and the server doesn't internally eliminate duplicates from the message set, it MUST NOT fail the conditional STORE operation for the second (or subsequent) occurrence of the message if the operation completed successfully for the first occurrence. For example, if the client specifies: a100 STORE 7,3:9 (UNCHANGEDSINCE 200012121230045) +FLAGS.SILENT (\Deleted) the server must not fail the operation for message 7 as part of processing "3:9" if it succeeded when message 7 was processed the first time. 4.3 MODSEQ message data item in FETCH Command This extension adds a MODSEQ message data item to the FETCH command. This allows clients to retrieve mod-sequence values for various metadata items for a range of messages in the currently selected mailbox. Once the client specifies the MODSEQ message data item in a FETCH request, the server MUST include the MODSEQ fetch response data items in all subsequent unsolicited FETCH responses. Syntax: MODSEQ <entry-names> The MODSEQ message data item, when used by the client in the FETCH command, takes a list of metadata items. For a flag <flagname> the corresponding entry-name has a form "/message/flags/system/<flagname>". An empty list requests the server to return only the per-message mod-sequence. The MODSEQ message data item causes the server to return MODSEQ fetch response data items. Syntax: MODSEQ ( <permsg-modsequence> <entry-name> <mod-sequence-value> ... ) MODSEQ response data items contain per-message mod-sequences and a possibly empty list of requested metadata items and their corresponding mod-sequences. Example: C: a FETCH 1 (MODSEQ ("/message/comment" "/message/flags/system/$MDNSent")) S: * 1 FETCH (MODSEQ (20000624140000 "/message/comment" 20000624140000 "/message/flags/system/$MDNSent" 20000624140000)) S: a OK Fetch complete 4.4 MODSEQ search criterion in SEARCH The MODSEQ criterion for the SEARCH command allows a client to search for the metadata items that were modified since a specified moment. Syntax: MODSEQ <entry-name> <mod-sequence-value> Messages that have modification values for metadata item <entry-name> which are equal to or greater than <mod-sequence-value>. This allows a client, for example, to find out which messages contain metadata items that have changed since the last time it updated its disconnected cache. If client specifies a MODSEQ criterion in a SEARCH command and the server returns a non-empty SEARCH result, the server MUST also return a MODSEQ response code in the tagged OK response. The MODSEQ is for all messages returned in untagged SEARCH results. See also section 4.6. Example: C: a SEARCH MODSEQ "/message/flags/system/\\draft" 20010320162338 ANNOTATION "/message/comment" "value" "IMAP4" S: * SEARCH 2 5 6 7 11 12 18 19 20 23 S: a OK [MODSEQ 2,5:7,11:12,18:20,23 20010917162338] Search complete In the above example, the message numbers of any messages containing the string "IMAP4" in the "value" attribute of the "/message/comment" entry and having a mod-sequence equal to or greater than 20010320162338 for flag "\Draft" are returned in the search results. Example: C: a SEARCH OR NOT MODSEQ "/message/flags/system/$MDNSent" 20010320162338 LARGER 50000 S: * SEARCH S: a OK Search complete, nothing found 4.5 MODSEQ Sort Criterion If a server implementing CONDSTORE also implements the SORT extension as defined by [SORT], it MUST also support sorting on per-message mod-sequence. Syntax: MODSEQ If client specifies a MODSEQ search (as per section 4.4) or sort criterion in the SORT command and the server returns a non-empty SORT result, the server MUST also return a MODSEQ response code in the tagged OK response which covers all messages returned in untagged SORT responses. See also section 4.6. Example: C: A282 SORT (SUBJECT MODSEQ) UTF-8 SINCE 1-Feb-2001 S: * SORT 2 81 83 84 82 882 S: A282 OK [MODSEQ 2,81:84,882 117] SORT completed Example: C: A283 SORT (SUBJECT REVERSE DATE) UTF-8 MODSEQ "/message/flags/system/\\Flagged" 21 S: * SORT 6 3 4 5 2 S: A283 OK [MODSEQ 2:6 125] SORT completed Example: C: A284 SORT (MODSEQ) KOI8-R OR NOT MODSEQ "/message/flags/system/$MDNSent" 20010320162338 SUBJECT "privet" S: * SORT S: A284 OK Sort complete, nothing found 4.6 MODSEQ Response code for successful FETCH, STORE and SEARCH and SORT Data: message set mod-sequence value The MODSEQ response code is sent in the following three cases: 1) A successful STORE UNCHANGEDSINCE command results in a MODSEQ response code being sent in a tagged OK response. This informs the client of the latest mod-sequence of all metadata items specified in the STORE command. The MODSEQ response code contains either a message set or a UID set. If sent in response to a STORE command, it contains the message set to which the mod-sequence applies. If sent in response to a UID STORE command, it contains the UID set to which the mod-sequence applies. Example: C: a103 STORE 9 (UNCHANGEDSINCE 200012121230045) +FLAGS.SILENT ($Forwarded) S: * 9 FETCH (MODSEQ (200012121231000 "/message/flags/system/$Forwarded" 200012121130046)) S: a103 OK [MODSEQ 9 200012121231000] Store completed 2) The MODSEQ response code in an untagged OK response MUST be sent by the server following a group of one or more unsolicited FETCH responses. This indicates that the client has received all updates to metadata items which have mod-sequence values less than or equal to the indicated mod-sequence value. A MODSEQ response code terminating a group of unsolicited FETCH responses always contains the message set to which the mod-sequence applies. Example: C: a103 UID FETCH 20138:* (FLAGS INTERNALDATE) S: * 101 FETCH (UID 20140 FLAGS (\Seen \Answered $Forwarded) INTERNALDATE "28-Oct-2001 15:44:25 +0300") S: * 12 FETCH (FLAGS (\Flagged) MODSEQ (200109121231000 "/message/flags/system/\\Flagged" 200109121231000)) S: * 16 FETCH (FLAGS (\Answered \Flagged) MODSEQ (200109121231000 "/message/flags/system/\\Flagged" 200109121231000 "/message/flags/system/\\Answered" 200109121231000)) S: * OK [MODSEQ 12,16 200109121231000] S: a103 OK Store completed 3) If a client specifies a MODSEQ criterion in a SEARCH command and the server returns a non-empty SEARCH result, the server MUST also return a MODSEQ response code in the tagged OK response. The MODSEQ response code MUST be for all messages which were returned in the untagged SEARCH response. The MODSEQ response code contains the message set to which the mod-sequence applies if it is in response to a SEARCH command; or the UID set if it is caused by a UID SEARCH command. 4) If client specifies a MODSEQ search or sort criterion in a SORT command and the server returns a non-empty SORT result, the server MUST also return a MODSEQ response code in the tagged OK response for all messages returned in the untagged SORT response. The MODSEQ response code contains the message set to which the mod-sequence applies if it is sent in response to a SORT command, or the UID set if it is caused by UID SORT. 4.7 HIGHESTMODSEQ status data items This document defines a new status data item: HIGHESTMODSEQ The highest mod-sequence value for any metadata item of any message in the mailbox. This is the same value that is returned by the server in the HIGHESTMODSEQ response code in OK untagged response (see section 4.1). Example: C: A042 STATUS blurdybloop (UIDNEXT MESSAGES HIGHESTMODSEQ) S: * STATUS blurdybloop (MESSAGES 231 UIDNEXT 44292 HIGHESTMODSEQ 200201011231777) S: A042 OK STATUS completed 5. Formal Syntax The following syntax specification uses the Augmented Backus-Naur Form (ABNF) notation as specified in [ABNF]. Non-terminals referenced but not defined below are as defined by [IMAP4]. Except as noted otherwise, all alphabetic characters are case- insensitive. The use of upper or lower case characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion. capability =/ "CONDSTORE" / "PERFLAGMODSEQ" mailbox-data =/ "STATUS" SP mailbox SP "(" [status-att SP status-value *(SP status-att SP status-value)] ")" status-value = number / mod-sequence-value store = "STORE" SP set store-modifiers SP store-att-flags store-modifiers = [ SP "(" 1*store-modifier ")" ] store-modifier = "UNCHANGEDSINCE" SP mod-sequence-value fetch-att =/ fetch-mod-sequence ;; modifies original IMAP4 fetch-att fetch-mod-sequence = "MODSEQ" SP entry-names fetch-mod-resp = "MODSEQ" SP "(" permsg-modsequence *(entry-name SP mod-sequence-value) ")" search-key =/ search-modsequence ;; modifies original IMAP4 search-key search-modsequence = "MODSEQ" SP entry-name SP mod-sequence-value resp-text-code =/ "HIGHESTMODSEQ" SP mod-sequence-value / "MODIFIED" SP set / "MODSEQ" SP set SP mod-sequence-value ;; set of message numbers for STORE/FETCH or ;; set of UIDs for UID STORE/UID FECTH entry-names = "(" *entry-name ")" ;; empty list means that only per-message ;; MODSEQ should be returned entry-name = '"' "/message/flags/system/" attr-flag '"' ;; each system or user defined flag <flag> ;; is mapped to "/message/flags/system/<flag>". ;; system IMAP flags must have two leading "\", ;; because "\" is an escape character. permsg-modsequence = mod-sequence-value ;; per message mod-sequence, if server ;; supports per flag mod-sequences, ;; this is the highest mod-sequence between ;; all metadata items mod-sequence-value = 1*DIGIT ;; Unsigned 64-bit integer (mod-sequence) ;; (0 <= n < 18,446,744,073,709,551,615) ;;Borrowed from IMAP4rev1 and modified accordingly: attr-flag = "\\Answered" / "\\Flagged" / "\\Deleted" / "\\Seen" / "\\Draft" / attr-flag-keyword / attr-flag-extension ;; Does not include "\Recent" attr-flag-extension = "\\" atom ;; Future expansion. Client implementations ;; MUST accept flag-extension flags. Server ;; implementations MUST NOT generate ;; flag-extension flags except as defined by ;; future standard or standards-track ;; revisions of this specification. attr-flag-keyword = atom ;;Extension to SORT sort-key =/ "MODSEQ" 6. Security Considerations There are no known security issues with this extension. 7. References [ABNF] Crocker, Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, Internet Mail Consortium, Demon Internet Ltd, November 1997. [IMAP4] Crispin, M., "Internet Message Access Protocol - Version 4rev1", RFC 2060, University of Washington, December 1996. [KEYWORDS] Bradner, "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, Harvard University, March 1997. [ACAP] Newman, Myers, "ACAP -- Application Configuration Access Protocol", RFC 2244, Innosoft, Netscape, November 1997. <ftp://ftp.isi.edu/in-notes/rfc2244.txt> [ANNOTATION] Gellens, R., Daboo, C., "IMAP ANNOTATE Extension", work in progress. <http://www.ietf.org/internet-drafts/draft-ietf-imapext-annotate-xx.txt> [SORT] Crispin, M., "Internet Message Access Protocol -- SORT Extension", work in progress. <http://www.ietf.org/internet-drafts/draft-crispin-imapext-sort-xx.txt> 8. Acknowledgments Some text was borrowed from "IMAP ANNOTATE Extension" by Randall Gellens and Cyrus Daboo, and "ACAP -- Application Configuration Access Protocol" by Chris Newman and John Myers. Many thanks to Randall Gellens for his comments on how CONDSTORE should interact with ANNOTATE extension and for thorough review of the document. Authors also acknowledge the feedback provided by Cyrus Daboo and Larry Greenfield. 9. Author's Addresses Alexey Melnikov mailto: Alexey.Melnikov@messagingdirect.com Steve Hole mailto: Steve.Hole@messagingdirect.com ACI WorldWide/MessagingDirect 900 10117 - Jasper Ave. Edmonton, Alberta, T5J 1W8, CANADA 10. Full Copyright Statement Copyright (C) The Internet Society 2001-2002. All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.