IMAP Paged SEARCH & FETCH Extension
draft-ietf-extra-imap-partial-03

Network Working Group                                        A. Melnikov
Internet-Draft                                                     Isode
Updates: 5267, 4731 (if approved)                         A. P. Achuthan
Intended status: Standards Track                          V. Nagulakonda
Expires: 8 June 2023                                              Yahoo!
                                                                L. Alves
                                                         5 December 2022

                  IMAP Paged SEARCH & FETCH Extension
                    draft-ietf-extra-imap-partial-03

Abstract

   The PARTIAL extension of the Internet Message Access Protocol (RFC
   3501/RFC 9051) allows clients to limit the number of search results
   returned, as well as to perform incremental (paged) searches.  This
   also helps servers to optimize resource usage when performing
   searches.

   This document extends PARTIAL SEARCH return option originally
   specified in RFC 5267.  It also clarifies some interactions between
   RFC 5267 and RFC 4731/RFC 9051.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 8 June 2023.

Copyright Notice

   Copyright (c) 2022 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

Melnikov, et al.           Expires 8 June 2023                  [Page 1]
Internet-Draft                IMAP PARTIAL                 December 2022

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

   This document may contain material from IETF Documents or IETF
   Contributions published or made publicly available before November
   10, 2008.  The person(s) controlling the copyright in some of this
   material may not have granted the IETF Trust the right to allow
   modifications of such material outside the IETF Standards Process.
   Without obtaining an adequate license from the person(s) controlling
   the copyright in such materials, this document may not be modified
   outside the IETF Standards Process, and derivative works of it may
   not be created outside the IETF Standards Process, except to format
   it for publication as an RFC or to translate it into languages other
   than English.

Table of Contents

   1.  Introduction and Overview . . . . . . . . . . . . . . . . . .   2
   2.  Document Conventions  . . . . . . . . . . . . . . . . . . . .   3
   3.  The PARTIAL extension . . . . . . . . . . . . . . . . . . . .   3
     3.1.  Incremental SEARCH and partial results  . . . . . . . . .   3
     3.2.  Interaction between PARTIAL, MIN, MAX and SAVE SEARCH
           return options  . . . . . . . . . . . . . . . . . . . . .   5
     3.3.  Extension to UID FETCH  . . . . . . . . . . . . . . . . .   6
     3.4.  Use of PARTIAL and CONDSTORE IMAP extensions together . .   7
   4.  Formal syntax . . . . . . . . . . . . . . . . . . . . . . . .   7
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   9
     6.1.  Changes/additions to the IMAP4 capabilities registry  . .   9
   7.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .   9
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  10
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  10

1.  Introduction and Overview

   This document defines an extension to the Internet Message Access
   Protocol [RFC3501] for performing incremental searches and fetches.
   This extension is compatible with both IMAP4rev1 [RFC3501] and
   IMAP4rev2 [RFC9051].

Melnikov, et al.           Expires 8 June 2023                  [Page 2]
Internet-Draft                IMAP PARTIAL                 December 2022

   The PARTIAL extension of the Internet Message Access Protocol (RFC
   3501/RFC 9051) allows clients to limit the number of search results
   returned, as well as to perform incremental (paged) searches.  This
   also helps servers to optimize resource usage when performing
   searches.

   This document extends PARTIAL SEARCH return option originally
   specified in RFC 5267.  It also clarifies some interactions between
   RFC 5267 and RFC 4731/RFC 9051.

2.  Document Conventions

   In protocol examples, this document uses a prefix of "C: " to denote
   lines sent by the client to the server, and "S: " for lines sent by
   the server to the client.  Lines prefixed with "// " are comments
   explaining the previous protocol line.  These prefixes and comments
   are not part of the protocol.  Lines without any of these prefixes
   are continuations of the previous line, and no line break is present
   in the protocol unless specifically mentioned.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   Other capitalised words are IMAP keywords [RFC3501][RFC9051] or
   keywords from this document.

3.  The PARTIAL extension

   An IMAP server advertises support for the PARTIAL extension by
   including the "PARTIAL" capability in the CAPABILITY response/
   response code.

3.1.  Incremental SEARCH and partial results

   The PARTIAL SEARCH return option causes the server to provide in an
   ESEARCH response a subset of the results denoted by the sequence
   range given as the mandatory argument.  The first result (message
   with the lowest matching UID) is 1; thus, the first 500 results would
   be obtained by a return option of "PARTIAL 1:500", and the second 500
   by "PARTIAL 501:1000".  This intentionally mirrors message sequence
   numbers.

Melnikov, et al.           Expires 8 June 2023                  [Page 3]
Internet-Draft                IMAP PARTIAL                 December 2022

   It is also possible to direct the server to start SEARCH from the
   latest matching (with the highest UID) message.  This can be done by
   prepeding "-" to the index.  For example -1 is the last message, -2
   is next to the last and so on.  Using this syntax helps server
   implementations to optimize their SEARCHes.

   A single command MUST NOT contain more than one PARTIAL or ALL search
   return option -- that is, either one PARTIAL, one ALL, or neither
   PARTIAL nor ALL is allowed.

   For SEARCH results, the entire result list MUST be ordered in mailbox
   order, that is, in UID or message sequence number order.

   Where a PARTIAL SEARCH return option references results that do not
   exist, by using a range which starts or ends higher (or lower) than
   the current number of results, then the server returns the results
   that are in the set.  This yields a PARTIAL return data item that
   has, as payload, the original range and a potentially missing set of
   results that may be shorter than the extent of the range.  If the
   whole range references results that do not exist, a special value
   "NIL" is returned by the server instead of the sequence set.

   Clients need not request PARTIAL results in any particular order.
   Because mailboxes may change, clients might wish to use PARTIAL in
   combination with UPDATE (see [RFC5267]) if the server also advertises
   CONTEXT=SEARCH capability, especially if the intent is to walk a
   large set of results; however, these return options do not interact
   -- the UPDATE will provide notifications for all matching results.

     // Let's assume that the A01 SEARCH without PARTIAL would return
     // 23764 results.
     C: A01 UID SEARCH RETURN (PARTIAL -1:-100) UNDELETED
       UNKEYWORD $Junk
     S: * ESEARCH (TAG "A01") UID PARTIAL (-1:-100 ...)
     // 100 most recent results in set syntax elided.
     S: A01 OK Completed.

Melnikov, et al.           Expires 8 June 2023                  [Page 4]
Internet-Draft                IMAP PARTIAL                 December 2022

     // Let's assume that the A02 SEARCH without PARTIAL would return
     // 23764 results.
     C: A02 UID SEARCH RETURN (PARTIAL 23500:24000) UNDELETED
       UNKEYWORD $Junk
     C: A03 UID SEARCH RETURN (PARTIAL 1:500) UNDELETED
       UNKEYWORD $Junk
     C: A04 UID SEARCH RETURN (PARTIAL 24000:24500) UNDELETED
       UNKEYWORD $Junk
     S: * ESEARCH (TAG "A02") UID PARTIAL (23500:24000 ...)
     // 264 results in set syntax elided,
     // this spans the end of the results.
     S: A02 OK Completed.
     S: * ESEARCH (TAG "A03") UID PARTIAL (1:500 ...)
     // 500 results in set syntax elided.
     S: A03 OK Completed.
     S: * ESEARCH (TAG "A04") UID PARTIAL (24000:24500 NIL)
     // No results are present, this is beyond the end of the results.
     S: A04 OK Completed.

3.2.  Interaction between PARTIAL, MIN, MAX and SAVE SEARCH return
      options

   This section only applies if the server advertises PARTIAL IMAP
   capability or CONTEXT=SEARCH [RFC5267], together with ESEARCH
   [RFC4731] and/or IMAP4rev2 [RFC9051].

   The SAVE result option doesn't change whether the server would return
   items corresponding to PARTIAL SEARCH result options.

   As specified in Section 3.1, it is an error to specify both PARTIAL
   and ALL result options in the same SEARCH command.

   When the SAVE result option is combined with the PARTIAL result
   option, and none of MIN/MAX/COUNT result options is present, the
   corresponding PARTIAL is returned, and the "$" marker would contain
   all messages returned by the PARTIAL result option.

   When the SAVE and PARTIAL result options are combined with the MIN or
   the MAX result option, and the COUNT result option is absent, the
   corresponding PARTIAL result and MIN/MAX is returned (if the search
   result is not empty), and the "$" marker would contain all messages
   returned by the PARTIAL result option together with the corresponding
   MIN/MAX message.

Melnikov, et al.           Expires 8 June 2023                  [Page 5]
Internet-Draft                IMAP PARTIAL                 December 2022

   If the SAVE and PARTIAL result options are combined with both MIN and
   MAX result options, and the COUNT result options is absent, the
   PARTIAL, MIN and MAX are returned (if the search result is not
   empty), and the "$" marker would contain all messages returned by the
   PARTIAL result option together with the MIN and the MAX messages.

   If the SAVE and PARTIAL result options are combined with the COUNT
   result option, the PARTIAL and COUNT are returned, and the "$" marker
   would always contain all messages found by the SEARCH or UID SEARCH
   command.

   The following table summarizes the additional requirement on ESEARCH
   server implementations described in this section.

          +==============================+=====================+
          | Combination of Result option |   "$" marker value  |
          +==============================+=====================+
          |         SAVE PARTIAL         |       PARTIAL       |
          +------------------------------+---------------------+
          |       SAVE PARTIAL MIN       |    PARTIAL & MIN    |
          +------------------------------+---------------------+
          |       SAVE PARTIAL MAX       |    PARTIAL & MAX    |
          +------------------------------+---------------------+
          |     SAVE PARTIAL MIN MAX     | PARTIAL & MIN & MAX |
          +------------------------------+---------------------+
          |    SAVE PARTIAL COUNT [m]    |  all found messages |
          +------------------------------+---------------------+

                                 Table 1

   Note for the table: '[m]' means optional "MIN" and/or "MAX"

3.3.  Extension to UID FETCH

   The PARTIAL extension also extends the UID FETCH command with a
   PARTIAL FETCH modifier.  The PARTIAL FETCH modifier has the same
   syntax as the PARTIAL SEARCH result option.  Presence of the PARTIAL
   FETCH modifier instructs the server to only return FETCH results for
   messages in the specified range.  It is useful when the sequence-set
   (first) parameter to the UID FETCH command includes unknown number of
   messages.

     // Returning information for the last 3 messages in the UID range
     C: 10 UID FETCH 25900:26600 (UID FLAGS) (PARTIAL -1:-3)
     S: * 12888 FETCH (FLAGS (\Seen) UID 25996)
     S: * 12889 FETCH (FLAGS (\Flagged \Answered) UID 25997)
     S: * 12890 FETCH (FLAGS () UID 26600)
     S: 10 OK FETCH completed

Melnikov, et al.           Expires 8 June 2023                  [Page 6]
Internet-Draft                IMAP PARTIAL                 December 2022

     // Returning information for the first 5 messages in the UID range
     C: 11 UID FETCH 25900:26600 (UID FLAGS) (PARTIAL 1:5)
     S: * 12591 FETCH (FLAGS (\Seen) UID 25900)
     S: * 12592 FETCH (FLAGS (\Flagged) UID 25902)
     S: * 12593 FETCH (FLAGS (\Answered) UID 26310)
     S: * 12594 FETCH (FLAGS () UID 26311)
     S: * 12595 FETCH (FLAGS (\Answered) UID 26498)
     S: 11 OK FETCH completed

3.4.  Use of PARTIAL and CONDSTORE IMAP extensions together

   This section is informative.

   The PARTIAL FETCH modifier can be combined with the CHANGEDSINCE
   FETCH modifier.

  // Returning information for the last 30 messages in the UID range
  // that have any flag/keyword modified since modseq 98305
  C: 101 UID FETCH 25900:26600 (UID FLAGS) (PARTIAL -1:-30 CHANGEDSINCE 98305)
  S: * 12888 FETCH (FLAGS (\Flagged \Answered) MODSEQ (98306) UID 25997)
  S: * 12890 FETCH (FLAGS () MODSEQ (98312) UID 26600)
  S: 101 OK FETCH completed

   The above example causes the server to first select the last 30
   messages and then only return flag changes for subset of these
   messages which have MODSEQ higher than 98305.

   Note that the order of PARTIAL and CHANGEDSINCE FETCH modifiers in
   the UID FETCH command is not important, i.e. the above example can
   also use "UID FETCH 25900:26600 (UID FLAGS) (CHANGEDSINCE 98305
   PARTIAL -1:-30)" command and it would result in the same responses.

4.  Formal syntax

   The following syntax specification uses the Augmented Backus-Naur
   Form (ABNF) notation as specified in [ABNF].

   Non-terminals referenced but not defined below are as defined by
   IMAP4 [RFC3501].

   Except as noted otherwise, all alphabetic characters are case-
   insensitive.  The use of upper or lower case characters to define
   token strings is for editorial clarity only.  Implementations MUST
   accept these strings in a case-insensitive fashion.

Melnikov, et al.           Expires 8 June 2023                  [Page 7]
Internet-Draft                IMAP PARTIAL                 December 2022

   SP                  = <Defined in RFC 5234>
   MINUS               = "-"

   capability          =/ "PARTIAL"
                          ;; <capability> from [RFC3501]

   modifier-partial    = "PARTIAL" SP partial-range

   partial-range-first = nz-number ":" nz-number
       ;; Request to search from oldest (lowest UIDs) to
       ;; more recent messages.
       ;; A range 500:400 is the same as 400:500.
       ;; This is similar to <seq-range> from [RFC3501],
       ;; but cannot contain "*".

   partial-range-last  = MINUS nz-number ":" MINUS nz-number
       ;; Request to search from newest (highest UIDs) to
       ;; oldest messages.
       ;; A range -500:-400 is the same as -400:-500.

   partial-range       = partial-range-first / partial-range-last

   search-return-opt   =/ modifier-partial
       ;; All conform to <search-return-opt>, from [IMAP-ABNF]/[RFC9051]

   search-return-data  =/ ret-data-partial

   ret-data-partial    = "PARTIAL"
                         SP "(" partial-range SP partial-results ")"
       ;; <partial-range> is the requested range.

   partial-results     = sequence-set / "NIL"
       ;; <sequence-set> from [RFC3501].
       ;; NIL indicates no results correspond to the requested range.

   tagged-ext-simple   =/ partial-range-last

   fetch-modifier      =/ modifier-partial

5.  Security Considerations

   This document defines an additional IMAP4 capability.  As such, it
   does not change the underlying security considerations of [RFC3501]
   and IMAP4rev2 [RFC9051].  The authors and reviewers believe that no
   new security issues are introduced with these additional IMAP4
   capabilities.

Melnikov, et al.           Expires 8 June 2023                  [Page 8]
Internet-Draft                IMAP PARTIAL                 December 2022

   This document defines an optimization that can both reduce the amount
   of work performed by the server, as well at the amount of data
   returned to the client.  Use of this extension is likely to cause the
   server and the client to use less memory than when the extension is
   not used.  However, as this is going to be new code in both the
   client and the server, rigorous testing of such code is required in
   order to avoid introducing of new implementation bugs.

6.  IANA Considerations

6.1.  Changes/additions to the IMAP4 capabilities registry

   IMAP4 capabilities are registered by publishing a standards track or
   IESG approved Informational or Experimental RFC.  The registry is
   currently located at:

      https://www.iana.org/assignments/imap4-capabilities

   IANA is requested to add definition of the PARTIAL extension with RFC
   XXXX as the reference.

7.  Acknowledgments

   This document was motivated by Yahoo! team and their questions about
   best client practices for dealing with large mailboxes.

   Editor of this document would like to thank the following people who
   provided useful comments or participated in discussions of this
   document: Timo Sirainen and Barry Leiba.

   This document uses lots of text from RFC 5267.  Thus work of the RFC
   5267 authors Dave Cridland and Curtis King is appreciated.

8.  References

8.1.  Normative References

   [ABNF]     Crocker, D., Ed. and P. Overell, Ed., "Augmented BNF for
              Syntax Specifications: ABNF", RFC 5234, January 2008,
              <https://www.rfc-editor.org/info/rfc5234>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

Melnikov, et al.           Expires 8 June 2023                  [Page 9]
Internet-Draft                IMAP PARTIAL                 December 2022

   [RFC3501]  Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION
              4rev1", RFC 3501, DOI 10.17487/RFC3501, March 2003,
              <https://www.rfc-editor.org/info/rfc3501>.

   [RFC4731]  Melnikov, A. and D. Cridland, "IMAP4 Extension to SEARCH
              Command for Controlling What Kind of Information Is
              Returned", RFC 4731, DOI 10.17487/RFC4731, November 2006,
              <https://www.rfc-editor.org/info/rfc4731>.

   [RFC5267]  Cridland, D. and C. King, "Contexts for IMAP4", RFC 5267,
              DOI 10.17487/RFC5267, July 2008,
              <https://www.rfc-editor.org/info/rfc5267>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC9051]  Melnikov, A., Ed. and B. Leiba, Ed., "Internet Message
              Access Protocol (IMAP) - Version 4rev2", RFC 9051,
              DOI 10.17487/RFC9051, August 2021,
              <https://www.rfc-editor.org/info/rfc9051>.

8.2.  Informative References

   [RFC7162]  Melnikov, A. and D. Cridland, "IMAP Extensions: Quick Flag
              Changes Resynchronization (CONDSTORE) and Quick Mailbox
              Resynchronization (QRESYNC)", RFC 7162,
              DOI 10.17487/RFC7162, May 2014,
              <https://www.rfc-editor.org/info/rfc7162>.

Authors' Addresses

   Alexey Melnikov
   Isode Limited
   Email: alexey.melnikov@isode.com
   URI:   https://www.isode.com

   Arun Prakash Achuthan
   Yahoo!
   Email: arunprakash@myyahoo.com

   Vikram Nagulakonda
   Yahoo!
   Email: nvikram_imap@yahoo.com

Melnikov, et al.           Expires 8 June 2023                 [Page 10]
Internet-Draft                IMAP PARTIAL                 December 2022

   Luis Alves
   Email: luis.alves@lafaspot.com

Melnikov, et al.           Expires 8 June 2023                 [Page 11]