Internet Draft                                             A. Melnikov
Document: draft-melnikov-imap-expunged                       Isode Ltd
Expires: July 2005                                        January 2005



  IMAP4 extension to CONDSTORE for reporting messages expunged since
                         last synchronization
                   draft-melnikov-imap-expunged-00

Status of this Memo

   By submitting this Internet-Draft, I certify that any applicable
   patent or other IPR claims of which I am aware have been disclosed,
   or will be disclosed, and any of which I become aware will be
   disclosed, in accordance with RFC 3668.

   Internet Drafts are working documents of the Internet Engineering
   Task Force (IETF), its Areas, and its Working Groups.  Note that
   other groups may also distribute working documents as Internet
   Drafts. Internet Drafts are draft documents valid for a maximum of
   six months.  Internet Drafts may be updated, replaced, or obsoleted
   by other documents at any time.  It is not appropriate to use
   Internet Drafts as reference material or to cite them other than as
   ''work in progress''.

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   A revised version of this draft document will be submitted to the
   RFC editor as a Standard Track RFC for the Internet Community.
   Discussion and suggestions for improvement are requested, and
   should be sent to ietf-imapext@imc.org and/or lemonade@ietf.org.
   Distribution of this memo is unlimited.


Abstract

   This document defines an extension to IMAP Conditional Store
   extension, which gives a disconnected client ability to receive
   notifications about messages expunged since last synchronization.


Table of Contents

  1. Conventions Used in this Document                              2
  2. Introduction                                                   2
  3. IMAP Protocol Changes                                          3
     3.1 EXPUNGE Command                                            3
     3.2 CLOSE Command                                              4
     3.3 "REPORTEXPUNGES" FETCH modifier                            5
     3.4 EXPUNGED Response                                          6
     3.5 Other issues                                               7
  4. Updated synchronization sequence                               7
  5. Formal Syntax                                                  9
  6. Security Considerations                                       10
  7. IANA Considerations                                           10
  8. References                                                    10
     8.1 Normative References                                      10
     8.2 Informative References                                    10
  9. Acknowledgments                                               10
  10. Author's Addresses                                           11
  11. Full Copyright Statement                                     11
  12. Intellectual Property                                        11
  13. Appendix A. Editorial.                                       12
     13.1 Change Log                                               12
     13.2 Open Issues for Discussion                               12
     13.3 To Do                                                    12


1.   Conventions Used in this Document

   In examples, "C:" and "S:" indicate lines sent by the client and
   server respectively.

   The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY"
   in this document are to be interpreted as defined in "Key words for
   use in RFCs to Indicate Requirement Levels" [KEYWORDS].

   <<Editorial comments and questions are enclosed like this>>

2.   Introduction

   The [CONDSTORE] extension gives a disconnected client ability to
   quickly synchronize flag (and annotation) changes for previously
   seen messages. In order for the client to discover which messages
   have been expunged since the last syncronization, the client still
   has to issue a UID FETCH or a UID SEARCH command. This document
   defines an extension to [CONDSTORE], that allows a compliant client
   to quickly learn about expunged messages.

   The Expunged Messages Notification extension is present in any
   IMAP4 implementation which advertises "X-DRAFT-I00-EXPUNGED" as one
   of the supported capabilities in the CAPABILITY command response.
   Any server returning the capability MUST also support the
   [CONDSTORE] extension.

   This document puts additional requirements on a server implementing
   the [CONDSTORE] extension. Each mailbox that supports persistent
   storage of mod-sequences, i.e. for which the server has sent
   HIGHESTMODSEQ untagged OK response code on a successful
   SELECT/EXAMINE, MUST increment per-mailbox mod-sequence when one or
   more message is expunged due to EXPUNGE,  UID EXPUNGE or CLOSE; the
   server MUST associate the incremented mod-sequence with the UID of
   the expunged message(s).
   <<Of course if multiple messages are expunged at once, the same mod-
   sequence can be used for all of them>>
   This change doesnÆt affect a client that only supports the
   CONDSTORE extension, however per-mailbox mod-sequence change due to
   expunges may force the client to send FETCH CHANGEDSINCE that will
   return no data, thus forcing additional round-trip. <<Is this Ok?>>


3.   IMAP Protocol Changes

3.1  EXPUNGE Command

   Arguments:  none

   Responses:  untagged responses: EXPUNGE

   Result:     OK - expunge completed
               NO - expunge failure: can't expunge (e.g., permission
                    denied)
               BAD - command unknown or arguments invalid

   This section updates definition of the EXPUNGE command described in
   section 6.4.3 of [IMAP4].

   The EXPUNGE command permanently removes all messages that have the
   \Deleted flag set from the currently selected mailbox.  Before
   returning an OK to the client, an untagged EXPUNGE response is sent
   for each message that is removed.

   If the server is capable of storing modification sequences for the
   selected mailbox, it MUST increment per-mailbox mod-sequence if at
   least one message was permanently removed due to the execution of
   the EXPUNGE command. For each permanently removed message the
   server MUST remember the incremented mod-sequence and corresponding
   UID. If at least one message got expunged, the server MUST send the
   updated per-mailbox modification sequence using the HIGHESTMODSEQ
   untagged response code.

   <<UID EXPUNGE? Should we just have a single section for EXPUNGE/UID
   EXPUNGE/CLOSE?>>

   Example:    C: A202 EXPUNGE
               S: * 3 EXPUNGE
               S: * 3 EXPUNGE
               S: * 5 EXPUNGE
               S: * 8 EXPUNGE
               S: * OK [HIGHESTMODSEQ 20010715194045319]
               S: A202 OK EXPUNGE completed

        Note: In this example, messages 3, 4, 7, and 11 had the
        \Deleted flag set.  See the description of the EXPUNGE
        response in [IMAP4] for further explanation.


3.2  CLOSE Command

   Arguments:  none

   Responses:  no specific responses for this command

   Result:     OK - close completed, now in authenticated state
               BAD - command unknown or arguments invalid

   This section updates definition of the CLOSE command described in
   section 6.4.2 of [IMAP4].

   The CLOSE command permanently removes all messages that have the
   \Deleted flag set from the currently selected mailbox, and returns
   to the authenticated state from the selected state.  No untagged
   EXPUNGE responses are sent.

   If the server is capable of storing modification sequences for the
   selected mailbox, it MUST increment per-mailbox mod-sequence if at
   least one message was permanently removed due to the execution of
   the CLOSE command. For each permanently removed message the server
   MUST remember the incremented mod-sequence and corresponding UID.
   If at least one message got expunged, the server MUST send the
   updated per-mailbox modification sequence using the HIGHESTMODSEQ
   untagged response code.

   Example:    C: A202 CLOSE
               S: * OK [HIGHESTMODSEQ 20010715194045319]
               S: A202 OK EXPUNGE completed


3.3  "REPORTEXPUNGES" FETCH modifier

   [CONDSTORE] has extended the syntax of the FETCH and UID FETCH
   commands to include an optional FETCH modifier.  This document
   defines a new UID FETCH modifier (note, it is NOT allowed with a
   FETCH command <<Should this be highlighted in ABNF?>>):

       REPORTEXPUNGES

   The REPORTEXPUNGES FETCH modifier MUST only be specified together
   with the CHANGEDSINCE FETCH modifier.
   <<alternative #1: extend syntax for the CHANGEDSINCE FETCH
   modifier.

   alternative #2: allow REPORTEXPUNGES to have own mod-sequence
   parameter.

   alternative #3: allow REPORTEXPUNGES by itself, this will mean
   report all expunged UIDs from the UID set specified in the UID
   FETCH.>>

   The REPORTEXPUNGES FETCH modifier instructs the server to also
   report all messages from the FETCH message set that were expunged
   and their associated expunge mod-sequence is bigger than the mod-
   sequence specified in the CHANGEDSINCE FETCH modifier.
   The expunged messages are reported using the EXPUNGED response as
   described in section 3.4.

   <<What if the server canÆt remember some expunged UIDs, for example
   the provided mod-sequence is too old? Can the server just return
   all UIDs not in the UID set? This can generate additional traffic,
   but should be Ok otherwise.>>

   Example: Without the REPORTEXPUNGES FETCH modifier a CONDSTORE-
   aware client must issue two commands to learn about flag changes,
   as well as messages expunged since the last synchronization:

       C: s100 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 12345)
       S: * 1 FETCH (UID 4 MODSEQ (65402) FLAGS (\Seen))
       S: * 2 FETCH (UID 6 MODSEQ (75403) FLAGS (\Deleted))
       S: * 4 FETCH (UID 8 MODSEQ (29738) FLAGS ($NoJunk $AutoJunk
   $MDNSent))
       S: s100 OK FETCH completed
       C: s101 UID SEARCH 1:*
       S: * SEARCH 4 6 7 8 10 12
       S: s101 OK search completed

   The second SEARCH response tells the client that the messages with
   UIDs 7, 10 and 12 are still present, but their flags havenÆt
   changed since the specified modification sequence.

   Using the REPORTEXPUNGES FETCH modifier it is sufficient to issue
   only a single command:

       C: s100 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 12345
   REPORTEXPUNGES)
       S: * 1 FETCH (UID 4 MODSEQ (65402) FLAGS (\Seen))
       S: * 2 FETCH (UID 6 MODSEQ (75403) FLAGS (\Deleted))
       S: * 4 FETCH (UID 8 MODSEQ (29738) FLAGS ($NoJunk $AutoJunk
   $MDNSent))
       S: * EXPUNGED 5,9
       S: s100 OK FETCH completed

   The EXPUNGED response tells the client that messages with UIDs 5
   and 9 have been expunged since the specified modification sequence.
   Messages with UIDs 1,2,3,4 and 11 have been expunged earlier.


3.4  EXPUNGED Response

   Contents:   list of UIDs

   The EXPUNGED response reports that the specified UIDs have been
   permanently removed from the mailbox. This response is similar to
   the EXPUNGE response [IMAP4], however it may return information
   about multiple messages and it returns UIDs, instead of message
   numbers.

   <<The EXPUNGED response also decrements the number of messages in
   the mailbox; it is not necessary to send an EXISTS response with
   the new value.>> <<It also affects message numbers?>>

   An EXPUNGED response MUST NOT be sent when no command is in
   progress, nor while responding to a FETCH, STORE, or SEARCH
   command.  This rule is necessary to prevent a loss of
   synchronization of message sequence numbers between client and
   server.  A command is not "in progress" until the complete command
   has been received; in particular, a command is not "in progress"
   during the negotiation of command continuation.

        Note: UID FETCH, UID STORE, and UID SEARCH are different
        commands from FETCH, STORE, and SEARCH.  An EXPUNGED
        response MAY be sent during a UID command.

   <<An alternative is to say that the EXPUNGED response is only
   allowed in response to FETCH/...>>

   The update from the EXPUNGED response MUST be recorded by the
   client.

   Example:    S: * EXPUNGED 5,7,9,15:21


3.5  Other issues

   <<Add CHANGEDSINCE SELECT/EXAMINE parameter?>>


4.   Updated synchronization sequence

   This section updates the description of optimized synchronization
   in section 6.1 of the [IMAP-DISC].

   An advanced disconnected mail client should use the EXPUNGED and
   [CONDSTORE] extensions when they are supported by the server. The
   client must cache the value from HIGHESTMODSEQ OK response code
   received on mailbox opening and update it whenever the server sends
   MODSEQ FETCH data items.

   If the client receives NOMODSEQ OK untagged response instead of
   HIGHESTMODSEQ, it MUST remove the last known HIGHESTMODSEQ value
   from its cache and follow more general instructions in section 3 of
   the [IMAP-DISC].

   When the client opens the mailbox for synchronization it first
   compares UIDVALIDITY as described in step d)1) in section 3 of the
   [IMAP-DISC]. If the cached UIDVALIDITY value matches the one
   returned by the server, the client MUST compare the cached value of
   HIGHESTMODSEQ with the one returned by the server. If the cached
   HIGHESTMODSEQ value also matches the one returned by the server,
   then the client MUST NOT fetch flags for cached messages, as they
   hasn't changed. If the value on the server is higher than the
   cached one, the client MAY use "SEARCH MODSEQ <cached-value>" to
   find all messages with flags changed since the last time the client
   was online and had the mailbox opened. Alternatively the client MAY
   use "FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>
   REPORTEXPUNGES)". The latter operation combines reporting expunged
   messages, searching for changed messages and fetching new
   information.

   In all cases the client still needs to fetch information about new
   messages (if requested by the user). If the client has used SEARCH
   MODSEQ, it will also have to discover which messages have been
   expunged.

   Step d) ("Server-to-client synchronization") in section 4 of the
   [IMAP-DISC] in the presence of the EXPUNGED & CONDSTORE extensions
   is amended as follows:

          d) "Server-to-client synchronization" - for each mailbox
          that requires synchronization, do the following:

            1a) Check the mailbox UIDVALIDITY (see section 4.1 of the
   [IMAP-DISC] for more details) with SELECT/EXAMINE/STATUS. If the
   UIDVALIDITY value returned by the server differs, the client MUST

               * empty the local cache of that mailbox;
               * "forget" the cached HIGHESTMODSEQ value for
                 the mailbox;
               * remove any pending "actions" which refer to UIDs in
                 that mailbox. Note, this doesn't affect actions
                 performed on client generated fake UIDs
                 (see section 5 of the [IMAP-DISC]);
               * skip steps 1b and 2-II;

            1b) Check the mailbox HIGHESTMODSEQ. If the cached value
   is the same as the one returned by the server, skip fetching
   message flags on step 2-II, i.e. the client only has to find out
   which messages got expunged.

            2) Fetch the current "descriptors";

               I)  Discover new messages.

               II) Discover changes to old messages using
                   "UID FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>
   REPORTEXPUNGES)".

            3) Fetch the bodies of any "interesting" messages that the
   client doesn't already have.

       Example (the UIDVALIDITY value is the same, but the
   HIGHESTMODSEQ value has changed on the server while the client was
   offline):

            C: A142 SELECT INBOX
            S: * 172 EXISTS
            S: * 1 RECENT
            S: * OK [UNSEEN 12] Message 12 is first unseen
            S: * OK [UIDVALIDITY 3857529045] UIDs valid
            S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
            S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
            S: * OK [HIGHESTMODSEQ 20010715194045007]
            S: A142 OK [READ-WRITE] SELECT completed

       after that:
            C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE
   20010715194032001 REPORTEXPUNGES)
            S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS
   (\Deleted))
            S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS
   ($NoJunk
                  $AutoJunk $MDNSent))
               ...
            S: * EXPUNGED 7,10:15
            S: A143 OK FETCH completed


5.   Formal Syntax

   The following syntax specification uses the Augmented Backus-Naur
   Form (ABNF) notation as specified in [ABNF].

   Non-terminals referenced but not defined below are as defined by
   [IMAP4] or [CONDSTORE].

   Except as noted otherwise, all alphabetic characters are case-
   insensitive.  The use of upper or lower case characters to define
   token strings is for editorial clarity only.  Implementations MUST
   accept these strings in a case-insensitive fashion.


     capability          =/ "X-DRAFT-I00-EXPUNGED"

     message-data        =/ expunged-resp

     expunged-resp       =  "EXPUNGED" SP known-uids

     known-uids          =  sequence-set
                             ;; sequence of UIDs, "*" is not allowed

     chgsince-fetch-mod  =  "REPORTEXPUNGES"
                             ;; REPORTEXPUNGES FETCH modifier conforms
                             ;; to the fetch-modifier syntax


6.   Security Considerations

   <<TBD>>


7.   IANA Considerations

   <<TDB>>


8.   References

8.1  Normative References

   [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
   Requirement Levels", RFC 2119, March 1997.

   [IMAP4], Crispin, M., "Internet Message Access Protocol - Version
   4rev1", RFC 3501, University of Washington, March 2003.

   [ABNF], DRUMS working group, Dave Crocker Editor, "Augmented BNF
   for Syntax Specifications: ABNF", Work in Progress.

   [CONDSTORE] Melnikov, A. and S. Hole, "IMAP Extension for
   Conditional STORE", draft-ietf-imapext-condstore-XX.txt.


8.2  Informative References

   [IMAP-DISC] Melnikov, A.  "Synchronization Operations For
   Disconnected Imap4 Clients", work in progress, draft-melnikov-imap-
   disc-XX.txt

   [TRANS-CAPA] Melnikov, A., "Transitional IMAP capabilities", work
   in progress, draft-melnikov-imap-transitional-capa-XX.txt


9.   Acknowledgments

   Thanks to Steve Hole, Cyrus Daboo and Michael Wener for encouraging
   me to write this document.


10.  Author's Addresses

   Alexey Melnikov
   Isode Limited
   5 Castle Business Village
   36 Station Road
   Hampton, Middlesex, TW12 2BX
   UK

   Email: Alexey.Melnikov@isode.com


11.  Full Copyright Statement

   Copyright (C) The Internet Society (2005).  This document is
   subject to the rights, licenses and restrictions contained in BCP
   78, and except as set forth therein, the authors retain all their
   rights.


   This document and the information contained herein are provided on
   an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
   REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
   THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
   ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
   PARTICULAR PURPOSE.


12.  Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed
   to pertain to the implementation or use of the technology described
   in this document or the extent to which any license under such
   rights might or might not be available; nor does it represent that
   it has made any independent effort to identify any such rights.
   Information on the procedures with respect to rights in RFC
   documents can be found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use
   of such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository
   at http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at ietf-
   ipr@ietf.org.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.


13.  Appendix A. Editorial.

   <<Note that this section will be deleted before publication>>

13.1 Change Log

   00   Initial Revision.


13.2 Open Issues for Discussion


13.3 To Do

  1.   Add section about rationale?