Network Working Group R. Austein
Internet Draft: IMAP4 Disconnected Access Epilogue Technology
Document: internet-drafts/draft-ietf-imap-disc-00.txt June 1994
SYNCHRONIZATION OPERATIONS FOR DISCONNECTED IMAP4 CLIENTS
Status of this Memo
This document is an Internet Draft. Internet Drafts are working
documents of the Internet Engineering Task Force (IETF), its Areas,
and its Working Groups. Note that other groups may also distribute
working documents as Internet Drafts.
Internet Drafts are draft documents valid for a maximum of six
months. Internet Drafts may be updated, replaced, or obsoleted by
other documents at any time. It is not appropriate to use Internet
Drafts as reference material or to cite them other than as a "working
draft" or "work in progress".
To learn the current status of any Internet-Draft, please check the
id-abstracts.txt listing contained in the Internet-Drafts Shadow
Directories on ds.internic.net, nic.nordu.net, ftp.isi.edu, or
munnari.oz.au.
This is a draft document of the IETF IMAP Working Group. A revised
version of this draft document will be submitted to the RFC editor as
an Informational RFC for the Internet Community. Discussion and
suggestions for improvement are requested, and should be sent to
imap@CAC.Washington.EDU. This document will expire before 31
November 1994.
This note attempts to address some of the issues involved in building
a disconnected IMAP4 client. In particular, it deals with the issues
of what might be called the "driver" portion of the synchronization
tool: the portion of the code responsible for issuing the correct set
of IMAP4 commands to synchronize the disconnected client in the way
that is most likely to make the human who uses the disconnected
client happy.
This memo is for informational use and does not constitute a
standard. Distribution of this memo is unlimited.
Austein [Page 1]
Internet Draft June 20, 1994
1. DESIGN PRINCIPLES
All mailbox state or content information stored on the disconnected
client should be viewed strictly as a cache of the state of the
server. The "master" state remains on the server, just as it would
with an interactive IMAP4 client. The one exception to this rule is
that information about the state of the disconnected client's cache
remains on the disconnected client: that is, unlike the equivilent
case in the DMSP protocol, the IMAP4 server is not responsible for
remembering the state of the disconnected IMAP4 client.
We assume that a disconnected client is a client that, for whatever
reason, wants to minimize the length of time that it is "on the
phone" to the IMAP4 server. Often this will be because the client is
using a dialup connection, possibly with very low bandwidth, but
sometimes it might just be that the human is in a hurry to catch an
airplane, or some other event beyond our control. Whatever the
reason, we assume that we must make efficient use of the network
connection, both in the usual sense (not generating spurious traffic)
and in the sense that we would prefer not to have the connection
sitting idle while the client and/or the server is performing
strictly local computation or I/O. Another, perhaps simpler way of
stating this is that we assume that network connections are
"expensive".
Practical experience with existing disconnected mail systems
(PCMAIL/DMSP) has shown that there is no single synchronization
strategy that is appropriate for all cases. Different humans have
different preferences, and the same human's preference will vary
depending both on external circumstance (how much of a hurry the
human is in today) and on the value that the human places on the
messages being transfered. The point here is that there is no way
that the synchronization program can guess exactly what the human
wants to do, so the human will have to provide some guidance.
Taken together, the preceeding two principles lead to the conclusion
that the synchronization program must make its decisions based on
some kind of configuration file provided by the human, but almost
certainly should not pause for I/O with the human during the middle
of the synchronization process. The human will almost certainly have
several different configurations for the synchronization program, for
different circumstances.
Automated support for helping naive humans write better configuration
files would be a good thing, but writing such tools is outside the
scope of this discussion.
Since a disconnected client has no way of knowing what changes might
Austein [Page 2]
Internet Draft June 20, 1994
have occured to the mailbox while it was disconnected, message
numbers are not useful to a disconnected client. All disconnected
client operations should be performed using UIDs, so that the client
can be sure that it and the server are talking about the same
messages during the synchronization process. It is permissible, but
probably not useful, for a disconnected client to use message numbers
once it has obtained a valid current mapping between UIDs and message
numbers.
2. OVERALL PICTURE OF SYNCHRONIZATION
The basic strategy for "normal" synchronization is pretty simple, and
closely follows the strategy used by a disconnected DMSP client
(terminology explained below):
a) Process any "actions" that were pending on the client;
b) Fetch the current list of "interesting" mailboxes;
c) For each mailbox, fetch the current "descriptors";
d) For each mailbox, fetch the bodies of any "interesting"
messages that the client doesn't already have.
Explanation:
a) "Actions" are queued requests that were made by the human to the
client's MUA software while the client was disconnected. Expected
requests are commands like COPY, STORE, EXPUNGE, CREATE. FETCH
commands may also show up as actions, if the MUA allows the human
to explicitly fetch the body of a particular message that was
skipped by the normal synchronization process. In general, any
IMAP4 command can show up as an action, as can a few things that
are not IMAP4 commands at all, such as requests to send a newly
composed message via SMTP.
The list of actions should be ordered. Eg, if the human deletes
message A1 in mailbox A, then expunges mailbox A, then deletes
message A2 in mailbox A, the human will expect that message A1 is
gone and that message A2 is still present but is now deleted.
By processing all the actions before proceeding with
synchronization, we avoid having to compensate for the local MUA's
changes to the server's state. That is, once we have processed
all the pending actions, the steps that the client must take to
synchronize itself will be the same no matter where the changes to
Austein [Page 3]
Internet Draft June 20, 1994
the server's state originated.
b) The set of "interesting" mailboxes pretty much has to be
determined by the human. What mailboxes belong to this set may
vary between different IMAP4 sessions with the same server,
client, and human.
c) "Descriptors" is a DMSP term, borrowed here because it's both
easier to use and more precise than a specific list of IMAP4 FETCH
data items. Conceptually, a message's descriptor is that set of
information that allows the synchronization program to decide what
protocol actions are necessary to bring the local cache to the
desired state for this message; since this decision is really up
to the human, this information probably includes a at least a few
header fields intended for human consumption. Exactly what will
constitute a descriptor depends on the client implementation. At
a minimum, the descriptor contains the message's UID and FLAGS.
Other likely candidates are the RFC822.SIZE and BODYSTRUCTURE data
items and the RFC-822 From:, To:, Date:, Subject:, and Message-ID:
header lines.
Note that this step is also where the client finds out about
changes to the flags of messages that the client already has in
its local cache, as well as finding out about messages in the
local cache that no longer exist on the server (ie, messages that
have been expunged).
d) "interesting" messages are those messages that the synchronization
program thinks the human wants to have cached locally, based on
the configuration file and the data retrieved in step (c).
The rest of this discussion will focus primarily on the
synchronization issues for a single mailbox.
3. DETAILS OF "NORMAL" SYNCHRONIZATION OF A SINGLE MAILBOX
The most common form of synchronization is where the human trusts the
integrity of the client's copy of the state of a particular mailbox,
and simply wants to bring the client's cache up to date so that it
accurately reflects the mailbox's current state on the server.
Let <lastseen> represent the highest UID that the client knows about
in this mailbox. Since UIDs are allocated in strictly ascending
order, this is simply the UID of the last message in the mailbox that
Austein [Page 4]
Internet Draft June 20, 1994
the client knows about. Let <lastseen+1> represent <lastseen>'s UID
plus one. Let <descriptors> represent a list consisting of all the
FETCH data item items that the implementation considers to be part of
the descriptor; at a minimum this is just the FLAGS data item.
With no further information, the client can issue issue the following
two commands:
tag1 UID FETCH <lastseen+1>:* <descriptors>
tag2 UID FETCH 1:<lastseen> FLAGS
The order here is significant. We want the server to start returning
the list of new message descriptors as fast as it can, so that the
client can start issuing more FETCH commands, so we start out by
asking for the descriptors of all the messages we know the client
cannot possibly have cached yet. The second command fetches the
information we need to determine what changes may have occurred to
messages that the client already has cached. Once the client has
issued these two commands, there's nothing more the client can do
with this mailbox until the responses to the first command start
arriving. A clever synchronization program might use this time to
fetch its local cache state from disk, or start the process of
synchronizing another mailbox.
Once the descriptors start arriving, the client can start issuing
appropriate FETCH commands for "interesting" messages or bodyparts
thereof. The decision on what is an "interesting" message is up to
the client software and the human. One easy criterion that should
probably be implemented in any client is whether the message is "too
big" for automatic retrieval, where "too big" is a parameter defined
in the client's configuration file.
It is important to note that fetching a message into the disconnected
client's local cache does NOT imply that the human has (or even will)
read the message. Thus, the synchronization program for a
disconnected client should always be careful to use the .PEEK
variants of the FETCH data items that implicitly set the \Seen flag.
Once the last descriptor has arrived and the last FETCH command has
been issued, the client simply needs to process the incoming fetch
items, using them to update the local message cache.
In order to avoid deadlock problems, the client must give processing
of received messages priority over issuing new FETCH commands during
this synchronization process. This may necessitate temporary local
queuing of FETCH requests that cannot be issued without causing a
deadlock. In order to achive the best use of the "expensive" network
connection, the client will almost certainly need to pay careful
attention to any flow-control information that it can obtain from the
underlying transport connection (usually a TCP connection).
Austein [Page 5]
Internet Draft June 20, 1994
4. SPECIAL CASE: DESCRIPTOR-ONLY SYNCHRONIZATION
For some mailboxes, fetching the descriptors might be the entire
synchronization step. Practical experience with DMSP has shown that
a certain class of mailboxes (eg, "archival" mailboxes) are used
primarily for long-term storage of important messages that the human
wants to have instantly available on demand but does not want
cluttering up the disconnected client's cache at any other time.
Messages in this kind of mailbox would be fetched exclusively by
explicit actions queued by the local MUA. Thus, the only
synchronization that is necessary for a mailbox of this kind is
fetching the descriptor information that the human will use to
identify messages that should be explicitly fetched.
Special mailboxes that receive traffic from a high volume, low
priority mailing list might also be in this catagory, at least when
the human is in a hurry.
5. SPECIAL CASE: FAST NEW-ONLY SYNCHRONIZATION
In some cases the human might be in such a hurry that s/he doesn't
care about changes to old messages, just about new messages. In this
case, the client can skip the UID FETCH command that obtains the
flags and UIDs for old messages (1:<lastseen>).
6. SPECIAL CASE: BLIND FETCH
In some cases the human may know (for whatever reason) that s/he
always wants to fetch any new messages in a particular mailbox,
unconditionally. In this case, the client can just fetch the
messages themselves, rather than just the descriptors, by using a
command like:
tag1 UID FETCH <lastseen+1>:* (FLAGS RFC822.PEEK)
7. SPECIAL CASE: OPTIMIZING "MOVE" OPERATIONS
Practical experience with IMAP, DMSP, and other mailbox access
protocols that support multiple mailboxes suggests that moving a
message from one mailbox to another is an extremely common operation.
In IMAP4 a "move" operation is really a combination of a COPY
operation and a STORE +FLAGS (\Deleted) operation. This makes good
protocol sense for IMAP, but it leaves a simple-minded disconnected
client in the silly position of deleting and possibly expunging its
cached copy of a message, then fetching an identical copy via the
network.
Austein [Page 6]
Internet Draft June 20, 1994
Fortunately, there is a relatively easy way around this problem. By
including the Message-ID: header and the INTERNALDATE data item as
part of the descriptor, the client can check the descriptor of a
"new" message against messages that are already in its cache, and
avoid fetching the extra copy. Of course, it's possible that the
cost of checking to see if the message is already in the local cache
may exceed the cost of just fetching it, so this technique should not
be used blindly. If the MUA implements a "move" command, it make
special provisions to use this technique when it knows that a
copy/delete sequence is the result of a "move" command.
Since it's theoretically possible for this algorithm to find the
wrong message (given sufficiently malignant Message-ID headers),
implementors should provide a way to disable this optimization, both
permanently and on a message-by-message basis.
Security Considerations
Security considerations ane not discussed in this memo.
Author's Address:
Rob Austein
Epilogue Technology Corporation
268 Main Street, Suite 283
North Reading, MA 01864
Phone: (617) 245-0804
FAX: (617) 245-8122
Email: sra@epilogue.com
Austein [Page 7]