E-Mail Retrieval Protocols

An Example POP Session

Again words in bold typeface are the POP commands you type in as a client:

    # telnet whoelse.com 110
    Trying 192.168.0.3...
    Connected to whoelse.com.
    Escape character is '^]'.
    +OK POP3 whoelse.com v2000.70 server ready
    HELO
    -ERR Unknown AUTHORIZATION state command
    USER yonsuk
    +OK User name accepted, password please
    PASS 12345
    +OK Mailbox open, 5 messages
    LIST +OK Mailbox scan listing follows
    1 325
    2 1250
    3 2145
    4 830
    5 525
    .
    RETR 5
    +OK 525 octets
    Return-Path: <wankyu@whatever.com>
    Received: from whatever.com (IDENT:wankyu@whatever.com[192.168.0.2])
            by mail.somewhere.com (8.9.3/8.9.3) with SMTP id WAA29446
            for yonsuk; Sun, 28 Jan 2001 23:18:09 +0900
    Date: Sun, 28 Jan 2001 23:18:09 +0900
    From: Wankyu Choi <wankyu@whatever.com>
    To: yonsuk@whoelse.com
    Message-Id: < F890755DE93ED411@ whatever.com>
    Subject: Just a Note

    Don't forget to bring your notebook tomorrow.
    Sleep tight.
    .
    DELE 5
    +OK Message deleted
    QUIT
    +OK Sayonara
    Connection closed by foreign host.
    #

Version 3 is the most recent implementation of POP. When we say POP, it refers to POP3 unless otherwise stated.

You initiate a POP session by connecting to the server. Unlike SMTP, the POP server responses start either with +OK indicating successful completion of a given command, or -ERR indicating errors. If you want to write a POP client program, you can use regular expressions to check whether a server response starts with +OK or -ERR. Note that there's no HELO command in POP. That's why the server reports -ERR Unknown AUTHORIZATION state command.

A POP session consists of three states: authorization, transaction, and update. The very moment a POP client gets connected to the server, it enters the authorization state. The USER and PASS commands are used to identify you. A successful authentication puts you into the transaction state where you can retrieve or delete messages.

In the example session above, the server reports that you have 5 messages in your mailbox. The LIST command lists the messages and their size in octets (a set of 8 bits). To view a particular message, you issue the RETR command with a message number as its argument: RETR 5 in this case. You can see the content of the message wankyu@whatever.com sent to yonsuk@whoelse.com.

The DELE command deletes a message specified by its message number argument. Note that the DELE command doesn't actually wipe out a given message. It simply marks it for deletion. Actual deletion occurs when you log out by issuing the QUIT command. Though not shown in the example session, to undelete a message, that is, to rid a message of the mark for deletion, the RSET command is used. The QUIT command ends the transaction state and the update state is initiated. In the update state, the POP server silently expunges e-mail messages marked for deletion. When done, the server logs you out. If no message is marked for deletion, however, the server ends a session without entering the update state.

You can check your e-mail, from any client anywhere, by telneting to your POP server with these simple commands.

Tags

The major difference between IMAP and other line-oriented protocols is that every command issued by a client should be preceded by a string of alphanumeric characters called tags. With tags, an IMAP client can figure out which server response is returned by what command. The IMAP client should generate unique tags during a session. The server, in turn, attaches the tag the client sends to its response. The server may also give responses without tags. Untagged responses are usually the resulting data a client command generated. Untagged responses are preceded by an asterisk (*).

Mailbox Formats

One of the biggest advantages of using IMAP is that you can have multiple mailboxes. A mailbox works like a file system. When you receive an e-mail message, it is put into your mailbox. The simplest form of a mailbox is a single plain text file with all e-mail messages put together within that file. On the UNIX platform, the user's main mailbox is typically found in /var/spool/mail/ directory with the same name as his account. The text file /var/spool/mail/wankyu, for example, is the user wankyu's mailbox storing his incoming e-mail.

The mbox format, referring to a single text file mailbox, appends messages successively. These are separated by a line delimiter starting with the word From. This is followed by a space and the sender's e-mail address, plus the date and the time the message was appended to the mailbox. This delimiter line is commonly called a From_line, and should not be confused with the From header line:

    From yonsuk@whoelse.com Sat Feb 17 17:31:28 2001 // A From_ line
    From: Yonsuk Song<yonsuk@whoelse.com> // A From header

A blank line is inserted after each message:

    From changsoo@neoqst.com Sat Feb 17 17:31:28 2001
       A verbatim copy of the first e-mail message

    From yongpil@neoqst.com Sat Feb 17 17:50:35 2001
       A verbatim copy of the second e-mail message

    From sungho@neoqst.com Sat Feb 17 18:19:30 2001
       A verbatim copy of the third e-mail message

There is a variant of the mbox format called MMDF, where each message is surrounded by lines containing four ctrl-As (^A^A^A^A), without any blank line separating each message.

As the number of your e-mail messages stored in your mailbox grows, a single text file model becomes way too inefficient to access and manage. Also, no two clients can access the same mailbox concurrently: one may lose its lock on the mailbox when another attempts to access it. To avoid this file-locking problem, every e-mail message needs to be stored as a separate file. The MH mailbox format consists of directories and each message is stored in a separate file. Most NNTP servers use a similar format to MH to manage news articles.

IMAP lets you create a hierarchy of multiple mailboxes in much the same way you manipulate file directories. Formats of the mailbox hierarchy vary depending on server implementations, but the simplest form is a mail file system format. In this format, each mailbox can be mapped to a directory or file. INBOX is always a directory.

Note that UW-IMAP server eventually appends messages to a single mbox format mailbox, whereas Cyrus server maintains every message as a separate file in a mailbox directory. Courier-IMAP server is unique in that it supports Qmail's Maildir mailbox format. With UW-IMAP, no two clients can access the same mailbox concurrently: that is, one may lose its lock on the mailbox if the other attempts to access it at the same time. Cyrus and Courier-IMAP servers don't suffer from this file-locking problem.

Example IMAP Session

An IMAP session is comprised of four distinct states. Since it supports multiple mailboxes, you need to select one to work with before you can access messages in it:

• Non-authenticated state: the client is connected to the server

• Authenticated state: the client is successfully authenticated

• Selected state: a mailbox has been successfully selected

• Logout state: the client issued the LOGOUT command or the server closed the connection

Let us now telnet to our localhost running an IMAP server:

    # telnet localhost imap
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    * OK [CAPABILITY IMAP4 IMAP4REV1 LOGIN-REFERRALS AUTH=LOGIN] localhost IMAP4rev1
    2000.287 at Tue, 30 Jan 2001 16:15:07 +0900 (KST)

The server greets you with an untagged response (note the asterisk at the start of the response).

You need to login if you want to do anything meaningful with the server. The LOGIN command is used with two arguments: your account name, also considered your mailbox name, and password. Remember that you need to prepend a tag string to each command:

    WCK001 LOGIN wankyu 12345
    * CAPABILITY IMAP4 IMAP4REV1 NAMESPACE IDLE MAILBOX-REFERRALS SCAN SORT THREAD=R
    EFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND
    WCK001 OK LOGIN completed

Note how the server issued a tagged response to your LOGIN command: "WCK001 OK LOGIN completed". You're not ready to list or read messages yet. You should first select a mailbox to work with. A special mailbox named INBOX is used for getting and storing all incoming messages. For the user wankyu, his INBOX is usually a directory in his home, for example, /home/wankyu/Mail/. With IMAP, you can create, delete, or rename mailboxes. INBOX, however, is the only mailbox you can't mess around with.

Let's select INBOX with the SELECT command and see how the server responds:

    WCK002 SELECT INBOX
    * 10 EXISTS
    * 6 RECENT
    * OK [UIDVALIDITY 980691555] UID validity status
    * OK [UIDNEXT 11] Predicted next UID
    * FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
    * OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent
    flags
    WCK002 OK [READ-WRITE] SELECT completed

The server reports that wankyu has 10 messages in his INBOX and 6 of them are recent and unread.

A message in an IMAP mailbox can be referred to either by its unique identifier (UID) or its sequence number. The message sequence number is a relative position from 1 to the number of messages in the mailbox. As each new message is added, it is assigned a message sequence number that is 1 higher than the number of messages in the mailbox before its addition. Message sequence numbers can be reassigned during a session for a number of reasons, for example, when a message is permanently removed from the mailbox. Therefore, they are not so reliable to work with between sessions.

On the other hand, UIDs persist across sessions but, unlike message sequence numbers, they are not necessarily contiguous, though UIDs are particulalry useful for maintaining a state from the previous session. A unique identifier validity value, 980691555 in this case, is associated with every mailbox to validate UIDs and is sent in a UIDVALIDITY response code at mailbox selection time.

Recent messages are the ones delivered into your mailbox since your last session, while "unseen" messages mean you haven't read them after they showed up in your mailbox. The server response includes a FLAGS line, used to mark the status of an e-mail message, indicating which flags the server supports. A flag is just like a file attribute, holding status information on the message in question. Each flag is explained in the following table:

You use the FETCH command to read a message. Let's read the fifth message in INBOX using the FETCH command. It takes multiple arguments with the first one being the message number. The rest of the arguments may vary depending on what you want to do with the command:

    WCK003 FETCH 5 BODY[1]
    * 5 FETCH (BODY[1] {61}
    Don't forget to bring your notebook tomorrow.
    Sleep tight.

    )
    WCK003 OK FETCH completed

With the BODY argument you may specify which part of the message you wish to retrieve: BODY[0] for the headers, or BODY[1] for the message body, for example. Let's see what headers look like in this message:

    WCK003 FETCH 5 BODY[0]
    * 5 FETCH (BODY[0] {452}
    Return-Path: <wankyu@whatever.com>
    Received: from whatever.com (IDENT:wankyu@whatever.com[192.168.0.2])
            by mail.somewhere.com (8.9.3/8.9.3) with SMTP id WAA29446
            for yonsuk; Sun, 28 Jan 2001 23:18:09 +0900
    Date: Sun, 28 Jan 2001 23:18:09 +0900
    From: Wankyu Choi <wankyu@whatever.com>
    To: yonsuk@whoelse.com
    Message-Id: < F890755DE93ED411@ whatever.com>
    Subject: Just a Note

    )
    WCK004 OK FETCH completed

You can even get a summary of the message structure with the BODYSTRUCTURE argument:

    WCK005 FETCH 5 BODYSTRUCTURE
    * 5 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 61
    2 NIL NIL NIL))
    WCK005 OK FETCH completed

Flags are retrieved much the same way:

    WCK006 FETCH 5 FLAGS
    * 5 FETCH (FLAGS (\Seen))
    WCK006 OK FETCH completed

Now you know the fifth message has been read (\Seen). You may combine these into a single command by surrounding arguments with parentheses:

    WCK007 FETCH 5 (BODYSTRUCTURE FLAGS)
    * 5 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 61
    2 NIL NIL NIL) FLAGS (\Seen))
    WCK007 OK FETCH completed

Let's see how we can create a mailbox. With a UW-IMAP server, the mailbox for the user wankyu resides in his home directory as a subdirectory called Mail:

    /home/wankyu/Mail/

If he creates another mailbox and gives it a name work, his home directory will have a file with that name:

    /home/wankyu/work/

Note that, to create a mailbox container, that is, a mailbox directory that holds other mailboxes, you need to append a trailing mailbox hierarchy delimiter. Some servers, including UW-IMAP, use a forward slash, "/", and others (Cyrus and Courier-IMAP) a single dot (.). For example, with UW-IMAP, you can create a mailbox container with the following:

    WKC008 CREATE work/
    WKC008 OK CREATE completed

The above IMAP command creates a mailbox container named work, which is a subdirectory in the user's home directory. On the other hand, the following command creates a mailbox text file named work:

    WKC008 CREATE work
    WKC008 OK CREATE completed

If a new mailbox is created under the work mailbox container, a mailbox file with the same name is made under the work subdirectory:

    /home/wankyu/work/Jan

If you walk through the path starting from the user's home directory, you get the mailbox name work/Jan.

Now we come out of the session by issuing the LOGOUT command:

    WCK009 LOGOUT
    * BYE whatever.com IMAP4rev1 server terminating connection
    WCK008 OK LOGOUT completed
    Connection closed by foreign host.
    #