Babyl files, slow as molasses

[Barry Margolin <barmar@think.com>]

    Date: Fri, 6 Nov 87 12:10 EST
    From: J. Scott Penberthy  <JSP@ibm.com>

    As it exists, the Symbolics mailer reads in all messages within each
    mail file.  If a mail file isn't in KBIN format, the mailer also must
    re-parse all that file's messages.  A better approach would use
    "virtual" messages.  In this case, the mailer would only have to read in
    a header from each mail file; this header would contain the pre-parsed
    summary line info for each message, plus a pointer to either some
    location in the mail file or to some other external file.  Only when a
    user requested to see a message would the mailer have to actually read
    in the entire message body.  The document examiner, for which Symbolics
    once won an award, maintains this "virtual copy" method for all
    documentation.  Why not use it for mail files?  Imagine the
    ramifications --> it would take only a few seconds to load a mail file
    with hundreds of messages, not an hour.

Most mail file formats require the mail reading application to read
through message N to find the beginning of message N+1, because it must
search for the message delimiter.  And even if each message were
prefaced with its size (does Babyl format allow this?) there is the
problem that file positioning operations must be made in units of the
file server's character set, but Zmail only knows the size of the
messages it is writing in Lispm characters.  Finally, some file server
protocols (TCP-FTP, in particular) don't permit arbitrary file
positioning.

The Document Examiner gets around the problem of positioning units by
storing its documents as binary files.  It can do this because there is
no need for non-Lispm applications to access these files; Babyl files
must be in the server host's native character set.  I don't think they
get around the second problem; I think you can't use the Document
Examiner if its data files are only accessible via TCP-FTP.

                                                barmar