babyl lossages

["A. Peter Blicher" <BLICHER@ibm.com>]

The SAIL machine at Stanford has a fairly nice, cheap way to handle mail
files (and all other files, for that matter).  The idea is that the file
is divided into logical pages, usually 1 message to a page.  The first
logical page of the file contains 1 line for each logical page, which is
identical to the first line of that logical page.  In addition, each
such line contains a pointer to where in the file that message resides,
and a number indicating its size.  Typically, at any given time only 1
logical page is "in core."  Whenever any change is made to a logical
page, its entry on the first page (known as the directory page) is
updated to reflect the change.  To avoid a lot of disk copying, logical
pages are allocated with some padding, so that small changes can be
accomodated without any disk moves.  All of this is implemented inside
the editor, and the editor is normally used to view a mail file.  To
compress the header information into the first line, the mailer concocts
a first line when it appends to the file (which is in fact where mail is
delivered to).  This might be viewed as rudimentary "parsing" and could
as well be done by the editor.  Consistency checks are done to make sure
the directory is valid for the file.

What the user sees is as follows.  Regardless of the size of his mail
file (and I've had them multi megabyte), editing the mail file gives a
display of the default logical page (usually the last not seen), in
about 1 second of real time (essentially the time to get the logical
page from disk and set up the editor).  The user has random access to
any other page with a couple of keystrokes, and display is almost
instantaneous.  Also, the user can look at the directory page (logical page
1), where he sees a 1 line entry for each message, much like the summary
window in zmail.  (The response times are for a KL-10 with a load factor
of say 1.0).  Macros exist for operations to messages associated with
pointing to the summary line.  The combination of this method with the
display editor and the display hardware are the fastest I've seen,
before or since.

A side benefit is that the same structure is useful with programs or
other structured things.  For example, each main function or program
subsection can be on its own logical page, hence the directory page
would have a list of, say, defuns (the first line of the logical page),
each of which would be randomly accessible by a keystroke or two.  This
is like some of the functionality of zmacs lisp mode, but for its
limited subset, easier to use, especially for the novice or browser.

All the elements to do this, I believe, available to the lispm, the
most important being random access within files.

------------

Another system, similar to the non-working-under-7.1 directory system,
is Brian Reid's extensions to the RAND mail system for Unix.  The main
apparent difference is that the user can control whether reparsing takes
place (i.e. whether a directory of messages should be updated).  It also
seems to be much faster on a vax 780 than zmail.  The main lossage with
this system as well as with the zmail directory system, is that you are
in deep trouble if you want to manipulate any of the mail files with the
editor or file utilities, since every message is in a separate file, and
the file names are random numbers.  A related problem is that maps over
many messages require many file openings, hence are slow, or else all
the files must be paged in at the beginning, which means you wait longly
the first time you want such a thing.

----------------

Therefore, I'd recommend a system such as the SAIL system, with the
bells and whistles of zmail.  The key is to use a chunking unit smaller
than a file, but still have random access.  The win is that a mail file
is easily looked at and operated on by the editor, so that operations
that the designers of the mail reader may not have foreseen can easily
be done.  Right now, babyl files apparently have no structure, so lose.
Kbin files have so much structure that the editor can't look at them
easily, so lose, and in any case are still too slow since there is no
paging.  Directory files can't easily be read with the editor, so lose.

In addition, I'd like it if zmail were essentially a mode of zmacs, so
that, for example, my zmacs customizations worked in zmail, and my zmail
customizations worked in zmacs (which is now only partially the case,
without much effort).  In principle, the zwei idea should take care of
this, so this criticism might be rephrased as saying that zmail needs
more than just bare zwei as its root.

--peter