LispM's crummy I/O performance (was RE: LispM Market Share)

[magerman@LINC.CIS.UPENN.EDU]

        [...misc deleted...]
       
           4. File read performance is crummy on the Lispms.  Since that's done
     so
              often, it leaves the impression that the Lispms are slow machines
    . 
       
       This has got to be one of the most frustrating things about the
       Symbolics for me.  The symbolics may not be an inherently slow machine
       but compared to a sparcstation (or even my amiga :-) it gives the feel
       of being quite sluggish (disk i/o, keyboard response in certain
       situations, changing windows, etc. etc.).  Of course, I use a Connection
       Machine so I'm not used to waiting around unless I *really* have to ;-).
       
     Lets get quantitative about this.
     ?  Why is LISPM I/O is slow?
     ?  What is a good way to measure relative performance given hardware
        differences? 

I have a quantitative response to how slow LispM I/O is and how much of
an obstacle it can be.  I started working on a connection machine with
a LispM front end this summer.  I wrote a simple program to perform a
much needed task: to recognize and count all sequences of length <= n
in a stream of tokens.  My data set included a million tokens, which
was contained in a 5meg file.

Now, anyone who has touched UNIX knows that reading in a 5meg file and
even allocating memory for 1 million tokens (which are represented as
integers) takes on the order of a few minutes (at most).  The LispM
took almost over 5 hours to read in the data set!  This is on a 3650
under 7.2.  What's more, the same LispM took almost 48 hours to dump
the data (via NFS to a disk on a SUN4)!
The ironic part of this is the CM took only about 15 seconds to
actually process the data.

I solved the same problem using C without the connection machine.  In
fact, on a HP Series 800, the serial solution takes only a few hours,
and it would take under an hour if not for paging problems at runtime.

I think that 1meg/hr is not an acceptable I/O rate.

-- David Magerman (magerman@linc.cis.upenn.edu)
University of Pennsylvania LINC Laboratory