Re: LISPM I/O performance

[kanderso%dino.bbn.com@RELAY.CS.NET]

  Here's the file I'm going to edit.  It was the biggest one I had readily
  available.
  
        arppro13.zuu.1   53 235644(8)    !   11/10/89 12:54:56 (11/10/89)   Kosma
  
       (scl:time (zwei:find-file #P"blaise:>kosma>dl>arppro13.zuu"))
  Evaluation of (ZWEI:FIND-FILE #P"BLAISE:>kosma>dl>arppro13.zuu") took 17.556589 seconds of elapsed time including:
    0.755 seconds processing sequence breaks,
    2.358 seconds in the storage system (including 0.032 seconds waiting for pages):
      0.365 seconds processing 1657 page faults including 1 fetches,
      1.994 seconds in creating and destroying pages, and
      0.000 seconds in miscellaneous storage system tasks.
  9,083 list, 505 structure words consed in WORKING-STORAGE-AREA.
  102,689 structure words consed in EDITOR-LINE-AREA.
  45 list, 32 structure words consed in *ZMACS-BUFFER-AREA*.
  NIL
  #<ZWEI:FILE-BUFFER "arppro13.zuu >kosma>dl BLAISE:" 43600407>
  ---> 
  
You might try wrapping (si:without-interrupts around your (time ...
form.  Also, file opening time is probably a significant part of this
test.  You might also try it on a file about twice the size.

  
  Then, I remembered that I just did that over the ethernet (TCP) so I
  tried again, first copying the file to my local machine; there was about
  a second difference overall:
  
  Evaluation of (ZWEI:FIND-FILE #P"KARL:>foo.foo") took 16.414146 seconds
  of elapsed time ...
  
  This works out to around 14K bytes/second, an abominable transfer rate,
  if you don't factor in overhead.  I don't know what zwei:com-read-file
  is doing internally, and I don't really care--it doesn't really matter.
  What DOES matter is that reading this file into ZMACS takes 16 to 17
  seconds, while reading the same file into emacs on my amiga takes 1 to 2
  seconds, and on our sparcstation loading it into gmacs, it's basically
  instantaneous (even I was surprized).

I think it would be fairer to try ftp-find-file in Gnu EMACS.
  
I think that ZWEI:FIND-FILE and LOAD should be optimized as much as
possible as these are probably the two most used I/O operations.  [See
some ideas on this below.]

      Anyway, I'll be very interested, after you make your program
      either use a binary file, or if you feel you can't do that,
      you use :STRING-LINE-IN and PARSE-INTEGER, or :READ-INPUT-BUFFER,
      some care about buffer boundaries, and PARSE-INTEGER, and then
      you come back and tell us:  "It takes X seconds, and that's too
      slow".  That will be good and useful, and I hope you do it.
  
  where does something like READ-LINE fit into all this? I've used that in
  some places.
  
READ-LINE cons a new line each time it is called.  For most
applications this is needless.  :STRING-LINE-IN will fill a line that
you provide it, so no consing is done.

I've suddenly found my self having to read in several larges files
prepeatedly.  For example:

-rw-rw-rw-  1 kanderso  5735077 Jan 23 00:57 rt.det.shrink

The file consists of 49340 lines containing the ascii representation
of 1085480 things as integers, single-floats, double-floats, and
symbols.  I read these in and create CLOS instances for each.  [I/O
may be slow, but i'm quite greatful i can do this.]

My first version used READ.  My second version used SI:XR-READ-THING
and internal read function.  This took 55 minutes to read in
[:element-type 'string-char] .  I then wrote my own version of
parse-integer [because the standard one conses when building
double-floats and bignums], and functions to parse floats and symbols.
Using these took 42 minutes about 30% less.