Re: Writing binary files in MCL...

[bill@cambridge.apple.com (Bill St. Clair)]

>   Well, I tried to ask this question once, but got no response, so
>I'll try to put it more simply:

One of the mailers or news processors on the way from there to here
was very slow. Your first message was posted on 6/13, your second on 6/17.
Neither arrived here until today: 6/22.

>I'm trying to write objects out and
>read them in in a binary file to increase speed and save space.  I'm
>using generic functions on the built-in classes/structure classes to
>handle the type specific encoding/decoding.  I'm using (write-byte..)
>and (read-byte..) to do the file I/O.  I like the generic function
>stuff; MCL seems pretty efficient in handling the method dispatch.  I
>was wondering, though, if there is a better/faster way to actually do
>the I/O?  I don't care if it's MCL-specific.  Any suggestions?

The problem with WRITE-BYTE, and all Common Lisp I/O for that matter,
is that to write a single character you call WRITE-BYTE, do generic
function dispatch on STREAM-WRITE-BYTE, then another function call on
MCL's low-level bufferred I/O support function. Common Lisp really needs
READ-ARRAY and WRITE-ARRAY functions. I've done this in a limited
way for 2.0 final, but my code won't work in 2.0b1.

MCL does have a facility for speeding up element-at-a-time I/O.
In 2.0b1 it only speeds up character file streams. In 2.0 final,
it also speeds up binary file streams. If you use these, you
can reduce the three function calls plus generic function dispatch
to a single function call.


STREAM-WRITER stream
  returns two values: WRITER & WRITER-ARG.
  (funcall WRITER WRITER-ARG char) is the same as (tyo char stream)
  except that it is much faster for some streams (file streams, in
  particular).

STREAM-READER stream
  returns two values: READER & READER-ARG.
  (funcall READER READER-ARG) is the same as (tyi stream)
  except that it is much faster for some streams (file streams, in
  particular).

Example:

(defun my-copy-file (from to &optional (if-exists :error))
  (with-open-file (from-stream from :direction :input)
    (with-open-file (to-stream (merge-pathnames to from)
                               :direction :output :if-exists if-exists)
      (multiple-value-bind (reader reader-arg) (stream-reader from-stream)
        (multiple-value-bind (writer writer-arg) (stream-writer to-stream)
          (loop
            (let ((char (funcall reader reader-arg)))
              (unless char
                (return))
              (funcall writer writer-arg char)))))
      (pathname to-stream))))

I compared times for copying a short file with four different methods:

1) MCL's COPY-FILE function
2) A version using my 2.0 final READ-ARRAY & WRITE-ARRAY functions
3) MY-COPY-FILE above
4) A version using READ-CHAR & WRITE-CHAR

The relative times for methods (1 2 3 4) were (1.0 1.22 1.66 4.34)