Re: File output performance

["Mark A. Tapia" <markt@dgp.toronto.edu>]

On Wed Jan 12, Bruce Lester asks how to speedup the file I/O in the
following code:

    (do ((ROW-NUMBER 1 (+ ROW-NUMBER 1)))
        ((> ROW-NUMBER ROWS)
         (close FILE-STREAM))
      
      (do ((COLUMN-NUMBER 1 (+ 1 COLUMN-NUMBER)))
          ((> COLUMN-NUMBER COLUMNS)
           (format FILE-STREAM "~%"))
        (format FILE-STREAM "~S" (nth COLUMN-NUMBER (assoc ROW-NUMBER 
                                                           (DATA
TABLE-SYMBOL))))
        (if (not (= COLUMN-NUMBER COLUMNS))
          (format FILE-STREAM ","))

It may be taking a long time to do the formatting. You
may speedup the code by using the stream writer and writer-arg.
I managed to reduce the file write time by adopting this
approach, buffering the output until the end and then using
writer and writer-arg to write out the data. This allowed me to
collect data from an experiment in real time and to write it out
between sessions. The time for writing was reduced to at most 1/5th.

Here's the code to retrieve the writer and writer-arg for the file
  (let (file)
    (setq file 
         (open (choose-new-file-dialog)
               :direction :output :if-exists :supersede))
    (multiple-value-bind (writer writer-arg)
               (stream-writer file)
       (fresh-line file)
          ...
You can then write to the file in binary form or character form.

Here are several examples:

1. format a s-expr and write the string to the file
(defun write-sexp (sexpr file writer writer-arg)
    (force-output file)
    (print-line writer writer-arg 
                (format nil "~s" sexpr)))

2.write one character
(funcall writer writer-arg (code-char (aref data time i))))

3. write a signed word
        (write-signed-word writer writer-arg dh file)


mark