More file output performance

[poeck@informatik.uni-wuerzburg.de (Karsten Poeck)]

More Data on slow/fast Lisp Mac File I/O:

For our application, a expert system shell, we need the ability
to save the contents of a knowledge base to file at run time for
end users. Since we need this feature for end users in saved images
where the compiler is disabled, we cannot use the #., compile file trick.

Up to now, we simply write a bunch of Lisp s-expressions
to a file, but this is awfully slow. 

For sake of simplicity lets assume that all our data is in propertylists
and we simply write s-expressions like
 (setf (symbol-plist <symbol>) '(<symbol-plist> <symbol>))
for every symbol into the file to save the knowledge base and simply load
the file afterwards.

Then I use something like
open file
over all symbols
     write s-expression for symbol to file
close file

to save the knowledge base as in the following
two functions save-knowledge-base and save-a-symbol

However this takes 35 seconds for a 135 K Text file.
Our knowledge bases are often more than 1 meg, so this is clearly a pain.

After the latest discussion about file IO I modified the save-a-symbol
function
to use the stream-writer-trick, but the speed is nearly the same and
tons of storage are allocated.

Since Fred-windows of 1 meg or more are saved in a couple of seconds
I suspect that the conversion of the s-expression to a string
done by format takes all the time. This conversion must be done
explicitly when I call (format nil "~S" ...) and in another way
when I do (format filestream "~S" ...)

And really, when I dont write anything on the hard-disk and only execute
the 
(format nil "(setf (symbol-plist '~a) ~S)~%" symbol (symbol-plist symbol))
to generate the strings it takes about 35 seconds and when
I omit the format and save a constant string it only takes 8 seconds.
So the given the string is fast enough, so
the problem is not the LISP-MAC-IO in itself. 

I have absolutely no idea have I could speed up the stuff and would
extremly
appreciate any hints. 

Karsten 

;;;; Various tests 

(defun save-knowledge-base (file)
  (with-open-file 
    (stream file
            :direction :output
            :if-does-not-exist :create)
    (multiple-value-bind 
      (function value)
      (stream-writer stream)
      ;should be do-all-knowledge-base-symbols; I simply save the same
symbol 'mgd12 500 times
      (dotimes (x 500)
        (save-a-symbol 'mgd12 stream function value)))))

(defun save-a-symbol (symbol stream function value)
  (declare (ignore function value))
  (format stream "(setf (symbol-plist '~a) '~S)~%" symbol (symbol-plist
symbol)))

#|
(setf (symbol-plist 'mgd12)
      '(Meta-Objekttyp M Parents1 (Mgd1) Actionrules (Ra5) Parents (Mgd1)
Prules (Rfb20) Range
        ("Innere Medizin" "Kfz-Diagnostik") Answertyp Oc Prompt
"Charakterisieren sie das Einsatzgebiet genauer!"
        Qmember (Qgd) Infotyp Basic Name "Einsatzgebiet"))
 

;Test Case
(let ((file (merge-pathnames "test.knowledgebase")))
  (when (probe-file file)
    (delete-file file))
  (time
   (save-knowledge-base file)))

->
(Save-Knowledge-Base File) took 41523 milliseconds (41.523 seconds) to run.
Of that, 5619 milliseconds (5.619 seconds) were spent in The Cooperative
Multitasking Experience.
 440 bytes of memory allocated.

|#

(defun save-a-symbol (symbol stream function value)
  (declare (ignore stream))
  (dovector (char  (format nil "(setf (symbol-plist '~a) '~S)~%" symbol
(symbol-plist symbol)))
    (funcall function value char)
    )
  )

#|
next test with stream-writer
(let ((file (merge-pathnames "test.knowledgebase")))
  (when (probe-file file)
    (delete-file file))
  (time
   (save-knowledge-base file)))

Save-Knowledge-Base File) took 44106 milliseconds (44.106 seconds) to run.
Of that, 6149 milliseconds (6.149 seconds) were spent in The Cooperative
Multitasking Experience.
 644616 bytes of memory allocated.

|#

(defun save-a-symbol (symbol stream function value)
  (declare (ignore stream function value))
  (dovector (char  (format nil "(setf (symbol-plist '~a) '~S)~%" symbol
(symbol-plist symbol)))
    (char= char char)
    )
  )

#|
(let ((file (merge-pathnames "test.knowledgebase")))
  (when (probe-file file)
    (delete-file file))
  (time
   (save-knowledge-base file)))

(Save-Knowledge-Base File) took 35502 milliseconds (35.502 seconds) to run.
Of that, 1548 milliseconds (1.548 seconds) were spent in The Cooperative
Multitasking Experience.
 644440 bytes of memory allocated.

|#


(defun save-a-symbol (symbol stream function value)
  (declare (ignore stream symbol))
  (dovector (char  "(setf (symbol-plist 'Mgd12) (Meta-Objekttyp M Parents1
(Mgd1) Actionrules (Ra5) Parents (Mgd1) Prules (Rfb20) Range (\"Innere
Medizin\" \"Kfz-Diagnostik\") Answertyp Oc Prompt \"Charakterisieren sie
das Einsatzgebiet genauer!\" Qmember (Qgd) Infotyp Basic Name
\"Einsatzgebiet\"))
")
    (funcall function value char)
    )
  )

#|
(let ((file (merge-pathnames "test.knowledgebase")))
  (when (probe-file file)
    (delete-file file))
  (time
   (save-knowledge-base file)))

(Save-Knowledge-Base File) took 7978 milliseconds (7.978 seconds) to run.
Of that, 753 milliseconds (0.753 seconds) were spent in The Cooperative
Multitasking Experience.
 440 bytes of memory allocated.

|#


#|
(time 
 (let ((stream (make-string-output-stream))
       (symbol 'mgd12))
   (dotimes (x 500)
     (format stream "(setf (symbol-plist '~a) '~S)~%" symbol (symbol-plist
symbol)))
   (get-output-stream-string stream)
   nil
   ))

(defun format-a-plist (symbol stream)
  (format stream "(setf (symbol-plist '")
  (format stream (string symbol))
  (format stream ")'(")
  (do ((plist (symbol-plist symbol)(cddr plist)))
      ((null plist))
    (format stream (string (first plist)))
    (format stream " ~S " (second plist)))
  (format stream "))~%")
  )

(time 
 (let ((stream (make-string-output-stream))
       (symbol 'mgd12))
   (dotimes (x 500)
     (format-a-plist symbol stream))
   (get-output-stream-string stream)
   nil
   ))


|#