Dumping data structures with circular references.

[Richard Lamson <rsl%max-fleischer.ila-sf.dialnet.symbolics.com@RELAY.CS.NET>]

[I am not addressing this one directly to RDP, since both of your addresses
bounced back to me yesterday.  What's the right way to send you mail?]

    Date: Fri, 3 Nov 89 09:19 PST
    From: Robert D. Pfeiffer <RDP@ALAN.LAAC-AI.Dialnet.Symbolics.COM>

    Many thanks for the replies so far (keep those cards and letters
    coming!).  I'm pursuing one additional item on this topic and it
    involves dumping the data in two passes.  The problem is that
    SYS:DUMP-FORMS-TO-FILE takes a filename rather than a stream and it
    doesn't allow me to say "append".  

Actually, even if you could "append", it writes SI:BIN-OP-EOF into the file
so loading the file would not read past the first EOF anyway.

    Anyone have an idea how best to solve this?  Maybe write two separate
    files with SYS:DUMP-FORMS-TO-FILE and then append them after the fact?
    Is it easy to append two binary files (or are there subtleties of the
    file format that will screw me)?  Or if I had the equivalent of
    SYS:DUMP-FORMS-TO-STREAM I guess I would be home free.  Then, I could
    simply do my own WITH-OPEN-FILE once and call SYS:DUMP-FORMS-TO-STREAM
    twice on it.

Here is a synopsis of what I did the last time I needed to do this:

  (defmethod (dump database) (filename)
    (si:writing-bin-file (bin-stream filename)
      (si:dump-attribute-list `(:package ,(package-name *package*) :mode :Lisp) bin-stream) 
      (do-all-objects (object)
	(si:dump-form-to-eval (creation-form object) bin-stream))
      (do-all-objects (object)
	(si:dump-form-to-eval (initialization-form object)))))

DO-ALL-OBJECTS does what you might expect, namely map over the entire
database, binding the given variable to each object in turn.

The CREATION-FORM method returns something like

  `(setf (gethash *database* ',unique-id) (make-instance ',flavor ...))

The INITIALIZATION-FORM method returns something like:

  `(let ((object (gethash *database* ',unique-id))
	 (prev (gethash *database* ',(object-unique-id previous-object)))
	 (next (gethash *database* ',(object-unique-id next-object)))
	 (other ',(object-unique-id other-object))
	 ...)		; Any other object references
     (setf (object-previous-object object) prev
	   (object-next-object object) next
	   (object-other-object object) other
	   ... 
	   (object-option-thing1 object) ',thing1
	   ...))

although in the interests of efficiency you could package this up into a
function.  It's also a good idea not to put in the QUOTE if you can avoid it,
since this can make your BIN file much larger.  I.e., if your unique IDs are
self-evaluating, or some of the other THING references are self-evaluating,
just put in ,THING instead of ',THING since the former takes fewer bytes in
the binary representation.