Issue: LOAD-OBJECTS (Version 2)

[David A. Moon <Moon@STONY-BROOK.SCRC.Symbolics.COM>]

Here is the updated version of this.  Sorry it took so long.
I will bring a few copies of this with me to the meeting.

Issue:         LOAD-OBJECTS

References:    none

Related issues: LOAD-TIME-EVAL,
                CONSTANT-COMPILABLE-TYPES,
                CONSTANT-CIRCULAR-COMPILATION

Category:      ADDITION

Forum:         Cleanup

Edit history:  Version 1, 2-Jan-89, by Moon (for discussion)
               Version 2, 13-Jan-89, by Moon (draft updated from discussion)

Problem description:

  Common Lisp doesn't provide any way to use an object of a user-defined
  type (defined with DEFCLASS or DEFSTRUCT) as a constant in a program
  compiled with COMPILE-FILE.  The problem is that LOAD has to be able
  to "reconstruct" an equivalent object when the compiled-code file is
  loaded, but the programmer has no way to tell LOAD how to do that.


Proposal (LOAD-OBJECTS:MAKE-LOAD-FORM):
          
  Define a new generic function named MAKE-LOAD-FORM, which takes one
  argument and returns two values.  The argument is an object that is
  referenced as a constant or as a self-evaluating form in a file being
  compiled by COMPILE-FILE.  The objective is to enable LOAD to
  construct an equivalent object.

  The first value, called the "creation form," is a form that, when
  evaluated at load time, should return an object that is equivalent to
  the argument.  The exact meaning of "equivalent" depends on the type
  of object and is up to the programmer who defines a method for
  MAKE-LOAD-FORM.  This is the same type of equivalence discussed
  in issue CONSTANT-COMPILABLE-TYPES.

  The second value, called the "initialization form," is a form that,
  when evaluated at load time, should perform further initialization of
  the object.  The value returned by the initialization form is ignored.
  If the MAKE-LOAD-FORM method returns only one value, the
  initialization form is NIL, which has no effect.  If the object used
  as the argument to MAKE-LOAD-FORM appears as a constant in the
  initialization form, at load time it will be replaced by the
  equivalent object constructed by the creation form; this is how the
  further initialization gains access to the object.

  Both the creation form and the initialization form can contain
  references to objects of user-defined types (defined precisely below).
  However, there must not be any circular dependencies in creation forms.
  An example of a circular dependency is when the creation form for the
  object X contains a reference to the object Y, and the creation form
  for the object Y contains a reference to the object X.  A simpler
  example would be when the creation form for the object X contains
  a reference to X itself.  Initialization forms are not subject to
  any restriction against circular dependencies, which is the entire
  reason that initialization forms exist.  See the example of circular
  data structures below.

  The creation form for an object is always evaluated before the
  initialization form for that object.  When either the creation form or
  the initialization form references other objects of user-defined types
  that have not been referenced earlier in the COMPILE-FILE, the
  compiler collects all of the creation forms together and collects all
  of the initialization forms together.  All of the creation forms are
  evaluated before any of the initialization forms.  The order of
  evaluation of the creation forms is unspecified except when the
  ordering is forced by data dependencies.  The order of evaluation of
  the initialization forms is unspecified.

  While these creation and initialization forms are being evaluated, the
  objects are possibly in an uninitialized state, analogous to the state
  of an object between the time it has been created by ALLOCATE-INSTANCE
  and it has been processed fully by INITIALIZE-INSTANCE.  Programmers
  writing methods for MAKE-LOAD-FORM must take care in manipulating
  objects not to depend on slots that have not yet been initialized.

  It is unspecified whether LOAD calls EVAL on the forms or does some
  other operation that has an equivalent effect.  For example, the
  forms might be translated into different but equivalent forms and
  then evaluated, they might be compiled and the resulting functions
  called by LOAD, or they might be interpreted by a special-purpose
  interpreter different from EVAL.  All that is required is that the
  effect be equivalent to evaluating the forms.

  COMPILE-FILE calls MAKE-LOAD-FORM on any object that is referenced as
  a constant or as a self-evaluating form, if the object's metaclass is
  STANDARD-CLASS, STRUCTURE-CLASS, any user-defined metaclass (not a
  subclass of BUILT-IN-CLASS), or any of a possibly-empty
  implementation-defined list of other metaclasses.  COMPILE-FILE will
  only call MAKE-LOAD-FORM once for any given object (compared with EQ)
  within a single file.

  It is valid for user programs to call MAKE-LOAD-FORM in other
  circumstances.

  The function MAKE-LOAD-FORM-USING-SLOTS can be useful in user-written
  MAKE-LOAD-FORM methods.  Its first argument is the object.  Its
  optional second argument is a list of the names of the slots to
  preserve; it defaults to all of the local slots.
  MAKE-LOAD-FORM-USING-SLOTS returns forms that construct an equivalent
  object using MAKE-INSTANCE and SETF of SLOT-VALUE for slots with
  values, or SLOT-MAKUNBOUND for slots without values, or using other
  functions of equivalent effect.  MAKE-LOAD-FORM-USING-SLOTS returns
  two values, thus it can deal with circular structures.

  The default MAKE-LOAD-FORM method for STANDARD-OBJECT signals an
  error.

  The default MAKE-LOAD-FORM method for STRUCTURE-OBJECT returns forms
  that construct an equivalent structure based on the structure name and
  the slot values.  This might be written using
  MAKE-LOAD-FORM-USING-SLOTS, but that is not required.


Examples:

  ;; Example 1
  (defclass my-class ()
     ((a :initarg :a :reader my-a)
      (b :initarg :b :reader my-b)
      (c :accessor my-c)))
  (defmethod shared-initialize ((self my-class) ignore &rest ignore)
    (unless (slot-boundp self 'c)
      (setf (my-c self) (some-computation (my-a self) (my-b self)))))
  (defmethod make-load-form ((self my-class))
    `(make-instance ',(class-name (class-of self))
                    :a ',(my-a self) :b ',(my-b self)))

  In this example, an equivalent instance of my-class is reconstructed
  by using the values of two of its slots.  The value of the third slot
  is derived from those two values.

  Another way to write the last form in the above example would have been

  (defmethod make-load-form ((self my-class))
     (make-load-form-using-slots self '(a b)))

  ;; Example 2
  (defclass my-frob ()
     ((name :initarg :name :reader my-name)))
  (defmethod make-load-form ((self my-frob))
    `(find-my-frob ',(my-name self) :if-does-not-exist :create))

  In this example, instances of my-frob are "interned" in some way.
  An equivalent instance is reconstructed by using the value of the
  name slot as a key for searching existing objects.  In this case
  the programmer has chosen to create a new object if no existing
  object is found; alternatively she could have chosen to signal an
  error in that case.

  ;; Example 3
  (defclass tree-with-parent () ((parent :accessor tree-parent)
                                 (children :initarg :children)))
  (defmethod make-load-form ((x tree-with-parent))
    (values
      ;; creation form
      `(make-instance ',(class-of x) :children ',(slot-value x 'children))
      ;; initialization form
      `(setf (tree-parent ',x) ',(slot-value x 'parent))))

  In this example, the data structure to be dumped is circular, because
  each parent has a list of its children and each child has a reference
  back to its parent.  Suppose make-load-form is called on one object in
  such a structure.  The creation form creates an equivalent object and
  fills in the children slot, which forces creation of equivalent
  objects for all of its children, grandchildren, etc.  At this point
  none of the parent slots have been filled in.  The initialization form
  fills in the parent slot, which forces creation of an equivalent
  object for the parent if it was not already created.  Thus the entire
  tree is recreated at load time.  At compile time, MAKE-LOAD-FORM is
  called once for each object in the true.  All of the creation forms
  are evaluated, in unspecified order, and then all of the
  initialization forms are evaluated, also in unspecified order.


Rationale:

  Only the programmer who designed a class can know the correct
  way to reconstruct objects of that class at load time, therefore
  the reconstruction should be controlled by a generic function.
  Using EVAL as the interface for telling LOAD what to do provides
  full generality.

  MAKE-LOAD-FORM returns two values so that circular structures can
  be handled.  If CONSTANT-CIRCULAR-COMPILATION is rejected,
  MAKE-LOAD-FORM will only return one value, although implementations
  that make an extension to support circular constants will probably
  also make the extension to accept two values from MAKE-LOAD-FORM.

  A default method, such as one that makes an object whose class has the
  same name and whose slots have equivalent contents, is not supplied
  for DEFCLASS-defined objects, because this is inappropriate for many
  objects and because it is easy to write for those objects where it is
  appropriate.  The function MAKE-LOAD-FORM-USING-SLOTS makes it even
  easier to write.

  MAKE-LOAD-FORM has a natural resemblance to PRINT-OBJECT, as a hook
  for the programmer to control the system's actions.

Current practice:

  Symbolics Flavors has something like this, but under a different name.
  The name Symbolics uses is not suitable for standardization.

  JonL reports that Lucid is getting more and more requests for this.

Cost to Implementors:

  This seems like only a few one-line changes in the compiled-code
  file writer and reader.  MAKE-LOAD-FORM-USING-SLOTS is a couple
  dozen lines of code, assuming the presence of the CLOS metaobject
  protocol or an implementation-dependent equivalent.

Cost to Users:

  None.

Cost of non-adoption:

  Serious impairment of the ability to use extended-type objects.  Each
  implementation will probably make up its own version of this as an
  extension.

Performance impact:

  None.

Benefits:

  See Cost of non-adoption.

Esthetics:

  No significant positive or negative impact.

Discussion:

  It would be possible to define an additional level of protocol that
  allows multiple classes to contribute to the reconstruction of an
  object, combining initialization arguments contributed by each class.
  Since a user can easily define that in terms of MAKE-LOAD-FORM without
  modifying the Lisp system, it is not being proposed now.

  Any type that has a read syntax is likely to appear as a quoted
  constant or inside a quoted constant.  Pathnames are one example, user
  programs often define others.  Also many implementations provide a way
  to create a compiled-code file full of data (rather than compiled Lisp
  programs), and such data probably include extended-type objects.

  Moon supports this.  David Gray and John Rose made major contributions
  to the discussion that produced this improved version 2 proposal.