Issue: LOAD-TIME-EVAL (Version 3)

[Kent M Pitman <KMP@STONY-BROOK.SCRC.Symbolics.COM>]

Well, I apologize for remaining silent on this issue for this long, but
little things were nagging at me about this issue and I haven't had time
to express them. I just couldn't vote in favor of it in the form in
which it was presented even though I agree strongly with the idea that
we should provide such a facility.

I finally wrote the following variant of Kempf's proposal, which has some
fairly substantive differences. Hopefully by reading through this, it will
become obvious to you what the nature of my uneasiness is.

-----
Issue:        LOAD-TIME-EVAL
References:   #, (p356),  EVAL-WHEN (pp69-70)
Category:     ENHANCEMENT/CHANGE
Edit history: 06-Jun-87, Version 1 by James Kempf,
	      17-Jul-87, Version 2 by James Kempf,
	      11-Nov-87, Version 3 by Kent Pitman
Status:       For Internal Discussion

Problem description:

 Common Lisp provides reader syntax (#,) which allows the programmer
 to designate that a particular expression within a program is to be
 evaluated early (at load time). Unfortunately, however, no access to
 this capability is available to programs which construct other
 programs without going through the reader.

 Also, CLtL is vague about whether the result of this early evaluation
 is re-evaluated at runtime.

Proposal (LOAD-TIME-EVAL:FOR-EVALUATION-ONLY):

 Define that
   #,form
 is permissible -only- in a for-evaluation position. Define it to be
 equivalent to:
   #.(MAKE-LOAD-TIME-EVAL 'form)

 Add a new function MAKE-LOAD-TIME-EVAL, defined as follows:

 MAKE-LOAD-TIME-EVAL form			[Function]

 Returns an object which remembers the given FORM and is specially
 recognized by the interpreter and compiler (in a for-evaluation
 position -only-) in the following way:

 If the object is seen by the interpreter for the first time, the
 FORM evaluated in a null lexical environment. The result of the
 evaluation is cached for later  use, and then returned as the
 result of evaluating the object. If the object is later seen again
 by the interpreter, the previously obtained result is immediately
 retrieved and returned as the result of evaluating the object.
 No re-evaluation occurs.

 If the object is seen by the file compiler (eg, COMPILE-FILE) in
 a for-evaluation position, the compiler processes FORM in such a way
 as to arrange for its evaluation at load time in a null lexical
 environment (independent of whether a value has already been cached
 by the interpreter). At runtime, the result of that evaluation will
 be treated as a literal constant.

 If the object is seen by the runtime compiler (ie, COMPILE), the
 compiler checks for a cached value which may have been produced by
 the interpreter. If one is found, it is used. If no such value is
 found, the runtime compiler will evaluate the FORM in a null
 lexical environment and use that value.  The value used will be
 treated as a literal constant in the code which is produced.

 Note that since some implementations are compiled-only (that is, they
 implement their interpreter using a compiler pre-pass) and some are
 interpreted-only (that is, they implement their compiler as a null
 operation and use only an interpreter), the question of whether the
 interpreter or the compiler will end up doing the processing is left
 somewhat vague. The programmer may assume only that the given FORM
 will be evaluated only once for each time it is loaded into a runtime
 environment.

Test Cases:

 (defmacro print-load-timestamp ()
   `(print ,(make-load-time-eval '`(load-timestamp ,(incf *foo*)
						   ,(get-universal-time)))))
 (defvar *foo* 0)
 (defun test-1 () (print-load-timestamp))
 (test-1)

 CLtL does not define this situation.
 Under this proposal, this code would print
   (LOAD-TIMESTAMP 1 <<a-universal-time>>)
 at the time the test case is loaded, whether interpreted or compiled.
 Subsequent calls to (TEST-1) should print the identical expression.

 (defun test-2 () (print #,'(+ 3 4)))

 CLtL does not adequately define this situation.
 Under this proposal, this would print (+ 3 4), whether interpreted
 or compiled.

 (defun test-3 () (print '#,'(+ 3 4)))

 Under CLtL, this would print (+ 3 4).
 Under this proposal, the behavior would be undefined.

Rationale:

 By making the description of MAKE-LOAD-TIME-EVAL be defined only in a
 for-evaluation situation, we eliminate the need for it to take an environment
 argument in order to allow it to guess whether an interpreter-style or
 compiler-style expansion is appropriate. A single expansion can be reliably
 correct.

 By making #, agree with MAKE-LOAD-TIME-EVAL in terms of where it may be used,
 we simplify the description of the resulting language.

 As discussed in more detail elsewhere in this proposal, the #, syntax is
 currently only reliably useful -inside- quoted structure, but this is
 unnecessarily complicated for most known uses. Since this proposal suggests
 a meaning for #, only -outside- quoted structure, it is an incompatible change
 but not one that will generally require vendors to immediately remove support
 for existing code.

Current practice:

 Although most implementations provide a substrate which would allow
 program-mediated access to load time evaluation in some way, the language
 only defines access through the sharpsign read syntax.

 No implementation is required to support TEST-1. Probably none support it in
 exactly the proposed form, though some may provide an analogous extension.

 Most implementations pretend to support TEST-2, though they do not all agree.
 Some compilers complain about the syntax of TEST-2, some arrange for a call
 to it to print (+ 3 4), and some may arrange for a call to it to print the
 result of evaluating (+ 3 4) -- ie, 7.

 Most or all implementations are believed to handle TEST-3 correctly, printing
 (+ 3 4).

Adoption Cost: 

 This proposal will require some implementations to change, but the cost of
 changing should not be high.

 A definition of MAKE-LOAD-TIME-EVAL which should suffice is:

 (DEFVAR *SQUID* (MAKE-SYMBOL "SQUID")
   "SQUID (Self-QUoting Internal Datum) is a Maclisp term.")
 (DEFUN MAKE-LOAD-TIME-EVAL (OBJECT)
   (LIST *SQUID* OBJECT NIL NIL))

 The game then is to make the evaluator do something like:

 (DEFUN SQUID-P (OBJECT) (AND (NOT (ATOM OBJECT)) (EQ (CAR OBJECT) *SQUID*)))
 (DEFVAR *YUCK* (MAKE-SYMBOL "YUCK"))
 (DEFUN *EVAL (OBJECT ENVIRONMENT)
   (COND ...
         ((SQUID-P OBJECT)
          (COND ((CADR OBJECT)
		 (IF (EQ (CADDR OBJECT) *YUCK*)
		     (ERROR "Does anyone think we should worry about this case?")
		     (CADDR OBJECT)))
	        (T
		 (SETF (CADDR OBJECT) *YUCK*)
		 (SETF (CADR OBJECT) T)
	         (SETF (CADDR OBJECT) (EVAL OBJECT)))))
	 ...))

 The compiler, of course, must use SQUID-P and something fairly analogous.

Cost of non-adoption: 

 There are numerous possible uses for this facility. Among them are:
  * Version control and system building facilities.
  * The Common Lisp Object System.
  * Language translators which desire to emulate "linking".
 While every implementation of Common Lisp could certainly provide an
 implementation specific facility, portability would suffer.

Benefits: 

 Portability and extended language power. The nature of the extension is
 such as to enable other extensions to be added more gracefully. The 
 Common Lisp Object System is a clear example.

Conversion Cost:

 The change to #, is an incompatible one, but vendors would be free to
 provide compatibility support for the old behavior for whatever period
 they deemed appropriate.

 In fact, the number of users of #, is likely to be quite small because
 it is a somewhat obscure and hard-to-explain feature, so conversion cost
 should not be a major issue.

Aesthetics:

 This proposal clarifies and regularizes existing parts of the language.
 Also, by adding program-accessible entry points to facilities already
 provided in a more contrived way, it makes the language easier to use.

Discussion:

 There is likely to be some controversy about this proposal, since
 there is no universally agreed upon formal processing model for
 Common Lisp.

 The cleanup committee seems to generally approve of the idea of a
 load-time-eval capability, but a number of the details seem to need
 ironing out.

 Pitman supports this re-draft of Kempf's proposal, but doesn't expect
 this to be the last word on the subject. Surprises are welcome.