Issue: LOAD-TIME-EVAL (Version 5)

[Kent M Pitman <KMP@STONY-BROOK.SCRC.Symbolics.COM>]

Well, this really needs a lot more polish, but I don't have more time to
spend on it today and I wanted to get it out for people to look at
before we meet next week. This is definitely a rough cut and I'm sure it
won't want to go to committee in this form, but at least now all the
proposals are under a single cover and we don't have to play leap-frog
with Moon removing my proposal or my removing his (unless one of us 
gives in).
-----
Issue:		LOAD-TIME-EVAL
References:	#, (p. 356),  (EVAL-WHEN (LOAD) ...) (p. 69-70)
Category:	ADDITION
Edit history:	06-Jun-87, Version 1 by James Kempf
		17-Jul-87, Version 2 by James Kempf
		12-Nov-87, Version 3 by Pitman (alternate direction)
		01-Feb-88, Version 4 by Moon
		  (from version 2 w/ edits suggested by Masinter)
		06-Jun-88, Version 5 by Pitman
		  (fairly major overhaul, merging versions 3 and 4)
Status:		For internal discussion

Problem description:

 Common Lisp provides reader syntax (#,) which allows the programmer
 to designate that a particular expression within a program is to be
 evaluated early (at load time) but to later be treated as a constant.
 Unfortunately, no access to this capability is available to programs
 which construct other programs without going through the reader.
    
 Some computations can be deferred until load time by use of EVAL-WHEN,
 but since EVAL-WHEN must occur only at toplevel, and since the nesting
 behavior of EVAL-WHEN is quite unintuitive, EVAL-WHEN is not a general
 solution to the problem of load-time computation of program constants.

 Also, CLtL is vague about whether the result of this early evaluation
 is re-evaluated at runtime. The meaning of #,exp in an for-evaluation
 position is unclear. Although CLtL doesn't come out and say so explicitly,
 portable code must currently use only '#,exp to get consistent behavior
 across implementations.

 CLtL is also vague on whether the result of a #, expression may be
 treated as a read-only constant by the loader (and hence shared with
 other programs). Users probably want some both read-only and modifiable
 load-time constants, so this may not be simply an issue of deciding on
 a single "right answer".

Proposal (LOAD-TIME-EVAL:QUOTED-MAGIC-TOKEN):

 Add a function MAKE-LOAD-TIME-CONSTANT, as described here:

 MAKE-LOAD-TIME-CONSTANT form env &optional read-only-p		[Function]

   FORM is a Lisp form. ENV is an environment of the sort received
   by the &ENVIRONMENT argument to a macro.

   When MAKE-LOAD-TIME-CONSTANT is called from the interpreter or the
   COMPILE function, it simply evaluates FORM in the null lexical
   environment and returns its value.  When MAKE-LOAD-TIME-CONSTANT is
   called during a file compilation, the result is a special object
   that is recognized at load time, when it occurs inside a constant.
   At load time, FORM is evaluated and its value is substituted for
   the object.

   MAKE-LOAD-TIME-CONSTANT uses its ENV argument and/or dynamic state
   to determine whether it is being called during a file compilation.
   Until Common Lisp is modified to specify the semantics of file
   compilation more precisely, this is necessarily implementation
   dependent.

   The READ-ONLY-P argument designates whether the result can be considered
   read-only constant. If NIL, the result must be considered ordinary,
   modifiable data. If T, the result is a read-only quantity which may, as
   appropriate, be copied into read-only space and/or shared with other
   programs.

 Specify that '(... #,exp ...) is equivalent to
 #.`(... ,(MAKE-LOAD-TIME-CONSTANT exp NIL T) ...).

 Rationale:

   This approach is the most compatible with existing practice.

 Cost to Implementors:

   The cost to implementors will depend on how #, is implemented.
   In some implementations, the primitives for implementing 
   MAKE-LOAD-TIME-CONSTANT may already exist, in others, more substantial
   changes may be required.

 Cost to Users:

   This change is upward compatible with user code.

 Benefits:

   It would be possible for macros to expand into load time constants.

 Examples: 

   Case QUOTED-MAGIC-TOKEN-1:

     (defmacro print-software-version (&environment env)
       `(quote ,(make-load-time-constant
		  '(format T "~A~%" (software-version))
		  env)))

     When interpreted or processed during invocation of COMPILE, this
     macro prints the value of (software-version) at macro expansion
     time and expands into (quote nil).  When macroexpanded during a
     file compilation, printing is deferred until the compiled file is
     loaded, and the constant is still (quote nil).

   Case QUOTED-MAGIC-TOKEN-2:

     (defmacro table-of-tables (&rest predicates &environment env)
       `(quote ,(mapcar #'(lambda (predicate)
			    `(,predicate
			      ,(make-load-time-constant
				 `(make-hash-table :test ',predicate)
				 env)))
			predicates)))

     (table-of-tables eql equal) expands into
     (quote ((eql #<table :test eql>) (equal #<table :test equal>)))
     except that when macroexpanded during a file compilation,
     the tables are not created until load time.  This example
     shows that the <object> returned by make-load-time-constant is
     recognized even when it is interior to a quoted constant.

Proposal (LOAD-TIME-EVAL:NEW-SPECIAL-FORM):
    
 Add a new special form, LOAD-TIME-CONSTANT, which has the following
 contract:

   LOAD-TIME-CONSTANT form &optional read-only-p	[Special Form]

   All processing of the FORM is deferred until the expression is
   in the "runtime" environment. Once that environment is available,
   FORM is evaluated in the null lexical environment and the result
   is both returned and saved for immediate access by the program
   on subsequent evaluations.

   In the interpreter, the FORM may be evaluated during pre-processing
   (if any) or dynamically when the LOAD-TIME-CONSTANT is first seen
   (in a non-pre-processing implementation). If the same LOAD-TIME-CONSTANT
   expression is later seen again by the interpreter, the previously
   obtained result is immediately retrieved and returned as the result
   of evaluating the object; no re-evaluation occurs.
    
   If the LOAD-TIME-CONSTANT expression is seen by the file compiler
   (eg, COMPILE-FILE), the compiler arranges for all semantic processing
   of FORM (including macro expansion) to occur at load time in a null
   lexical environment (independent of whether any value has been cached
   for interpreter use). At runtime, the result of that evaluation will
   be treated as an immediate quantity; no re-evaluation occurs.
    
   If a LOAD-TIME-CONSTANT expression is seen by the runtime compiler
   (eg, COMPILE), the compiler checks for a cached value which may have
   been produced by the interpreter. If one is found, it is used. If no
   such value is found, the runtime compiler will evaluate the FORM in
   a null lexical environment and use that value.  The value used will be
   treated as an immediate quantity in the code which is produced; no
   re-evaluation occurs.   
    
   Note that since some implementations are compiled-only (that is, they
   implement their interpreter using a compiler pre-pass) and some are
   interpreted-only (that is, they implement their compiler as a null
   operation and use only an interpreter), the question of whether the
   interpreter or the compiler will end up doing the processing is left
   somewhat vague. The programmer may assume only that the given FORM
   will be evaluated only once for each time it is loaded into a runtime
   environment.
    
   Note, however, that in the case of quoted code (processed by explicit
   use of EVAL), each call to EVAL is treated like a load. Caching may not
   be implemented by having LOAD-TIME-CONSTANT displace its source level
   call. So while
     (DEFVAR *FOO* 1)
     (DEFUN FOO () (LOAD-TIME-CONSTANT (INCF *FOO*)))
   will increment *FOO* only once,
     (DEFUN FOO () (EVAL '(LOAD-TIME-CONSTANT (INCF *FOO*))))
   will increment *FOO* once each time FOO is called.

   The READ-ONLY-P argument designates whether the result can be considered
   read-only constant. If NIL, the result must be considered ordinary,
   modifiable data. If T, the result is a read-only quantity which may, as
   appropriate, be copied into read-only space and/or shared with other
   programs. (Because this is a special form, this argument is -not- evaluated
   and only the literal symbols T and NIL are permitted.)

 Make #,exp be equivalent to (LOAD-TIME-CONSTANT exp T). As such, it
 would -always- appear in a for-evaluation position, and never inside quoted
 structure.

 Rationale:
    
   By making the description of LOAD-TIME-CONSTANT defined as a special
   form, we eliminate the need for it to take an environment argument.

   By making #, agree with LOAD-TIME-CONSTANT in terms of where it may be
   used, we simplify the description of the resulting language.

   As discussed in more detail elsewhere in this proposal, the #, syntax
   is currently only reliably useful -inside- quoted structure, but this
   is unnecessarily complicated for most known uses. Since this proposal
   suggests a meaning for #, only -outside- quoted structure, it is an
   incompatible change (though not one that would necessarily require
   vendors to immediately remove support for existing code).

 Cost to Implementors:

   This is an incompatible change to the program interface.
   The cost is not trivial, but is not particularly high.
   Most of the "hard" substrate required to support this proposal
   probably already exist; in most cases, what needs to change is
   only the way in which the substrate is presented to the
   programmer.

   Some code-walkers would have to be taught about this new
   special form. Such changes would likely be trivial.

 Cost to Users:

   Very few users probably use #, right now. A very small amount of code
   might need to be rewritten (and recompiled).

   The following kinds rewrites exemplify the total amount of work needed
   in the few situations that use #, currently:
      '#,exp           => #,exp
      '(... #,exp ...) => #,`(... ,exp ...)
      `(... #,exp ...) => `(... ,#,exp ...)

   Some user-defined code-walkers would have to be taught about
   this new special form. Such changes would also be likely be trivial.

   Although change to #, is an incompatible one, vendors would be free to
   provide compatibility support for the old behavior for whatever period
   they deemed appropriate. #, is equivalent to, but not the same as,
   (LOAD-TIME-CONSTANT exp T), so it might in fact expand into
   (SYSTEM::LOAD-TIME-CONSTANT exp T) which might be both a macro that
   expanded into (LOAD-TIME-CONSTANT exp T). SYSTEM::LOAD-TIME-CONSTANT
   might be a magic token which was treated specially within quoted 
   structure as an unportable extension during a transition period.

 Benefits:

   Relatively consistent interpreter/compiler treatment of this special form
   would be possible.

   Pretty printing expressions in which LOAD-TIME-CONSTANT occurred would be
   possible.

   An expression such as `(... #,(... , ...) ...) would be meaningful.

   Manipulating expressions involving uses of LOAD-TIME-CONSTANT would be
   possible. Currently, calling READ is enough to trigger resolution of the
   constant, so a program doing file-to-file source code rewrites must
   either use a special readtable or resign itself to premature resolution
   of load time constants.

   Expressions involving quoted LOAD-TIME-CONSTANT expressions would be
   possible to quote. Currently, the #, feature is syntactic, not semantic,
   and so is not sensitive to levels of quotation. You can refer to a
   load time constant by writing '#,exp but you cannot refer in turn to that
   expression which refers to a load time constant by writing ''#,exp .
   Under this NEW-SPECIAL-FORM proposal #,exp and '#,exp and ''#,exp (and
   so on) are all usefully distinct.

   It would be possible to use , in a #, expression. For example, the expression:

 Examples:

   Case NEW-SPECIAL-FORM-1:

     (defmacro print-load-timestamp ()
       `(print (load-time-constant
		 `(load-timestamp ,(incf *foo*) ,(get-universal-time))
		  t)))
     (defvar *foo* 0)
     (defun test-1 () (print-load-timestamp))
     (test-1)
      
     CLtL does not define this situation.
     Under this proposal, this code would print
	(LOAD-TIMESTAMP 1 <<a-universal-time>>)
     at the time the test case is loaded, whether interpreted or compiled.
     Subsequent calls to (TEST-1) should print the identical expression.
     Currently, no known implementation supports the proposed behavior.

  Case NEW-SPECIAL-FORM-2:

     (defun test-2 () (print #,'(+ 3 4)))
    
     CLtL does not adequately define this situation.
     Under this proposal, this would print (+ 3 4), whether interpreted
     or compiled.

     Currently, some compilers complain about the syntax, some arrange for
     it to print (+ 3 4), and some arrange forit to print the result of
     (+ 3 4), or 7.
    
  Case NEW-SPECIAL-FORM-3:

     (defun test-3 () (print '#,'(+ 3 4)))
    
     Under CLtL, this would print (+ 3 4).
     Under this proposal, the behavior would be undefined.
     Currently, most implementations support the proposed behavior.
    
  Case NEW-SPECIAL-FORM-4:

     (pprint '(+ x #,(sqrt x)))

     prints something re-readable like
	(+ X #,(SQRT 2)) or (+ X (LOAD-TIME-CONSTANT (SQRT 2) T))
     but not something like
	(+ X 1.4142135)
     Currently, no implementation is known to already support the
     proposed behavior, but in principle it is possible for a valid
     implementation to already do so.

   Case NEW-SPECIAL-FORM-5:

     (defmacro foo (x y)
       `(member ,x #,`(foo ,,y baz)))
     (macroexpand '(foo *bar*)) => ??

     Under CLtL, this situation is not adequately defined.
     Under this proposal, the macroexpansion would be 
     (member foo #,`(foo ,*bar* baz))
     Currently, this triggers a read error such as "comma not in backquote"
     or "SYSTEM::BACKQUOTE-COMMA undefined function" in most
     implementations.

Current practice:

 Although most implementations provide a substrate which would allow
 program-mediated access to load time evaluation in some way, the language
 only defines access to this substrate through the sharpsign read syntax.

Costs of Non-Adoption: 

 There are numerous possible uses for this facility. Among them are:
   * Version control and system building facilities.
   * The Common Lisp Object System.
   * Language translators which desire to emulate "linking".
  While every implementation of Common Lisp could certainly provide an
  implementation specific facility capable of supporting such facilities,
  portability of such facilities would suffer.
  
Benefits:

 Portability and extended language power.  The nature of each proposed
 extension is such as to enable other extensions to be added more
 gracefully. The Common Lisp Object System is a clear example.

Aesthetics:
    
 These proposals fill a hole in the spectrum of alternatives for
 deferring evaluation until a compiled file is loaded. Currently, code
 which is read by the reader can arrange for it to be done, as can
 top level code, but embedded code cannot. As such, these proposals
 clarify and regularize existing parts of the language. Also, by
 adding program-accessible entry points to facilities already provided
 in a more contrived way, it makes the language easier to use.

Discussion:
    
 There is likely to be some controversy about this proposal, since
 there is no universally agreed upon formal processing model for
 Common Lisp.
    
 The cleanup committee seems to generally approve of the idea of a
 load-time-eval capability, but a number of the details seem to need
 ironing out.
    
 Moon supported a previous draft of QUOTED-MAGIC-TOKEN. In this draft,
 KMP changed the presentation and also added the READ-ONLY-P argument
 in order to make it sit nicely with the alternate proposal, 
 NEW-SPECIAL-FORM. It's more than slightly possible that after all this
 editing, Moon will have some problems with this version and want to
 submit a refined draft.

 Pitman supports NEW-SPECIAL-FORM.

 Rees has expressed strong support for the idea of implementing #, as a 
 new special form rather than perpetuating the current state of affairs.
 He had some input into the high-level content of this proposal, though
 he hasn't reviewed any drafts. This paragraph is intended primarily to
 incite him to say something definitive one way or the other.