Issue COMPILER-LET-CONFUSION, version 1

[sandra%defun@cs.utah.edu (Sandra J Loosemore)]

Here is a first pass at a writeup for this issue.  I've included four
different proposals which I hope cover all the possibilities.  (I also
hope my writeup is unbiased enough to live up to KMP's exacting
standards!)

As usual, comments and complaints are welcome.

-Sandra



Issue:		COMPILER-LET-CONFUSION
References:	CLtL p. 112
Category:	CHANGE/CLARIFICATION
Edit History:   V1, 27 Sept 1988, Sandra Loosemore (initial version)
Status:		**DRAFT**


Problem Description:

The description of the COMPILER-LET special form in CLtL is confusing
to many people.  There are no examples provided to make it clear how it
is supposed to be used.

The description of how COMPILER-LET works in the interpreter only
makes sense for implementations which perform macroexpansion in
parallel with evaluation.  In an implementation which performs macro
expansion in a prepass and follows CLtL literally in making
COMPILER-LET behave "exactly like LET with all the variable bindings
implicitly declared SPECIAL", it would not work at all for its stated
purpose, "communication among complicated macros".

Subtle bugs can be introduced because of the different handling of the
variable bindings in the interpreter and the compiler.  In compiled
code, the bindings are only lexically visible during the expansion of
macros at compile time, while in interpreted code the bindings have
dynamic scope and may also be seen during ordinary evaluation if
evaluation and macroexpansion happen concurrently.

Further compatibility problems can result from the value forms being
evaluated in a null lexical environment in the compiler and the
ordinary lexical environment in the interpreter.



Proposal COMPILER-LET-CONFUSION:CLARIFY-STATUS-QUO:

  Change the documentation of COMPILER-LET to include more discussion of
  its intended purpose, including examples.
  
  Change the description of COMPILER-LET's behavior in the interpreter
  to indicate that it is not "exactly like LET with all the variable
  bindings implicitly declared SPECIAL".  Instead, clarify that
  processing of COMPILER-LET by the interpreter happens at the same time
  as macroexpansion; the value forms are evaluated sequentially
  in a null lexical environment, bound to the special variables, and
  that these bindings are available during the expansion of macro calls
  in the body.
  
  Mention that there may be compatibility problems between compiled and
  interpreted code if the interpreter performs macroexpansion in
  parallel with evaluation.

  
  Rationale:

  This proposal formalizes what appears to be the behavior of most
  implementations.
  

  Current Practice:
  
  COMPILER-LET is rarely used.
  
  
  Cost to implementors:

  Minimal.

  
  Cost to users:
  
  None.  Any code that would break under the new rules for the interpreted
  behavior of COMPILER-LET would already be broken when compiled.


  Benefits:
  
  The description of COMPILER-LET would make more sense for implementations
  that perform macroexpansion during a prepass in the interpreter.  Some of
  the compatibility problems between interpreted and compiled code would be
  removed.



Proposal COMPILER-LET-CONFUSION:REQUIRE-PREPASS:

  Disallow interpreters from performing macroexpansion concurrently with
  evaluation.  Macroexpansion must be performed in a preprocessing
  codewalk.

  Clarify that COMPILER-LET is processed by the interpreter or compiler
  (the "processor") at the same time that macros are expanded.  The
  processor sequentially evaluates each of the value forms in a null
  lexical environment, and the special variables are bound to these values
  during the expansion of any macro calls in the body.
  

  Rationale:
  
  This proposal forces the interpreter to treat both macroexpansion and
  COMPILER-LET in the same way that the compiler does.
  
  
  Current Practice:
  
  COMPILER-LET is rarely used.
  
  
  Cost to implementors:
  
  For implementations that do not implement their interpreters using a
  prepass, this would be a fairly substantial change.
  
  
  Cost to users:
  
  Strictly speaking, this is a compatible change to the language.  Since
  CLtL makes no guarantees about the time of macroexpansion, any user
  code that depends upon macroexpansion happening concurrently with
  evaluation is nonportable.  Some users, however, dislike
  implementations with preprocessing interpreters, and want to be able
  to freely redefine macros as well as functions.

  
  Benefits:
  
  An area of incompatibility between compiled and interpreted code is
  eliminated.  This proposal would also address some other compatibility
  problems beyond those relating to COMPILER-LET (such as issue
  LOAD-TIME-EVAL).


Proposal COMPILER-LET-CONFUSION:ELIMINATE:

  Remove COMPILER-LET from the language.
  
  
  Rationale:
  
  Some people think that COMPILER-LET is ugly.  Removing it from the
  language is a much simpler solution than an elaborate proposal to
  assign it consistent semantics in the interpreter and compiler.
  
  
  Current Practice:
  
  COMPILER-LET is rarely used.  
  
  
  Cost to implementors:
  
  Minimal.  Implementations could continue to support COMPILER-LET as
  an extension.
  
  
  Cost to users:
  
  People who use COMPILER-LET would have to rewrite their programs to use
  some other construct.  Most uses of COMPILER-LET for communication between
  macros can be handled using MACROLET instead.
  
  I have been able to do this quite easily for all of the examples which I
  have seen so far.  For example:
  
    (defvar *local-type-declarations* '())
     
    (defmacro local-type-declare (declarations &body forms)
      `(compiler-let ((*local-type-declarations* 
                        (append ',declarations *local-type-declarations*)))
         ,@forms))
     
    (defmacro typed-var (var)
       (let ((type (assoc var *local-type-declarations*)))
         (if type `(the ,(cadr type) ,var) var)))
     
    (defun f (x y)
      (local-type-declare ((x fixnum) (y float))
        (+ (typed-var x) (typed-var y))))
     
    
  can be rewritten as:
  
    (defmacro local-type-declare (declarations &body forms)
        (local-type-declare-aux declarations forms))
    
    (eval-when (eval compile load)
        (defun local-type-declare-aux (declarations forms)
    	`(macrolet ((typed-var (var)
    			(let ((type  (assoc var ',declarations)))
    			    (if type `(the ,(cadr type) ,var) var)))
    		    (local-type-declare (new-declarations &body new-forms)
    			(local-type-declare-aux
    			    (append new-declarations ',declarations)
    			    new-forms)))
    	     ,@forms)))
    
    
  The MACROLET versions are usually somewhat more complicated than the
  COMPILER-LET versions, but not substantially so unless there are a large
  number of macros involved.
  
  
  Benefits:
  
  Having one less special form would simplify the language.  An area of
  incompatibility between compiled and interpreted code would be
  eliminated.



Proposal COMPILER-LET-CONFUSION:REDEFINE:

  Clarify the description of COMPILER-LET to emphasize that it is used to
  make bindings of special variables available when macro calls appearing in
  the body are expanded.
  
  Clarify that COMPILER-LET is processed by the interpreter or compiler
  (the "processor") at the same time that macros are expanded.  The
  processor sequentially evaluates each of the value forms in a null
  lexical environment, storing the variable/value pairs in the
  environment.
  
  Extend the description of environment objects captured with &ENVIRONMENT 
  to state that they also contain information about bindings indicated with 
  COMPILER-LET.
  
  Clarify that MACROEXPAND and MACROEXPAND-1 use the information stored in
  the environment argument to actually perform the special bindings of
  the variables to the values.
  
  Clarify that the special bindings indicated by COMPILER-LET are also
  visible during the evaluation of value forms of lexically nested 
  COMPILER-LET constructs.  The special bindings are not present during
  any other compile-time evaluation of the body, such as evaluation of
  (EVAL-WHEN (COMPILE) ...) forms.
  
  
  Rationale:
  
  This proposal provides consistent semantics for COMPILER-LET in both the
  compiler and interpreter.  It will work just as well in interpreters that
  perform macroexpansion in parallel with execution, as in those that perform
  macroexpansion in a prepass.
  
  
  Current Practice:
  
  COMPILER-LET is rarely used.
  
  
  Cost to implementors:
  
  Implementors will have to change their interpreters and compilers to
  have COMPILER-LET store things in the environment.  The representation
  of environment objects may need to change slightly.  Implementations
  in which &ENVIRONMENT does not return a complete environment object
  will have to be changed.  MACROEXPAND and MACROEXPAND-1 will have to
  be changed to rebind the variables from the environment.
  
  
  Cost to users:
  
  User programs that perform code walks may have to be changed; the
  changes would roughly parallel those listed above.

  
  Benefits:
  
  The treatment of this special form by the interpreter is made consistent
  with its treatment by the compiler.




Discussion:

Can any of you implementors who have an interpreter that perform
macroexpansion in a preprocessing phase comment on how you handle
COMPILER-LET?

Proposal COMPILER-LET-CONFUSION:CLARIFY-STATUS-QUO is the alternative
which is the most consistent with current practice and requires the
least change for both implementors and users.  However, it addresses only
some of the problems which can lead to incompatibility between compiled
and interpreted code.

Proposal COMPILER-LET-CONFUSION:REQUIRE-PREPASS would ensure
consistent semantics for macroexpansion as well as processing of
COMPILER-LET.  On the negative side, it is a substantial change for
some implementations.

Proposal COMPILER-LET-CONFUSION:ELIMINATE is by far the simplest
alternative, but it would also has the greatest impact on users.

Proposal COMPILER-LET-CONFUSION:REDEFINE also guarantees consistent
semantics for COMPILER-LET between the interpreter and compiler.  The
disadvantages of this proposal are that COMPILER-LET remains
complicated and potentially confusing, and it requires changes in both
user and system code.
-------