issue COMPILER-LET-CONFUSION, V3

[sandra%defun@cs.utah.edu (Sandra J Loosemore)]

Here is the latest version.  As I said I would do earlier, all but
proposal ELIMINATE have been removed.  This is another issue I would
like to put on a mail ballot in December, so please say something soon
if you have problems with the writeup. 

-Sandra



Issue:		COMPILER-LET-CONFUSION
References:	CLtL p. 112
Category:	CHANGE
Edit History:   V1, 27 Sep 1988, Sandra Loosemore (initial version)
                V2, 04 Oct 1988, Sandra Loosemore (add another example)
		V3, 31 Oct 1988, Sandra Loosemore (only proposal ELIMINATE)


Problem Description:

The description of the COMPILER-LET special form in CLtL is confusing
to many people.  There are no examples provided to make it clear how it
is supposed to be used.

The description of how COMPILER-LET works in the interpreter only
makes sense for implementations which perform macroexpansion in
parallel with evaluation.  In an implementation which performs macro
expansion in a prepass and follows CLtL literally in making
COMPILER-LET behave "exactly like LET with all the variable bindings
implicitly declared SPECIAL", it would not work at all for its stated
purpose, "communication among complicated macros".

Subtle bugs can be introduced because of the different handling of the
variable bindings in the interpreter and the compiler.  In compiled
code, the bindings are only lexically visible during the expansion of
macros at compile time, while in interpreted code the bindings have
dynamic scope and may also be seen during ordinary evaluation if
evaluation and macroexpansion happen concurrently.

Further compatibility problems can result from the value forms being
evaluated in a null lexical environment in the compiler and the
ordinary lexical environment in the interpreter.



Proposal COMPILER-LET-CONFUSION:ELIMINATE:

  Remove COMPILER-LET from the language.
  
  
  Rationale:
  
  Some people think that COMPILER-LET is ugly.  Removing it from the
  language is a much simpler solution than an elaborate proposal to
  assign it consistent semantics in the interpreter and compiler.
  
  
  Current Practice:
  
  COMPILER-LET is rarely used.  
  
  
  Cost to implementors:
  
  Minimal.  Implementations could continue to support COMPILER-LET as
  an extension.
  
  
  Cost to users:
  
  People who use COMPILER-LET would have to rewrite their programs to use
  some other construct.  Most uses of COMPILER-LET for communication between
  macros can be handled using MACROLET instead.
  
  I have been able to do this quite easily for all of the examples which I
  have seen so far.  For example:
  
    (defvar *local-type-declarations* '())
     
    (defmacro local-type-declare (declarations &body forms)
      `(compiler-let ((*local-type-declarations* 
                        (append ',declarations *local-type-declarations*)))
         ,@forms))
     
    (defmacro typed-var (var)
       (let ((type (assoc var *local-type-declarations*)))
         (if type `(the ,(cadr type) ,var) var)))
     
    (defun f (x y)
      (local-type-declare ((x fixnum) (y float))
        (+ (typed-var x) (typed-var y))))
     
    
  can be rewritten as:
  
    (defmacro local-type-declare (declarations &body forms)
        (local-type-declare-aux declarations forms))
    
    (eval-when (eval compile load)
        (defun local-type-declare-aux (declarations forms)
    	`(macrolet ((typed-var (var)
    			(let ((type  (assoc var ',declarations)))
    			    (if type `(the ,(cadr type) ,var) var)))
    		    (local-type-declare (new-declarations &body new-forms)
    			(local-type-declare-aux
    			    (append new-declarations ',declarations)
    			    new-forms)))
    	     ,@forms)))
    
    
  The MACROLET versions are usually somewhat more complicated than the
  COMPILER-LET versions, but not substantially so unless there are a large
  number of macros involved.

  Another approach for converting old code is to define a
  COMPILER-LET-like macro to explicitly make the special variable
  bindings available during the expansion of a specified set of macros.
  The following macro FAKE-COMPILER-LET makes the special variable
  bindings visible during the expansion of the named macros, and during
  the evaluation of nested FAKE-COMPILER-LETs.

    ;;; Imitation COMPILER-LET.  Second argument is a list of macros which
    ;;;    are to see the special bindings; these must have been defined using
    ;;;    DEFMACRO, not MACROLET (because expansion takes place in null
    ;;;    lexical environment).
    
    (defmacro fake-compiler-let (binding-forms macros &body body)
        (expand-fake-compiler-let binding-forms macros body))
    
    (eval-when (eval compile load)
        (defun expand-fake-compiler-let (binding-forms macros body)
            (let* ((vars    (mapcar #'(lambda (b)
                                          (if (consp b) (car b) b))
                                    binding-forms))
                   (vals    (mapcar #'(lambda (b)
                                          (if (consp b) (eval (cadr b)) nil))
                                    binding-forms))
                   (binders (mapcar #'(lambda (var val)
                                          `(,var ',val))
                                    vars vals))
                   (defs    (mapcar #'(lambda (m)
                                          `(,m (&whole w)
                                               (let ,binders
                                                   (declare (special ,@vars))
                                                   (macroexpand-1 w))))
                                    macros)))
                `(macrolet ((fake-compiler-let (binding-forms macros
						    &body body)
                                (let ,binders
                                    (declare (special ,@vars))
                                    (expand-fake-compiler-let
                                        binding-forms macros body)))
                            ,@defs)
                     ,@body)))
        )
    
    
    ;;; Example to illustrate nesting behavior
    
    (eval-when (eval compile)
        (defvar *depth* 0)
        (defmacro current-depth ()
            *depth*)
        )
    
    (fake-compiler-let ((*depth* (1+ *depth*)))
                       (current-depth)
        (format t "First value = ~s~%" (current-depth))
        (fake-compiler-let ((*depth* (1+ *depth*)))
                           (current-depth)
            (format t "Second value = ~s~%" (current-depth))))
  

  Benefits:
  
  Having one less special form would simplify the language.  An area of
  incompatibility between compiled and interpreted code would be
  eliminated.



Discussion:

Three other proposals on this issue were also considered.  One of
these, CLARIFY-STATUS-QUO, becomes the default if proposal ELIMINATE
is rejected.  Essentially, it would clarify that processing of
COMPILER-LET by the interpreter happens at the same time as
macroexpansion; the value forms are evaluated sequentially in a null
lexical environment, bound to the special variables, and that these
bindings are available during the expansion of macro calls in the
body.  Some mention should also be made in the standard of the
compatibility problems between compiled and interpreted code.

The other two proposals were REQUIRE-PREPASS (to require all intepreters
to perform macroexpansion and processing of COMPILER-LETs in a code-walking
prepass) and REDEFINE (to assign somewhat different semantics to
COMPILER-LET).  Neither of these proposals appeared to have very much
support.

Proposal COMPILER-LET-CONFUSION:ELIMINATE appears to have the most
support among members of the compiler committee; JonL (and others at
Lucid), Loosemore, Van Roggen, Dalton, and Perdue have all expressed
support for the idea of getting rid of COMPILER-LET.  Since this is an
incompatible change, it is felt that a substantial effort should be
made to help users convert old code.
-------