Re: issue COMPILER-LET-CONFUSION, version 6

[Jeff Dalton <jeff%aiai.edinburgh.ac.uk@NSS.Cs.Ucl.AC.UK>]

> Problem Description:
> 
>  The description of the COMPILER-LET special form in CLtL is confusing
>  to many people.  There are no examples provided to make it clear how it
>  is supposed to be used. The only description which is offered is overly
>  concrete, which have led to confusion about the intent of COMPILER-LET,
                   â??â??â??â?? has
>  and about its implementability.
>  
>  The intent of COMPILER-LET was to permit information to be communicated
>  between macros by use of dynamic variables at macroexpansion time.
>  It was not necessary to the intended uses of COMPILER-LET that such
>  variables ever be bound at execution time.  
>  
>  Unfortunately, probably because some implementations did not primitively
>  support COMPILER-LET at the time CLtL was written, an exception was 
>  permitted to make COMPILER-LET `more or less work' in interpreters:

Is it really "interpreters" (verses some unspecified other thing), or
macroexpansion time verses executiontime?

>  the COMPILER-LET variables were permitted to be bound at execution time.
>  The problem was further compounded by the fact that CLtL presented this
>  exception as part of COMPILER-LET's contract rather than as an 
>  implementation note, and by the fact that no examples of actually using
>  COMPILER-LET correctly are provided.
>  
>  Subtle bugs can be introduced because of the different handling of the
>  variable bindings in the interpreter and the compiler.  In compiled
>  code, the bindings are only lexically visible during the expansion of
>  macros at compile time, while in interpreted code the bindings have
>  dynamic scope and may also be seen during ordinary evaluation if
>  evaluation and macroexpansion happen concurrently.

Concurrently?

>  Further compatibility problems can result from the value forms being
>  evaluated in a null lexical environment in the compiler and the ordinary
>  lexical environment in the interpreter.

There's nothing here about occasions such as
  (compiler-let ((*v* 1))
    #'(lambda () (m)))
where M is a macro that refers to *V*, but the lambda-expr isn't
macroexpanded until it's finally called, and so parhpas after the
dynamic extent of the *V* binding has ended.  Isn't this part of
the "problem"?

> Background and Analysis:

>  [...]

>  Opinion is divided as to which is more understandable.  Some
>  people find the COMPILER-LET idiom more understandable, while others
>  find it just as natural to use MACROLET or SYMBOL-MACROLET.

There's also the problem that the COMPILER-LET may not work, due
to the extent problem noted above.

> Proposal (COMPILER-LET-CONFUSION:REPAIR):

>     [...]

>     In interpreters which do not do a semantic-prepass, it is necessary
>     to fully macroexpand the body. Assuming the presence of a
>     SYSTEM::MACROEXPAND-ALL primitive, the definition of COMPILER-LET
>     could look like:
>       (DEFMACRO COMPILER-LET (BINDINGS &BODY FORMS &ENVIRONMENT ENV)
>         (SETQ BINDINGS ;; Assure no non-atom bindings
> 	      (MAPCAR #'(LAMBDA (BINDING) 
> 		          (IF (ATOM BINDING) (LIST BINDING) BINDING))
> 		      BINDINGS))
>         (PROGV (MAPCAR #'CAR BINDINGS)
> 	       (MAPCAR #'CDR BINDINGS)
> 	  (SYSTEM::MACROEXPAND-ALL `(PROGN ,@FORMS) ENV)))
>     This reduces the problem of writing a program capable of doing a
>     full macroexpansion. Many systems already have such a facility.
>     Pitman wrote such a facility in Cloe Runtime in order support 
>     SYMBOL-MACROLET (before it was christened a special form); it was
>     about 750 lines of relatively straightforward, well-commented code.

The need for a prepass (why "semantic prepass"?) isn't explained,
and there's nothing in the problem description (about the extent issue,
say) that indicates there's a problem the prepass solves.

>   Cost to Users:
> 
>     Code currently depending on this feature is either non-existent or
>     already not portable (due to wide variation in implementation 
>     strategy for COMPILER-LET).
> 
>     Most users will probably be happy for any interpretation which offers
>     them a future shot at portability.
> 
>     Some users have indicated they dislike interpreters which do a semantic
>     prepass, because they like to be able to dynamically redefine macros
>     while debugging.

Can macros be expanded to something that records the COMPILER-LET
nevironment so that they could be correctly re-expanded later?  For
example, suppose macro expansions were cached (in a hash table, say)
and the original code left in place.  Then, if the macro is redefined,
the expansion is recomputed.  The C-LET environment could be saved
along with the expansion.  Would that work?  

> Proposal (COMPILER-LET-CONFUSION:ELIMINATE):
> 
>   Remove COMPILER-LET from the language.
>   
>   Rationale:
> 
>     Some people think that having one less special form would simplify the
>     language.  The revised COMPILER-LET semantics, which require
>     COMPILER-LET to make special bindings which are only lexically visible

"only lexically visible" isn't quite right.  It sounds like you're
saying they have lexical scope, but they don't because they're visible
to macro functions.  So I think they have indefinite scope and dynamic
extent, like all special variables, but the extent is macroexpansion
time rather than execution time.

>     within its body, are not shared by any other feature in the language,
>     and require a fairly complex implementation technique.  There are
>     other constructs which are strictly lexical that can be readily used
>     to solve the same kinds of problems that COMPILER-LET is intended to
>     be used for.

>     [...]


> Current Practice:
>   
>  Some implementations have implemented the description in CLtL. 
>  Users of those implementations (quite reasonably) can't figure how to 
>  use COMPILER-LET and so don't use it much.
> 
>  Some implementations (the ones from which COMPILER-LET originally came)
>  continue to use their pre-CLtL semantics. These semantics are useful, though
>  incompatible with CLtL (which they largely consider to simply be in error).

I always find this mysterious because I don't know what the "pre CLtL"
semantics was (if it's diffeerent).

>  Users of those implementations probably use COMPILER-LET somewhat more 
>  often since it has an intelligible behavior, but their code is not portable
>  since it relies on behaviors which are either contrary to or not guaranteed
>  by CLtL.

>  [...]

-- Jeff