Issue: LET-TOP-LEVEL (version 1)

[Jon L White <edsel!jonl@labrea.Stanford.EDU>]

re:     This is perhaps a bad example of what you are trying to illustrate, 
        for it needlesly violates the "global environment" constraint of 
        CLtL p145.
    On the contrary, violation of that constraint was precisely the topic
    I was trying to illustrate.  Removal of that constraint was the subject
    under discussion. 
        Since you are merely trying to show up a flaw in the "pre-processor"...
    That is not at all what I was trying to do.  

Yes, Dave, I know that was not the main point of this series of discussions;
and I'm fully aware that you were trying to find another example where the
extent of the lexical environment for the expander function is critical.
I'm sorry for characterizing your comments so flippantly.  What I was trying 
to point out to you was that the problems of pre-processors occur long before
you get to the captured lexical environment problem.  Since defmacros in 
"the same form" as their clients don't in general work right during this 
kind of pre-processing, I think it's moot to talk about the additional 
problems of capturing lexical variables in such macros.

Thus your example:
	(defun foo (x) (macrolet ((baz (x y)
			(print `(baz ,x ,y expanded))
			`(list ,x ,y)))
		         (lambda (z) (baz x z))))
nicely illustrates the need for indefinite extent of the lexical environment
containing the argument to FOO;  but the example called "nonsensical program"
fails to make a similar point so strikingly, since attempts to compile or 
pre-process it break down (in the typical pre-processor) regardless of the
lexical variable questions.


re:     (defun tryit () 
	  (compiler-let ((cnt 0))
	    (compiler-let ((throw-away
			     (progn 
			       (defmacro do-it-once (x) (incf cnt) `(QUOTE ,x))
			       (defmacro total-count () `(QUOTE ,cnt))
			       t)))
	      (defun foo (x)
		(do-it-once x) 
		(do-it-once x)
		(total-count)))))

    I don't understand how this is relevant, since compiler-let is an operation
    on dynamic variables, not lexical variables.

Ah, I thought this little cleverity might go over peoples' heads.  The
issue being illustrated is how 'compiler-let' is a loophole, permitting 
backdoor calls to EVAL during the simple compilation of a function.  THERE 
IS NO EXPECTATION THE VARIABLES USED ARE LEXICAL.  The time-of-evaluation
issues related to "pre-processing" and/or compilation can to some degree be 
circumvented by this loophole;  thus there is no absolute need to put macros 
in a file (or separate lexical environment) in order to get them to expand 
"right". 

Now it certainly would be a different question if the variable CNT in this
TRYIT example were referenced freely by the DEFUN for FOO.  Then it would
be all very interesting to ask what the program might mean if the binding 
for CNT were with a LET rather than a compiler LET (but the binding for 
THROW-AWAY would still be a COMPILER-LET).  This might, I think, illustrate
the weird situation that you have been hinting at, where there is a mismatch
of times at which a lexical environment must exist.  [Note, of course, that
there is no "file" involved, so that not all of the issues called "Toplevel"
are restrictd to file compilations.]

However, your original "nonsensical program" doesn't have this property:
      (let ((x (big-computation-that-returns-22/7)))
	(defmacro foo (y) `(list ,x ,y))
	(defun bar (z) (foo (1+ z))))
Notice that at eval time, (FOO (1+ Z)) expands as the equivalent:
	(defun bar (z) (list 22/7 (1+ z)))
and this form doesn't have any pointers to the lexical environment outside 
of the defun.  And, conceivably, if "TopLevel" were to "pass thru" a LET, 
then this would be the same situation at compile time.

IF instead, you had give the example as:
	(defun bar (z) (list x (foo (1+ z))))
THEN it would illustrate the issue to be clarified, for then the 
macroexpansion would lead to:
	(defun bar (z) (list x (list 22/7 (1+ z))))
and "x" is the captured lexical variable whose compile-time/load-time
existence is in question.  But I don't see this inconsistency as much
worse than the fact that macro expansions will "freeze" values at compile
time which would be different had the expansion function been run at a 
later time (say, load time).


re: You are correct in (implicitly) pointing out that this claim of Pavel's
    should have been limited to the case where the non-null closed-over
    environment (really the captured free identifiers) came from a load-time
    binding rather than a compile-time binding. 

This clarification of the "claim" reduces its contention to the situation
where there is something like the ambiguity of compile-time/load-time just
mentioned.  The TRYIT example says nothing about the reduced claim.
However, it still leaves open the question of what to do at compile time.


As I recall the discussions on this matter over the past years, I see:

  (1) Many users who claim that having the defmacro expander function be
      treated like any other form (i.e., its lexical environment is the 
      same as the surrounding context) is a very useful capabililty that
      they full well understand how to use without causing problems;

  (2) Attempts to illustrate an ambiguous extent situation for the lexical
      environment of defmacros are not "garden variety" code, and are in 
      general subsidary to the problems of specifying even when the global 
      environment is to be changed [i.e., just when is the macro-function 
      "cell" named by a defmacro to be updated?].

To me, this looks like mounting evidence that retracting the special 
limitation on defmacro would be in the best interests of the community.
On the other hand, I see stability as an even more paramount concern,
particularly since the user who wants the capability currently denied
him for defmacro can simply "hand-expand" the defmacro into a setf of
macro-function.  No easy choice, again.



-- JonL --


P.S.: I also don't see future extensions to "TopLevel" being done in such a
      way so that it will "pass thru" a LET.  It's nice to contemplate this,
      but I suspect it will cause more problems than it solves.