Issue: PROCLAIM-LEXICAL

[Jonathan A Rees <JAR@AI.AI.MIT.EDU>]

Issue:        PROCLAIM-LEXICAL
References:   variables (p55), scope/extent (p37), global variables (p68),
	      declaration specifiers (p157)
Category:     CLARIFICATION/ENHANCEMENT
Edit history: Revision 2 by JAR 04/28/87
Status:       Very preliminary.

Description of problem:

  CLtL pp. 55-56 implies that if a name (symbol) is not proclaimed or
  declared special, then a free reference to that name is a reference to
  the special variable of that name, while a LAMBDA-binding of that name
  indicates a binding of the lexical variable of that name.  This would
  mean that the following program is legal and that (TST) => 4:

    (defun tst ()
      (setq x 3)
      (funcall (let ((x 4)) #'(lambda () x))))

  However, if you feed this program to many Common Lisp compilers
  (including Symbolics's and DEC's), a warning message will be
  produced for the SETQ, saying something like "Warning: X not
  declared or bound, assuming special."

  These warnings, unlike the annotations of undefined functions (which
  occur only at the end of a compilation), are presented so
  prominently that a user would be hard put to say that a program
  which elicited such warning messages was "correct" in that
  implementation.  Unlike the situation with unused variables, there is
  no possible declaration one can write which suppresses the warning
  messages.

  This disagreement between theory and practice should be mended
  somehow.

Proposal (PROCLAIM-LEXICAL:CURRENT-PRACTICE):

  Change the language definition (page 55?) to say that it is an error
  for there to be a free reference or assignment to a name unless a
  SPECIAL proclamation or declaration is in effect for that name.

  This would legitimize the behavior of current implementations.

Proposal (PROCLAIM-LEXICAL:BY-THE-BOOK):

  Shame implementors into going by the book.  Implementations should
  simply stop intimidating users who want to write code like this.
  "Apparently unbound variable" warnings should be given the same
  purely advisory status that "apparently undefined function" warnings
  now enjoy.  The exact meaning of this is of course implementation-
  dependent.

Proposal (PROCLAIM-LEXICAL:GENERAL):

  Introduce a new declaration specifier, LEXICAL, which is dual to the
  SPECIAL declaration specifier; it may appear in proclamations and
  declarations.

  A name may be proclaimed either lexical or special, but not both.

  A free reference or assignment to a name is an error if there is
  neither a SPECIAL nor a LEXICAL proclamation or declaration in
  effect for the name.  A LAMBDA-binding in the absence of a
  declaration or proclamation binds the lexical variable.

  The global lexical environment and the global dynamic environment
  are identical.  I.e. an assignment to the global binding of a
  lexical variable will be reflected in the observed global value of
  the dynamic variable of the same name, and vice versa.

  Example:

    (proclaim '(lexical x))
    (proclaim '(special y))
    (setq x 1 y 2)

    (defun tst ()
      (let ((x 3) (y 4))
	(locally (declare (special x) (lexical y))
	  (list x y
	        (funcall (let ((x 5) (y 6))
			   #'(lambda () (list x y))))))))

    (tst) => (1 2 (5 4))

Proposal (PROCLAIM-LEXICAL:RESTRICTED):

  Same as PROCLAIM-LEXICAL:GENERAL, but with the following restriction:
  If a name is proclaimed lexical, then it is an error for there to be
  a special declaration of the same name.  For example, the special
  declaration of X in the example is an error.


Cost of adopting change:

  CURRENT-PRACTICE: Ostensibly none, although implementations which
  don't signal this as an error should be explicitly encouraged
  to do so.  It ought to be signalled in interpreted code as well.

  BY-THE-BOOK: This would be an easy fix to the error reporting and
  bookkeepping components of existing compilers.  Of course it is not
  a change to the language, so there is no impact on portable code.

  GENERAL: This would be straightforward to implement, and is perhaps even
  already present, in implementations that use deep binding for
  special variables.  In shallow bound implementations there's a
  problem because two "global" value cells are needed: one for the
  current dynamic binding, and one for the global lexical binding;
  and when no dynamic binding is in effect, they must somehow be
  "coalesced" so that SETQ's are reflected in both.

  RESTRICTED: This is specifically designed to address the
  implementation problems of GENERAL.  Having a dynamic binding and
  having a global lexical binding are mutually exclusive.

Benefits:

  CURRENT-PRACTICE is incompatible with Scheme, which explicitly
  allows a program to have both a global binding and LAMBDA bindings
  of the same variable.  Therefore, adopting any of the other three
  proposals would be a major step towards reducing the
  incompatibilities between the two languages, since both code
  conversion and embedded scheme implementations would be made easier
  and more graceful.

  GENERAL and RESTRICTED have the additional advantage that, in an
  implementation that uses deep binding for dynamic variables, a
  lexical proclamation can be used to achieve significant performance
  gains on references and assignments to global variables.  A LEXICAL
  proclamation would also allow some desirable redundancy, both
  for documentation and for error checking.

Cost of converting existing code:

  CURRENT-PRACTICE: Some programs may rely on the CLtL semantics;
  these would have to be changed if what they're doing now becomes
  officially erroneous.  People might find consistent enforcement
  of this rule rather surprising and frustrating.

  GENERAL and RESTRICTED are both upwards compatible.  Of course,
  code-walking utilities would have to be taught about the new
  feature, since it affects the basic semantics of the language.

Aesthetics:

  The special/lexical business is generally one of CL's most insidious
  and disgusting aspects, and really needs a much more thorough
  overhaul than is proposed above (e.g. there ought to be a separate
  DYNAMIC-LET or SPECIAL-LET which does dynamic binding).  But this is
  probably practically and politically infeasible.  Given that, I
  don't think any of these proposals has clear advantages or
  disadvantages from the point of view of simplicity or elegance of
  description, except that GENERAL is somewhat easier to describe
  than RESTRICTED.

Discussion:

  The four proposals indicate the range of possibilities.  After some
  discussion, we ought to be able to prune this down, of course.

  JAR thinks that CURRENT-PRACTICE is unacceptable, BY-THE-BOOK is
  unsatisfying but somewhat better, and GENERAL has unsolved implementation
  problems, even though it seems the most powerful, symmetric, and elegant.
  Thus RESTRICTED seems preferable.

  This proposal ignores the question of what to do about an analogue
  of DEFVAR or DEFPARAMETER for global lexicals.  Interlisp and PSL both
  have something along these lines (DEFGLOBAL), don't they?

  Given the presence of lexical proclamations in the language, it's
  still not clear whether a free, undeclared, unproclaimed reference
  should be an error, and if it's not an error, whether it should refer
  to the lexical binding or the special binding.  I suggest that it not
  be an error, and I don't really care what its meaning should be (since
  in the situation I care about I'm not going to be dynamically binding
  the variable anyhow, so it doesn't matter).  The RESTRICTED semantics
  would imply that such a reference would have to be to the special
  variable.

  [By the way, I don't like this use of the term "variable," but I'm
  trying to be consistent with CLtL p. 55.  Note that according to this
  it doesn't make any sense to talk about "declaring a variable to be
  special" since any variable is already either special or lexical; it
  is names which can be so declared.  The book is definitely NOT
  consistent about this, and ought to be fixed.]