Issue: PROCLAIM-LEXICAL (Version 5)

[Masinter.pa@Xerox.COM]

JAR didn't come thru with a rewrite, so this is a last minute attempt to
construct one. What I mainly did was to omit the summary of Moon's
description of the "cell" model.

I'd like to present this to X3J13 even in its current interim state. We
will discuss it on Monday. (BTW, I intend to try to have a follow up
get-together Monday evening to go over and confirm the status of various
issues before the Tuesday meeting. Please check the hotel announcement
board (I assume there is one.)).

!
Issue:        PROCLAIM-LEXICAL
References:   variables (p55), scope/extent (p37),
              global variables (p68),
	         declaration specifiers (p157)
Category:     ENHANCEMENT
Edit history: Version 2 by Rees 28-Apr-87
              Version 3 by Moon 16-May-87
              Version 4 by Masinter 27-Oct-87
              Version 5 by Masinter 14-Nov-87

Problem Description:

The distinction between special and lexical variables is one of the more
confusing aspects of Common Lisp to those used to programming in Scheme.
There is no way to "undo" a SPECIAL proclaimation. Several aspects of
variable declarations (including whether the variable is a constant,
never rebound, and the like) are impossible in portable Common Lisp,
although they exist in several implementations. There are many
combinations of declarations allowed and not allowed.

CLtL pp. 55-56 implies that if a name (symbol) is not proclaimed or
declared special, then a free reference to that name is a reference to
the special variable of that name, while a LAMBDA-binding of that name
indicates a binding of the lexical variable of that name.  This would
mean that the following program is legal and that (TST) => 4:

    (defun tst ()
      (setq x 3)
      (funcall (let ((x 4)) #'(lambda () x))))

However, if you feed this program to many Common Lisp compilers, a
warning message will be produced for the SETQ, saying something like
"Warning: X not declared or bound, assuming special." These warnings are
prominent; it is hard to believe  that a program which elicited such
warning messages was "correct" in that implementation. This is a
disagreement between theory and practice.

Proposal (PROCLAIM-LEXICAL:ADD-LEXICAL-GLOBAL-CONSTANT):

Introduce new declaration specifiers, LEXICAL, GLOBAL, and CONSTANT,
which are mutually exclusive with the SPECIAL declaration specifier.
All four may be used as proclamations; only SPECIAL and LEXICAL may be
used as declarations.

A name may be proclaimed only one of LEXICAL, SPECIAL, GLOBAL, or
CONSTANT.  A name is said to be unproclaimed if it has not been
proclaimed to be any of these four.

A free reference or assignment to a name is an error if it is
unproclaimed and undeclared. (While we expect many programming
environments to allow such references in interactive programming, no
portable program should rely on free references to unproclaimed and
undeclared variables; compiler warnings when encountering them are
encouraged.) 

A LAMBDA-binding in the absence of a declaration or proclamation binds
the lexical variable.

SPECIAL proclamations and declarations behave as defined in CLtL.

LEXICAL proclamations have no effect other than to make the name cease
to be unproclaimed.  LEXICAL declarations shadow all enclosing
declarations and proclamation of any of these four types.  LEXICAL
declarations have the same scoping rules as SPECIAL declarations.

GLOBAL proclamations make it an error to bind the name.

CONSTANT proclamations make it an error to bind the name and an error to
assign to the name.

DEFCONSTANT is defined in terms of SETQ and the CONSTANT proclamation.
All keyword symbols are automatically proclaimed CONSTANT.

A free reference or assignment accesses the same value regardless of the
declaration or proclamation.  This is called the global value. SPECIAL
binding alters the global value within its extent. (Multiple process and
multiple processor systems will have to make their own definitions of
the extent of a SPECIAL binding, as noted on p.38 of CLtL--this proposal
does not address that.)

There is only one global value for a name and it is used by all free
references, all free assignments, and all SPECIAL bindings.

It is an error to re-proclaim a variable to a different class. (We
expect this issue to revisited by the compiler committee, see discussion
below.)
 
Example:

  (proclaim '(lexical x))
  (proclaim '(special y))
  (setq x 1 y 2)

  (defun tst ()
     (let ((x 3) (y 4))
	(locally (declare (special x) (lexical y))
	  (list x y
	        (funcall (let ((x 5) (y 6))
			   #'(lambda () (list x y))))))))

    (tst) => (1 4 (5 4))

Note that the second element of the list is 2. That is because the
special binding of y changes the global value to 4, and the
declared-lexical reference to y accesses the global value, since there
is no surrounding lexical binding.

Current Practice:

No implementations have all of these proclaimations, although some have
LEXICAL and some have GLOBAL.

Cost of adopting change:

This proposal requires no fundamental changes to implementations; the
superficial changes are to expose all of the primitive concepts which
are already available to programmers. Compilers and interpreters will
need to support LEXICAL as a declaration. Checking that variables
proclaimed as GLOBAL are not rebound is possible but not required.

Referencing or assigning to an unproclaimed and undeclared name "is an
error", not "signals an error", which allows but does not require an
implementation to issue a warning.  This is a change from current
language but does not mandate a change from current practice. 

Benefits:

LEXICAL proclamation enhances compatibility with Scheme. GLOBAL
proclamation allows more efficient deep-bound implementations.

Cost of converting existing code:

This change is upward compatible and will affect no existing code except
for code-walkers and other programs that analyze code.

Aesthetics:

The distinction between special and lexical variables is one of the more
confusing aspects of Common Lisp to those used to programming in Scheme.
Some might prefer to have a separate syntax (instead of LET with special
declarations) to do dynamic bindings. However, for the most part, such
changes are orthogonal to the issues raised here. 

Making primitive concepts explicitly available enhances aesthetics.

Discussion:

The original Common Lisp designers had difficulty with this issue;
putting in these features was part of the original intent, but time
didn't allow.

This proposal and many other proposals were discussed before this
version was arrived at.

There are four things that we would like to be able to proclaim about a
variable name (aside from TYPE):
 - the default kind of binding if no declaration specifies which kind
 - the name is not misspelled so don't warn about free references
 - it is illegal to specially bind the global cell, because it is
   a constant or because we want to optimize the performance of a
   deep-binding implementation
 - it is illegal to store into the global cell, because it is a constant

It is already possible in Common Lisp to proclaim all four of these
things, but some of them are mixed up with other concepts and not
separately available.

  LEXICAL  -- does nothing other than "not misspelled"
  SPECIAL  -- changes the default kind of binding from lexical to
special
  GLOBAL   -- makes it illegal to specially bind the global cell
  CONSTANT -- makes it illegal to store into the global cell
              CONSTANT implies GLOBAL because special-binding is storing

SPECIAL exists now.  LEXICAL is its logical complement.  CONSTANT exists
now but only buried inside the DEFCONSTANT macro.  GLOBAL exists now but
only when implied by DEFCONSTANT. This proposal makes them all
explicitly available.

It is appropriate to make CONSTANT and GLOBAL proclamations change the
default kind of binding to "illegal", rather than leaving it lexical, to
avoid unforseen interactions.

Is it legal to change the proclaimation of a variable?

We want to allow a variable to be re-proclaimed for two reasons: to
correct proclaimations issued in error by the user without having to end
the Lisp session and start over, and to make it easier to merge programs
written by two different programmers.  (The latter reason is suspect --
they shouldn't be in the same package anyway -- but there are times when
a quick fix is extremely handy.)  On that other hand, we want the
compiler to be able to wire certain things in tight as a result of these
proclamations, so we need to make clear that if you proclaim something
to be GLOBAL, compile some code, then proclaim it to be SPECIAL and then
compile some more code the rebinds this variable, you may not get what
you expect. Same with changing the proclaimation of a constant.

The current considerations of the compiler committee might give a
framework for explaining what the rules are within that framework.
However, given the current state of things, it is best to say that it
"is an error" to re-proclaim a variable into a different class -- this
says that portable code cannot do this and count on the result -- but
that implementations are strongly urged to allow this re-proclamation as
a way of correcting erroneous proclamations, perhaps issuing a warning
or signalling a correctable error whenever a proclamation actually gets
changed.