Issue: PROCLAIM-LEXICAL (Version 4)

[Masinter.pa@Xerox.COM]

I'm still not very happy with this, if only because the proposed
solution is not really motivated well by the problem described.

I attempted to reword the various discussion elements to reflect later
comments and agreements.

We let this one slip. 

!
Issue:        PROCLAIM-LEXICAL
References:   variables (p55), scope/extent (p37),
              global variables (p68),
	         declaration specifiers (p157)
Category:     ENHANCEMENT
Edit history: Version 2 by Rees 28-Apr-87
              Version 3 by Moon 16-May-87
              Version 4 by Masinter 27-Oct-87

Problem Description:

CLtL pp. 55-56 implies that if a name (symbol) is not proclaimed or
declared special, then a free reference to that name is a reference to
the special variable of that name, while a LAMBDA-binding of that name
indicates a binding of the lexical variable of that name.  This would
mean that the following program is legal and that (TST) => 4:

    (defun tst ()
      (setq x 3)
      (funcall (let ((x 4)) #'(lambda () x))))

However, if you feed this program to many Common Lisp compilers, a
warning message will be produced for the SETQ, saying something like
"Warning: X not declared or bound, assuming special."

These warnings, unlike the annotations of undefined functions (which
occur only at the end of a compilation), are presented so prominently
that a user would be hard put to say that a program which elicited such
warning messages was "correct" in that implementation.  Unlike the
situation with unused variables, there is no possible declaration one
can write which suppresses the warning messages. This disagreement
between theory and practice should be mended somehow.

There is no way to "undo" a SPECIAL proclaimation. Several aspects of
variable declarations (including whether the variable is a constant,
never rebound, and the like) are impossible in portable Common Lisp,
although they exist in several implementations.

Proposal (PROCLAIM-LEXICAL:ADD-LEXICAL-GLOBAL-CONSTANT):

Introduce new declaration specifiers, LEXICAL, GLOBAL, and CONSTANT,
which are mutually exclusive with the SPECIAL declaration specifier.
All four may be used as proclamations; only SPECIAL and LEXICAL may be
used as declarations.

A name may be proclaimed only one of LEXICAL, SPECIAL, GLOBAL, or
CONSTANT.  A name is said to be unproclaimed if it has not been
proclaimed to be any of these four.

A free reference or assignment to a name is an error if it is
unproclaimed and undeclared. (While we expect many programming
environments to allow such references in interactive programming, no
portable program should rely on free references to unproclaimed and
undeclared variables; compiler warnings when encountering them are
encouraged.) 

A LAMBDA-binding in the absence of a declaration or proclamation binds
the lexical variable.

SPECIAL proclamations and declarations behave as defined in CLtL.

LEXICAL proclamations have no effect other than to make the name cease
to be unproclaimed.  LEXICAL declarations shadow all enclosing
declarations and proclamation of any of these four types.  LEXICAL
declarations have the same scoping rules as SPECIAL declarations.

GLOBAL proclamations make it an error to bind the name.

CONSTANT proclamations make it an error to bind the name and an error to
assign to the name.

DEFCONSTANT is defined in terms of SETQ and the CONSTANT proclamation.
All keyword symbols are automatically proclaimed CONSTANT.

A free reference or assignment accesses the same value regardless of the
declaration or proclamation.  This is called the global value. SPECIAL
binding alters the global value within its extent. (Multiple process and
multiple processor systems will have to make their own definitions of
the extent of a SPECIAL binding, as noted on p.38 of CLtL--this proposal
does not address that.)

There is only one global value for a name and it is used by all free
references, all free assignments, and all SPECIAL bindings.

It is an error to re-proclaim a variable to a different class. (We
expect this issue to revisited by the compiler committee, see discussion
below.)
 
Example:

  (proclaim '(lexical x))
  (proclaim '(special y))
  (setq x 1 y 2)

  (defun tst ()
     (let ((x 3) (y 4))
	(locally (declare (special x) (lexical y))
	  (list x y
	        (funcall (let ((x 5) (y 6))
			   #'(lambda () (list x y))))))))

    (tst) => (1 4 (5 4))

Note that the second element of the list is 2. That is because the
special binding of y changes the global value to 4, and the
declared-lexical reference to y accesses the global value, since there
is no surrounding lexical binding.

Current Practice:

No implementations have all of these proclaimations, although some have
LEXICAL and some have GLOBAL.


Cost of adopting change:

This proposal requires no fundamental changes to implementations; the
superficial changes are to expose all of the primitive concepts which
are already available to programmers. Compilers and interpreters will
need to support LEXICAL as a declaration. Checking that variables
proclaimed as GLOBAL are not rebound is possible but not required.

Referencing or assigning to an unproclaimed and undeclared name "is an
error", not "signals an error", which allows but does not require an
implementation to issue a warning.  This is a change from current
language but does not mandate a change from current practice. 

Benefits:

LEXICAL proclamation enhances compatibility with Scheme. GLOBAL
proclamation allows more efficient deep-bound implementations.

Cost of converting existing code:

This change is upward compatible and will affect no existing code except
for code-walkers and other programs that analyze code.

Aesthetics:

The distinction between special and lexical variables is one of the more
confusing aspects of Common Lisp to those used to programming in Scheme.
Some might prefer to have a separate syntax (instead of LET with special
declarations) to do dynamic bindings. However, for the most part, such
changes are orthogonal to the issues raised here. 

Making  primitive concepts explicitly available can only enhance
aesthetics.

Discussion:

The original Common Lisp designers had difficulty with this issue;
putting in these features was part of the original intent, but time
didn't allow.

The following discussion carefully uses different words for kinds of
"cells" and kinds of "bindings". Lack of differentiation here has led to
a lot of confusion. Terminology in this area differs in writings about
Lisp and other programming language.

"Cells" are conceptual locations that remember a value and can be read
and written.  Sometimes these conceptual locations are implemented by
actual memory locations, and sometimes they aren't, but that's an
implementation detail and is irrelevant here.  (Sometimes these are
called variables, but other times the word "variable" means something
else, so for the sake of clarity I'm not going to use the word
"variable".)

Common Lisp has several kinds of cells. Local cells have lexical scope,
while global cells have global scope and can be referenced from anywhere
except where they are shadowed by a local cell referenced by the same
name. 

Common Lisp has two kinds of binding:  Lexical binding creates a new
cell.  Special binding saves the value of a cell, gives it a new value
during the extent of the binding, and later restores the old value.  We
know that special binding saves and restores because of what other
functions see when they read the cell.  (This should not be confused by
issues of shallow-bound versus deep-bound implementation, which are
irrelevant here: cells are not the same thing as memory locations.)

In principle, all cells have indefinite extent: they exist as long as a
reference to them is possible.  However, for most local cells, in the
absense of a closure, no reference to the cell is possible once the
invocation which created the cell exits; in those cases, the cell has
dynamic extent.

Common Lisp has a fairly complicated set of rules for how to determine
what cell is referenced when a program mentions a variable name.
("lexically free" means outside of all bindings of that name, regardless
of whether those bindings are special or lexical.)

  1. If the reference is lexically free, use the global cell.
  2. Otherwise, in the absence of a SPECIAL declaration use the cell
that
     was affected by the innermost binding; if that binding was special,
     use the global cell; if it was lexical, use the local cell created
     by that binding.
  3. Otherwise, the SPECIAL declaration forces use of the global cell.

We certainly do not want to introduce a third kind of cell, because that
would lead to a great deal of confusion.  Nor do we want to introduce a
third kind of binding.  What we -are- trying to accomplish is to make
the primitive concepts orthogonally available.  An example of a problem
we have right now that needs to be fixed is that there is no way to say
that a name is not misspelled without simultaneously saying that
bindings of that name should be special rather than lexical.  These
concepts should be available as separate constructs.

There are four things that we would like to be able to proclaim about a
variable name (aside from TYPE):
 - the default kind of binding if no declaration specifies which kind
 - the name is not misspelled so don't warn about free references
 - it is illegal to specially bind the global cell, because it is
   a constant or because we want to optimize the performance of a
   deep-binding implementation
 - it is illegal to store into the global cell, because it is a constant

It is already possible in Common Lisp to proclaim all four of these
things, but some of them are mixed up with other concepts and not
separately available.  Let's pick names for the four kinds of
proclamations, and let's also agree that any proclamation serves the
"not misspelled" purpose.  Let's further agree that these four
proclamations are mutually exclusive.

  LEXICAL  -- does nothing other than "not misspelled"
  SPECIAL  -- changes the default kind of binding from lexical to
special
  GLOBAL   -- makes it illegal to specially bind the global cell
  CONSTANT -- makes it illegal to store into the global cell
              CONSTANT implies GLOBAL because special-binding is storing

SPECIAL exists now.  LEXICAL is its logical complement.  CONSTANT exists
now but only buried inside the DEFCONSTANT macro.  GLOBAL exists now but
only when implied by DEFCONSTANT. This proposal makes them all
explicitly available.

It is appropriate to make CONSTANT and GLOBAL proclamations change the
default kind of binding to "illegal", rather than leaving it lexical, to
avoid unforseen interactions.

Fahlman added the following on question of overriding an old
proclaimation by issuing a new one:

We want to allow a variable to be re-proclaimed for two reasons: to
correct proclmations issued in error by the user, without having to kill
off the Lisp and start over, and to make it easier to merge programs
written by two different programmers.  (The latter reason may be bogus
-- they shouldn't be in the same package anyway -- but there are times
when a quick fix is extremely handy.)  On that other hand, we want the
compiler to be able to wire certain things in tight as a result of these
proclamations, so we need to make clear that if you proclaim something
to be GLOBAL, compile some code, then proclaim it to be SPECIAL and then
compile some more code the rebinds this variable, you may not get what
you expect. Same with unconstantifying a constant.

The current considerations of the compiler committee might give a
framework for explaining what the rules are within that framework.
However, given the current state of things, it is best to say that it
"is an error" to re-proclaim a variable into a different class -- this
says that portable code cannot do this and count on the result -- but
that implementations are strongly urged to allow this re-proclamation as
a way of correcting erroneous proclamations, perhaps issuing a warning
or signalling a correctable error whenever a proclamation actually gets
changed.

The cleanup committee considered numerous alternatives to this proposal,
including one where global lexical binding was separate from global
dynamic binding, various rules for not allowing a global variable to be
shadowed by a special declaration and the like.