(another) revised version of the defining macros proposal

[sandra@cs.utah.edu (Sandra J Loosemore)]

Here is yet another revision to the defining macros proposal.  The only
thing that has changed is the wording of the DEFCONSTANT section.  I hope
that this time I have at least managed to make the language concise enough
so that we can agree on what it says, even if we don't all agree *with*
what it says.

-Sandra


Issue:		COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS
References:	CLtL pages 66-70, 143
Category:	CLARIFICATION
Edit history:   V1, 07 Oct 1987 Sandra Loosemore
                V2, 15 Oct 1987 Sandra Loosemore
                V3, 15 Jan 1988 Sandra Loosemore
		V4, 06 May 1988 Sandra Loosemore
		V5, 20 May 1988 Sandra Loosemore
		V6, 09 Jun 1988 Sandra Loosemore


Problem Description:

Standard programming practices assume that, when calls to defining
macros such as DEFMACRO and DEFVAR are processed by COMPILE-FILE,
certain side-effects occur that affect how subsequent forms in the
file are compiled.  However, these side-effects are not mentioned in
CLtL, except for a passing mention that macro definitions must be
``seen'' by the compiler before it can compile calls to those macros
correctly.  In order to write portable programs, users must know
exactly which defining macros have compile-time side-effects and what
those side-effects are. 

Inter-file compilation dependencies are distinct from, and not
addressed by, this issue. 


Proposal: COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS:CLARIFY

(1) Clarify that defining macros such as DEFMACRO or DEFVAR, appearing
within a file being processed by COMPILE-FILE, normally have
compile-time side effects which affect how subsequent forms in the
same file are compiled.  A convenient model for explaining how these
side effects happen is that the defining macro expands into one or
more EVAL-WHEN forms, and that the calls which cause the compile-time
side effects to happen appear in the body of an (EVAL-WHEN (COMPILE)
...) form.  This is also the recommended implementation technique. 

(2) The affected defining macros and their specific side effects are
as follows:
 
DEFTYPE: The body of a DEFTYPE form must be evaluable at compile time.
If the expansion of a DEFTYPE'd type specifier is also a valid type
specifier at compile time, then the DEFTYPE'd type specifier is also
considered to be fully defined at compile time and must be recognized
within subsequent type declarations. 

DEFMACRO, DEFINE-MODIFY-MACRO:  Macro definitions must be stored at compile
time, so that occurences of the macro later on in the file will be expanded
correctly.  The body of the macro (but not necesarily its expansion) must 
be evaluable at compile time.
 
DEFUN: An implementation may choose to store information about the
function for the purposes of compile-time error-checking (such as
checking the number of arguments on calls), or to enable the function
to be expanded inline.  Portable code should not rely on DEFUN making
the function definition available at compile time.
 
DEFVAR, DEFPARAMETER: The compiler must recognize that the variables
named by these forms have been proclaimed special.  The initial value
form must not be evaluated at compile time. 
 
DEFCONSTANT: The compiler must recognize the symbol as being constant
(for example, to suppress warnings about references to the symbolic
constant as an unbound variable or to enable warnings about binding or
SETQ'ing the constant in the code being compiled).  Neither evaluation
of the value-form or SETQ'ing of the symbol may occur at compile-time.
However, if the (unevaluated) value-form is CONSTANTP, the compiler is
allowed to build assumptions about the value of the constant into
programs being compiled, as described on p. 68-69 of CLtL.
 
DEFSETF, DEFINE-SETF-METHOD: SETF methods must be available during the
expansion of calls to SETF later on in the file.  The body of
DEFINE-SETF-METHOD and the complex form of DEFSETF must be evaluable
at compile time, although the expansions need not be. 
 
DEFSTRUCT:  The structure type name must be recognized as a valid type name
in declarations, as for DEFTYPE.  The structure slot accessors must be made
known to SETF.  In addition, further DEFSTRUCT definitions should be able
to :INCLUDE a structure type defined earlier in the file being compiled.
The functions which DEFSTRUCT generates, and the #S reader syntax, may or
may not be available at compile time.

(3) The compile-time side effects may cause information about the
definition to be stored differently than if the defining macro had
been processed in the "normal" way (either interpretively or by loading
the compiled file).

In particular, the information stored by the defining macros at
compile time may or may not be available to the interpreter (either
during or after compilation), or during subsequent calls to COMPILE or
COMPILE-FILE.  For example, the following code is nonportable because
it assumes that the compiler stores the macro definition of FOO where
it is available to the interpreter:

    (defmacro foo (x) `(car ,x))
    (eval-when (eval compile load)
        (print (foo '(a b c))))

A portable way to do the same thing would be to include the macro definition
inside the EVAL-WHEN:

    (eval-when (eval compile load)
        (defmacro foo (x) `(car ,x))
        (print (foo '(a b c))))



Rationale:

The proposal reflects standard programming practices.  The primary
purpose of the proposal is to make an explicit statement that CL
supports the behavior that most programmers expect and many
implementations already provide.


Current Practice:

Many (probably most) Common Lisp implementations, including VaxLisp
and Lucid Lisp, are already largely in conformance.  

In VaxLisp, macro definitions that occur as a side effect of compiling
a DEFMACRO form are available to the compiler (even on subsequent calls
to COMPILE or COMPILE-FILE), but are not available to the interpreter
(even within the file being compiled).
 
Kyoto Common Lisp is a notable offender.  By default, KCL evaluates *all*
top level forms as they are compiled, which is clearly in violation of the
behavior specified on p 69-70 of CLtL.  There is a flag to disable the
compile-time evaluation, but then macros such as DEFMACRO, DEFVAR, etc. do
not make their definitions available at compile-time either.


Cost to implementors:

Making the defining macros expand into EVAL-WHENs to store the required
information is a simple and recommended implementation technique.  The
intent of the proposal is specifically not to require the compiler to
have special knowledge about each of these macros.


Cost to users:

Since CLtL does not specify whether and what compile-time side-effects
happen, any user code which relies on them is, strictly speaking,
nonportable.  In practice, however, most programmers already expect
the behavior described in this proposal and will not find it to be
an incompatible change.


Benefits:

Adoption of the proposal will provide more definite guidelines on how to
write programs that will compile correctly under all CL implementations.


Discussion:

Reaction to an earlier version of this proposal on the CL mailing list was
overwhelmingly positive.

It has been suggested that this proposal should also include PROCLAIM.
However, since PROCLAIM is not a macro, its compile-time side effects
cannot be handled using the EVAL-WHEN mechanism.  A separate proposal
seems more appropriate. 

There has also been a suggestion that DEFCONSTANT should always
evaluate the value provided.  The behavior specified in this proposal
makes DEFCONSTANT similar to DEFVAR and DEFPARAMETER, while allowing
the user to explicitly ask for compile-time evaluation using the #. read
macro.

Item (3) allows for significant deviations between implementations.
While there is some sentiment to the effect that the compiler should
store definitions in a manner identical to that of the interpreter,
other people believe strongly that compiler side-effects should be
completely invisible to the interpreter.  The author is of the opinion
that since this is a controversial issue, further attempts to restrict
this behavior should be considered as separate proposals.

-------