18. The Compiler

18.1 The Basic Operations of the Compiler
The purpose of the Lisp compiler is to convert Lisp functions into programs in the Lisp Machine's instruction set, so that they will run more quickly and take up less storage. Compiled functions are represented in Lisp by FEFs (Function Entry Frames), which contain machine code as well as various other information. The format of FEFs and the instruction set are explained in LINK:(section-fef-format). There are three ways to invoke the compiler from the Lisp Machine. First, you may have an interpreted function in the Lisp environment which you would like to compile. The function compile is used to do this. Second, you may have code in an editor buffer which you would like to compile. The EINE editor has commands to read code into Lisp and compile it. Third, you may have a program (a of function definitions and other forms) written in a file on the file system. The compiler can translate this file into a QFASL file. Loading in the QFASL file is like reading in the source file, except that the functions in the source file will be compiled. The qc-file function is used for translating source files into QFASL files.
18.2 How to Invoke the Compiler
compile symbol
symbol should be defined as an interpreted function (its definition should be a lambda-expression). The compiler converts the lambda-expression into a FEF, saves the lambda-expression as the :previous-expr-definition and :previous-definition properties of symbol , and changes symbol 's definition to be the FEF. (See fset-carefully , LINK:(fset-carefully-fun). (Actually, if symbol is not defined as a lambda-expression, compile will try to find a lambda-expression in the :previous-expr-definition property of symbol and use that instead.)
uncompile symbol
If symbol is not defined as an interpreted function and it has a :previous-expr-definition property, then uncompile will restore the function cell from the value of the property. This "undoes" the effect of compile .
qc-file filename &optional output-file load-flag in-core-flag package
The file filename is given to the compiler, and the output of the compiler is written to a file whose name is filename except with an FN2 of "QFASL". The input format for files to the compiler is described on this link. Macro definitions and special declarations created during the compilation will be undone when the compilation is finished. The optional arguments allow certain modifications to this procedure. output-file lets you change where the output is written. package lets you specify in what package the source file is to be read. Normally the system knows, or asks interactively, and you need not supply this argument. load-flag and in-core-flag are incomprehensible; you don't want to use them.
qc-file-load filename
qc-file-load compiles a file and then loads it in.
See also the disassemble function (LINK:(disassemble-fun)), which lists the instructions of a compiled function in symbolic form. The compiler can also be run in Maclisp on ITS. On the MIT-AI machine, type :LISPM1;QCMP. It will type out "READY" and leave you at a Maclisp top level. Then type (qc-file filename) , expressing filename in Maclisp form. 
Example:
(qc-file '((lispm) foo >))
18.3 Input to the Compiler

The purpose of qc-file is to take a file and produce a translated version which does the same thing as the original except that the functions are compiled. qc-file reads through the input file, processing the forms in it one by one. For each form, suitable binary output is sent to the QFASL file so that when the QFASL file is loaded the effect of that source form will be reproduced. The differences between source files and QFASL files are that QFASL files are in a compressed binary form which reads much faster (but cannot be edited), and that function definitions in QFASL files have been translated from S-expressions to FEFs. So, if the source contains a (defun ...) form at top level, then when the QFASL file is loaded, the function will be defined as a compiled function. If the source file contains a form which is not of a type known specially to the compiler, then that form will be output "directly" into the QFASL file, so that when the QFASL file is loaded that form will be evaluated. Thus, if the source file contains (setq x 3) , then the compiler will put in the QFASL file instructions to set x to 3 at load time. However, sometimes we want to put things in the file that are not merely meant to be translated into QFASL form. One such occasion is top level macro definitions; the macros must actually get defined within the compiler in order that the compiler be able to expand them at compile time. So when a macro form is seen, it should (sometimes) be evaluated at compile time, and should (sometimes) be put into the QFASL file. Another thing we sometimes want to put in a file is compiler declarations. These are forms which should be evaluated at compile time to tell the compiler something. They should not be put into the QFASL file. Therefore, a facility exists to allow the user to tell the compiler just what to do with a form. One might want a form to be:
Put into the QFASL file (translated), or not.
Evaluated within the compiler, or not.
Evaluated if the file is read directly into Lisp, or not.
Two forms are recognized by the compiler to allow this. The less general but Maclisp compatible one is declare ; the completely general one is eval-when . An eval-when form looks like 

(eval-when times-list 
	   form1 
	   form2 
           ...)
The times-list may contain any of the symbols load , compile , or eval . If load is present, the form s are written into the QFASL file to be evaluated when the QFASL file is loaded (except that defun forms will put the compiled definition into the QFASL file instead). If compile is present, the form s are evaluated in the compiler. If eval is present, the form s are evaluated when read into Lisp; this is because eval-when is defined as a special form in Lisp. (The compiler ignores eval in the times-list .) For example, (eval-when (compile eval) (macro foo (x) (cadr x))) would define foo as a macro in the compiler and when the file is read in interpreted, but not when the QFASL file is fasloaded. For the rest of this section, we will use lists such as are given to eval-when , e.g. (load eval) , (load compile) , etc. to describe when forms are evaluated. A declare form looks like (declare form1 form2 ...) . declare is defined in Lisp as a special form which does nothing; so the forms within a declare are not evaluated at eval time. The compiler does the following upon finding form within a declare : if form is a call to either special or unspecial , form is treated as (load compile) ; otherwise it is treated as (compile) . If a form is not enclosed in an eval-when nor a declare , then the times at which it will be evaluated depend on the form. The following table summarizes at what times evaluation will take place for any given form seen at top level by the compiler.
(eval-when times-list form1 ...)
times-list
(declare (special ...)) or (declare (unspecial ...))
(load compile)
(declare anything-else )
(compile)
(special ...) or (unspecial ...)
(load compile eval)
(macro ...) or (defstruct ...)
(load compile eval)
(comment ...)Ignored
(begf ...) or (endf ...)
Ignored but may one day put something in the QFASL file.
(compiler-let ((var val ) ...) body ...)
At (compile eval) time, processes the body with the indicated variable bindings in effect. Does nothing at load time.
(local-declare (decl decl ... ) body ...)
Processes the body in its normal fashion, with the indicated declarations added to the front of the list which is the value of local-declarations .
anything-else
(load eval)
Sometimes a macro wants to return more than one form for the compiler top level to see (and to be evaluated). The following facility is provided for such macros. If a form 
(progn (quote compile) form1  form2  ...)
is seen at the compiler top level, all of the form s are processed as if they had been at compiler top level. (Of course, in the interpreter they will all be evaluated, and the (quote compile) will harmlessly evaluate to the symbol compile and be ignored.)
eval-when Special Form
An eval-when form looks like 
(eval-when times-list  form1  form2  ...)
If one of the element of times-list is the symbol eval , then the form s are evaluated; otherwise eval-when does nothing. But when seen by the compiler, this special form does the special things described above.
declare Special Form
declare does nothing, and returns the symbol declare . But when seen by the compiler, this special form does the special things described above.
18.4 Compiler Declarations
This section describes functions meant to be called during compilation, and variables meant to be set or bound during compilation, by using declare or local-declare .
local-declare Special Form
A local-declare form looks like 
(local-declare (decl1  decl2  ...)
   form1 
   form2 
   ...)
Each decl is consed onto the list local-declarations while the form s are being evaluated (in the interpreter) or compiled (in the compiler). There are two uses for this. First, it can be used to pass information from outer macros to inner macros. Secondly, the compiler will specially interpret certain decl s as local declarations, which only apply to the compilations of the form s. It understands the following forms:
(special var1 var2 ... )
The variables var1 , var2 , etc. will be treated as special variables during the compilation of the form s.
(unspecial var1 var2 ... )
The variables var1 , var2 , etc. will be treated as local variables during the compilation of the form s.
(macro name lambda (x) body )
name will be defined as a macro during the compilation of the form s. Note that the cddr of this item is a function.
special Special Form
(special var1 var2 ...) causes the variables to be declared to be "special" for the compiler.
unspecial Special Form
(unspecial var1 var2 ...) removes any "special" declarations of the variables for the compiler.
The next three declarations are primarily for Maclisp compatibility.
*expr Special Form
(*expr sym1 sym2 ...) declares sym1 , sym2 , etc. to be names of functions. In addition it prevents these functions from appearing in the list of functions referenced but not defined printed at the end of the compilation.
*lexpr Special Form
(*lexpr sym1 sym2 ...) declares sym1 , sym2 , etc. to be names of functions. In addition it prevents these functions from appearing in the list of functions referenced but not defined printed at the end of the compilation.
*fexpr Special Form
(*fexpr sym1 sym2 ...) declares sym1 , sym2 , etc. to be names of special forms. In addition it prevents these names from appearing in the list of functions referenced but not defined printed at the end of the compilation.
There are some advertised variables whose compile-time values affect the operation of the compiler. The user may set these variables by including in his file forms such as 
(declare (setq open-code-map-switch t))
run-in-maclisp-switch Variable
If this variable is non-nil , the compiler will try to warn the user about any constructs which will not work in Maclisp. By no means will all Lisp machine system functions not built in to Maclisp be cause for warnings; only those which could not be written by the user in Maclisp (for example, *catch , make-array , value-cell-location , etc.). Also, lambda-list keywords such as &optional and initialized prog variables will be mentioned. This switch also inhibits the warnings for obsolete Maclisp functions. The default value of this variable is nil .
obsolete-function-warning-switch Variable
If this variable is non-nil , the compiler will try to warn the user whenever an "obsolete" Maclisp-compatibility function such as maknam or samepnamep is used. The default value is t .
allow-variables-in-function-position-switch Variable
If this variable is non-nil , the compiler allows the use of the name of a variable in function position to mean that the variable's value should be funcall 'd. This is for compatibility with old Maclisp programs. The default value of this variable is nil .
open-code-map-switch Variable
If this variable is non-nil , the compiler will attempt to produce inline code for the mapping functions (mapc , mapcar , etc., but not mapatoms ) if the function being mapped is an anonymous lambda-expression. This allows that function to reference the local variables of the enclosing function without the need for special declarations. The generated code is also more efficient. The default value is T.
all-special-switch Variable
If this variable is non-nil , the compiler regards all variables as special, regardless of how they were declared. This provides full compatibility with the interpreter at the cost of efficiency. The default is nil .
inhibit-style-warnings-switch Variable
If this variable is non-nil , all compiler style-checking is turned off. Style checking is used to issue obsolete function warnings and won't-run-in-Maclisp warnings, and other sorts of warnings. The default value is nil . See also the inhibit-style-warnings macro, which acts on one level only of an expression.
retain-variable-names-switch Variable
This controls whether the generated FEFs remember the names of the variables in the function; such information is useful for debugging (the arglist function uses it, see LINK:(arglist-fun)), but it increases the size of the QFASL file and the FEFs created. The variable may be any of
nilNo names are saved.
argsNames of arguments are saved.
allNames of arguments and &aux variables are saved.
The default value of this symbol is args , and it should usually be left that way.
compiler-let Macro
(compiler-let ((variable value)...) body...) , syntactically identical to let , allows compiler switches to be bound locally at compile time, during the processing of the body forms. 
Example:
(compiler-let ((open-code-map-switch nil))
          (map (function (lambda (x) ...)) foo))
will prevent the compiler from open-coding the map . When interpreted, compiler-let is equivalent to let . This is so that global switches which affect the behavior of macro expanders can be bound locally.
inhibit-style-warnings Macro
(inhibit-style-warnings form) prevents the compiler from performing style-checking on the top level of form . Style-checking will still be done on the arguments of form . Both obsolete function warnings and won't-run-in-Maclisp warnings are done by means of the style-checking mechanism, so, for example, 
(setq bar (inhibit-style-warnings (value-cell-location foo)))
will not warn that value-cell-location will not work in Maclisp, but 
(inhibit-style-warnings (setq bar (value-cell-location foo)))
will warn, since inhibit-style-warnings applies only to the top level of the form inside it (in this case, to the setq ).
18.5 Compiler Source-Level Optimizers
The compiler stores optimizers for source code on property lists so as to make it easy for the user to add them. An optimizer can be used to transform code into an equivalent but more efficient form (for example, (eq obj nil) is transformed into (null obj) , which can be compiled better). An optimizer can also be used to tell the compiler how to compile a special form. For example, in the interpreter do is a special form, implemented by a function which takes quoted arguments and calls eval . In the compiler, do is expanded in a macro-like way by an optimizer, into equivalent Lisp code using prog , cond , and go , which the compiler understands. The compiler finds the optimizers to apply to a form by looking for the compiler:optimizers property of the symbol which is the car of the form. The value of this property should be a list of optimizers, each of which must be a function of one argument. The compiler tries each optimizer in turn, passing the form to be optimized as the argument. An optimizer which returns the original form unchanged (and eq to the argument) has "done nothing", and the next optimizer is tried. If the optimizer returns anything else, it has "done something", and the whole process starts over again. This is somewhat like a Markov algorithm. Only after all the optimizers have been tried and have done nothing is an ordinary macro definition processed. This is so that the macro definitions, which will be seen by the interpreter, can be overridden for the compiler by an optimizer.
18.6 Files that Maclisp Must Compile
Certain programs are intended to be run both in Maclisp and in Lisp Machine Lisp. These files need some special conventions. For example, such Lisp Machine constructs as &aux and &optional must not be used. In addition, eval-when must not be used, since only the Lisp Machine compiler knows about it. All special declarations must be enclosed in declare s, so that the Maclisp compiler will see them. It is suggested that you turn on run-in-maclisp-switch in such files, which will warn you about a lot of bugs. The macro-character combination "#Q" causes the object that follows it to be visible only when compiling for the Lisp Machine. The combination "#M" causes the following object to be visible only when compiling for Maclisp. These work only on subexpressions of the objects in the file, however. To conditionalize top-level objects, put the macros if-for-lispm and if-for-maclisp around them. (You can only put these around a single object.) The if-for-lispm macro turns off run-in-maclisp-switch within its object, preventing spurious warnings from the compiler. The #Q macro-character does not do this, since it can be used to conditionalize any S-expression, not just a top-level form. There are actually three possible cases of compiling: you may be compiling on the Lisp Machine for the Lisp Machine; you may be compiling in Maclisp for the Lisp Machine (with :LISPM1;QCMP); or you may be compiling in Maclisp for Maclisp (with COMPLR). (You can't compile for Maclisp on the Lisp Machine because there isn't a Lisp Machine Lisp version of COMPLR.) To allow a file to detect any of these conditions it needs to, the following macros are provided:
if-for-lispm Macro
If (if-for-lispm form) is seen at the top level of the compiler, form is passed to the compiler top level if the output of the compiler is a QFASL file intended for the Lisp Machine. If the Lisp Machine interpreter sees this it will evaluate form (the macro expands into form ).
if-for-maclisp Macro
If (if-for-maclisp form) is seen at the top level of the compiler, form is passed to the compiler top level if the output of the compiler is a FASL file intended for Maclisp (e.g. if the compiler is COMPLR). If the Lisp Machine interpreter sees this it will ignore it (the macro expands into nil ).
if-for-maclisp-else-lispm Macro
If (if-for-maclisp-else-lispm form1 form2) is seen at the top level of the compiler, form1 is passed to the compiler top level if the output of the compiler is a FASL file intended for Maclisp; otherwise form2 is passed to the compiler top level.
if-in-lispm Macro
On the Lisp Machine, (if-in-lispm form) causes form to be evaluated; in Maclisp, form is ignored.
if-in-maclisp Macro
In Maclisp, (if-in-maclisp form) causes form to be evaluated; on the Lisp Machine, form is ignored.
When you have two definitions of one function, one conditionalized for one machine and one for the other, indent the first "(defun " by one space, and the editor will put both function definitions together in the same file-section. In order to make sure that those macros and macro-characters are defined when reading the file into the Maclisp compiler, you must make the file start with a prelude, which will have no effect when you compile on the real machine. The prelude can be found in "AI: LMDOC; .COMPL PRELUD"; this will also define most of the standard Lisp Machine macros and reader macros in Maclisp, including defmacro and the backquote facility. Another useful facility is the form (status feature lispm) , which evaluates to t when evaluated on the Lisp machine and to nil when evaluated in Maclisp.