Re: Lexical closures (bug?)

[Siskind.pa@XEROX.COM]

    Date: Mon, 15 Aug 88 14:58:32 pdt
    From: lbaum@BOEING.COM (Larry Baum)

    We were working thru the exercises in Charniak, Riesbeck & McDermot, 
    Artificial Intelligence Programming, and tried the following:

    (defun compose (&rest functions)
      #'(lambda (x)
          (do ((fns functions (cdr fns))
               (value x (funcall (car fns) value)))
              ((null fns) value))))

    This should return the composition of the functions passed as
    arguments; e.g. (compose #'1+ #'1+) should return a closure which should
    increment its argument by 2:

    (setq 2+ (compose #'1+ #'1+))
    (funcall 2+ 4) => 6

    However, when I try this on my 36xx, with either Genera 7.1 or 7.2, the
    control stack overflows.  In fact if I print the "(car fns)" it is
    apparnet that in each iteration what is being funcalled is not the next
    function, but the entire closure!

    Any ideas?

    Larry Baum

According to the specification of Symbolics Common Lisp (and I think
CLtL in general) this is not a bug. The problem is that the list created
implicitly by the &rest argument functions is created on the stack. When
the function compose returns, this list (of functions) disappears and
is unavailable when the lexical closure created by compose is later called.
This closure references the stack pointer as it exists later on and finds
implementation and time dependent garbage there so its behavior is bogus.
The solution is to copy all lists which are passed as &rest arguments if they
need to exists beyond the scope of the calling functions as in:

(defun compose (&rest functions)
 (let ((functions (copy-list functions)))   ;<----
   #'(lambda (x)
       (do ((fns functions (cdr fns))
            (value x (funcall (car fns) value)))
           ((null fns) value)))))

This holds true not just for &rest lists which need to be referenced
by a created lexical closure but also for any other way you create
a pointer to a &rest argument as in:

(defun foo (&rest elements)
 (setq bar elements))

(foo 1 2 3 4)

The above will lose and you need:

(defun foo (&rest elements)
 (setq bar (copy-list elements)))

Note that the documentation clearly specifies that this is an error.
You could have two potential complaints. First, that this should not be
an error in the first place. It turns out that consing &rest arguments
on the stack is a major optimization alieviating a lot of GC work. The
claim is that explicitly calling copy-list when the optimization is
inapprpriate is a small price to pay for the minority of times it is
needed to allow the optimization to be the default for the majority
of times that it is appropriate. Second, that the compiler or microcode
should report the error. Now, I admit that this is a breach of safety
not to detect and report such an error but I guess that that requires
a lot of overhead which many implementors felt not worthwhile. This is
one of the few ways you can get screwed on a lispm while writing fairly
straightforward code. Perhaps Symbolics should document all such ways
and print them in large font bold print on the first page of the manual
so that people don't get caught by this kind of gatcha again.
        Hope this helps,
        Jeff
-------