Issue: UNWIND-PROTECT-CLEANUP-NON-LOCAL-EXIT (Version 2)

[Masinter.pa@Xerox.COM]

My habit of starting alphabetically and not getting all the way done has
caused this issue to be neglected. I thought I would start from the end
this pass. This is an important issue, it seems to be in the way of
producing a valid portable error system. 

It took me several hours to try to put together a version that
summarized the arguments that took place last May. I hope I have
captured them. 

We now also have some new members; do we have any new opinions? (If
anyone wants my archive on this or any issue, let me know and I will
store it.)

!
Issue:          UNWIND-PROTECT-CLEANUP-NON-LOCAL-EXIT
References:     UNWIND-PROTECT (p140, p142, p39)
                Issue IGNORE-ERRORS, Draft error proposal.
Category:       CLARIFICATION/CHANGE
Edit history:   Version 1 by Pitman   27-Feb-87
                Version 2 by Masinter 24-Oct-87

Problem Description:

If a non-local return is done while in the cleanup form of an
UNWIND-PROTECT, the behavior is not always well-defined.

There are three basic cases:

Situation 0. Transfer to another point within the cleanup form.
   (UNWIND-PROTECT 3 (BLOCK NIL (RETURN 4)) (PRINT 'XXX))

There is no ambiguity about how this form is to be interpreted.
Effectively: 

      . 3 evaluates to itself, which is queued for return
        from the UNWIND-PROTECT. 
      . The BLOCK expression is entered, 4 is returned to
	it and discarded because this is a not-for-value 
	situation.
      . XXX is printed, XXX is returned by the PRINT and
	that value is discarded because this is a not-for-value
	situation.
      . The 3 which was yielded earlier is retrieved and
	returned as the value of the UNWIND-PROTECT.

Situation 1. Transfer to a point inside the point to which control 
    would have transferred.
    (CATCH 'FOO
      (CATCH 'BAR
	  (UNWIND-PROTECT (THROW 'FOO 3)
	    (THROW 'BAR 4)
	    (PRINT 'XXX))))

    This is a subject of controversy because:
    . 3 evaluates to itself and is saved by THROW which begins
      searching for tag FOO. 
    . 4 evaluates to iself and is saved by THROW which begins
      searching for tag BAR.
    . Disagreement exists as to whether it is an error if the
      BAR tag is not found within the local dynamic scope of
      the UNWIND-PROTECT cleanup form containing (THROW 'BAR 4)
      but is found within the scope of the target of the 
      pending THROW (to FOO).

Situation 2. Transfer to a point outside the point to which return would
    already have been. For example:
    (CATCH 'BAR
      (CATCH 'FOO
	(UNWIND-PROTECT (THROW 'FOO 3)
	  (THROW 'BAR 4)
	  (PRINT 'XXX))))
    This is a subject of controversy because:
    . 3 evaluates to itself and is saved by THROW which begins
      searching for tag FOO. 
    . 4 evaluates to iself and is saved by THROW which begins
      searching for tag BAR.
    . Disagreement exists as to whether it is an error if the
      BAR tag is not found within the local dynamic scope of
      the UNWIND-PROTECT cleanup form containing (THROW 'BAR 4)
      but is found outside the scope of the target of the 
      pending THROW (to FOO).

What is the appropriate behavior for situation 1 and situation 2 and
similar ones? For example, suppose that when WITH-OPEN-FILE tries to
close the file upon unwinding, it signals an error, and the condition
handler also attempts to throw? The question applies to all non-local
transfers, whether performed by THROW, RETURN-FROM, RETURN, GO.

There is general agreement about Situation 2, but three proposals for
Situation 1.

Proposal (UNWIND-PROTECT-CLEANUP-NON-LOCAL-EXIT:2-EXIT):

Where an UNWIND-PROTECT cleanup form attempts a non-local exit to a
point outside the original non-local exit, control is passed to the
outer exit (and the pending original non-local exit is discarded.)

In Situation 2, the value 4 is returned from the (CATCH 'BAR ...); XXX
is not printed.

[In no case will UNWIND-PROTECT cleanup forms ever be attempted more
than once.]

Proposal (UNWIND-PROTECT-CLEANUP-NON-LOCAL-EXIT:1-RETURN-INNER):

While processing UNWIND-PROTECT cleanup forms, a non-local exit to a
point outside of the scope of the UNWIND-PROTECT, but still within the
dynamic scope of of the target of the original non-local exit.... 

... the second non-local exit succeeds, and the original pending exit is
discarded. 

In Situation 1, the pending seek for tag FOO is discarded by the second
THROW to BAR and the value 4 is transfered to (CATCH 'BAR ...), which
returns 4. The (CATCH 'FOO ...) then returns the 4 because its first
argument has returned normally. XXX is not printed.

Proposal (UNWIND-PROTECT-CLEANUP-NON-LOCAL-EXIT:1-RETURN-OUTER):

... the second non-local exit instead continues the original (more
extensive) one. In Situation 1, the second THROW to BAR is discarded in
lieu of continuing the THROW to FOO.

Proposal (UNWIND-PROTECT-CLEANUP-NON-LOCAL-EXIT:1-SIGNAL-ERROR):

... signals an error.

(If the IGNORE-ERRORS cleanup proposal or the proposed error/signal
system be adopted, it would be an error which would not be ignored by
IGNORE-ERRORS, for reasons outlined below.)

In situation 1, the second THROW to BAR would signal an error.  

Implementation notes:

There are several ways of implementing 1-SIGNAL-ERROR; this example is
merely given to illustrate: 

1. Every CATCH, and every TAGBODY and BLOCK which might have non-local
exits via GO, RETURN or RETURN-FROM creates a marker with a validity
state, initially true.

2. When a non-local exit (via THROW, GO, RETURN or RETURN-FROM) plans to
return control past a validity marker, it sets its validity state to
false, before any state is removed from the stack and any cleanup form
is evaluated. (For THROW, this happens after the target CATCH is found,
since, if there is no matching catch, the error is signalled in the
dynamic environment of the THROW).

3. If the validity state of the target of a non-local exit is false,  an
(non-ignorable) error is signalled.

4. When a non-local exit evaluates an unwind-protect cleanup form, it
first removes it from the list of such forms, so that with any
subsequent non-local exit it won't be evaluated again.

Rationale:


Current Practice:

Some implementations generate garbage code in situations 2 and 3.  Some
have differing behavior compiled and interpreted.

Most that have implementations seem to implement 2-EXIT, but there is
some divergence in Situation 1. For example, Spice Lisp and Xerox
implement 1-RETURN-INNER, while Symbolics implements 1-SIGNAL-ERROR.
 
Some compiled implementations agree with 1C and 2C. Some just output
garbage code (probably because lots of people don't anticipate this
case, which is admittedly quite rare).


Adoption Cost:

While require some compiler modifications in some implementations. In
some cases, that work was in order anyway since compilers may currently
be doing nothing particularly useful or defensible with the code in
question.  There is some sentiment that 1-RETURN-INNER is the simplest
to implement. 

Benefits:

No matter which proposal is adopted, having this situation uniformly
treated seems critical.

Programs which do this accidentally should behave the same on all
systems so that bugs can be detected and fixed very early rather than
being found later on a system which disagrees.

Programs which do this on purpose generally are trying to do something
fairly intricate and really need to be able to depend on it being
uniformly treated. A portable error/signal system and debugger may be
among these.

1-RETURN-INNER's motiviation is simplicity and aesthetics;
1-SIGNAL-ERROR and 1-RETURN-OUTER's motivation are robustness and
convenience. These are discussed here.

1-RETURN-INNER is more intuitive to programmers who reason about
programs in terms of continuation passing. It falls out of the normal
scoping rules as a consequence of the fact that the cleaup code is
evaluated in the lexical and dynamic environment in which the
UNWIND-PROTECT form appears. The action of THROW is usefully described
by saying that it is just like any other function. It happens to discard
the current continuation,  run some cleanup things (like variable
unbindings and UNWIND-PROTECT actions), and transfer control elsewhere
in the program. In doing so, the function uses data structure primitives
not generally available to other programs, but it is not linguistically
different and receives no special exemption with regard to THROWs or
other non-local transfers of control done within its execution. A THROW
 from within an UNWIND-PROTECT cleanup is not different than one in any
other code; it discards the ongoing action (stack unwinding) and
replaces it by another action (as it happens, another stack unwinding).
The previous unwind is never resumed.

1-SIGNAL-ERROR and 1-RETURN-OUTER complicate the language semantics but
improve environment reliability. For example, given

  (LOOP
    (CATCH 'FOO
      (UNWIND-PROTECT (LOOP)
        (THROW 'FOO T))))

With 1-RETURN-INNER there is no way of exiting such a form.
1-SIGNAL-ERROR would prevent programmers from getting into this
unfortunate situation.

1-RETURN-OUTER would guarantee that, upon a non-local exit out of a
computation, the   computation will be exited, one way or another. 


Conversion Cost:

Most user programs don't do this so the cost of converting existing code
is probably minimal. (There is some evidence that there are programs
that expect the behavior of 1-RETURN-INNER, so that the cost of
conversion is even lower in that case.)

Discussion:

1-SIGNAL-ERROR (and 1-RETURN-OUTER) prevent a serious environment bug,
where it is possible to create loops which cannot be aborted with
interrupts, the debugger or the like. 
The error signalled in 1-SIGNAL-ERROR must not be one caught by
IGNORE-ERRORS; otherwise

(loop
    (ignore-errors
      (unwind-protect (loop)
        (error))))

would have similar non-termination capabilities. 

Some argue that the situation that 1-SIGNAL-ERROR attempts to prevent is
perfectly valid: while it would be for a programmer to get into this,
the user has pretty clearly asked to have a loop that cannot be exited
and deserves to have that request carried out.  "We create syntax in a
language not to allow us to say the things that are obvious, but so that
we can say things that are not obvious. If we didn't need to say things
that were not obvious, we'd just jumble all the words together and
assume people would figure out some uniquely determined, obviously
useful interpretation. In fact, we make careful rules to allow us to say
bizarre things not so we can say bizarre things all the time, but so
that if there comes a time to say such things, we won't be at a loss for
words."

One real example offered was a top-level, where the Common Lisp
implementation's abort character would return to that level. 

 (DEFUN MY-TOPLEVEL ()
   (PROG ()
     LOOP (UNWIND-PROTECT (REALLY-MY-TOPLEVEL)
			  (GO LOOP))))

so that people who'd invoked my toplevel couldn't get back to toplevel
Lisp. While it might be rare, it's not unthinkable that someone could
really write this and mean what they said...

Some do not credit the seriousness of the "environment bug"; generally,
programming environments may need some way of aborting such a program
without executing the offending cleanup form, but it is an environment
problem which should have an environment solution. There are many other
ways a malicious Lisp user could blow the system out
of the water, e.g.  (makunbound '*terminal-io*), although those are
admittedly not valid Common Lisp programs.

Moon supported 1-SIGNAL-ERROR or 1-RETURN-OUTER.

Fahlman said he is inclined to support 1-SIGNAL-ERROR over
1-RETURN-OUTER.

Masinter, Pitman and Steele indicated support for 1-RETURN-INNER. 


All endorse 2-EXIT.