loop vs. sequence functions

[trwrb!smpvax1!jrg@ucbvax.Berkeley.EDU]

At Inference, we've been using an iteration facility based on the Interlisp
iterator facility heavily for several years.  We provide a public, supported
version of this to our customers as part of ART, our expert system building
tool.  The iterator facility is much like the LOOP facility, to my under-
standing, though I believe that the LOOP facility may clean up a few of the
less-clean aspects of using iterators.  My point is that we've had a large
user base writing Common Lisp programs in an environment where a good iteration
facility is available.  They still use sequence functions.  Many write 
iterations, but almost all use sequence functions in addition, often in
preference to iteration.  Sometimes this is based on the judgement that the
sequence functions are easier to read.  Sometimes this is based on the
(almost always mistaken) belief that the Lisp compiler will turn a sequence
operation with appropriate declarations into better code than the iteration
expands into.  More often the reason is the former, for sake of clarity.
I'm arguing that in our experience, a good iteration facility did not eliminate
use of the sequence functions (else I wouldn't have proposed the :target
argument).

In addition, I have another motive in this.  Part of the problem is one of
education.  People take courses that teach them Common Lisp syntax.  These
courses rarely teach how to write reasonably efficient programs -- at least
the students don't come away able to write efficient programs.  These courses
tend to focus on teaching what Common Lisp functions do and what the various
arguments are used for, then ask students to write programs using them to
show that they understand the explanation.  They usually don't get into
performance considerations to any significant degree.  Perhaps if we added
a :target argument for re-using conses, it would be taught and explained as
a way to avoid consing were that is appropriate.  This would provide a
tangible syntactic mechanism that would be taught and explained in a manner
that made it clear to students that storage was being allocated when sequence
functions are used in the straightforward manner versus when the :target
argument was used.  Granted, this depends on the structure of the course,
etc, but at least there would be some piece of syntax that could be matched
with teaching this important concept.

Another part of the proposal was sort of an addendum to address this same
issue in non-sequence, list, or string functions.  This part of the proposal
has parts that are analogous to the :target argument and thus either redundant
or not based on the expected utility of LOOP.  But other parts would not be
addressed by LOOP, namely the make-string-output-stream function.

Just to throw some fuel onto the fire, an alternative to the :target argument
would be to support a FREE-SEQUENCE function that returned a sequence to Lisp
for later re-use.  When the Lisp could make use of this previously-allocated
storage rather than cons new storage, it would, particularly in the sequence,
list, and string functions in preference to new allocation.  The advantage
to this is that it adds less syntactic baggage to the language.  The dis-
advantage is that is makes it less obvious where the underlying implementa-
tion is re-using storage, leaving much to the implementation.  For products
such as ART, which are marketed as minimizing the need for garbage collection,
this might pose a problem since we need to be very sure that the implementation
is re-using storage as much as possible.  Of course, we could have both...