Complexity, yet again

["Scott E. Fahlman" <Fahlman@C.CS.CMU.EDU>]

In reply to: Robert W. Kerns <RWK at SCRC-YUKON.ARPA>

I may respond to your other points later, after I've had time to peruse
the latest edition of the Symbolics documentation.  I made reference to
the Gray manual because I've got a copy of it at home and because the
flavors documentation seemed, upon cursory inspection, to be the same as
before, modulo the addition of even more hairy features (whoppers!).

In this note, I would like to respond to one point you made that I think
may be important and very relevant to this discussion: the one about
wanting things to be modifiable forever because you never reboot your
machine.  (Back when I had a 3600, I used to reboot it at least every
half hour because I couldn't get it out of SHEET-LOCK and OUTPUT-HOLD,
but that's another story.)

I think you may have put your finger on a cultural difference that few
people recognize.  Some people like to run one Lisp world for as long as
possible, which means that late binding and the changeability of
everything in the middle of a session looms very large in their
thinking; others like to blow away a lisp job and start over ("reboot"
on a system like yours) whenever things get messed up.  This is not so
much an individual difference as a function of what a given machine and
system encourage you to do.

If I mess up a bunch of data structures or decide that I want to totally
rearrange the hierarchy of flavor types, it doesn't bother me much to go
back, fix the source file, recompile, and start off again in a clean
Lisp.  Being able to change such things while running is nice, but you
pay for this flexibility in one of two ways: either things are compiled
somewhat less tensely than if you tell the compiler that you're not
going to surprise it, or else you compile things to the max and make sure
that whenever something changes, everything that has to be fixed is
fixed.

Flavors wants full runtime modifiability and the absolute maximum in
efficiency, so it makes the latter choice.  Well, almost...some things
would just be too expensive to fix up, so they don't get fixed; others
are fixable, but only at a cost the probably doesn't want to pay.  Once
there are a few of these exceptions, you can't just tell the user to
change what he likes and to just trust the system to keep everything
consistent.  The user has to more or less understand how the fixing-up
process works and what its limitations are.  And this, I think is the
biggest unavoidable cognitive load on the would-be user of flavors.
Because of the combination of maximum compilation, arbitrary load order,
and runtime flexibility, the consistency machinery is very hairy, and
there's no simple lie that you can tell the user that will allow him to
ignore it all.  I'm pretty sure that this is a fundamental problem and
not a documentation issue.

Is that much complexity inevitable in a system with the power of
flavors?  Well, I think that the design choices that were made were the
natural ones in the MIT Lisp Machine/Symbolics culture: you don't want
to reboot, so everything must be modifiable; everything must always be
compiled to the max for best efficiency; minimizing user-visible
complexity is a laudable goal, but when push comes to shove runtime
flexibility and efficiency win.  On its own terms, given that set of
goals, flavors succeeeds admirably.

I don't like flavors because I have different goals.  Being able to
patch up EVERYTHING at runtime and loading things in completely
arbitrary orders is not so important to me.  Keeping the complexity down
is more important.  (That makes me a wimp.  Actually, I'm only worried
about all these naive users...)

I agree with Moon, who has said that an object-oriented system must in
the end produce code that runs just about as fast as if you used more
conventional techniques.  Otherwise, nobody will use it for anything
serious.  But I am willing to divide my life into phases: when I'm
messing around and doing exploratory programming, I'm willing to suffer
a modest loss of efficiency -- say, a factor of two overall.  When I've
got the system the way I want it, then I want to be able to compile out
all the flexibility to get the very last drop of performance.  the
excess complexity of flavors comes from wanting both of these things AT
THE SAME TIME.  I'm willing to accept one or the other at any given
time, and to recompile and reload the universe when the time comes to
switch modes.

By the way, I am not saying that CommonLoops does better according to
these goals -- just that they still have an opportunity to do so,
depending on how various issues get resolved.

-- Scott