CommonLoops

["Scott E. Fahlman" <Fahlman@C.CS.CMU.EDU>]

[This is in reply to the message from Weinreb asking what I meant when I
said that Flavors throws a lot of complexity at the user who wants to do
something simple.  For some reason, I can no longer get mail through to
hosts like SCRC-QUABBIN or SCRC-STONY-BROOK.  Since the query was CC'ed
to this mailing list, I'll reply via the list.]

Dan,

Dave Moon asked me more or less the same question, and I sent him a
lengthy reply a couple of days ago.  (I don't know if it got there,
however -- I've heard nothing back from Moon.)  I guess since the
question has now been raised in public, I may as well send my reply to
this whole list.  It follows -- the stuff after the dashed line.

Before we get to that, let me make a couple of preliminary remarks in
response to the questions you raised:

First, I suspect that for any "simple" situation that I describe to you,
you will be able to show me a straightforward way to implement the
desired thing in flavors.  That's not the point.  The point is that in
order for a beginner to feel that he has sufficient control of flavors
to find this simple solution, he has to understand an awful lot of hairy
machinery, most of which isn't relevant to the problem at hand but which
is not far enough out of the way that a beginner can confidently ignore
it.  The image I have (admittedly an exaggeration) is of a minefield: if
I want to get from point A to point B, you can show me a simple path;
for the beginner, however, the hard part is in knowing where NOT to
step.  Commonloops (so far) has fewer mines and they are all in a couple
of well-defined areas, not sprinkled around all over.

Second, I have not tried to sort out the elements of this apparent
complexity: how much of it is fundamental in the design of flavors, how
much is due to ugly add-ons that have grown up over the years, how much
is the fault of the documentation, and how much is due to the lack of
any readable flavor-based code that I can look at for inspiration, and
how much is due to the fact that, in order to play with this system, I
have to use a machine (the 3600) whose user interface I don't get along
with at all.

Third, let me just note that I am not the only one who finds Flavors to
be excessively complex.  Ask around.  Once people have invested an
intensive month or two in learning flavors, they seem to find it a
reasonable or even indispensible tool, and these people seem to forget
that they ever found it confusing.  But my sense of the community is
that these people are greatly outnumbered by people who who have tried
to learn about flavors (perhaps in a less intense way) and have decided
that it is an ungodly hairy system that they never want to use if there
is any alternative.  I've thought hard about whether this is just a
mental block on my part -- perhaps an over-reaction to a couple of bad
experiences -- but if that is the case, a lot of other people share this
same block.  So the question is not whether I'm right or wrong about
flavors being hairy; the question is why so many people THINK it is
hairy, and what (i anything) can be done about that.

Here's the message to Moon:

---------------------------------------------------------------------------

Dave,

I'm writing this when I really should be doing a bunch of other stuff,
so this may not end up being as coherent as I would like.  But anyway,
here are some of my thoughts on why Flavors seems gratuitously hairy to
me -- more hairy than, for example, CommonLoops.  Of course, hair is
in the eye of the beholder (ugh!).

First, some general views on what makes a software system seem simple or
complex.

1. If a system has a frequently-used part that is (or could be) relatively
clean and simple and a much hairier part that is needed only
occasionally, it is important that the casual user be able to totally
ignore the hairy part.  The document should be in two layers: here's the
part of the system you are going to need right away, and here is heavy
stuff that you might someday want to learn about, once you are totally
comfortable with the basic stuff.  And the system itself should be
organized in such a way that you will not accidentally wander (or be
dragged) deeper into the woods than you really want to go; the hair
should be totally out of your way until you explicitly ask for it.

2. Some users just worry about the surface semantics of a system, but
most good programmers are thinking all the time about efficiency and
choices that affect efficiency.  That means that you must have some
mental model of how things are implemented.  This model can be
oversimplified in some areas or it can even be a total lie, as long as
it gives the programmer a good qualitative feel for what various things
cost.  If a system's underlying model is clean and obvious, or if an
adequate implementation model can be suggested by the documentation,
then the programmers using this system will feel much more secure; if
not, then the programmers will feel that the system is out of control.

3. When a system is already very hard to understand, a few inelegant
constructs that increase the level of visual clutter can make a big
difference -- the straw that broke the camel's back.

I think that the Flavors system (as described in the Gray edition of the
Chine Nual, which is the version I happen to have handy) does rather
poorly by all of these criteria.  I won't dwell on the small
inelegancies, except to note that the system has a bunch of these.  For
example, the gettable/settable/initiable options for instance variables
should be on by default, with with an option to turn them off; as it is,
even very simple flavor definitions must be cluttered up with
odd-looking keywords.  There are several things like this that increase
the general level of clutter and make flavors code LOOK hairier than it
is.

The documentation is a big part of the problem with flavors, I think.
There are a LOT of complicated little options and features and
specialized functions (such as Instantiate-Flavor) that are all mixed in
with the stuff that the average user needs to know.  It may or may not
be the case that each and every one of these is necessary for doing
advanced system programming in flavors (some of them look like real
kludges), but if they must be around they should be put in a special
"hairy flavor wizards only" section of the manual.  The first few times
I tried to read the document, I started to falter when I hit
Instantiate- Flavor, and I was totally confused by the time I made it to
the section on Defflavor options (which finished me off).

Examples are very important -- without them, the documentation has to be
perfect -- and there are essentially none of them in the manual.  Worse,
there don't seem to be any relatively simple flavor-ized programs for
people to look at anywhere in the world.  At least, we got no response
when we asked for pointers to such things.  So people have to read the
manual and then start writing their own code without a template, or else
they have to look at something like the Symbolics window system.  That
is not a good way to persuade people that flavors are simple.

I should also mention that the few times I have tried to read
production-quality flavors code (in various parts of the 3600 software),
I have had a very difficult time finding the places where some code
actually does something; most methods seem to just do a Send to
somewhere else, along with doing a tiny bit of what has to be done.  I
would suggest that an important goal of good coding style in these
systems is to make it easy for the user to find where the action really
is, or else to clearly specify what the total effect of calling a
particular method is.

I think that the most fundamental problem is that flavors tries too hard
to combine late binding and modifiability of EVERYTHING with the goal of
compiling everything in the tightest possible way, with no wasted cycles
or levels of indirection that might make the late binding less complex.
These are incompatible goals, and by trying too hard to achieve both,
Flavors ends up with an explosion both in the complexity of the language
as seen by the user and in the underlying implementation (which, as I
said above, the users will want to understand at some level).  I agree
with the view that users will not use an object-oriented system for real
applications if it costs too much at runtime.  However, I would much
prefer a less complex system that retains total flexibility at some
modest cost in efficiency, and which also allows the user to say "I am
done with modifying the hierarchy of flavors and methods, so now compile
this system to the max."  It is the attempt to get both flexibility and
maximum efficiency AT THE SAME TIME that is the source of so much hair.

The method-combination stuff is the most obvious example of this.
Instead of going with <some code> (run-super) <some more code>, it is
deemed important to smash the superior method into the middle of the
function and to get it all compiled into one lump.  But then there is
the problem of undoing all this and recompiling if some loser goes in
and changes the type hierarchy.  The result is a lot of user-visible
complexity, funny exceptions about wrappers and whoppers, and a very
complex underlying model for those users who want to understand things
at that level.

If, instead, we pay for the extra method lookup implicit in the
run-super, everything works in the obvious way.  On top of that we add a
cache (which makes things run faster without changing the user's model
much) and a finalize-and-compile mode that makes everything maximally
efficient but rules out further changes in the hierarchy (or in that
part of it that is above the function in question).  It is the
compiler's job to make sure that whatever it does is functionally
equivalent to what the non-compiled version would do, and in a situation
declared to be stable, it can do this without a lot of extra coaching by
the user.  That seems much simpler to me than :before, :after, wrappers,
recompile-flavor, and so on.  Similarly, it seems to me (I haven't
thought this through) that some judicious use of an occasional level of
indirection might make it possible to get sufficient flexibility in
load-time order without all the major gyrations that flavors seems to go
through now; a super-compile mode could again be used to eliminate this
extra cost when the user says that the system is now in its final form.

Let's look at CommonLoops by way of contrast.  Again more examples are
needed.  The current document is not cluttered by hairy special-case
options; perhaps these will have to be added later, but maybe they can
be segregated from the main writeup.  All of the extensibility is hidden
in the meta-class stuff, which the typical user will never need to get
into -- one trap-door into the hairy regions rather than fifty.  The
underlying mechanisms appear to be rather simple, though they need to
be explained better.  A cache does not add conceptual complexity because
it "does the right thing"; the same with a super-compile mode to crush
out the last drops of inefficiency once the system is stable (if indeed
they add that).  The function-call syntax is familiar and intuitive;
while there may be some complex cases where one must think carefully
about which method is the most specific, in all of the cases that will
really come up this is obvious.

The question, of course, is whether the CommonLoops thing can remain as
simple as it now appears to be while it develops the necessary hooks and
options for doing various kinds of real world work.  Maybe it can't, and
maybe as an experienced implementor of these things you see this more
clearly than I do.  But right now it looks very clean, and Flavors looks
like a nightmare by comparison.

I don't know how clear any of this is, and I'm sure that you will think
that some of this is off base.  I've tried to give you a general idea of
why flavors LOOKS hairy to me, without thinking too hard about which of
these concerns are really justified and which are just misconceptions
and lack of experience on my part.  But I guess it is the perceptions
that are of interest here.  What it adds up to for me is that I'm eager
to try CommonLoops and I hope I never have to write anything serious in
flavors.

-- Scott