Re: Benchmarking the Explorer vs. the Symbolics

[Brad Miller <miller@acorn.cs.rochester.edu>]

    Date: Wed, 24 Dec 86 11:29 PST
    From: Mabry Tyson <TYSON@SRI-STRIPE.ARPA>

    It is difficult to have good benchmarks.  Code and usage tend to be
    optimized for the machine and conditions you are used to. [...]

Absolutely. I made no claim that my benchmark was in any sense "good", only
that it was "representative" of what many of us here do. (mainly me, perhaps)

	[...]

    The only true test of any system is how productive it makes you.
    Benchmarks are like deciding who is a better person by how fast they
    knit.  Obviously Symbolics thinks that the Common Lisp, Dynamic Windows,
    new Flavors, Who Calls Database, and all the other features will make
    their customers more productive.  In some cases, some customers will
    probably disagree.

I think one thing we (as customers) have to do is give the manufacturers some
feedback though. I made a somewhat feeble attempt, but at least I made the
attempt. I don't see much use of these mailing lists, personally I think these
are the sorts of observations that more customers should be making! One
concern of mine is that while it may be a generally neat thing to add lots of
new features, if the utility of those features are questionable, so is the
marginal benefit.

Who-calls is a good example. [That does explain one of the performance flukes,
by the way - thanks for pointing it out, I had forgotten about it.]
Personally, I think that for the few times I use it, I would rather build the
database at THAT time, than on every load. I may use who-calls twice a month,
and I load files every few minutes. Dynamic windows may be another good
instance (perhaps the entire new accept/presentation system?). While I have on
occasion found it nice, it's marginal utility may not be worth the cost in
significantly lowered performance.  That may be particularly true of the
presentation system - I don't write user interfaces as often as I just try to
use the machine - why make me pay all the time to make 1% of my job easier?

Overall, I think it may be a good thing to keep in mind that different users
have different levels of expertese and different needs from the system.
Advanced users should not have to pay the price of making the machine easy to
use for novices. Novices should not have to have an unlearnable machine to
make the machine powerful for experts. No one should have to pay a performance
penalty for features they don't really need or at least rarely use, until they
START to use them... Obviously this is an idealization...

You make the point that perhaps I should be comparing the explorers with
release 6.1 or even 5.2 and you may be right, if I were trying to compare
processor speed, given a more feature-for-feature identical system. But that
wasn't the point. We can all look at the raw benchmarks and see the symbolics
engine is faster, for a variety of reasons. What I want to know is how does it
do when it's doing MY job. That doesn't mean, either, that I wouldn't be more
happy with the slower machine, if it had features I really wanted. It's just
yet another datapoint. I simply found it somewhat amusing that the symbolics
was now doing SO MUCH in system software that it was slower actually executing
most (of our) applications than the explorer, and it starts off with a
processor roughly 2x faster! For some people this price will be too high...
and I was curious as to why it happened and perhaps voicing a hope that
symbolics might make things a little more efficient....

At any rate, I don't want anyone to think that this is a tirade against 7.0,
or anything, it's not. For the most part, I think 7.0 is a vast improvement
over 6.1 and I'm glad we have it. My only minor quibble in this regard might
be the overall number of things that changed... as others have mentioned. I'm
just trying to give hopefully useful constructive feedback in my own immodest
way. And end up with a machine or two that makes me more productive!

Brad Miller
------
miller@cs.rochester.edu
miller@acorn.cs.rochester.edu
miller@rochester.arpa