Further speed comparisons of AKCL and Lucid

[Jon L White <edsel!jonl@labrea.stanford.edu>]

Bob --

After your first broadcast of "benchmark" results to this list, I called
into question your methodology.  I'm wondering if the same critique applies
to the results you recently sent out:

    Date: Fri, 27 May 88 11:51:32 CDT
    From: boyer@rascal.ics.utexas.edu (Bob Boyer)
    To: kcl@cli.com
    Subject: Further speed comparisons of AKCL and Lucid

    MAXIMA is Bill Schelter's Common Lisp version of DOE Macsyma.  
    . . . 
    TESTS       AKCL     LUCID    EXPLORER I      
    . . . 
    [1,13s]    278.63    388.14    511.1


In particular, you mentioned that Bill Schelter had been hired specifically
to "fix up" your theorem-prover program and KCL to run fast.  One must
assume that since MAXIMA is his work, it too has been written in such a way 
to utilize the specific performance enhancements you have put into ACKL.  
I'm sure that you would like to know that your efforts are improving KCL 
are succeeding; and since it has been obvious in previous interchanges that 
you are not interested in tuning your programs to run in Sun Common Lisp, 
then I wonder how unbiased these numbers can be?

The previous benchmark interchange showed abundantly that changes can be 
made in source code that will take advantage of the way one implementation 
is built, while prejudicing it adversely against another.  Apart from 
simply using the data-structures and functions explicitly tuned in one 
implementation but not in the other, there is the question of which compiler 
declarations are appropriate for each "stock hardware" Lisp.  These are your
conclusions -- not just my own words.

Both you and Bill also confessed about the previous interchange that neither
of you knew much of anything about the declarations appropriate for Lucid's
compiler.  [I'm aware that you have only one copy of the Sun Common Lisp 
documentation, and that it not always in a convenient place for you two
to look at.]  On the other hand, you have your familiar AKCL sources "on
line", and you can more easily figure out how to embellish code to take best
advantage of it.   While I certainly wouldn't begrudge anyone the opportunity
to use the tools he is best familiar with -- indeed, this is the quickest 
route to completing one's research, regardless of moderate differences in 
computer speed --  I would expect a higher degree of effort from someone 
making a public comparative analysis.

Without any interest or motivation in putting a "best foot forward" for the 
systems you compare AKCL against, it is not surprising that you can find at
least some programs, in at least some modes of compilation, that will run 
significantly slower in those others.  Contrast this, if you will, against 
the methodology Dick Gabriel used when carrying out the Lisp Timings project:
he contacted a correspondent at each vendor or supplier, who prepared the 
common benchmark sources as appropriate for that implementation.  Admittedly,
the benchmarks themselves were biased strongly in favor of PDP10 MacLisp; but
at least the comparisons were with each implementation's "best foot" foward,
rather than with the MacLisp "best foot" compared with, say, the PSL "worst 
foot".

Incidentally, a junior-level person at Lucid was given the task of trying
to reproduce your benchmark numbers from the previous interchange.  He had 
only a Sun3/160 rather than the Sun3/280 that you had;  but the spectrum of 
numbers didn't exactly parallel the numbers you sent out.  He was using
*exactly* the sources just as you sent them, and did not "tweak" or tune them
in any way.  Of about 10 runs he made, Lucid's timing was a factor of two or 
so faster than AKCL's timing on eight of them; on the remaining two, AKCL was 
noticeably faster, but not a factor of two faster.  I do not yet know how 
these number would look if Lucid-specific declarations were added to the code.

The reason I feel compelled to reply to you about this matter is that since
your first mailing out of comparisons, I have heard friends at an independent
AI company saying "...we have now heard that KCL is faster than LUCID...".
Note that there were no qualifications or restrictions placed on this 
generalization -- it was a simple statement of alleged fact.  That hardly
seems an accurate summary of the results directly observable here at Lucid.


Do you really feel that you want to lend you name as the support for this 
now-circulating, *unqualified* rumor?



-- JonL --


P.S.: "an independent AI company saying" -- names withheld to protect the
      innocent.