[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Further speed comparisons of AKCL and Lucid
After your first broadcast of "benchmark" results to this list, I called
into question your methodology. I'm wondering if the same critique applies
to the results you recently sent out:
Date: Fri, 27 May 88 11:51:32 CDT
From: email@example.com (Bob Boyer)
Subject: Further speed comparisons of AKCL and Lucid
MAXIMA is Bill Schelter's Common Lisp version of DOE Macsyma.
. . .
TESTS AKCL LUCID EXPLORER I
. . .
[1,13s] 278.63 388.14 511.1
In particular, you mentioned that Bill Schelter had been hired specifically
to "fix up" your theorem-prover program and KCL to run fast. One must
assume that since MAXIMA is his work, it too has been written in such a way
to utilize the specific performance enhancements you have put into ACKL.
I'm sure that you would like to know that your efforts are improving KCL
are succeeding; and since it has been obvious in previous interchanges that
you are not interested in tuning your programs to run in Sun Common Lisp,
then I wonder how unbiased these numbers can be?
The previous benchmark interchange showed abundantly that changes can be
made in source code that will take advantage of the way one implementation
is built, while prejudicing it adversely against another. Apart from
simply using the data-structures and functions explicitly tuned in one
implementation but not in the other, there is the question of which compiler
declarations are appropriate for each "stock hardware" Lisp. These are your
conclusions -- not just my own words.
Both you and Bill also confessed about the previous interchange that neither
of you knew much of anything about the declarations appropriate for Lucid's
compiler. [I'm aware that you have only one copy of the Sun Common Lisp
documentation, and that it not always in a convenient place for you two
to look at.] On the other hand, you have your familiar AKCL sources "on
line", and you can more easily figure out how to embellish code to take best
advantage of it. While I certainly wouldn't begrudge anyone the opportunity
to use the tools he is best familiar with -- indeed, this is the quickest
route to completing one's research, regardless of moderate differences in
computer speed -- I would expect a higher degree of effort from someone
making a public comparative analysis.
Without any interest or motivation in putting a "best foot forward" for the
systems you compare AKCL against, it is not surprising that you can find at
least some programs, in at least some modes of compilation, that will run
significantly slower in those others. Contrast this, if you will, against
the methodology Dick Gabriel used when carrying out the Lisp Timings project:
he contacted a correspondent at each vendor or supplier, who prepared the
common benchmark sources as appropriate for that implementation. Admittedly,
the benchmarks themselves were biased strongly in favor of PDP10 MacLisp; but
at least the comparisons were with each implementation's "best foot" foward,
rather than with the MacLisp "best foot" compared with, say, the PSL "worst
Incidentally, a junior-level person at Lucid was given the task of trying
to reproduce your benchmark numbers from the previous interchange. He had
only a Sun3/160 rather than the Sun3/280 that you had; but the spectrum of
numbers didn't exactly parallel the numbers you sent out. He was using
*exactly* the sources just as you sent them, and did not "tweak" or tune them
in any way. Of about 10 runs he made, Lucid's timing was a factor of two or
so faster than AKCL's timing on eight of them; on the remaining two, AKCL was
noticeably faster, but not a factor of two faster. I do not yet know how
these number would look if Lucid-specific declarations were added to the code.
The reason I feel compelled to reply to you about this matter is that since
your first mailing out of comparisons, I have heard friends at an independent
AI company saying "...we have now heard that KCL is faster than LUCID...".
Note that there were no qualifications or restrictions placed on this
generalization -- it was a simple statement of alleged fact. That hardly
seems an accurate summary of the results directly observable here at Lucid.
Do you really feel that you want to lend you name as the support for this
now-circulating, *unqualified* rumor?
-- JonL --
P.S.: "an independent AI company saying" -- names withheld to protect the