[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [spr2500] more benchmarks
Your problem report has been given tracking id spr2500.
Please use it in the subject line of any correspondences
regarding your problem report.
>> I'm confused.
>> I've been experimenting with AKCL and the gabriel benchmarks from
>> rascal.ics.utexas.edu. I'm finding Allegro Common LISP 3.1.13 running
>> on a Solbourne 5/501 (sparc) to be from 2 to 4 times slower than AKCL!
Sorry this has taken so long, but it has become a challenge to sort out
all of the issues. No doubt when Stewart Elliot phoned you Tuesday
and told you where your initial error was, he also told you all about
Franz Inc's philosophy on type inference (we tend to be conservative,
which thus tends to make our lisp more unbreakable than others on stock
hardware), our optimizations (we do a good job of optimization with
proper declarations and speed/safety settings) and our service (we have
a reputation for fast response to bugs AND performance problems which
include patches and/or help with declarations).
I took the triang.cl that you sent in one of your messages and tested it
against our current triang.cl (whose text I will send to you privately).
Please note that just because Kevin Layer's time stamp is in your version
doesn't mean that the stamp is accurate; someone may have taken out many
declarations which would have otherwise helped Allegro CL, and these
changes would probably not affect the timestamp.
The test I ran was on an RS/6000 model 520. I used this machine because
it is the only one of our machines that has akcl on it. The Allegro CL
compiler front-end is essentially the same one that is on the sparc, so
you should be able to get equivalent comparisons. All tests were made
with your makefile and command file (test-help.lsp) with modifications
to the makefile as the tests proceeded:
Your original triang.cl:
Orig Opt Fix
Allegro CL 148.09 18.23 17.9
akcl 36.05 35.94 35.94
Our current triang.cl:
Orig Opt Fix
Allegro CL 109.87 20.04 18.05
akcl 118.42 117.52 117.52
Explanation of tests:
Orig: Tested with everything as originally given, using different
Opt: Modified makefile for the compile phase to perform the optimize
proclamation, which was mistakenly placed in the run phase.
Fix: Included a patch for an inefficient #'elt compilation (no effect
on akcl results).
1. Your inquiry caused me to find an optimization bug in a call to elt,
which is generated by a transformation of the map call in our version
of triang.cl. I was able to make a patch that brought us back down to
essentially the same result as we get using your version. Thanks!!!
2. Allegro CL does not trust declarations if the trust-declarations-switch
returns nil, which by default happens whenever speed is not greater than
safety. This accounts for the larger times in test #1. Akcl seems to
trust all declarations. The question is: What happens if in a real
application (which may have some bugs) the programmer declares a value
to be a certain type but gives the lisp some other type?
3. Allegro CL optimizes svref only if our verify-non-generic-switch
returns t (which only happens if speed is 3 and safety is 1 or 0).
Akcl seems to always optimize the non-generic svref call (i.e. it
assumes that since you are calling svref you will always send it a
simple-vector). These behaviors are evidenced by the fact that your
triang.cl uses svref and ours uses aref. Again, the question is:
what happens in general code when a user passes something other than
a simple-vector to svref?
4. Given proper declarations and proclamations, Allegro CL optimizes aref
the same as svref. Akcl does not seem to know how to optimize aref.
I must claim ignorance of akcl here, and welcome you to show me how
to declare akcl to optimize aref.
In one of your later mail messages to Stewart, after you had realized
that optimization is important:
compiling with speed 3 and safety 0 made a big difference in all but
one file. This one is triang-mod (I'm including it for you). I get
about 53 seconds with allegro and 35 seconds with akcl on my 386 at home!
I looked at triang-mod, and noticed some instances where your declarations
could have given more optimization. In particular, you have declared the
arrays as (type (vector fixnum) ...). Vectors are certainly one-dimensional,
but vectors which are adjustable, are displaced to another array, or have
a fill pointer cannot be implemented as "simple" vectors, so the above
declaration would not allow Allegro CL to generate as efficient code as
if you had said (type simple-vector ...).
I will send you version of triang.cl that I used to run these tests.
You might note that the file looks much more like the one defined in
"Performance and Evaluation of Lisp Systems" (Gabriel, 1985). You might
also note that there are some new declarations, AND some declarations
were actually taken out. We try to do this as much as possible because
as we improve our type propagation system to allow less declarations to
be used in benchmark source code, a correponding improvement occurs wrt
how much declaration is needed in application code.
I also noticed a difference in the benchmark files included with allegro
common lisp. They did things like change arefs to svrefs and added
a few declarations. When I changed the arefs *back to* svrefs (I will
note that K. layer provided the triang.cl file in the "rascal" gabriel
benchmarks that is four days earlier than what is on the allegro tape)
I got a performance of about 14 seconds with akcl and 10 seconds with
allegro. So in most cases allegro does alright. I wonder if it should
do better than 29% better than akcl. That is shouldn't a native compiler
do about 50% better than a compiler going through C?
Yes, and the numbers above indicate that with proper declarations, it is
twice as fast.
I hope this helps you understand what had happened in your benchmarking
efforts. If you have any more questions or comments, please don't
hesitate to send me mail, and to cc firstname.lastname@example.org so that you can still
be helped if I am not available.
Duane Rettig, Franz Inc. 1995 University Avenue, Suite 275
duane@Franz.COM (internet) Berkeley, CA 94704
uunet!franz!duane (uucp) Phone: (415) 548-3600; FAX: (415) 548-8253