Re: [spr2500] more benchmarks

[duane@Franz.COM (Duane Rettig)]

Your problem report has been given tracking id spr2500.
Please use it in the subject line of any correspondences
regarding your problem report.

>> I'm confused.
>> I've been experimenting with AKCL and the gabriel benchmarks from
>> rascal.ics.utexas.edu.  I'm finding Allegro Common LISP 3.1.13 running
>> on a Solbourne 5/501 (sparc) to be from 2 to 4 times slower than AKCL!

Joel:
  Sorry this has taken so long, but it has become a challenge to sort out
all of the issues.  No doubt when Stewart Elliot phoned you Tuesday 
and told you where your initial error was, he also told you all about
Franz Inc's philosophy on type inference (we tend to be conservative,
which thus tends to make our lisp more unbreakable than others on stock
hardware), our optimizations (we do a good job of optimization with
proper declarations and speed/safety settings) and our service (we have
a reputation for fast response to bugs AND performance problems which
include patches and/or help with declarations).

I took the triang.cl that you sent in one of your messages and tested it
against our current triang.cl (whose text I will send to you privately).
Please note that just because Kevin Layer's time stamp is in your version
doesn't mean that the stamp is accurate; someone may have taken out many
declarations which would have otherwise helped Allegro CL, and these
changes would probably not affect the timestamp.

The test I ran was on an RS/6000 model 520.  I used this machine because
it is the only one of our machines that has akcl on it.  The Allegro CL
compiler front-end is essentially the same one that is on the sparc, so
you should be able to get equivalent comparisons.  All tests were made
with your makefile and command file (test-help.lsp) with modifications
to the makefile as the tests proceeded:

Your original triang.cl:
              Orig     Opt      Fix
Allegro CL   148.09   18.23    17.9
akcl          36.05   35.94    35.94

Our current triang.cl:
              Orig     Opt      Fix
Allegro CL   109.87   20.04    18.05
akcl         118.42   117.52  117.52

Explanation of tests:
Orig: Tested with everything as originally given, using different
      triang.cl files.

Opt: Modified makefile for the compile phase to perform the optimize
     proclamation, which was mistakenly placed in the run phase.

Fix: Included a patch for an inefficient #'elt compilation (no effect
     on akcl results).

Notes:
1. Your inquiry caused me to find an optimization bug in a call to elt,
   which is generated by a transformation of the map call in our version
   of triang.cl.  I was able to make a patch that brought us back down to
   essentially the same result as we get using your version.  Thanks!!!

2. Allegro CL does not trust declarations if the trust-declarations-switch
   returns nil, which by default happens whenever speed is not greater than
   safety.  This accounts for the larger times in test #1.  Akcl seems to
   trust all declarations.  The question is: What happens if in a real
   application (which may have some bugs) the programmer declares a value
   to be a certain type but gives the lisp some other type?

3. Allegro CL optimizes svref only if our verify-non-generic-switch
   returns t (which only happens if speed is 3 and safety is 1 or 0).
   Akcl seems to always optimize the non-generic svref call (i.e. it
   assumes that since you are calling svref you will always send it a
   simple-vector).  These behaviors are evidenced by the fact that your
   triang.cl uses svref and ours uses aref.  Again, the question is:
   what happens in general code when a user passes something other than
   a simple-vector to svref?

4. Given proper declarations and proclamations, Allegro CL optimizes aref
   the same as svref.  Akcl does not seem to know how to optimize aref.
   I must claim ignorance of akcl here, and welcome you to show me how
   to declare akcl to optimize aref.


In one of your later mail messages to Stewart, after you had realized
that optimization is important:

    Stewart,
    compiling with speed 3 and safety 0 made a big difference in all but
    one file.  This one is triang-mod (I'm including it for you).  I get
    about 53 seconds with allegro and 35 seconds with akcl on my 386 at home!

I looked at triang-mod, and noticed some instances where your declarations
could have given more optimization.  In particular, you have declared the
arrays as (type (vector fixnum) ...).  Vectors are certainly one-dimensional,
but vectors which are adjustable, are displaced to another array, or have
a fill pointer cannot be implemented as "simple" vectors, so the above
declaration would not allow Allegro CL to generate as efficient code as
if you had said (type simple-vector ...).

I will send you version of triang.cl that I used to run these tests.
You might note that the file looks much more like the one defined in
"Performance and Evaluation of Lisp Systems" (Gabriel, 1985).  You might
also note that there are some new declarations, AND some declarations
were actually taken out.  We try to do this as much as possible because
as we improve our type propagation system to allow less declarations to
be used in benchmark source code, a correponding improvement occurs wrt
how much declaration is needed in application code.

    I also noticed a difference in the benchmark files included with allegro
    common lisp.  They did things like change arefs to svrefs and added
    a few declarations.  When I changed the arefs *back to* svrefs (I will
    note that K. layer provided the triang.cl file in the "rascal" gabriel
    benchmarks that is four days earlier than what is on the allegro tape)
    I got a performance of about 14 seconds with akcl and 10 seconds with
    allegro.  So in most cases allegro does alright.  I wonder if it should
    do better than 29% better than akcl.  That is shouldn't a native compiler
    do about 50% better than a compiler going through C?

Yes, and the numbers above indicate that with proper declarations, it is
twice as fast.

I hope this helps you understand what had happened in your benchmarking
efforts.  If you have any more questions or comments, please don't
hesitate to send me mail, and to cc bugs@franz.com so that you can still
be helped if I am not available.

Duane Rettig, Franz Inc.        1995 University Avenue, Suite 275
duane@Franz.COM (internet)      Berkeley, CA  94704
uunet!franz!duane (uucp)        Phone: (415) 548-3600; FAX: (415) 548-8253