Re: timings

[Charles Frankston <CBF at MIT-MC>]

It is rather obvious that the timings you distributed are wall times for
the Lisp Machine, whereas the Vax and MC times count only time spent
directly executing code that is considered part of Macsyma.  Ie. the
Vax and MC times exclude not only garbage collection, but operating system
overhard, disk i/o and/or paging, time to output characters to terminals, etc.

I submit comparing wall times with (what the Multics people call) "virtual
CPU" time, is not a very informative excercise.  I'm not sure if the Lisp
Machine has the facilities to make analagous measurements, but everyone
can measure wall time, and in some ways thats the most useful comparison.
Is anyone willing to try the same benchmarks on the Vax and MC with just
one user on and measureing wall times?

Also, are there yet any Lisp machines with greater than 256K words?  No
one would dream of running Macsyma on a 256K word PDP10 and I presume that
goes the same for a 1 Megabyte Vax.  The Lisp Machine may not have a time
sharing system resident in core, but in terms of amount of memory needed
for operating system overhard, the fanciness of its user interface
probably more than makes up for that.  I'll bet another 128K words of
memory would not be beyond the point of diminishing returns, insofar
as running Macsyma.

Lastly, the choice of examples.  Due to internal Macsyma optimizations,
these examples have a property I don't like in a benchmark.  The timings
for subsequent runs in the same environment differ widely from previous
runs.  It is often useful to be able to factor out setup times from a
benchmark.  These benchmarks would seem to run the danger of being dominated
by setup costs.  (Eg. suppose disk I/O is much more expensive on one system;
that is probably not generally interesting to a Macsyma user, but it could
dominate benchmarks such as these.)

I would be as interested as anyone else in seeing the various lisp systems
benchmarked.  I hope there is a reasonable understanding in the various
Lisp communities of how to do fair and accurate, else the results will be
worse than useless, they will be damaging.