benchmarks

[barmar@Think.COM (Barry Margolin)]

    Date: Wed, 28 Feb 90 13:37 EST
    From: Jeffrey Mark Siskind <Qobi@ZERMATT.LCS.MIT.EDU>

    I think that you missed my point. I know what CPU time is and I know why it is
    reported and why people use it. I just reiterate my point that personally I
    find elapsed wall clock time to be the most appropriate measure of system
    performance. I know that it can vary depending on many factors such as system
    load, network load, and paging quirks. BUT ALL OF THESE AFFECT HOW LONG I MUST
    WAIT TO GET MY RESULT. And that is all that counts to me. 

If you're trying to optimize a program, you need to be more precise than
this.  When you're trying to figure out which part of the program needs
to be optimized you don't want random factors to stear you the wrong
way.  You want repeatable results that have some relationship to the
program you're timing.

							      Besides, few people
    use timesharing systems anymore today. Most people use single user machines
    (even if they happen to be Unix boxes which can support timesharing, they are
    apportioned one per person and typically run one user task at a time) so most
    of the arguments of using CPU time to get load invariant benchmark data are
    overweighed by the response time argument.

We're starting to buy X terminals now, and we also usually have many
users on our large Vaxes.  I think X terminals are going to signal the
rebirth of timesharing; until X terminals came along you had to use
expensive single-user workstations in order to get good windowing.  Even
before we got the X terminals, many of our users were using Sun 3/50's
as X terminals.  Configuring single-user workstations with enough memory
and swap space to run Lisp is expensive.

And even when single-user workstations are being used as such,
occasionally they are timeshared.  For instance, administrators
sometimes need to login to workstations to debug problems or edit
configuration files.  I wouldn't consider these processes to be the
normal system overhead, and if the administrator is doing this while the
user is trying to benchmark the administrator's behavior should be
factored out of the timings.

These random effects can often be factored out by running the benchmark
multiple times and throwing out extreme times.  The Symbolics metering
facility has some stuff that does this kind of analysis automatically.

                                                barmar